Redis Source Code Analysis () --- Redis's 11 excellent designs

It took about a month to classify Redis code from the very beginning, from the struct structure analysis to the final analysis of the main program, the code of each major module breaks through and learns from each other one by one. In short, it has gained a lot. For a long time, I haven't had the patience to thoroughly learn a framework. Learning a framework is only a small part of it, it is really hard to thoroughly understand the principles behind it. At the final stage of this study, it is time to get some good results. I have summarized some of the better code and design ideas I have summarized over the past month, I originally wanted to build ten brilliant designs, but later I felt that each point was wonderful. I still made 11 excellent designs, and I told you to open the research. There is no language here, focusing on a programming idea and design, I hope everyone can understand it well. (The following sorting does not matter. I just sorted it in chronological order. The link below is related articles I wrote. If you want to learn more, please click here)

1. Implementation of hyperloglog base statistics algorithm (http://blog.csdn.net/androidlushangderen/article/details/40683763 ). Speaking of this, the funny thing is that I initially thought it was a type of log, just like slowLog, and later I realized that this is a basic statistics algorithm, similar algorithms include LLC, and HLLC is his upgraded version.

2. Re-implementation of zmalloc memory allocation (http://blog.csdn.net/androidlushangderen/article/details/40659331 ). The author of Redis apparently made preparations for memory allocation. It would not be silly to call the mallo and free methods of the system. People made a small encapsulation here, this makes it easier for managers to control the system memory. The following is a declaration of a small struct. You may understand this.

/* call zmalloc to apply for size of space */
Void * zmalloc (size_t size) {
// the malloc function is actually called
Void * PTR = malloc (size + PREFIX_SIZE);

// if the result of the application is null, oom has occurred. Call oom's processing method
If (! PTR) zmalloc_oom_handler (size);
# ifdef HAVE_MALLOC_SIZE
// updates the size of used_memory
Update_zmalloc_stat_alloc (zmalloc_size (PTR));
Return PTR.
# the else
* (PTR) (size_t *) = the size;
Update_zmalloc_stat_alloc (size + PREFIX_SIZE);
PTR + PREFIX_SIZE return (char *);
# endif
}

3. multi transaction operation (http://blog.csdn.net/androidlushangderen/article/details/40392209 ). The transaction operations in Redis gave me a new look. When I was doing this design, I used the key and watch key concepts, A key maintains a list of all watch clients. A Client also has all the keys it monitors. If a key is touched, all the next operations on clients that have the same knowledge about this key are invalid. Please click the link below for details.

4. redis-benchmark performance testing (http://blog.csdn.net/androidlushangderen/article/details/40211907 ). Redis has a performance statistics concept here, which is a relatively high sense. Compared with calling many latency methods, Redis judges the latency to see whether the performance is good or bad.

5.zip map compression structure design (http://blog.csdn.net/androidlushangderen/article/details/39994599 ). Redis tries its best to deal with memory processing. ziplist compression lists and zipmap compression graphs are very typical designs. An integer variable of the int64 type is directly placed in the body of the regular structure, which occupies 8 bytes. However, in general, the values we save are relatively small, one byte is enough, and seven bytes are wasted. Therefore, the zip compression series structure can be used to dynamically allocate bytes to cope with different situations. This design is brilliant, to determine the location of the key-value, the offset is located based on the length reserved earlier.

6. sparkline micro line diagram redesign (http://blog.csdn.net/androidlushangderen/article/details/39964591 ). The emergence of sparkline in Redis should help me to improve literacy. People can output a table similar to a line chart in the form of strings, and use the samples of many Sample songs collected, this class is mostly used for analysis and statistics, such as latency. c latency analysis class is used.

7. Implement memory management (http://blog.csdn.net/androidlushangderen/article/details/40716469) for object reference count ). We know that there are two methods to manage the lifecycle of an object. One is the root search method (this is used in JVM), and the other is the reference counting method, redis provides us with the implementation of this method. The following is the implementation of adding and removing references to objects:

/* robj object increase or decrease the reference count, increment the value of refcount in robj */
Void incrRefCount robj * o {
// increment the value of refcount in robj
O - > refcount++;
}
/* decrement the reference count in robj, and when the reference reaches 0, release the object */
Void decrRefCount robj * o {
// if the previous reference count is <=0, an exception has occurred
If (o->refcount <= 0) redisPanic("decrRefCount against refcount <= 0");
If (o->refcount == 1) {
// if the previous reference count is 1, decrement it again, just in time to be referenced by any object, so you can release the object
The switch (o - > type) {
Case REDIS_STRING: freeStringObject (o); Break;
Case REDIS_LIST: freeListObject (o); Break;
Case REDIS_SET: freeSetObject (o); Break;
Case REDIS_ZSET: freeZsetObject (o); Break;
Case REDIS_HASH: freeHashObject (o); Break;
Default: redisPanic (" Unknown object type "); Break;
}
Zfree (o);
} else {
// for other reference counts of >1, you simply decrement the reference count as usual
O - > refcount -;
}
}

The focus is to reduce the implementation of referenced methods.

8. fork sub-process implementation background program (http://blog.csdn.net/androidlushangderen/article/details/40266579 ). Fork creates sub-threads to implement the operations of background programs. I was able to see this for the first time. I didn't know how to use fork before. This time, I really got a boost in knowledge. The key point is that the fork method is processed differently in the subthread and the return value in the parent thread. The parent thread returns the PID number of the subthread and returns 0 in the subthread.

/* RBD save operation in the background */
Int rdbSaveBackground (char * filename) {
Pid_t childpid;
Long long start;

If (server rdb_child_pid! = 1) return REDIS_ERR;

Server. Dirty_before_bgsave = server. Dirty;
Server. Lastbgsave_try = time (NULL);

Start = ustime ();
// use fork() to create a child process to implement the RDB save operation
// at this point, there are two processes executing the code of this function, and the pid returned by the sub-procedure is 0,
// so the following code will be executed. The code returned in the parent process is the child's pid, not zero, so the code for the else branch will be executed
// putting the return -1 in the parent process means the creation of the child process failed
If ((childpid = fork()) == 0) {
// the code judged in this if is the operation performed after the child thread
Int retval;

Child / * * /
CloseListeningSockets (0);
RedisSetProcTitle (" redis - RDB - bgsave ");
// this is the rdbSave() operation
Retval = rdbSave (filename);
If (retval == REDIS_OK) {
Size_t private_dirty = zmalloc_get_private_dirty ();

If (private_dirty) {
RedisLog (REDIS_NOTICE,
RDB: %zu MB of memory used by copy-on-write,
Private_dirty/(1024 * 1024));
}
}
ExitFromChild ((retval = = REDIS_OK)? 0:1);
} else {
// performs subsequent operations on the parent thread
Parent / * * /
Server. Stat_fork_time = ustime () - start;
Server.stat_fork_rate = (double) zmalloc_used_memory() * 1000000 / server.stat_fork_time/(1024*1024*1024); /* GB per second
LatencyAddSampleIfNeeded (" fork ", for server stat_fork_time / 1000);
If (childpid == -1) {
Server. Lastbgsave_status = REDIS_ERR;
RedisLog (REDIS_WARNING,"Can't save in background: fork: %s",
The strerror (errno));
Return REDIS_ERR;
}
RedisLog (REDIS_NOTICE,"Background saving started by pid %d",childpid);
Server. Rdb_save_time_start = time (NULL);
Server. Rdb_child_pid = childpid;
UpdateDictResizePolicy ();
Return REDIS_OK;
}
Return REDIS_OK; / * unreached * /
}

9. Convert the long type to the String type method (http://blog.csdn.net/androidlushangderen/article/details/40649623 ). In the past, we have done a lot of algorithm implementations for converting string values to numeric values. Maybe your function is implemented, but the efficiency is high. But when we are dealing with long numbers, efficiency may be lower. Redis provides us with a good idea here. At ordinary times, we calculate/10, and then % 1o calculates the remainder. People come directly to a/100, then, you can directly map the string array to the remainder value for calculation. The algorithm is as follows;

/* Convert a long long into a string. Returns the number of
* characters needed to represent the number.
* If the buffer is not big enough to store the string, 0 is returned.
*
* Based on the following article (that apparently does not provide a)
* novel approach but only publicizes an already used technique):
*
* https://www.facebook.com/notes/facebook-engineering/three-optimization-tips-for-c/10151361643253920
*
* Modified in order to handle signed integers since the original code was
* designed for unsigned integers. */
/* long long converts to string */
Int ll2string(char* DST, size_t dstlen, long long svalue) {
Static const char digits[201] =
"0001020304050607080910111213141516171819"
"2021222324252627282930313233343536373839"
"4041424344454647484950515253545556575859"
"6061626364656667686970717273747576777879"
"8081828384858687888990919293949596979899";
Int negative;
Unsigned long long value;

/* The main loop works with 64bit unsigned integers for simplicity, so
* we convert the number here and remember if it is negative
I'm going to do plus and minus here
If (svalue < 0) {
If (svalue! = LLONG_MIN) {
Value = - svalue;
} else {
Value = (((unsigned long long) LLONG_MAX)+1;
}
Negative = 1;
} else {
Value = svalue;
Negative = 0;
}

/ * Check length. * /
Uint32_t const length = digits10(value)+negative;
If (length >= dstlen) return 0;

/ * Null term. * /
Uint32_t next = length;
DST [next] = '\ 0';
Next,;
While (value >= 100) {
// convert the values
Int const I = (value % 100) * 2;
The value / = 100;
The residual value of // I is replaced by the corresponding digit in the digits array
DST [next] = digits[I + 1];
DST [next-1] = digits[I];
Next - = 2;
}

* Handle last 1-2 digits. */
If (value < 10) {
DST [next] = '0' + (uint32_t) value;
} else {
Int I = (uint32_t) value * 2;
DST [next] = digits[I + 1];
DST [next-1] = digits[I];
}

/ * Add sign. * /
If (negative) DST [0] = '-';
Return length;
}

10. Regular expression implementation algorithm (http://blog.csdn.net/androidlushangderen/article/details/40649623 ). We usually use many regular expressions. How do we know how regular expressions are implemented through simple pattern process matching? Why? It can represent any character and then match it later. * it represents all characters. It is not so easy to implement such an algorithm. Redis has implemented this algorithm, it's a treasure.

* Glob-style pattern matching
Support for glob-style wildcard formats, such as * for any one or more characters,? Represents any character, and [ABC] represents any letter in square brackets. * /
Int stringmatchlen(const char *pattern, int patternLen,
Const char *string, int string len, int nocase)
{
While (patternLen) {
The switch (pattern [0]) {
Case: '*'
While (pattern[1] == '*') {
// if ** appears, it must match
Pattern++;
PatternLen -;
}
If (patternLen = = 1)
Return 1; / * * / match
While (stringLen) {
If (stringmatchlen (+ 1, the pattern patternLen - 1,
String, stringLen nocase))
Return 1; / * * / match
String++;
StringLen -;
}
Return 0; / * * / no match
Break;
Case '? ':
If (stringLen = = 0)
Return 0; / * * / no match
/ * because? Can represent any character, so move the matching character one further character */
String++;
StringLen -;
Break;
Case '[' :
{
Int the not match;

Pattern++;
PatternLen -;
Not = pattern[0] == '^';
If (not) {
Pattern++;
PatternLen -;
}
Match = 0;
While (1) {
If (pattern[0] == '\ ') {
// if an escape character is encountered, the pattern character moves back one position
Pattern++;
PatternLen -;
If (the pattern [0] = = string [0])
Match = 1;
} else if (pattern[0] == ']') {
// until I meet another bracket, then stop
Break;
} else if (patternLen == 0) {
The pattern -;
PatternLen++;
Break;
} else if (pattern [1] = = '-' && patternLen > = 3) {
Int start = pattern [0];
Int the end = pattern [2];
Int c = string [0];
If (start > end) {
Int t = start;
Start = end;
End = t;
}
If (nocase) {
Start = tolower (start);
End = tolower (end);
C = tolower (c);
}
The pattern + = 2;
PatternLen - = 2;
If (c >= start && c <= end)
Match = 1;
} else {
If (! Nocase) {
If (the pattern [0] = = string [0])
Match = 1;
} else {
If (tolower ((int) pattern [0]) = = tolower ((int) string [0]))
Match = 1;
}
}
Pattern++;
PatternLen -;
}
If (not)
Match =! The match;
If (! The match)
Return 0; / * * / no match
String++;
StringLen -;
Break;
}
Case '\ \' :
If (patternLen >= 2) {
Pattern++;
PatternLen -;
}
/ * fall through * /
Default:
If there are no key characters in the regular expression, compare */ directly
If (! Nocase) {
If (the pattern [0]! = the string [0])
// unequal, direct mismatch
Return 0; / * * / no match
} else {
If (tolower ((int) pattern [0]). = tolower ((int) string [0]))
Return 0; / * * / no match
}
String++;
StringLen -;
Break;
}
Pattern++;
PatternLen -;
If (stringLen == 0) {
The pattern while (* = = '*') {
Pattern++;
PatternLen -;
}
Break;
}
}
If (patternLen == 0 && stringLen == 0)
// if the length of matching characters and pattern characters is reduced to 0, the match is successful
Return 1;
Return 0;
} 

11. Redis's drand48 () random algorithm re-implementation (http://blog.csdn.net/androidlushangderen/article/details/40582189 ). The implementation of the Redis random algorithm, as the last of the 11 major designs, does not mean how bad this design is compared with the previous one, because I think it is more characteristic, I will append it. Because the author of Redis considers that the random algorithm may have different characteristics in different operating systems, math is not recommended. the rand () method is implemented based on the drand48 () algorithm. What is drand48 ()? Click the link.

Well, the above is the excellent design in Redis. In fact, there are still many traces of excellent code in Redis. Due to limited space, the readers are waiting to learn and discover it by themselves.

Redis Source Code Analysis () --- Redis's 11 excellent designs