Redis source code analysis (I) Memory Management

I. redis Memory Management

Redis is a memory-based Key-value database, and its memory management is very important. In order to shield the differences between different platforms and count the memory usage, redis encapsulates memory allocation functions at a layer. in the program, zmalloc and zfree functions are used in a unified manner. The source code of these functions is src/zmalloc. H and src/zmalloc. in the c file, click the source code here.

Ii. redis memory management source code analysis

Redis encapsulation is to avoid differences between the underlying platform. At the same time, it is convenient for you to implement relevant functions. We can use src/zmalloc. the macro definitions in the H file are used to analyze how redis shields underlying platform differences. zmalloc. the macro description in H is as follows:

#if defined(USE_TCMALLOC)#define ZMALLOC_LIB ("tcmalloc-" __xstr(TC_VERSION_MAJOR) "." __xstr(TC_VERSION_MINOR))#include <google/tcmalloc.h>#if (TC_VERSION_MAJOR == 1 && TC_VERSION_MINOR >= 6) || (TC_VERSION_MAJOR > 1)#define HAVE_MALLOC_SIZE 1#define zmalloc_size(p) tc_malloc_size(p)#else#error "Newer version of tcmalloc required"#endif#elif defined(USE_JEMALLOC)#define ZMALLOC_LIB ("jemalloc-" __xstr(JEMALLOC_VERSION_MAJOR) "." __xstr(JEMALLOC_VERSION_MINOR) "." __xstr(JEMALLOC_VERSION_BUGFIX))#include <jemalloc/jemalloc.h>#if (JEMALLOC_VERSION_MAJOR == 2 && JEMALLOC_VERSION_MINOR >= 1) || (JEMALLOC_VERSION_MAJOR > 2)#define HAVE_MALLOC_SIZE 1#define zmalloc_size(p) je_malloc_usable_size(p)#else#error "Newer version of jemalloc required"#endif#elif defined(__APPLE__)#include <malloc/malloc.h>#define HAVE_MALLOC_SIZE 1#define zmalloc_size(p) malloc_size(p)#endif#ifndef ZMALLOC_LIB#define ZMALLOC_LIB "libc"#endif...#ifndef HAVE_MALLOC_SIZEsize_t zmalloc_size(void *ptr);#endif

Through the macro preprocessing above, we can find that redis performs the following preprocessing to shield the differences between different systems (databases:

AIf Google's tc_malloc library exists in the system, use the tc_malloc family function to replace the original malloc family function.

BIf the jemalloc library of Facebook exists in the system, use the je_malloc family function to replace the original malloc family function.

CIf the current system is a MAC system, use the memory allocation function in <malloc/malloc. h>.

DIn other cases, a fixed-length field is assigned at the same time before each allocated space segment to record the size of the allocated space.

Tc_malloc is a set of open-source memory management libraries processed by Google and implemented in C ++. The homepage is here. Tcmalloc assigns a local cache for each thread. Small allocation can be directly satisfied by the local cache of the thread. If necessary, the object will be moved from the central data structure to the local cache of the thread. At the same time, regular garbage collection will be used to migrate the memory from the local cache of the thread to the central data structure. This article provides a detailed introduction to tcmalloc.

Jemalloc is also a memory creation management library, and its founder Jason Evans is also a very famous developer in FreeBSD. For details, see here. Jemalloc aggregates many technologies verified during the use of malloc. Ignore the details. From the perspective of architecture, the best part is arena and thread cache.

The reader must be certain that the system does not have malloc. Why is there such a memory management library ?? Because the fragmentation rate of the classic libc distributor is relatively high, you can view the analysis in this article. If you are not familiar with memory fragmentation, refer to this article, here is an example of how malloc and free work. For a summary of the ptmalloc, tcmalloc, and jemalloc memory allocation policies, click here.

The following describes the memory management functions encapsulated by redis. src/zmalloc. H has related declarations.

void *zmalloc(size_t size);//mallocvoid *zcalloc(size_t size);//callocvoid *zrealloc(void *ptr, size_t size);/reallocvoid zfree(void *ptr);//freechar *zstrdup(const char *s);size_t zmalloc_used_memory(void);void zmalloc_enable_thread_safeness(void);void zmalloc_set_oom_handler(void (*oom_handler)(size_t));float zmalloc_get_fragmentation_ratio(void);size_t zmalloc_get_rss(void);size_t zmalloc_get_private_dirty(void);void zlibc_free(void *ptr);

Now we will mainly introduce the redis memory allocation function void * zmalloc (size_t size). The corresponding declaration form is as follows:

void *zmalloc(size_t size) {    void *ptr = malloc(size+PREFIX_SIZE);    if (!ptr) zmalloc_oom_handler(size);#ifdef HAVE_MALLOC_SIZE    update_zmalloc_stat_alloc(zmalloc_size(ptr));    return ptr;#else    *((size_t*)ptr) = size;    update_zmalloc_stat_alloc(size+PREFIX_SIZE);    return (char*)ptr+PREFIX_SIZE;#endif}

After reading the source code, we found a prefix_size macro. Its macro definition format is as follows:

/* zmalloc.c */#ifdef HAVE_MALLOC_SIZE#define PREFIX_SIZE (0)       #else  #if defined(__sun)#define PREFIX_SIZE (sizeof(long long))#else                         #define PREFIX_SIZE (sizeof(size_t))#endif#endif

Combined with SRC/zmalloc. h has related macro description. We found that since tc_malloc, je_malloc, and malloc Function Families on Mac platforms provide functions for calculating the size of allocated space (tc_malloc_size, je_malloc_usable_size, and malloc_size ), therefore, you do not need to allocate a space record size separately. On Linux and Sun platforms, record the size of allocated space. For Linux, use sizeof (size_t) to set long field records; for Sun systems, use sizeof (long) to set long field records, and the prefix_size macro in the corresponding source code.

What is prefix_size used?

To count the memory occupied by the current process. In zmalloc. C, there is a static variable:

static size_t used_memory = 0;

This variable records the total memory occupied by the process. This variable must be updated every time the memory is allocated or released (of course thread-safe ). When allocating memory, you need to specify the amount of memory allocated. However, when the memory is released (for memory libraries that do not provide the malloc_size function), you cannot know the size of the space to be released by pointing to the memory to be released. At this time, the prefix_size mentioned above takes effect, and the space size can be obtained through the recorded content. (In Linux, there are also corresponding functions to obtain the size of the allocated memory space. For details, see here ).

Through the source code of zmalloc, we can find that the allocated space code is void * PTR = malloc (size + prefix_size). Obviously, the allocated space size is: Size + prefix_size, for tc_malloc or je_malloc or MAC systems, the prefix_size is 0. When the allocation fails, corresponding error handling occurs.

We have already said that redis uses the used_memory variable to count the memory occupied by the current process. Therefore, when allocating and releasing the memory, We need to update the value of used_memory immediately, corresponding to the redis source code:

#ifdef HAVE_MALLOC_SIZE    update_zmalloc_stat_alloc(zmalloc_size(ptr));    return ptr;#else    *((size_t*)ptr) = size;    update_zmalloc_stat_alloc(size+PREFIX_SIZE);    return (char*)ptr+PREFIX_SIZE;#endif

The above code has macro preprocessing # ifdef have_malloc_size is obviously the case where we mentioned above that the malloc_size function is provided by tc_malloc je_malloc Mac and so on, we can easily know the size of the allocated memory through the unified malloc_size function. But how does redis handle functions that do not provide the malloc_size function? Look at the source code above # The code below else is actually present, and its corresponding memory structure is as follows:

Prefix-size

Memory size

The allocated memory is preceded by a fixed-size pref_size space, which is used to record the size of the memory segment. The size occupied by the size is known and is the length of the size_t type, therefore, you can specify the current memory block size by * (size_t *) PTR) = size. After each memory allocation, the actual address pointer returned is the address pointing to the memorysize (char *) PTR + prefix_size, it can easily calculate the actual memory header address to release the memory.

Redis uses the update_zmalloc_stat_alloc (_ n ,__ size) and update_zmalloc_stat_free (_ n) macros to update the used_memory variable when allocating or releasing the memory. Update_zmalloc_stat_alloc is defined as follows:

#define update_zmalloc_stat_alloc(__n) do {     size_t _n = (__n);     if (_n&(sizeof(long)-1)) _n += sizeof(long)-(_n&(sizeof(long)-1)); \     if (zmalloc_thread_safe) {         update_zmalloc_stat_add(_n);     } else {         used_memory += _n;     } } while(0)

Redis writes the update operation into a macro in the form of efficiency considerations.

In the above Code

A, if (_ N & (sizeof (long)-1) _ n + = sizeof (long)-(_ N & (sizeof (long)-1 ));
Alignment is mainly considered to ensure that the new _ n is a multiple of sizeof (long.

B, if (zmalloc_thread_safe ){\
Update_zmalloc_stat_add (_ n );\

}

Assume that multiple threads exist in the process and ensure thread security zmalloc_thread_safe, the lock is required when updating the variable. Use the macro have_atomic to select the corresponding synchronization mechanism.

The implementation of zmalloc_calloc, zmalloc_free, and so on will not detail the details. For details, refer to the source code.

Finally, we will describe the zmalloc_get_rss () function.
This function is used to obtain the RSS of a process. Why does Shenma use RSS? It is called resident set size, which refers to the actual use of physical memory (including the memory occupied by the shared library ). In Linux, the PID can be obtained by reading the/proc/Pid/STAT file system. The PID is the process Number of the current process. The number of bytes read is not the number of bytes, but the number of pages in the memory. The system calls sysconf (_ SC _pagesize) to obtain the memory page size of the current system. After obtaining the RSS of a process, you can calculate the size of the memory fragments of the current data. You can use RSS to divide the size by used_memory. RSS includes all memory usage of processes, including code, shared libraries, and stacks. Where are the memory fragments? As we have already explained above, we usually consider efficiency and memory alignment. Therefore, fragments are generated here. Compared with the memory usage of malloc in traditional glibc, Qualcomm often uses other memory library systems. In redis, jemalloc is used instead of simple malloc by default. The source file src/makefile contains the following code:

ifeq ($(uname_S),Linux)MALLOC=jemalloc

We can know that jemalloc is used by default in Linux, and related libraries deps/jemalloc are included in the source code published by redis.

In general, redis allocates memory independently, allocates memory in real time based on built-in algorithms when requests arrive, and controls memory management completely. It's simple, but it's actually powerful.

Exam:

Http://blog.ddup.us /? P = 136