Memory Management for redis source code parsing

The content of zmalloc. H is as follows:

1 void *zmalloc(size_t size);2  void *zcalloc(size_t size);3  void *zrealloc(void *ptr, size_t size);4  void zfree(void *ptr);5  char *zstrdup(const char *s);6 size_t zmalloc_used_memory(void);7  void zmalloc_enable_thread_safeness(void);8  float zmalloc_get_fragmentation_ratio(void);9 size_t zmalloc_get_rss(void);10 size_t zmalloc_allocations_for_size(size_t size);11 12  #define ZMALLOC_MAX_ALLOC_STAT 256

Just a few lines .This is the memory management interface of redis. Zmalloc, zcalloc, zrealloc, and zfree correspond to malloc, calloc, realloc, and free in the C library respectively. .Zstrdup is used to generate a copy of a string. The following functions are used to obtain memory usage information, which will be detailed later.
Before reading the source code of zmalloc. C, let's take a look at how redis manages memory. .To facilitate memory management, apsaradb for redis stores the memory size in the header of the memory block after a memory block is allocated.
Real_ptr is the pointer returned by redis after calling malloc. .Redis stores the size and size of the memory block into the header. The size occupies a known memory size, which is a length of the size_t type, and then returns the ret_ptr .When the memory needs to be released, ret_ptr is passed to the memory management program. Through ret_ptr, the program can easily calculate the value of real_ptr, and then pass real_ptr to free to release the memory. .
Redis records all memory allocations. Redis defines an array with the length zmalloc_max_alloc_stat. Each element of the array represents the number of memory blocks allocated by the current program, and the size of the memory block is the subscript of the element. .In the program, this array is zmalloc_allocations. Zmalloc_allocations [16] indicates the number of allocated 16-bytes memory blocks. .The static variable used_memory in zmalloc. C is used to record the total size of the currently allocated memory. .
The following code is analyzed.
First, define the macro prefix_size:

1 #ifdef HAVE_MALLOC_SIZE2  #define PREFIX_SIZE (0)3  #else4  #if defined(__sun)5  #define PREFIX_SIZE (sizeof(long long))6  #else7  #define PREFIX_SIZE (sizeof(size_t))8  #endif9  #endif

If have_malloc_size is defined, the prefix_size is 0. .Here, have_malloc_size is used to determine whether the system has the malloc_size function. This have_malloc_size macro is in config .H is defined as follows:

1 /* Use tcmalloc‘s malloc_size() when available.2 * When tcmalloc is used, native OSX malloc_size() may never be used because3 * this expects a different allocation scheme. Therefore, *exclusively* use4 * either tcmalloc or OSX‘s malloc_size()! */5  #if defined(USE_TCMALLOC)6 #include <google/tcmalloc.h>7 #if TC_VERSION_MAJOR >= 1 && TC_VERSION_MINOR >= 68 #define HAVE_MALLOC_SIZE 19 #define redis_malloc_size(p) tc_malloc_size(p)10 #endif11 #elif defined(__APPLE__)12 #include <malloc/malloc.h>13 #define HAVE_MALLOC_SIZE 114 #define redis_malloc_size(p) malloc_size(p)15 #endif

If Google's tcmalloc library is used, redis_malloc_size corresponds to the tc_malloc_size function of the tcmalloc library. .If it is compiled on Apple's Mac, redis_malloc_size corresponds to malloc_size. .The redis_malloc_size function is to obtain the size of the memory block pointed to by the parameter P. .The tcmalloc library is in Google's Google-perftools library. It is said that the Library is incredibly efficient in memory management. .However, this database is written in C ++, while redis is written in C. It is still a bit unawesome to both databases. . ..

If the malloc_size function is not available, prefix_size is defined with the length of the long type on the Solaris System, and the length of the size_t is used in other systems..

Next, define the following macros. These macros are used to map the allocation functions in the library to the standard library if the tcmalloc library is used. .The following functions can directly use the name of the standard library function. You do not need to change it when changing the database.

1/* explicitly override malloc/free etc when using tcmalloc. */2 # If defined (use_tcmalloc) 3 # define malloc (size) tc_malloc (size) 4 # define calloc (count, size) tc_calloc (count, size) 5 # define realloc (PTR, size) tc_realloc (PTR, size) 6 # define free (PTR) tc_free (PTR) 7 # Two macros under endif are used to update the zmalloc_allocations array. Update_zmalloc_stat_alloc is used to update the allocated size when the memory is allocated, and update_zmalloc_stat_free is used to delete the corresponding records when the memory is released. 1 # define update_zmalloc_stat_alloc (_ n ,__ size) do {2 size_t _ n = (_ n); 3 size_t _ stat_slot = (_ size <zmalloc_max_alloc_stat )? _ SIZE: zmalloc_max_alloc_stat; 4 If (_ N & (sizeof (long)-1) _ n + = sizeof (long)-(_ N & (sizeof (long) -1); 5 If (zmalloc_thread_safe) {6 pthread_mutex_lock (& used_memory_mutex); 7 used_memory + = _ n; 8 zmalloc_allocations [_ stat_slot] ++; 9 pthread_mutex_unlock (& used_memory_mutex); 10} else {11 used_memory + = _ n; 12 zmalloc_allocations [_ stat_slot] ++; 13} 14} while (0)

The first parameter _ n of update_zmalloc_stat_alloc is the actual memory size obtained from the system, and the second parameter is the memory size requested by the program. .Update_zmalloc_stat_alloc first checks the subscripts of the memory size requested by the program in the zmalloc_allocations array. .If the memory size is greater than the length of the zmalloc_allocations array-1, the corresponding subscript is the last one. Then, alignment the actually allocated memory size to an integer multiple of the long length (malloc usually considers alignment, and the actually allocated memory size will also vary depending on alignment, will be introduced later) .Finally, record the actual allocation size in used_memory and add one at the zmalloc_allocations location. .If thread security is set here, two static variables should be locked before recording. .

1 #define update_zmalloc_stat_free(__n) do { 2 size_t _n = (__n); 3 if (_n&(sizeof(long)-1)) _n += sizeof(long)-(_n&(sizeof(long)-1)); 4 if (zmalloc_thread_safe) { 5 pthread_mutex_lock(&used_memory_mutex); 6 used_memory -= _n; 7 pthread_mutex_unlock(&used_memory_mutex); 8 } else { 9 used_memory -= _n; 10 } 11 } while(0)

Update_zmalloc_stat_free and update_zmalloc_stat_alloc are similar, but only the used_memory value is reduced. .
For zmalloc, zarloc, zrealloc, and zfree functions, it is just a simple encapsulation of functions in the standard library. .In addition to calling the memory allocation function of the standard library (or tcmalloc Library), the work done is to make appropriate records for each memory allocation and release. .If the system has the malloc_size function, you can directly call the two macros above. .If you do not have the malloc_size function, you must record the size of the memory block in the prefix_size area of the allocated memory header. .The code is very simple: 1 * (size_t *) PTR) = size; Convert PTR to a pointer of the size_t type, and then assign the size value to the memory to which it points .I don't think it is necessary to distinguish the system when prefix_size is defined, because the type of memory size is size_t. .The previous macro judgment is a bit redundant. .The size here is the size of the returned memory area, excluding the header of the storage size.
Two steps are required to read the memory block size: 1 realptr = (char *) ptr-PREFIX_SIZE;
2 oldsize = * (size_t *) realptr); The ret_ptr is passed in by the program, and real_ptr is obtained by subtracting prefix_size. .Read the size from the memory pointed to by real_ptr.
The other functions are also very straightforward. .The last section describes the zmalloc_get_rss () function. This function is used to obtain the RSS of a process. Why does Shenma use RSS? Google Reader? Apparently not. . .It is called resident set size, which refers to the actual use of physical memory (including the memory occupied by the shared library) .In Linux, you can obtain it by reading the/proc/Pid/STAT file. The PID is the process Number of the current process. .The number of bytes read is not the number of bytes, but the number of pages in the memory. The system calls sysconf (_ SC _pagesize) to obtain the memory page size of the current system. .UNIX systems seem to be directly available through task_info, which is much simpler than Linux systems.
After obtaining the RSS of a process, you can calculate the size of the memory fragments of the current data. Use RSS to divide it by used_memory. .RSS contains all the memory usage of processes, including code, shared libraries, and stacks. However, because redis usually consumes much more data in the memory than the memory occupied by the data, this simple calculation is more accurate. .
There is a problem here: How much memory is allocated for the program, and where is the memory fragmentation? In fact, when calling malloc, malloc does not allocate memory strictly according to the parameter value. .For example, if a program requests only one byte of memory, malloc will not allocate only one byte. Generally, based on memory alignment and other considerations, malloc will allocate four bytes .In this way, if the program requests a large number of 1 byte memory, it will actually use 4 times of the requested memory. It is a waste of malloc for small memory allocation. .Therefore, fragments are generated here.
In general, redis's memory management is simple and crude, and there is no complicated reference counting technology such as Shenma. .However, in many cases, simplicity is often efficient and reasonable. Redis memory usually contains several GB of data. This method is fast and the statistical results are accurate. .   Coding trick:

#define update_zmalloc_stat_alloc(__n) do { \    size_t _n = (__n);     if (_n&(sizeof(long)-1)) _n += sizeof(long)-(_n&(sizeof(long)-1));     if (zmalloc_thread_safe) {         update_zmalloc_stat_add(_n);     } else {         used_memory += _n;     } } while(0)

First, round up the low position of _ n, and then change _ n to a multiple of sizeof (long). For example, for a 32-bit system, sizeof (long) = 100 (Binary ), after _ n is rounded up, both the lower two digits become 0. Alignment between allocated space and memory