Read Redis C Programming II: Memory allocation

Memory allocation is a core issue for C programs, and many open source software will customize their own memory allocation strategy for their software needs, and Redis is no exception. In general, however, Redis is not specifically about memory allocation, and the most important feature of its memory allocation strategy is the addition of statistical information. After all, Redis is a memory database, knowing how much memory is being used, and how much memory is available is a problem that it needs to focus on. Let's look at what's inside Zmalloc.

First, inside the zmalloc.h.

#if defined (use_tcmalloc) #define ZMALLOC_LIB ("tcmalloc-" __xstr (tc_version_major) "." __xstr (Tc_version_minor)) # Include <google/tcmalloc.h> #if (tc_version_major = = 1 && tc_version_minor >= 6) | | (Tc_version_major > 1) #define have_malloc_size 1#define zmalloc_size (P) tc_malloc_size (p) #else #error "newer VERSION of Tcmalloc required "#endif #elif defined (use_jemalloc) #define ZMALLOC_LIB (" jemalloc-"__xstr (jemalloc_version_major ) "." __xstr (Jemalloc_version_minor) "." __xstr (Jemalloc_version_bugfix)) #include <jemalloc/jemalloc.h> #if ( Jemalloc_version_major = = 2 && jemalloc_version_minor >= 1) | | (Jemalloc_version_major > 2) #define have_malloc_size 1#define zmalloc_size (P) je_malloc_usable_size (p) #else #error "Newer version of Jemalloc required" #endif #elif defined (__apple__) #include <malloc/malloc.h> #define Have_malloc _size 1#define zmalloc_size (P) malloc_size (p) #endif #ifndef zmalloc_lib#define zmalloc_lib "libc" #endif

Here you can see, for redis memory allocation, there are several optional strategies, Google's Tmalloc,facebook Jemalloc, both of which are the excellent implementation of malloc, the difference. More two malloc discussions can be found in blog http://blog.sina.com.cn/s/blog_51df3eae01016peu.html. Alternatively, you can choose libc native malloc, but you need to add 4 bytes to the memory allocation length before each memory allocation to achieve the Malloc_size functionality. The source code has already brought the source code of the Jemalloc, can see the official or relatively respected use Jemalloc.

The most important part of the following is how memory usage statistics are implemented:

#if defined (__atomic_relaxed) #define UPDATE_ZMALLOC_STAT_ADD (__n) __atomic_add_fetch (&used_memory, (__n), __ atomic_relaxed) #define UPDATE_ZMALLOC_STAT_SUB (__n) __atomic_sub_fetch (&used_memory, (__n), __ATOMIC_RELAXED) # Elif defined (have_atomic) #define UPDATE_ZMALLOC_STAT_ADD (__n) __sync_add_and_fetch (&used_memory, (__n)) #define Update_zmalloc_stat_sub (__n) __sync_sub_and_fetch (&used_memory, (__n)) #else # define UPDATE_ZMALLOC_STAT_ADD (__     N) do {pthread_mutex_lock (&used_memory_mutex);     Used_memory + = (__n); Pthread_mutex_unlock (&used_memory_mutex);     } while (0) #define UPDATE_ZMALLOC_STAT_SUB (__n) do {pthread_mutex_lock (&used_memory_mutex);     Used_memory-= (__n); Pthread_mutex_unlock (&used_memory_mutex);     } while (0) #endif # define UPDATE_ZMALLOC_STAT_ALLOC (__n) do {size_t _n = (__n);     if (_n& (sizeof (long)-1)) _n + = sizeof (long)-(_n& (sizeof (long)-1));     if (Zmalloc_thread_safe) {update_zmalloc_stat_add (_n); } ElSe {used_memory + = _n;     }} and (0) #define UPDATE_ZMALLOC_STAT_FREE (__n) do {size_t _n = (__n);     if (_n& (sizeof (long)-1)) _n + = sizeof (long)-(_n& (sizeof (long)-1));     if (Zmalloc_thread_safe) {update_zmalloc_stat_sub (_n);     } else {used_memory-= _n; }} while (0)

Most of the systems we use already provide added subtraction functions for thread safety, such as __atomic_add_fetch, __atomic_sub_fetch, or __sync_add_and_fetch,__sync_sub_and_ Fetch, if not implemented also does not matter, Zmalloc inside provides the above alternative scheme, and uses the macro function to unify together. In the alternative we see a more interesting phenomenon, you allocate n size of memory, but in the statistics is not added N, but if (_n& (sizeof (long)-1)) _n + = sizeof (long)-(_n& (sizeof (long) )-1)) such a way. This is why, after looking at the data, this is the memory alignment, so-called memory alignment, is the system in the actual allocation of memory for efficiency considerations, will allocate some memory to the pointer. For a 32-bit machine, 4-byte alignment can increase the speed of the CPU access, such as a long variable, if the 4-byte boundary is stored, then the CPU reads two times, so the efficiency is low. However, using a 1-byte or 2-byte alignment in a 32-bit machine can cause variable access speed to be reduced. So this takes into account the processor type, and also the compiler type. The default is 4-byte alignment in VC, and GNU GCC is also the default 4-byte alignment. So in the default environment, the actual allocation size is calculated as 4-byte alignment.

A static variable is designed to indicate whether thread safety is enabled, the amount of memory that a representative uses, and a lock that is used by the thread lock for statistics.

static size_t used_memory = 0;
static int zmalloc_thread_safe = 0;
pthread_mutex_t Used_memory_mutex = Pthread_mutex_initializer;

The system realizes four core allocation function zmalloc,zcalloc,zrelloc,zfree, corresponding to the standard malloc,calloc,relloc,free. and memory space statistics, such as thread safety control functions, this part of the source code is relatively easy to understand not elaborate. In addition to the ZMALLOC_GET_RSS, its role is to directly query the memory usage under the Proc/pid/state file of the UNIX series system (this is part of the process all allocated memory includes the global tatic and the variables in the stack). And divide to get their memory usage efficiency. Finally is Zmlloc_private_diry, it is the query system in the Proc/self/smap, that is, the process fork after the change, just fork out is Private_clean, the change is Private_dirty, The role here is to query the child process memory increment in the Redis related program process

Read Redis C Programming II: Memory allocation