Redis source code interpretation of memory management ———— Zmalloc file

This article mainly records I am looking at the Redis source code of some understanding and ideas. Due to limited skill, there is definitely a malfunction, so. I hope the master gives a correction.

The first article is a memory-related introduction. Because I like to look at some components first, then look at the overall process.

Let's start with the code.

Header file

Primarily provides basic memory allocation and release capabilities

void *zmalloc (size_t size);//primary provides memory allocation and release of the underlying function void *zcalloc (size_t size); void *zrealloc (void *ptr, size_t size); void Zfree (void *ptr), char *zstrdup (const char *s), size_t zmalloc_used_memory (void), void zmalloc_enable_thread_safeness ( void); void Zmalloc_set_oom_handler (void (*oom_handler) (size_t)); float zmalloc_get_fragmentation_ratio (void); size_ T zmalloc_get_rss (void); size_t zmalloc_get_private_dirty (void); void Zlibc_free (void *ptr); #ifndef have_malloc_ sizesize_t zmalloc_size (void *ptr); #endif

0. Preface

In this code we can see the Have_atomic macro definition, what is the function of it.

This file mainly provides the management of Tcmalloc and Jemalloc memory.

Tcmalloc is part of Google Perftool. Unlike a typical memory pool, which deals directly with the OS, the OS recycles when memory is idle (the STL memory pool is not recycled), and at the same time uses TLS (thread local storage) to manage the pool of memory, avoiding the need to synchronize the allocated memory within a thread.

Jemalloc and Tcmalloc are similar, the author Jason Evans is free BSD developer, performance and usage is not comparable with Tcmalloc.

Tcmalloc is more convenient to integrate with Google Perftool for performance evaluation.

Zmalloc primarily provides encapsulation of the malloc function, assuming that the malloc function is glibc, then the allocated memory is the length + memory to be allocated. Then put the head into the size

1. Request Memory

void *zmalloc (size_t size) {    void *ptr = malloc (size+prefix_size);    if (!ptr) zmalloc_oom_handler (size), #ifdef have_malloc_size    update_zmalloc_stat_alloc (Zmalloc_size (PTR));    return ptr; #else    * ((size_t*) ptr) = size;    Update_zmalloc_stat_alloc (size+prefix_size);    Return (char*) ptr+prefix_size; #endif}

Update_zmalloc_stat_alloc (Zmalloc_size (PTR)); is to align its own memory first, assuming that long is a 4-bit aligned to an integer multiple of 4. Then record the size of the memory into a global variable

For example, the following code

#define UPDATE_ZMALLOC_STAT_ALLOC (__n) do {     size_t _n = (__n);     if (_n& (sizeof (long)-1)) _n + = sizeof (long)-(_n& (sizeof (long)-1));     if (zmalloc_thread_safe) {         update_zmalloc_stat_add (_n);     } else {         used_memory + = _n;}     } while (0)

Memory has many different ways of doing it, and he is also very interesting.

The use of long can be either 64 machine or 32-bit machine. Assume that a mutually exclusive lock is used in a multithreaded situation.

Assuming that no macro have_malloc_size is defined, it is four bytes memory in front of the memory. Finally, the start address of the memory is returned.
Similarly

void *zcalloc (size_t size)

This method encapsulates the function of the Calloc function.
2. Request Memory again
Code such as the following

void *zrealloc (void *ptr, size_t size) {#ifndef have_malloc_size    void *realptr; #endif    size_t oldsize;    void *newptr;    if (ptr = = NULL) return zmalloc (size), #ifdef have_malloc_size    oldsize = Zmalloc_size (PTR);    Newptr = ReAlloc (ptr,size);    if (!newptr) zmalloc_oom_handler (size);    Update_zmalloc_stat_free (oldsize);    Update_zmalloc_stat_alloc (Zmalloc_size (newptr));    return newptr; #else    realptr = (char*) ptr-prefix_size;    oldsize = * ((size_t*) realptr);    Newptr = ReAlloc (realptr,size+prefix_size);    if (!newptr) zmalloc_oom_handler (size);    * ((size_t*) newptr) = size;    Update_zmalloc_stat_free (oldsize);    Update_zmalloc_stat_alloc (size);    Return (char*) newptr+prefix_size; #endif}

The method is to request memory of the specified size again for a specified amount of memory. The address has not changed.

3. Freeing up memory

void Zfree (void *ptr) {#ifndef have_malloc_size    void *realptr;    size_t oldsize; #endif    if (ptr = = NULL) return; #ifdef have_malloc_size    update_zmalloc_stat_free (Zmalloc_size ( PTR));    Free (PTR); #else    realptr = (char*) ptr-prefix_size;    oldsize = * ((size_t*) realptr);    Update_zmalloc_stat_free (oldsize+prefix_size);    Free (realptr); #endif}

4. Copy memory, encapsulates the memcpy function

Char *zstrdup (const char *s) {    size_t L = strlen (s) +1;    Char *p = Zmalloc (l);    memcpy (p,s,l);    return p;}

5 getting the RSS physical memory valueBy looking at the 23rd line of the process's/proc/pid/stat file.

size_t Zmalloc_get_rss (void) {    int page = sysconf (_sc_pagesize);    size_t RSS;    Char buf[4096];    Char filename[256];    int fd, Count;    Char *p, *x;    snprintf (filename,256, "/proc/%d/stat", Getpid ());    if (fd = open (filename,o_rdonly)) = =-1) return 0;    if (read (fd,buf,4096) <= 0) {        close (FD);        return 0;    }    Close (FD);    p = buf;    Count = 23; /* RSS is the 24th field In/proc/<pid>/stat *    /while (P && count--) {        p = strchr (P, ');        if (p) p++;    }    if (!p) return 0;    x = STRCHR (P, ');    if (!x) return 0;    *x = ' + ';    rss = Strtoll (p,null,10);    RSS *= page;    return RSS;}

6. Memory Utilization

/* fragmentation = rss/allocated-bytes */float zmalloc_get_fragmentation_ratio (void) {    return (float) zmalloc_get_ RSS ()/zmalloc_used_memory ();}

7. Atomic additions under multi-threading

#ifdef have_atomic#define Update_zmalloc_stat_add (__n) __sync_add_and_fetch (&used_memory, (__n)) #define Update_ Zmalloc_stat_sub (__n) __sync_sub_and_fetch (&used_memory, (__n)) #else # define UPDATE_ZMALLOC_STAT_ADD (__n) do {     Pthread_mutex_lock (&used_memory_mutex);     Used_memory + = (__n);     Pthread_mutex_unlock (&used_memory_mutex); } while (0) #define UPDATE_ZMALLOC_STAT_SUB (__n) do {     pthread_mutex_lock (&used_memory_mutex);     Used_memory-= (__n);     Pthread_mutex_unlock (&used_memory_mutex); } while (0) #endif

The memory management of the Redis memory using Google and Jemalloc is very efficient.
The external interface is also more concise.
Worth learning.

Many other articles. Welcome attention:

Redis source code interpretation of memory management ———— Zmalloc file