Memory leakage caused by glibc memory allocation mechanism"

Author: Chuanhui
| Reprinted, but the original source, author information, and copyright statement of the article must be indicated in hyperlink form

Link: http://www.nosqlnotes.net/archives/105

A memory module in the database-like system we are developing has a suspected "memory leak" problem. The phenomenon is as follows: the memory module has not been returned after the memory is released.
System, for example, the memory occupied by the memory module is 10 Gb. After the memory is released, you can use the top command or/proc/Pid/status to check whether the occupied memory is still 10 Gb.
5 GB, sometimes 3 GB, etc. The memory release behavior is uncertain.

First, let's talk about the memory management mechanism of the memory module. Our memory management is very simple. We use a global fixed-length memory pool. Each memory block is 64 KB. If the applied memory is smaller than or equal to 64 KB
Obtain a memory block from the idle linked list of the memory pool, and return the idle linked list when the memory is released. If the applied memory is larger than 64 KB, it is directly obtained through the operating system malloc and free. Some data
The structure involves the management of many small objects, such as hash tables and B-tree. These data structures obtain the memory from the global memory pool and then organize the data according to the characteristics of the data structure. To increase memory application/release
To reduce lock conflicts, and keep an 8 MB memory block for each thread separately. Each thread preferentially obtains the memory from the exclusive 8 MB memory block of the thread, when the exclusive memory is insufficient, it is obtained from the global memory pool.

Since all our memory application/release operations need to be performed through the global memory pool, we add the memory statistics function for each sub-module in the global memory pool: when each sub-module applies for memory
The module ID is sent to the global memory pool for statistics. After the problem is reproduced, it is found that the statistical results of the global memory pool are as expected. Therefore, it is suspected that the operation is performed by the operating system or glibc.

The memory management mechanism of glibc in Linux is roughly as follows:

From the operating system perspective, the memory allocation of processes is completed by two system calls: BRK and MMAP. BRK is the highest address pointer _ edata to the high address of the Data Segment (. Data)
Push, MMAP finds a free space in the virtual address space of the process. Among them, the memory allocated by MMAP is released by munmap, and the memory will be immediately returned to the operating system when it is released;The memory allocated by BRK can be released only after the high address memory is released.
.
That is to say, if you have applied for two blocks of memory A and B through BRK, before B is released, a cannot be released and it is still occupied by the process. It is suspected that "Memory leakage" is detected through top check ". By default
Memory Allocation equal to kb calls MMAP/mummap, and memory requests smaller than kb call sbrk (you can set m_mmap_threshold to call
). For details about the memory management mechanism, refer to the articles shared by Baidu.
.

Our memory module applies for/releases the memory in 2 MB. We should use MMAP and munmap to allocate and release the memory, the memory will not be occupied by the process after it is released. With the help of kernel colleagues, after a long period of analysis and positioning, we found the new features of glibc:M_mmap_threshold can be dynamically adjusted
. M_mmap_threshold is dynamically adjusted between kb to 32 MB (32-bit) or 64 MB (64-bit). After each request and release a 2 MB memory, m_mmap_threshold is adjusted to a value between 2 m and 2 m + 4 K. For details, refer to the patch description of glibc.
). For example:

Char * no_used = new char [2*1024*1024];

Memset (no_used, 0xfe, 2*1024*1024 );

Delete [] no_used;

// The m_mmap_threshold value is adjusted to a value between 2 m and 2 m + 4 K. For subsequent applications, <= 2*1024*1024 of memory blocks will go through sbrk instead of MMAP.

After learning about this phenomenon, we found the original "Memory leakage"
Because the value of m_mmap_threshold is dynamically adjusted, the subsequent 2 MB memory application is implemented through sbrk, And the sbrk can be released only after the high address memory is released.
. You can explicitly set m_mmap_threshold or m_mmap_max to disable the dynamic adjustment feature of m_mmap_threshold to avoid the above questions.
Question.

Of course, MMAP calls may lead to page breaks in the process. To improve performance, the common practices are as follows:

1. Change the dynamic memory to static. For example, use the memory pool technology or assign a certain size to each thread at startup. For example, 8 MB memory will be used directly in the future;

2. Disable MMAP memory calling and disable glibc memory tightening to return the memory to the system. glibc is equivalent to implementing a memory pool function. You only need to add two lines of code when the process starts:

Mallopt (m_mmap_max, 0); // disable malloc from calling MMAP to allocate memory

Mallopt (m_trim_threshold, 0); // disable memory indent. The memory applied by the sbrk will not be returned to the operating system after it is released.

Highlights:

In the process of tracing the "Memory leakage" problem, try to use the glibc hook function (malloc hook
)

Measure the memory size of malloc and free: Apply for 8 more bytes when malloc is used, of which 4 are the record length and 4 are the records.
When magic_num, malloc, and free are used, the amount of memory applied and released by the process is counted. The practice shows that no matter whether the custom hook function is locked or not, the malloc and free hook functions are
Multi-thread running is not normal, and other students also find the same problem (malloc hook multithreading Problem
).

 

 

Http://stackoverflow.com/questions/2020997/using-glibc-malloc-hooks-in-a-thread-safe-manner