"Heap fragmentation" and Solutions

 

Recently, I used C to develop a server software. The Code requires frequent memory allocation, and the third-party libraries GLIB and libCurl also have different memory allocation methods.

In the development process, I found a problem, that is, under the stress test, the program memory consumption continues to grow until 3 GB of virtual storage space is used up, thus malloc fails. In my code, the memory is released in a timely manner after use, so this phenomenon should not occur.

Therefore, it is suspected that there is a memory leak, But no matter valgrind or dmalloc is used, the memory leakage issue cannot be detected.

Distress ......

In desperation, I doubt whether there are heap fragments. The so-called "heap fragmentation" means a large amount of memory is allocated in the heap. These memories are used up and released, but a small part of the memory is not released at the top of the "Heap, as a result, the entire "Heap" space is very large. No valuable information was found for "heap fragmentation.

Later I read "Flying" (blog http://www.cublog.cn/u/30686/index.html) Summary of a document about Linux memory.

 

The following code verifies the "memory holes" or more precisely "heap fragments": (after running, send the USR1 signal to the process multiple times. After each signal is sent, cat/proc/[pid]/maps to observe heap changes)

 

# Include <stdlib. h>
# Include <stdio. h>
# Include <string. h>
# Include <unistd. h>
# Include <signal. h>
# Include <malloc. h>

Int memsize = 1024*4;

Void sig_handler (INT sig)
...{
}

Int main (INT argc, char ** argv)
...{

Signal (SIGUSR1, sig_handler );

Pause ();

Char * P [11];
Int I;

For (I = 0; I <10; I ++ )...{
P [I] = malloc (memsize );
Strcpy (P [I], "123 ");

}

P [10] = malloc (memsize );
Strcpy (P [10], "123 ");


Pause ();

Printf ("free first 10 block ");

For (I = 0; I <10; I ++ )...{
Free (P [I]);
}

Pause ();

Printf ("free last 1 block ");

Free (P [10]);

Pause ();

Return 0;
}

 

 

In this Code,

First, allocate 11 4 K space blocks.

Then, release the first 10 blocks of space,

Through/proc/[pid]/maps, we can see that the heap space has not changed, because the last space has not been released. This forms a "memory hole ", from the programmer's perspective, I think that only 4 K space is used, but from the operating system's perspective, I think you have used 44 K space.

After releasing 11th blocks of space, we can see that the heap space is reduced, but the 44 K space is not returned to the operating system. This involves the implementation of glibc memory management, for more information, see Linux memory optimization.

Finally, let's talk about a solution to "heap fragmentation:

When malloc () is allocating memory, BRK () is used to allocate <128 kb of memory on the heap; for memory ≥ 128 kb, mmap2 () is used to map another space; for the Space Generated by mmap2 (), free () will be released without the "heap fragmentation" problem. Therefore, one solution is "try to use the memory allocation method of mmap2 ".

Therefore, you need to lower the threshold for using mmap2 () to allocate memory for malloc (), which can be set through mallopt (m_mmap_threshold. Mallopt () is an API not made public by glibc.

In this way, I lowered the threshold of mmap2 () to 32 KB (because a large number of 32 KB memory blocks are allocated in my code ), test results show that the memory usage is maintained at a low value.