Linux under GLIBC memory management __linux

I don't remember the references.

1 Background Introduction

A suspected "memory leak" problem: malloc application of memory, free release after the operating system has not been returned, such as memory module occupied by 10GB of memory, after the release of memory, through the top command or/proc/pid/status view occupied memory is sometimes still 10G, Sometimes 5G, sometimes 3G, etc, the behavior of memory release is uncertain. 2 malloc ()/free (), mmap (), BRK (), user program-->glibc the relationship between -->linux

1) malloc ()/free () is the two memory allocation/release function commonly used in C language, but ANSI C does not specify how they should be implemented specifically. So on each system level platform (Windows, Mac, Linux, etc.), the underlying memory manipulation of the function is not the same.
2 under Linux, malloc ()/free () is the implementation of the GLIBC Library is responsible for, it will be based on a certain strategy and the underlying system communication (call the system API), so the user program will not directly interact with the Linux kernel, Instead, it is managed by GLIBC, which provides the default memory manager. The relationship is: User program---->GLIBC---->linux kernel.
3) glibc uses Ptmalloc as the implementation of its memory manager.

* BRK Chunk list can be released linearly from top. The chunk, which is released in the middle, cannot be returned to the OS, but is chained to the Bins/fast bins container.

* Mmap allocated memory, equal to directly from the physical memory mapped a piece over, the release of this memory, you can directly return to the OS.

* A piece of memory for request is allocated by BRK or by mmap, which is determined by the GLIBC strategy mechanism. There is a threshold, you can adjust this policy, the default is less than threshold allocated by BRK, greater than or equal to be allocated by MMAP, threshold default is 128KB. GLIBC specifically from that version to support the dynamic adjustment threshold need to investigate. By default, on a 64-bit system, BRK can be dynamically tuned to 128kb to 32MB. The adjustment strategy can be summed up as follows: found that the available memory of the top can release more than 256KB, the threshold will be adjusted to 256kb, and so on until 32MB.

glibc uses two kinds of user program memory management mechanism, considering with the system bottom communication cost, if the direct manipulation large amount of small block memory, frequently communicates with the system call, may reduce the program the running efficiency. The small block of memory into the BRK maintenance of the heap, equal to the implementation of a cache, used up can save up, then can be returned to the system together. But because of its implementation, only one pointer to the top of the heap is maintained. Therefore, you want the memory return system to be returned from the top down, in turn. Under this mechanism, if the heap has a block of memory has been occupied, the bottom of all the memory is useless, but this design determines that memory can not be returned to the system at this time, so there is a "hole (Hole)" problem. This design is especially problematic for user programs that frequently request/release small chunks of memory due to business requirements. Although glibc has developed this somewhat "tough" memory management program, however, some methods are provided to allow the relevant thresholds to be adjusted (threshold), although it is not possible to interfere with the management of memory, but these methods can be used to determine "how big, How small" and "how much to return" Such problems as these. 3 mallopt () and Malloc_trim (0) 1) mallopt is a function that regulates the relevant thresholds

URL: http://man7.org/linux/man-pages/man3/mallopt.3.html

#include < malloc.h > int mallopt (int param, int value); M_mmap_threshold for allocations greater than or equal to the limit specified (in bytes) by m_mmap_threshold that can ' t b E Satisfied from the "free" list, the Memory-allocation functions employ mmap (2) instead of increasing of the program break
Using SBRK (2).  allocating memory using MMAP (2) has the significant advantage this allocated memory blocks can always be Indepe 
Ndently released back to the system. 
(By contrast, the heap can being trimmed only if memory are freed at the top end.)  On the other hand, there are some disadvantages to the "use of" mmap (2): deallocated space isn't placed on the free list 
For reuse by later allocations; Memory May is wasted because mmap (2) allocations must be page-aligned; And the kernel must perform the expensive task of zeroing out memory via allocated (2). Balancing these factors leads to a default setting of 128*1024 for the m_mmap_threshold. The lower LiMIT for this parameter is 0.

The upper limit is default_mmap_threshold_max:512x1024 on 32-bit systems or 4x1024x1024xsizeof (long) on 64-bit systems. Note:nowadays, GLIBC uses a dynamic mmap threshold by default. The initial value of the threshold is 128x1024 but when blocks larger than the current threshold and less than or Equa
 L to Default_mmap_threshold_max are freed, the THRESHOLD are adjusted upwards to the size of the freed block. When dynamic Mmap thresholding was in effect, the threshold for trimming the heap was also dynamically to be adjusted
 E The dynamic mmap threshold. Dynamic adjustment of the mmap threshold is disabled if any of the M_trim_threshold, M_top_pad, M_mmap_threshold, or m_m
 Map_max parameters is set.

The mallopt () function adjusts parameters the behavior of the Memory-allocation functions (3). The param argument specifies the parameter to is modified, and value specifies the new value for that parameter.

The following values can be specified for param:

m_check_action

Setting This parameter controls I glibc responds when various kinds of programming to errors are detected (e.g., freeing th e same pointer twice). The 3 least significant bits (2, 1, and 0) of the value assigned to this parameter determine the glibc behavior, as follow S

Bit 0 If This bit is set,

Then print a one-line message on stderr that provides details about the error. The message starts with the string "* * * GLIBC detected * * *" followed by the program name, the name of the Memory-allocati On function in which the error is detected, a brief description of the error, and the memory address where the error is Detected.

Bit 1 If This bit is set,

Then, after printing any error message specified by bit 0, which is terminated by calling abort (3). In glibc versions since 2.4, if bit 0 are also set, then, between printing the error message and aborting, the program also Prints a stack trace in the manner of BackTrace (3), and prints the process ' s memory mapping in the style of/proc/[pid]/m APS (Proc (5)).

Bit 2 (since glibc 2.4),

This bit has a effect only if bit 0 is also set. If This bit is set, then the "one-line message describing" The error is simplified to contain just the name of the function Where the error was detected and the brief description of the error.

The remaining bits in value are ignored.

combining the above details, the following numeric values are meaningful for m_check_action:

0 Ignore error conditions; Continue execution (with undefined results).
1 Print a detailed error message and continue execution.
2 Abort the program.
3 Print detailed error message, stack trace, and memorymappings, and abort.
5 Print A simple error message and continue execution.
7 Print Simple error message, stack trace, and memory mappings, and abort.

Since glibc 2.3.4, the default value for the M_check_action parameter is 3. In glibc version 2.3.3 and earlier, the default value is 1. Using a nonzero m_check_action value can be useful because otherwise a crash could happen much later, and the true cause of The problem is then very hard to track down.

M_mmap_max

This parameter specifies the maximum number of allocation requests, the May is simultaneously serviced using MMAP (2). This parameter exists because some systems have a limited number of internal tables for use by Mmap (2), and using the more tha n a few of them may degrade performance.

The default value is 65,536, a value which has no special significance and which servers only as a safeguard. Setting This parameter to 0 disables the "Use of Mmap" (2) for servicing large allocation.

M_mmap_threshold

For allocations greater than or equal to the limit specified (in bytes) by m_mmap_threshold that can ' t be satisfied fro M The free list, the memory-allocation functions employ mmap (2) instead of increasing of the program break using SBRK (2).

Allocating memory using MMAP (2) has the significant advantage this allocated memory blocks can always be independently Released to the system. (By contrast, the heap can being trimmed only if memory are freed at the top end.) On the other hand, there are some disadvantages to the "use of" mmap (2): deallocated space isn't placed on the free list fo R reuse by later allocations; Memory May is wasted because mmap (2) allocations must be page-aligned; And the kernel must perform the expensive task of zeroing out memory via allocated (2). Balancing these factors leads to a default setting of 128*1024 for the m_mmap_threshold. The lower limit for this parameter is 0. The upper limit is default_mmap_threshold_max:512*1024 on 32-bit systems or 4*1024*1024*sizeof (long) on 64-bit systems. Note:nowadays, GLIBC uses a dynamic mmap threshold by default. The initial value of the threshold is 128*1024, but when blocks larger than the current threshold and less than or equal t o DefauLt_mmap_threshold_max are freed, the THRESHOLD is adjusted upward to the size of the freed block. When dynamic Mmap thresholding was in effect, the threshold for trimming the heap was also dynamically to be adjusted The dynamic mmap threshold. Dynamic adjustment of the mmap threshold is disabled if any of the M_trim_threshold, M_top_pad, M_mmap_threshold, or M_mma P_max parameters is set.

M_mxfast (since glibc 2.3)

Set the upper limit for memory allocation requests of that are using "satisfied". (The measurement unit for this parameter is bytes.) Fastbins are storage areas that hold deallocated blocks of memory of the same size without merging adjacent free blocks. Subsequent reallocation of blocks of the same size can be handled very to quickly by allocating to the fastbin, although me Mory fragmentation and the overall memory footprint of the program can increase. The default value is 64*sizeof (size_t)/4 (i.e., parameter, 32-bit architectures). The range for this parameter are 0 to 80*sizeof (size_t)/4. Setting M_mxfast to 0 disables the use of Fastbins.

M_perturb (since glibc 2.4)

If This parameter are set to a nonzero value, then bytes of allocated memory (+ other than allocations via Calloc (3)) are INI Tialized to the complement of the ' value in ' least significant byte of value, and when allocated memory is released Usin G Free (3), the freed bytes are set to the least significant byte of value. This can is useful the for detecting errors where programs incorrectly rely on allocated memory being initialized to zero, or Reuse values in memory that has already been freed.

M_top_pad

This is parameter defines the amount of padding to employ when calling SBRK (2) to modify the program break. (The measurement unit for this parameter is bytes.) This parameter has a effect in the following Circumstances:when the program break is increased, then M_top_pad bytes Added to the SBRK (2) request. When the heap was trimmed as a consequence of calling free (3) (for the discussion of M_trim_threshold) is preserved in the top of the heap.

In either case, the amount of padding are always rounded to a system page boundary. Modifying M_top_pad is a trade-off between increasing the number of system calls (when the parameter was set low) and Wasti ng unused memory at the top of the heap, when the parameter was set high. The default value for this parameter is 128*1024.

M_trim_threshold

When the amount of contiguous free memory in the top of the heap-grows sufficiently large, free (3) employs SBRK (2) to Rele ASE this memory the system. (This can is useful in programs which continue to execute for a long period after freeing a significant amount of memory.) The M_trim_threshold parameter specifies the minimum size (in bytes), that's, the memory must reach before SBRK (2) is used to trim the heap.

The default value for this parameter is 128*1024. Setting m_trim_threshold to-1 disables trimming completely. Modifying M_trim_threshold is a trade-off between increasing the number of system calls (when the parameter is set low) D wasting unused memory at the top of the heap (when the parameter was set high).

Environment variables

A number of environment variables can be defined to modify some of the same parameters as are by controlled (). Using These variables has the advantage, the source code of the program, need not is changed. To is effective, these variables must is defined before the the "the" a memory-allocation function. (If the same parameters are adjusted via Mallopt (), then the mallopt () settings take precedence.) For the security reasons, which variables are ignored in Set-user-id and Set-group-id.

The environment variables are as follows (note the trailing underscore in the end of the ' name of each variable):

Malloc_check_

This environment variable controls the same parameter as mallopt () m_check_action. If this variable be set to a nonzero value, then a special implementation of the Memory-allocation functions is used. (This is accomplished using the Malloc_hook (3) feature.) This implementation performs additional error checking, but is slower than the standard set of memory-allocation functions . (This implementation does is not detect all possible errors; memory leaks can still occur.)

The value assigned to this environment variable should am a single digit, whose meaning are as described for m_check_action . Any characters beyond the initial digit are ignored. For security reasons, the effect of Malloc_check_ was disabled by default for Set-user-id and Set-group-id. However, if the File/etc/suid-debug exists (the content of the file is irrelevant) then Malloc_check_ also has a effect For Set-user-id and Set-group-id programs.

Malloc_mmap_max_

Controls the same parameter as mallopt () M_mmap_max.

Malloc_mmap_threshold_

Controls the same parameter as mallopt () M_mmap_threshold.

Malloc_perturb_

Controls the same parameter as mallopt () m_perturb.

Malloc_trim_threshold_

Controls the same parameter as mallopt () M_trim_threshold.

Malloc_top_pad_

Controls the same parameter as mallopt () M_top_pad. 2) Malloc_trim ()

Responsible to tell glibc in the BRK maintenance of the heap queue, the top of the heap left a lot of space, the rest of the spare space to return to the system. It cannot return memory other than the top of the heap.

URL: http://man7.org/linux/man-pages/man3/malloc_trim.3.html

       #include <malloc.h>
       int malloc_trim (size_t  pad);

The Malloc_trim () function attempts to release free memory in the top of the heap (by calling SBRK (2) with a su itable argument).

The pad argument specifies the amount of free spaces to leave untrimmed in the top of the heap. If This argument is 0, only the minimum amount of memory are maintained at the top of the heap (i.e., one page or less). A nonzero argument can be used to maintain some trailing spaces at the top of the heap in order to allow future To is made without has to extend the heap with SBRK (2). 4 Resolution Method

From the operating system's perspective, the memory allocation of the process is accomplished by two system calls: BRK and mmap. BRK is to push the top address pointer of the data segment (. data) _edata to the high address, mmap to find an idle one in the virtual address space of the process. Where the Mmap allocated memory is released by Munmap, the operating system is immediately returned when memory is released, and the BRK allocated memory needs to wait until the high address memory is released before it can be released. In other words, if A and b two blocks of memory have been applied successively through BRK, A is impossible to release before B is released, and is still occupied by the process, viewing the suspect "memory leak" through the top. By default, memory allocations greater than or equal to 128KB call Mmap/mummap, and memory requests that are less than 128KB call SBRK (which can be adjusted by setting m_mmap_threshold). GLIBC:M_mmap_threshold can be adjusted dynamically . The value of the M_mmap_threshold is dynamically adjusted between 128KB to 32MB (32-bit) or 64MB (64-bit), and the value of M_mmap_threshold is adjusted to a value between 2M and 2M after each request and release of a size 2MB memory . For example:

char* no_used = new CHAR[2 * 1024 * 1024]; memset (no_used, 0xFE, 2 * 1024 * 1024); Delete[] no_used;

In this way, the value of the M_mmap_threshold is adjusted to a value between 2M and 2M + 4K, and subsequent requests for <= 2 * 1024 * 1024 of the memory block will go sbrk rather than MMAP, and SBRK will need to wait until the high address memory is released before the low address memory can be released.

You can explicitly set the M_mmap_threshold or M_mmap_max to turn off the features of the m_mmap_threshold dynamic adjustment to avoid these problems.

Of course, Mmap calls can cause the process to generate page faults, which are common in order to improve performance:

1 Change the dynamic memory to static, such as using the memory pool technology or when starting to assign a certain size to each thread, such as 8MB of memory, after the direct use;

2) Prohibit mmap memory calls, prohibit glibc memory tightening to return the memory system, glibc equivalent to implement a memory pool function. You only need to add two lines of code when the process starts:

mallopt (M_mmap_max, 0);//disable malloc call MMAP Allocate memory

mallopt (m_trim_threshold, 0);//disable memory indent, sbrk the requested memory will not be returned to the operating system after release