GLIBC Memory Management Knowledge Point Summary

These days looking at the content of glibc memory management module, feel a lot of harvest, here to do a simple summary, so that the knowledge point review.

Let's introduce the relevant background. A project team has encountered a glibc memory explosion when developing a database-like NoSQL system. Accordingly, after a series of investigation, they raised a number of questions, namely:

1.GLIBC under what circumstances will memory not be returned to the operating system? What are the constraints on how 2.GLIBC memory is managed? What kind of memory allocation scenario is appropriate.

3. The memory management in our system is inconsistent with GLIBC's memory management constraints.

4.GLIBC is how to manage memory.

With these questions, I went to see the glibc ptmalloc2 source, summed up a number of knowledge points.

First, let's look at a diagram of the default memory layout of a Linux process under x86_64:

As shown in the figure above, for the AMD64 system, the memory layout uses the classic layout: the. Text segment is first loaded, then the. Data section, and finally the. BSS segment. The top 128TB is used by the kernel and cannot be accessed directly by the application. The stack of applications begins to grow downward from a high address.

The space between the BSS segment and the stack is free, the free space is divided into two parts, one is heap, and the other is the mmap mapping area. The mmap region grows relative to the stack area.

On different Linux kernels and machines, the starting position of the mmap area is generally different. And under the current kernel default configuration, the process's stack and mmap map regions do not start with a fixed address, and the values are different at each startup. Of course, you can use the following command:
sudo sysctl-w kernel.randomize_va_space

Change the value of the global variable randomize_va_space to 0 so that the stack and mmap map areas of the process start at a fixed location.

After a quick look at the default memory layout for Linux processes under X86_64, let's look at some
Related functions for operating system memory allocation: The heap and mmap:heap and mmap mapping zones are virtual memory spaces that can be used freely by the user program, but they are not mapped to memory space at the beginning and are inaccessible. Access to this space can cause segmentation fault before the kernel requests that the space be allocated. Users can manage heap and mmap mapping areas directly using system calls, but more often, programs use the malloc functions and free functions provided by C to dynamically allocate and release memory. Heap action-related functions: System call BRK () and C library functions Sbrk (). The malloc function family of glibc is to invoke the SBRK (), sbrk () function to map the virtual address space to memory in the kernel management for use by the malloc function.
The C language dynamic memory allocation basic function is malloc (), the implementation on Linux (System call) is by invoking the simple BRK (). Mmap Map Zone action-related functions: The UNIX process can use the MMAP function to create new virtual storage areas and map files or objects to those areas. Files are mapped to multiple pages, and if the size of the file is not the sum of all the pages, the space that is not used by the last page will be zeroed. Munmap performs the reverse operation and deletes the object mappings for a specific address area. The function is defined as follows:

#include <sys/mman.h>
void *mmap (void *addr, size_t length, int port, int flags, int fd, off_t offset);
int Munmap (void *addr, size_t length);

Delayed allocation of memory: note that the physical mapping of this address is only established when an address is actually accessed. When a user applies for memory, the Linux kernel allocates only a linear area (that is, virtual memory) and does not allocate actual physical memory; only when the user uses this memory does the kernel allocate specific physical pages to the user, which takes up valuable physical memory. When released, release the linear area, find the corresponding physical page, and release it all.

Note: When you do not know how much memory each part of the program will need, the system memory space is limited, and the memory requirements are changed, then the memory management program is required to allocate and reclaim memory. The more dynamic The program is, the more important memory management is, and the more important is the choice of the memory allocation program.

Memory management methods: There are many methods of memory management, they have advantages and disadvantages, have their own best case, where I will only list some of the main advantages and disadvantages of the method and the comparison of scenarios.

1.C-style memory management program:

The C-style memory management program mainly implements malloc and free functions. The Memory management program adds additional virtual memory to the process primarily by calling BRK () or mmap ().

Advantages:

A. Life cycle is easy to manage when it is limited to the memory of the current function  B. The changes in the
structure of the program are not required.

Disadvantage: A program that is not suitable for managing a long memory lifetime.

Common situations: Doug Lea Malloc, Ptmalloc, BSD Malloc, Hoard, Tcmalloc.

2. Pool-type memory management: A memory pool is a half memory management method. The memory pool helps some programs to automate memory management, which goes through certain stages, each with its own memory pool assigned to a specific stage of the process. At the end of each phase, all memory is freed at once.

Advantages:

A notable advantage is that memory allocation and recycling are faster. Because each time is done in a pool, the allocation can be done in O (1) time, and the amount of time needed to free up the memory pool is similar.
B. The error-handling pool can be allocated in advance so that the program can recover when normal memory is depleted.
C. An application can manage memory simply. There are very easy to use standard implementations.

Disadvantages: A lot of disadvantages, personally feel that pool-type memory management is generally only applicable to specific, there are obvious memory pool characteristics of the program. Common scenarios: Many Web services processes, such as Apache.

3. Reference counter

4. Garbage collector: The garbage collector is a dynamic storage allocator that automatically frees allocated blocks that the program no longer needs. In the context of the C program, the application calls the malloc, but never calls free, and the garbage collector periodically recognizes the garbage block and calls it accordingly, putting the blocks back in the idle list. Many types of garbage collectors need to know the planning of internal pointers to data structures, and they must be part of the language itself in order to run correctly.

Advantages:

Never worry about the dual release of memory or the life cycle of an object.
B. With some collectors, you can use the same APIs as regular allocations.

Disadvantages:

A. Inability to interfere when memory is freed.
B. Slow
C. Bugs caused by garbage collection errors are difficult to debug.
d. Do not set the pointer no longer used to NULL, there will be a memory leak (very easy to forget, right.) ）

Common scenarios: An important part of modern language systems such as Java, ML, and Perl. Well, the first part of the brief description ends with an overview of Ptmalloc memory Management:

The 1.ptmalloc implements malloc (), free (), and a set of other functions to provide support for dynamic memory management.

2. The Distributor is between the user program and the kernel, which responds to the user's allocation request, requests memory from the operating system, and then returns it to the user program.

3. In order to maintain efficient allocation, the allocator typically allocates a single memory that is larger than the user's request and manages the memory with some sort of algorithm.

4. The user releases the memory and does not return it immediately to the operating system, and the allocator manages the freed free space. When responding to user assignment requirements, the allocator will first find a suitable memory in the free space for the user and allocate a new memory when the free space is not found.
5. In order to design an efficient allocator, there should be a number of factors to consider, such as the amount of memory occupied by the allocator itself.