Detailed parsing of Linux memory usage methods

I am a programmer, so I am here to explain the use of Linux memory in a programmer's perspective.

When it comes to memory management, the two concepts that flash in our minds are virtual memory, with physical memory. These two concepts are mainly derived from the support of the Linux kernel.

Linux has a two-level memory management level, a linear zone, similar to 00c73000-00c88000, corresponding to virtual memory, which does not actually occupy actual physical memory; The first level is a specific physical page that corresponds to the physical memory on our machine.

Here to mention a very important concept of memory delay allocation. When a user requests memory, the Linux kernel allocates only a linear region (that is, the virtual memory), and does not allocate actual physical RAM, only when the user uses the memory, the kernel allocates a specific physical page to the user, which takes up valuable physical memory . The kernel frees the physical page by releasing the linear area, finding its corresponding physical page, and releasing it all out of the process .

char *p=malloc( 2048 ) //这里只是分配了虚拟内存2048，并不占用实际内存。 strcpy(p,” 123 ”) //分配了物理页面，虽然只是使用了3个字节，但内存还是为它分配了2048字节的物理内存。 free(p) //通过虚拟地址，找到其所对应的物理页面，释放物理页面，释放线性区。

We know that the user's process and kernel are running at different levels, and the communication between the process and the kernel is done through system calls. Processes in the application and release of memory, mainly through the brk,sbrk,mmap,unmmap of these system calls (all virtual memory), the parameters passed mainly the corresponding virtual memory.

Note that in a process that only accesses virtual memory, it is virtually invisible to the use of internal nuclear physics memory, which is completely transparent to the process .

GLIBC Memory Manager

So each time we call malloc to allocate a chunk of memory, we make the appropriate system calls?

The answer is no, here I'm going to introduce a new concept to the GLIBC memory manager.

We know that the functions such as malloc and free are all library functions included in the GLIBC library, so let's just imagine how inefficient the program would be if every memory operation was invoked by a system call.

In fact, GLIBC uses a wholesale and retail approach to managing memory . GLIBC requests a chunk of memory (virtual memory) each time it is called by the system, and when the process requests memory, GLIBC takes a chunk out of the memory it has acquired to the process.

GLIBC for the heap memory request greater than 128k memory request, glibc mmap to the kernel to apply for memory, that is, malloc is implemented by the mmap, which does not guarantee that the memory address increases upward; less than 128k uses BRK, malloc calls the system to call BRK to implement wholesale virtual memory to the kernel, which is correct for it. The 128k threshold can be set by GLIBC's library function.

The memory manager is facing difficulties

When we write a program, the size of the memory block is irregular, and there are frequent applications and releases, which inevitably produce memory fragments. and memory fragments, directly resulting in large chunks of memory requests can not be satisfied, and thus more occupy the system resources, if the pieces are organized, it will increase the load of the CPU, many are conflicting indicators, here I will not elaborate.

When we were writing the program, we had two conceptual heap and stack when we were involved in memory. Traditionally, stack's memory address grew downward, and the memory address of the heap grew upward.

The function malloc and free, primarily for heap operations, are controlled by the programmer to access the memory autonomously.

The memory address of the heap grows upward here, which is not exactly true.

GLIBC for the heap memory request greater than 128k memory request, glibc in mmap way to the kernel to request memory, this does not guarantee that the memory address increases upward, less than 128k is the use of BRK, for it is correct. The 128k threshold can be set by GLIBC's library function.

Here I'll start with a large memory application, which corresponds to the mmap system call.

For large memory requests, GLIBC directly uses the MMAP system call to divide the virtual address for the process to use alone , freeing the memory with Unmmap system calls when the Block memory is released, which does not create a memory fragment in the middle of the process.

For small-block memory requests, after the program starts, the process obtains an address at the bottom of the heap, and each time the process makes a memory request, glibc increases the heap top up to expand the memory space, which is what we call the heap address up. When you operate on these small chunks of memory, there is a problem with memory fragments. In fact, the BRK and SBRK system call is to adjust the heap top address pointer.

So when is the heap memory released?

When Glibc discovers that the heap top has a continuous 128k space is idle, it will be called through the BRK or SBRK system to adjust the location of the heap top and return the occupied memory to the system. At this point, the kernel frees up the physical memory that is occupied by removing the corresponding linear zone.

Let me tell you a memory hole problem:

A scene, the heap top has a piece of memory in use, and the following is a large amount of contiguous memory has been released, then this block of memory can be released? Can the corresponding physical memory be released?

I'm sorry, I can't.

That is to say, as long as the top portion of the application memory is still occupied, I release more memory below, will not be returned to the system, still occupy the physical memory. Why is that?

This is primarily related to the kernel when dealing with the heap, it is too simple, it can only adjust the heap top pointer to adjust the linear area of the adjustment program, but only by adjusting the linear area to free memory. So as long as the heap top is not reduced, the memory consumed will not be released.

Ask a question:

char *p=malloc( 2 ); free(p)

Why request memory, need two parameters, one is the memory size, one is the return pointer, and when the memory is freed, but only the memory pointer?

This is mainly related to the memory management mechanism of GLIBC. In glibc, a chunk structure is maintained for each piece of memory. GLIBC when allocating memory, glibc fills in the size of the block of memory in the chunk structure, then the memory allocated to the process.

chunk ------size p------------ content

When the process frees memory, the size of the block memory can be found as long as the pointer-4 is released.

Note: glibc allocates a minimum of 16 bytes when making memory requests so that the chunk structure can be maintained.

Debugging tools provided by GLIBC:

To facilitate debugging, GLIBC provides the user with a hook (hook) for functions such as malloc, such as __malloc_hook

The corresponding is a function pointer,

void * function (size_t size, const void *caller);

Where caller is the recipient of the call to malloc return value (the address of a pointer). There are also __malloc_initialize_hook function pointers, which are only called once (when dynamic memory is allocated for the first time). (malloc.h)

Some statistics using malloc (SVID extension) can be stored with a struct mallinfo, callable.

struct mallinfo mallinfo ( void );

How to detect memory leakage? GLIBC provides a function

void Mtrace (void) and its reaction void muntrace (void)

This will depend on an environment variable malloc_trace refers to the file, some information recorded in the file

Used to detect memory leakage, the essence of which is the installation of the aforementioned hooks. These functions are generally used

#ifdef debugging packages to reduce overhead in a non-debug state. The resulting file is said to be not recommended to read,

Instead, use the Mtrace program (Perl script for parsing). Here is a simple example to illustrate the process, which is

SOURCE program:

#include #include #include intmain( int argc, char *argv[] ) {    int *p, *q ;    #ifdef DEBUGGING    mtrace( ) ;    #endif    p = malloc( sizeof( int ) ) ;    q = malloc( sizeof( int ) ) ;    printf ( "p = %p\nq = %p\n" , p, q ) ;    *p = 1 ;    *q = 2 ;    free( p ) ;    return 0 ; }

Very simple program, where Q is not released. After we set the environment variable and touch out the file

The results of the implementation are as follows:

p = 0x98c0378q = 0x98c0388

The contents of the file are as follows

= Start @./test30:[ 0x8048446 ] + 0x98c0378 0x4 @./test30:[ 0x8048455 ] + 0x98c0388 0x4 @./test30:[ 0x804848f ] - 0x98c0378

Here I basically finished, we write the program when the data part of the memory use problem.

Memory consumed by code

Data part of memory, then we write the program is not also occupy memory?

In Linux, the loading of the program involves two tools, linker and loader. Linker mainly involves the use of dynamic link library, loader mainly involves the loading of software.

1. EXEC executes a program
2. Elf is now very popular in the format of the executable file, which divides the program run by two segments, a segment is executable code snippet, it is read-only, execute, and the other segment is a data segment, which is read-write and cannot be executed.
3. Loader will start, through the mmap system call, the code side and data segments mapped into memory, in fact, it is allocated virtual memory, note that this time, not occupy the physical memory, only the program executed to the appropriate place, the kernel will allocate physical memory for it.
4. Loader will go to find the link library that the program relies on, first see if the link library is mapped into memory, if you do not use Mmap, the code snippet and the data segment are mapped into memory, otherwise it simply joins the process's address space. This means that the memory address space for libraries such as GLIBC is exactly the same.

So a 2M program, when executed, does not mean that it allocates 2M of physical memory, which is related to the amount of code it is running, and the dynamic-link library it relies on.

The difference between linking a dynamic link library during a run and linking a dynamic library during compilation

We call the dynamic link library in two ways: one is to compile, to indicate the dependent dynamic link library, so that loader can be at the time of the program launch, all the dynamic links are mapped into memory, one is in the process of running, through Dlopen and Dlfree way to load the dynamic link library, Dynamic Link insecure library is dynamically loaded into memory.

These two ways, from the programming point of view, the first is the most convenient, the efficiency of the impact is not small, there are some differences in memory usage.

In the first way, the code for a library, which runs once, consumes physical memory and then consumes physical memory until the process terminates, even if it is not used again.

The second way, the library code occupies the memory, can be released by Dlfree Way, returned to the physical memory.

This difference is primarily related to processes that have a long lifespan but occasionally invoke various libraries. If this is the case, it is recommended to call the dynamic-link library in the second way.

Memory-intensive measurements

To measure how much memory a process consumes, Linux provides a convenient way for us to provide all the information we have in the/proc directory, and in fact the top tools are also available here to obtain the appropriate information.

/proc/meminfo 机器的内存使用信息 /proc/pid/maps pid为进程号，显示当前进程所占用的虚拟地址。 /proc/pid/statm 进程所占用的内存 [[email protected] ~]# cat /proc/self/statm 654 57 44 0 0 334 0

Output interpretation

CPU and CPU0 ... The meaning of each parameter of each row (in the first example) is:

Parameter Interpretation/proc//status

Size (pages) 任务虚拟地址空间的大小 VmSize/ 4 Resident(pages) 应用程序正在使用的物理内存的大小 VmRSS/ 4 Shared(pages) 共享页数 0 Trs(pages) 程序所拥有的可执行虚拟内存的大小 VmExe/ 4 Lrs(pages) 被映像到任务的虚拟内存空间的库的大小 VmLib/ 4 Drs(pages) 程序数据段和用户态的栈的大小 （VmData+ VmStk ） 4 dt(pages) 04

View Machine available memory

/proc/ 28248 />free total used free shared buffers cached Mem: 1023788 926400 97388 0 134668 503688 -/+ buffers/cache: 288044 735744 Swap: 1959920 89608 1870312

When we look at the idle memory of the machine with the free command, we find that the value of the. This is mainly because, in Linux there is such a thought, the memory is not white, so it as far as possible cache and buffer some data to facilitate the next use. But in fact, the memory is also available for immediate use.

So free memory =free+buffers+cached=total-used

To view the memory used by the process

Looking at the memory used by a process is a very confusing thing. Because we write the program, it is necessary to use the dynamic link library, add it to their own address space, but/PROC/PID/STATM statistics, will be the memory of these dynamic link library is also simple to calculate in.

The problem with this is that the dynamic link library takes up some of the memory that other programs use, but it's up to you. Your program contains sub-processes, and some of the memory that is reused by dynamic link libraries is repeated.

Therefore, it is very difficult to accurately evaluate the memory occupied by a program, and it may be useful for us to write a module to accurately calculate the memory occupied by a certain virtual address. (T002)

Detailed parsing of Linux memory usage methods