Linux process footprint Memory detailed __linux

Presumably the students who wrote the program on Linux have an experience of how much memory the process consumes, or how much memory (physical memory) your program occupies at runtime. Usually we can see how much memory is consumed by the process through the top command. Here we can see three important indicators of virt, res, and SHR, and what do they mean by their respective representatives? This is the problem that this article needs to discuss with you. Of course, if you go a little deeper, you might ask where the physical memory that the process occupies is used. At this point the top command may not give you the answer you want, but we can analyze the Smaps file provided by the proc file system, which provides an exhaustive list of the usage of the physical memory occupied by the current process.

This blog is divided into three parts altogether. The first part briefly expounds the two important concepts of virtual memory and resident memory; The second part explains the practical reference meaning of the three parameters of Virt, res and SHR in the top command. The last part introduces the format of the Smaps file, By analyzing the Smaps file we can learn more about the usage of process physical memory, such as how much space the Mmap file occupies, how much space is consumed by dynamic memory, how much space the function call stack consumes, and so on.

Two concepts about memory

To understand the output of the top command with regard to memory usage, we must first figure out the two concepts of virtual memory (fictitious Memory) and resident memory (resident Memory).

"Virtual Memory"

The first thing to emphasize is that virtual memory differs from physical memory, although both contain memory words but they belong to two different levels of concept. A process that consumes a large amount of virtual memory space does not mean that the program's physical memory must also occupy a large footprint. Virtual memory is a logical memory space concept that is carefully designed by the operating system kernel to manage process address spaces (management). The pointers in our program are actually the addresses in this virtual memory space. For example, after writing a C + + program, we need to use g++ to compile, and the address of the compiler is actually the address of the virtual memory space. Because the program is not yet running, how to talk about the physical memory space address. Any instruction or data that may need to be used in the process of running a program must be in the virtual memory space. Since virtual memory is a logical (imaginary) memory space, in order to allow the program to run on the physical machine, there must be a mechanism for mapping the imaginary virtual memory space to the physical memory space (the space on the real RAM memory strip). This is actually what the page map table in the operating system is doing. The kernel maintains a separate page mapping table for each process in the system ... The basic principle of the page map is to map a part of the virtual memory space that needs to be accessed through the page map to a physical memory space in the process of running the program, so that the CPU can access a corresponding address on the physical memory through the mechanism of the Lookup page mapping table when it accesses the virtual memory address.       Page is the basic unit that the virtual memory space maps to the physical memory space. Figure 1 below illustrates the interrelationships between virtual and physical memory spaces, which are associated with the page table. Where the shaded portions of the virtual memory space are mapped to portions of the physical memory space corresponding to the same coloring. The gray portion of the virtual memory space indicates that there is no corresponding part in the physical memory space, which means that the gray part is not mapped into the physical memory space. It is also guided by the guiding principle of "on-demand mapping", because virtual memory space is so large that many of them may not need to be accessed at all in the course of a program's operation, so there is no need to map these parts of the virtual memory space to physical memory space. So far it has been basically explained what virtual memory is. To sum up, virtual memory is an imaginary memory space in which the part of the virtual memory space that needs to be accessed in the course of the program is mapped to the physical memory space. A large virtual memory space can only indicate that the program is in the process of being able to access more space, does not mean that the physical memory space occupies large.

"Resident Memory"

Resident memory, as the name suggests, refers to the physical memory that is mapped to the process virtual memory space. In Figure 1 above, the portion of the system's physical memory space that is shaded is memory-resident. For example, A1, A2, A3, and A4 are the resident memory of process A; B1, B2, and B3 are the resident memory of process B. The resident memory of a process is the physical memory that is actually occupied by the process. In general, the process we are talking about consumes a lot of memory, which is to say how much memory resides rather than how much virtual memory is consumed. Because of the large virtual memory does not mean that the physical memory consumption is large.

About the two concepts of virtual memory and resident memory we're talking about here. Here's a look at what virt, res, and SHR mean in the top command. The meaning of Virt, res, and SHR in the top command it is easy to explain the meaning of virt after figuring out the concept of virtual memory. Virt represents the size of the process virtual memory space. The sum of all the spaces in the A1, A2, A3, A4, and gray sections corresponds to process a in Figure 1. That is, the virt contains parts that have been mapped to the physical memory space and that have not been mapped to the physical memory space. Res means the size of the part of the process virtual memory space that has been mapped to the physical memory space. Corresponding to the process a in Figure 1 is the sum of the A1, A2, A3, and A4 several partial spaces. So, looking at how much memory the process consumes during the run should look at the RES value rather than the Virt value. Finally, look at the meaning of shr. SHR is the abbreviation for share (shared), which represents the amount of shared memory consumed by the process. In Figure 1 above, we see that the A4 in process a virtual memory space and the B3 in the process B virtual memory space are mapped to the A4/B3 part of the physical memory space. I think it's strange. Why is this happening? In fact, the programs we write depend on many external dynamic libraries (. So), such as libc.so, libld.so, and so on. These dynamic libraries are only saved/mapped in memory, and if the dynamic library is needed for a process to run, the dynamic loader maps the memory to the virtual memory space of the corresponding process. This can also occur when multiple advances communicate with each other in a way that is shared memory. As a result, the virtual memory space of different processes is mapped to the same physical memory space. This part of the physical memory space is actually shared by multiple processes, so we call them shared memory, expressed in SHR. The memory that a process consumes is its own exclusive memory in addition to the memory shared by other processes. So the size of the process exclusive memory is calculated only by subtracting the SHR value from the value of Res.

Smaps files for the process

The View command is: The pid/smaps of the cat/proc/process

With the top command we can already see the virtual space size (virt) of the process, the physical memory (RES) consumed, and the memory shared with other processes (SHR).  But that's all, if I want to know how the virtual memory space of the process is distributed, such as how much space the heap occupies, how much space the file map (mmap) occupies, and how much space the stack occupies. Whether the process has memory swapped to swap space, and if so, the size of the swap. Mmap How many pages of Open data files in memory are dirty pages (Dirty page) are not written back to disk. Mmap How many pages are currently in memory in the Open data file and how many pages are still not loaded into page Cahe on the disk. Wait a minute

None of these questions can be answered by the top command, but sometimes these are the questions we need to answer when we analyze and optimize the performance bottlenecks of the program. Fortunately, the world has a much bigger problem-solving approach than the problem itself. Linux provides a smaps file for each process through the proc file system, and by analyzing the file we can answer the questions raised above.

In the Smaps file, each record (shown in Figure 2 below) represents a contiguous area of the process virtual memory space. The first line from left to right indicates the address range, the permission ID, the mapping file offset, the device number, the Inode, and the file path. Detailed explanations can be found in understanding-linux-proc-id-maps.

The next 8 fields have the following meanings: Size: Indicates how large the mapped region is in the virtual memory space. RSS: Indicates how much space is currently occupied by this map area in physical memory. Shared_clean: The size of the page that is not overwritten with other processes. Shared_dirty: The size of the overwritten page that is shared with other processes. Private_clean: The size of a private page that has not been overwritten. Swap: Represents mmap memory (also called anonymous memory, such as malloc dynamically allocated memory) due to insufficient physical memory to swap to the size of the swap space. PSS: The virtual memory Area shares the amount of physical memory that is used after the calculation (some memory is shared with other processes, such as mmap in). For example, the portion of the physical memory mapped by the zone is also mapped by another process, and the portion of the physical memory is 1000KB, and the process is allocated half of the memory, that is, pss=500kb.

Figure 2. Smaps File Example

With SMAP so detailed about the virtual memory space to the physical memory space mapping information, I believe you have been able to analyze the file to answer the 4 questions raised above.

Finally, I hope that all readers will be able to read this article on the process of virtual memory and physical memory has a clearer understanding, and can more accurately understand the top command about memory output, and finally through the Smaps file to further analyze the process using memory