[Go] Observe the memory consumption of the process

ext.: http://www.cnblogs.com/bravery/archive/2012/06/27/2560611.html Overview

Presumably the students who wrote the program on Linux had an experience of how much memory was being consumed by the analysis process, or asked the question-how much memory (physical memory) did your program consume at run time? Usually we can see how much memory the process is using with the top command. Here we can see three important indicators of virt, res, and SHR, what do they mean by each other? This is the question that this article needs to discuss with everybody. Of course, if you go deeper, you might ask where the physical memory used by the process is going to be. The top command may not give you the answer you want, but we can analyze the Smaps file provided by the proc file system, which gives an exhaustive list of the usage of the physical memory currently occupied by the process.

This blog has a total of three sections. The first part briefly elaborates the two important concepts of virtual memory and resident memory; The second part explains the practical reference meaning of the three parameters of Virt, res and SHR in the top command; The last part introduces you to the format of the Smaps file, By analyzing the Smaps file we can learn more about the usage of the process's physical memory, such as how much space the Mmap file occupies, how much space is consumed by dynamic memory, how much space is consumed by the function call stack, and so on.

two concepts about memory

To understand the output of the top command with regard to memory usage, we must first clarify the two concepts of virtual memory and resident memory (resident memories).

• Virtual memory

The first thing to emphasize is that virtual memory differs from physical memory, although both contain memory words but they belong to two different levels of concept. The large amount of virtual memory used by the process does not mean that the physical memory of the program must also occupy a large space. Virtual memory is a logical memory space concept designed by the operating system kernel in order to manage the process address space (management). The pointers in our program are actually the addresses in this virtual memory space. For example, after we have written a C + + program, we need to use g++ to compile, when the compiler adopts the address is actually the virtual memory space address. Because the program is not running yet, what about the physical memory space address? any instruction or data that may need to be used in the course of running the program must be in the virtual memory space . Since virtual memory is a logical (false) memory space, in order for the program to run on a physical machine, there must be a mechanism to map the imaginary virtual memory space to the physical memory space (the real space on the RAM memory bar). This is actually what the page table in the operating system does. The kernel maintains a separate page mapping table for each process in the system . The basic principle of a page map is to map a piece of virtual memory space that needs to be accessed during a program's run through a page map to a physical memory space, so that the CPU accesses the corresponding virtual memory address by using the mechanism of the Lookup page mapping table to access a corresponding address on the physical memory. Page is the basic unit of virtual memory space mapping to physical memory space.

1 illustrates the relationship between virtual memory space and physical memory space, which is associated with a page table. The portions of the virtual memory space that are shaded are mapped to the same shaded portion of the physical memory space, respectively. The gray part of the virtual memory space indicates that there is no corresponding part in the physical memory space, which means that the gray part is not mapped to the physical memory space. This is also the spirit of "on-demand mapping", because the virtual memory space is very large, many of them may not need to access at the time of the program run, so there is no need to map these parts of the virtual memory space to the physical memory space.

So far, we have basically explained what virtual memory is. To summarize, virtual memory is an imaginary memory space in which portions of virtual memory space that need to be accessed are mapped to physical memory space during the program's run. Large virtual memory space can only indicate that the program can be accessed during the process of larger space, does not mean that the physical memory space consumption is also large.

Figure 1. Virtual memory space to physical memory space mapping

• Memory resident

Resident memory, as the name implies, is the physical memory that is mapped to the process virtual memory space. In 1, the portion of the system's physical memory space that is shaded is memory-resident. For example, A1, A2, A3, and A4 are memory-resident of process A; B1, B2, and B3 are memory-resident of Process B. The memory in which the process resides is the actual physical memory occupied by the process. In general, how much memory is consumed by the process we're talking about, in fact, how much of the memory resides, not how much virtual memory. Because virtual memory is large, it does not mean that it occupies large physical memory.

About the two concepts of virtual memory and resident memory we're talking about this. In the next section, let's look at what the Virt, res, and SHR in the top command mean respectively.

meaning of Virt, res, and SHR in the top command

It is easy to explain the meaning of virt after figuring out the concept of virtual memory. Virt represents the size of the process virtual memory space. The corresponding to process a in Figure 1 is A1, A2, A3, A4, and the sum of all the space in the gray section. That is, Virt contains the sum of parts that have been mapped to physical memory space and that have not yet been mapped to physical memory space.

The meaning of res refers to the size of the part of the process virtual memory space that has been mapped to the physical memory space. The corresponding to process a in Figure 1 is A1, A2, A3, and A4 the sum of several parts of space. So, to see how much memory the process occupies during the run, it should look at the value of res instead of the value of Virt.

Finally, let's look at what the shr means. SHR is an abbreviation for share (shared), which represents the amount of shared memory that the process consumes. In 1 we see that the B3 in A4 and process B virtual memory space in process a virtual memory space are mapped to the A4/B3 portion of the physical memory space. It's strange to see. Why is there such a situation? In fact, the program we write will depend on many external dynamic libraries (. So), such as libc.so, libld.so, and so on. These dynamic libraries only save/Map a copy in memory, and the dynamic loader maps this memory to the virtual memory space of the corresponding process if it is required by a process to run. This can also occur when multiple advances communicate with each other through shared memory. As a result, the virtual memory space for different processes will be mapped to the same physical memory space. This part of the physical memory space is actually shared by multiple processes, so we call them shared memory, expressed in SHR. The memory that a process consumes is its own exclusive memory, in addition to the memory shared by other processes. So to calculate the size of the process exclusive memory, simply subtract the SHR value with the value of Res.

the Smaps file for the process

　　With the top command we can see the virtual space size (VIRT) of the process, the physical memory Used (RES), and the memory (SHR) shared with other processes. But that's all, if I want to know the following questions:

1. The distribution of virtual memory space for a process, such as how much space the heap occupies, how much space the file mapping (mmap) occupies, and how much space the stack occupies?
2. Does the process have the memory that is swapped to swap space, and if so, the size that is swapped out?
3. Mmap How many pages in memory are dirty pages (Dirty page) that have not been written back to disk in the data file?
4. Mmap How many pages are currently in memory in the Open data file, and how many pages are still not loaded into page Cahe in the disk?
5. Wait a minute

None of these questions can be answered by the top command, but sometimes these are the questions we need to answer when we analyze and optimize the performance bottleneck of the program. Fortunately, the problem-solving approach in the world is much more than the problem itself. Linux provides a smaps file for each process through the proc file system, and by analyzing the file we can answer each of the questions raised above.

In the Smaps file, each record (as shown in 2) represents a contiguous area of the process virtual memory space. The first line, from left to right, indicates the address range, the permission ID, the mapping file offset, the device number, the Inode, and the file path. A detailed explanation can be found in understanding-linux-proc-id-maps.

The next 8 fields have the following meanings:

• Size : Indicates how large the mapped area is in the virtual memory space.
• Rss: indicates how much space the map area currently occupies in physical memory　　　　
• Shared_clean: The size of the page that has not been overwritten with other processes
• Shared_dirty: the size of the rewritten page that is shared with other processes
• Private_clean: the size of the private page that has not been overwritten.
• Private_dirty: the size of the private page that has been overwritten.
• Swap: represents non-mmap memory (also known as anonymous memory, such as malloc dynamically allocated RAM) due to the size of swap to swap space because of insufficient physical memory.
• PSS: The virtual memory area is divided into the amount of physical memory used after calculation (some memory is shared with other processes, such as mmap in). For example, the part of the physical memory mapped by the region is also mapped by another process, and the size of the physical memory is 1000KB, then the process is allocating half of the memory, that is, pss=500kb.

Figure 2. A record in the Smaps file

With SMAP so detailed about the virtual memory space to the physical memory space mapping information, I believe you have been able to analyze the file to answer the 4 questions raised above.

Finally hope that all readers can read this article on the process of virtual memory and physical memory has a clearer understanding, and can more accurately understand the top command about the output of memory, and finally through the Smaps file to further analyze the situation of the process using memory.

[Go] Observe the memory consumption of the process