Top Command Memory footprint profiling

Original: http://yalung929.blog.163.com/blog/static/203898225201212981731971/

Introduction: The top command, as one of the most commonly used performance analysis tools under Linux, can monitor and collect CPU,IO , and memory usage of the process. For example, we can Use the top command to get how much virtual memory (VIRT), physical memory (RES), shared memory is used by a process ( SHR).

Recently encountered a consulting problem, a product doing performance analysis needs to get the actual size of the physical memory of the process (excluding parts shared with other processes), seemingly simple problem, but after research and analysis, found behind a lot of stories ...

1 VIRT RES SHRthe exact meaning

Three memory metrics,vrit,RES,SHR What exactly does it mean? Who can tell us? Mans page? a Linux expert? SUSE engineer? Linus? Who can tell the most correct answer? No one! Because only the source code is the most correct answer.

Then let's go to the source code, which is the biggest benefit of open source software.

First of all, the source of these three data is certainly the kernel, the related data structure of the process must be maintained by the kernel. Then top as a user-space program, in order to obtain the kernel space data, it needs to be obtained through the system interface (API). While the proc file system is a way to exchange data between Linux kernel space and user space, and is a very important way, this and the Windows more likely to be based on function calls in different forms.

When you call the system function read to read an ordinary file, the kernel executes the corresponding file system code to transfer the file contents from disk to you.

When you call the system function Read read a proc file, the kernel executes the corresponding proc File system code from the kernel's data structure to send you the relevant content. the proc file does not have a relationship with the disk. It's just the system interface.

and a process related information,Linux all through the /proc/<pid>/ within the file told us.

Below, you can use common file reading and writing tools, such as cat to get a variety of information about the process. This is much more flexible and much richer than the way a function is called.

Back to our question, thetop command shows the process information that is certainly also obtained through proc , because there is no other way, no system function can do this thing,top It is also impossible to fetch data over the user layer directly from the kernel.

With the above information, you can quickly find the key code from the top source:

Aha,statm file:

Based on The order of SSCANF, the first value is VIRT, the second value is RES, and the third value is SHR!

Wait, as if the value is not on,top shows the shr is 344k, and statm gives 86 !

Let's look at a key line of code:

STATM shows the number of pages, andtop shows the KB. X86 , a page is 4KB,* 4 = 344. That's right!

As a result, we found the most critical entry, and then retrace to see how the kernel generated statm file content. ~~

The PROC_PID_STATM function is responsible for generating The contents of the Statm file, and when you print the statm file using the Cat command, This function in the kernel executes.

Proc_pid_statm Gets the mm_struct data structure of the process, which is the memory descriptor of the process, through which it can get all the information of the process memory usage and mapping.

Further investigating the TASK_STATM function, you can see:

The first value (VIRT) is MM->TOTAL_VM, that is, thetotal size of the process virtual memory, which is clearer, as long as the process requests the RAM, either malloc or a stack or a global, this value will be counted;

The second value (RES) is a mm->file_rss+mm->anon_rss;

The third value (SHR) is mm->file_rss.

RES to and SHR , the kernel divides the physical memory into two parts, some of which are mapped to files, and some are anonymous memory that is not mapped to a file.

But why is file_rss called gkfx ? Should be an indicative representation that this part of the memory may be shared. But it doesn't mean it's really shared. So what counts into file_rss? By looking up the relevant code, we find ( there may be omissions):

L Code snippet for the program.

L Code snippet for dynamic library.

L file mappings made by Mmap.

L Anonymous mapping through mmap , but indicates the map_shared property.

L shared memory requested by Shmget.

That is, the process consumes physical memory in the above way, and counts into file_rss, which is the SHR field of top . We see that in general, these memory are present in a shared way. However, if a dynamic library is used by only one process, its code snippet is not shared.

In turn to look at the content of Anon_rss Statistics, whether it must be exclusive? No, such as A child process after the new fork, because of the copy on write mechanism, is shared with the parent process before the page is modified. This part of the value is not reflected in the SHR field of the top command .

In summary, the SHR field displayed by the top command does not accurately describe the amount of memory that the process shares with other processes, and there is an error.

So how do I get the correct amount of shared memory for the process?

2 Get the process exact amount of shared memory

We note that within the proc/<pid> that describes the process information, there is a smaps file that shows information about all the memory segments, including Shared_clean Shared_dirty Private_clean Private_dirty: several fields.

Find the relevant code, you can see, a page if the number of maps >=2 counted into shared_* ; if =1 is counted into Private_ *. (The dirty page counts into the *_dirty, otherwise counts into the *_clean)

The sum of the shared_* values of all segments within the smaps file is the amount of shared memory that is process accurate!

The sum of the private_* values of all the segments in the smaps file is the total amount of exclusive memory that is process accurate!

3 Summary

Through the above analysis, we can get the following conclusions:

The L Top command parses /PROC/<PID>/STATM statistics VIRT and RES and SHR The field value.

L VIRT is the total amount of virtual memory requested.

L RES is the sum of the physical memory used by the process.

L SHR is the sum of the physical memory "mapped to a file" in RES . Including:

The code snippet for the program.

The code snippet for the dynamic library.

file mappings made through Mmap.

Anonymous mappings made through Mmap, but the map_shared attribute is indicated.

The shared memory requested by Shmget.

The shared_* in L/proc/<pid>/smaps is the physical memory >=2 The number of mappings in RES .

The private_* in L/proc/<pid>/smaps is the physical memory =1 The number of mappings in RES .

Top Command Memory footprint profiling