How does Linux process memory look at the Virt RES shr value displayed by the Anatomy top command

Introduction: The top command, as one of the most commonly used performance analysis tools under Linux, can monitor and collect CPU, IO, and memory usage of the process. For example, we can use the top command to get how much virtual memory (VIRT), physical memory (RES), Shared memory (SHR) a process uses.

Recently encountered a consulting problem, a product doing performance analysis needs to get the actual size of the physical memory of the process (excluding parts shared with other processes), seemingly simple problem, but after research and analysis, found behind a lot of stories ...

1 VIRT RES The exact meaning of SHR

Three memory metrics, what exactly does vrit,res,shr mean? Who can tell us? Mans page? A Linux expert? SuSE engineer? Linus? Who can tell the most correct answer? No one! Because only the source code is the most correct answer.

Then let's go to the source code, which is the biggest benefit of open source software.

First of all, the source of these three data is certainly the kernel, the related data structure of the process must be maintained by the kernel. Then top as a user-space program, in order to obtain the kernel space data, it needs to be obtained through the System interface (API). The proc file system is a way to exchange data between Linux kernel space and user space, and is a very important way, and Windows prefers to be based on function calls in a different form.

When you call the system function read to read an ordinary file, the kernel executes the corresponding file system code to transfer the file contents from disk to you.

When you call the system function read read a proc file, the kernel executes the corresponding proc file system code from the kernel's data structure to send you the relevant content. The proc file does not have a relationship with the disk. It's just the system interface.

and a process related information, Linux all through the/proc/<pid>/within the file told us.

Below, you can use common file reading and writing tools, such as cat to get a variety of information about the process. This is much more flexible and much richer than the way a function is called.

Back to our question, the top command shows the process information that is certainly also obtained through Proc, because there is no other way, there is no system function can do this thing, top can not go over the user layer directly take the kernel to get the data.

With the above information, you can quickly find the key code from the top source:

Aha, STATM file:

Based on the order of SSCANF, the first value is Virt, the second value is res, and the third value is shr!

Wait, as if the value is not on, top shows the SHR is 344k, and STATM gives is 86!

Let's look at a key line of code:

STATM shows the number of pages, and top shows the KB. X86, one page is 4kb,86 * 4 = 344. That's right!

As a result, we found the most critical entry, and then retrace to see how the kernel generated STATM file content. ~~

The PROC_PID_STATM function is responsible for generating the contents of the Statm file, which is performed by the kernel when you print the Statm file using the Cat command.

PROC_PID_STATM gets the MM_STRUCT data structure of the process, which is the memory descriptor of the process, through which it can get all the information of the process memory usage and mapping.

Further investigating the TASK_STATM function, you can see:

The first value (VIRT) is MM->TOTAL_VM, that is, the total size of the process virtual memory, this is clearer, as long as the process to request the RAM, whether it is malloc or stack or global, will count into this value;

The second value (RES) is a mm->file_rss+mm->anon_rss;

The third value (SHR) is Mm->file_rss.

Res to and SHR, the kernel divides the physical memory into two parts, some of which are mapped to files, and some are anonymous memory that is not mapped to a file.

But why is File_rss called GKFX? Should be an indicative representation that this part of the memory may be shared. But it doesn't mean it's really shared. So what counts into File_rss? By looking up the relevant code, we find (there may be omissions):

L Code snippet for the program.

L Code snippet for dynamic library.

L File mappings made by Mmap.

L Anonymous mapping through Mmap, but indicates the Map_shared property.

L Shared memory requested by Shmget.

That is, the process consumes physical memory in the above way, and counts into File_rss, which is the shr field of top. We see that in general, these memory are present in a shared way. However, if a dynamic library is used by only one process, its code snippet is not shared.

In turn to look at the content of Anon_rss statistics, whether it must be exclusive? No, such as a child process after the new fork, because of the copy on write mechanism, is shared with the parent process before the page is modified. This part of the value is not reflected in the SHR field of the top command.

In summary, the SHR field displayed by the top command does not accurately describe the amount of memory that the process shares with other processes, and there is an error.

So how do I get the correct amount of shared memory for the process?

2 Get the process exact amount of shared memory

We note that within the proc/<pid> that describes the process information, there is a smaps file that shows information about all the memory segments, including Shared_clean Shared_dirty Private_clean Private_dirty: Several fields.

Find the relevant code, you can see, a page if the number of mappings >=2 count into shared_*; if = 1 counts to private_*. (The dirty page counts into the *_dirty, otherwise counts into the *_clean)

The sum of the shared_* values of all segments within the Smaps file is the amount of shared memory that is process accurate!

The sum of the private_* values of all the segments in the Smaps file is the total amount of exclusive memory that is process accurate!

3 Summary

Through the above analysis, we can get the following conclusions:

The L Top command parses/PROC/<PID>/STATM statistics Virt and res and SHR field values.

L Virt is the total amount of virtual memory requested.

L RES is the sum of the physical memory used by the process.

L SHR is the sum of the physical memory "mapped to a file" in Res. Including:

The code snippet for the program.

The code snippet for the dynamic library.

File mappings made through Mmap.

Anonymous mappings made through Mmap, but the map_shared attribute is indicated.

The shared memory requested by Shmget.

The shared_* in L/proc/<pid>/smaps is the physical memory >=2 the number of mappings in Res.

The private_* in L/proc/<pid>/smaps is the physical memory =1 the number of mappings in Res.

How does Linux process memory look at the Virt RES shr value displayed by the Anatomy top command