Analysis of virtual memory

I haven't updated my blog for a long time, and I 've been confused by the virtual memory recently. I 've been turning to Google for almost half a month and trying to get a clue, but I don't know much about the basic knowledge, let's look at what virtual space is. This article focuses on the analysis of virtual space occupied by a process.

First look at top

Top-18:34:57 up 4 days, 4 Users, load average: 0.00, 0.00, 0.00
Task: 114 total, 1 running, 113 sleeping, 0 stopped, 0 zombie
CPU (s): 0.2% us, 0.2% Sy, 0.0% Ni, 99.7% ID, 0.0% wa, 0.0% hi, 0.0% Si, 0.0% St
Mem: 4051424 k total, 3612920 K used, 438504 K free, 200040 K Buffers
Swap: 8385920 k total, 0 K used, 8385920 K free, 3114324 K cached

Here we will focus on swap. When the physical memory space is insufficient, the content of the virtual space of the process will be placed in the SWAp partition on the hard disk,

[Root @ web-dev-146 ~] # Cat/proc/SWAPs
Filename type size used priority
/Dev/sda3 partition 8385920 0-2

It is obvious that 8 GB is divided into hard disk partitions to store process virtual space content data.

For Shared Memory ing, the page missing exception handler first searches for the target page in the SWAp cache (the physical page that conforms to address_space and offset). If it is found, the address is directly returned. If it is not found, determines whether the page is in the SWAp area. If the page is in, perform a swap operation. If neither of the above conditions is met, the handler allocates a new physical page, and insert it into the page cache. The process will eventually update the process page table.

We will analyze the role of swap cache through the instance later.

The size in SWAp cached is

Free

Total used free shared buffers cached
Mem: 4051424 3613052 438372 0 200072 3114368
-/+ Buffers/cache: 298612 3752812
Swap: 8385920 0 8385920

It indicates that reading files occupies such a large amount of physical space and is used to cache opened files.

There is no need to release the operating system from the cache as much memory as needed, so you don't have to worry about it, however, through our example, we can see that swap cache is growing because we use file ing and read the file content.

You can manually release it through Echo 3>/proc/sys/Vm/drop_caches

Write a C program

# Include <sys/types. h>
# Include <sys/STAT. h>
# Include <fcntl. h>
# Include <stdio. h>
# Include <unistd. h>
# Include <sys/Mman. h>

Int main ()
{
Int FD;
Int I;
If (FD = open ("/home/MongoDB/shard200/fs_media_recommend.8", o_rdwr | o_creat, s_irwxu) <0 ){
Printf ("open file wrong! ");
Return 0;
}

Struct stat file_stat;
If (fstat (FD, & file_stat) <0)
{
Printf ("fstat wrong ");
Return 0;
}

Int * start_fp;
If (maid = (int *) MMAP (null, 2146435072, prot_read | prot_write, map_shared, FD, 0) = map_failed)
{
Printf ("MMAP wrong ");
Return 0;

}
For (I = 0; I <10000000; I ++ ){
Printf ("% d \ n", start_fp [I]);
}
/* Msync (start_fp, file_stat.st_size, ms_async );

If (munmap (start_fp, file_stat.st_size) <0)
{
Printf ("munmap wrong ");
Return 0;
}*/
Sleep (100000 );
}

Here we use MMAP file ing, And we can analyze the virtual address information of the process for such a long time.

Top-18:43:07 up 4 days, 4 Users, load average: 0.15, 0.05, 0.02
Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie
Cpu0: 0.0% us, 0.0% Sy, 0.0% Ni, 100.0% ID, 0.0% wa, 0.0% hi, 0.0% Si, 0.0% St
Cpu1: 0.0% us, 0.0% Sy, 0.0% Ni, 100.0% ID, 0.0% wa, 0.0% hi, 0.0% Si, 0.0% St
Mem: 4051424 k total, 3612804 K used, 438620 K free, 200052 K Buffers
Swap: 8385920 k total, 0 K used, 8385920 K free, 3114332 K cached
Cumulative time on
PID user PR Ni virt res shr s % CPU % mem time + swap code data command
16955 root 16 0 2050 m 25 m 25 M r 0.0 0: 25. 84 0.6g 4 2.0 test9

When you run the modified program, you will find that the res of the process is increasing, but virt, swap, and the above swap have not changed at all.

You can try to get rid of the code that I read from the ing. Once virt and swap have determined the size, the size is allocated with virtual space after the process is loaded, of course it is not really divided into 2 GB space, because the swap corresponding to the virtual space has not changed, it means that the content of the virtual space has not been written to the disk, in fact, let's take a look at the process virtual space analysis.

[Root @ web-dev-146 ~] # Ps-Ef | grep test9
Root 16968 9017 19 00:00:01 pts/2./test9
Root 16970 13123 0 00:00:00 pts/1 grep test9
[Root @ web-dev-146 ~] # Cat/proc/16968/maps
00400000-00401000 R-XP 00000000/root/test9
00600000-00601000 RW-P 00000000/root/test9
3c5a400000-3c5a41c000 R-XP 00000000 1405202/lib64/ld-2.5.so
3c5a61b000-3c5a61c000 r -- p 0001b000 08:02 1405202/lib64/ld-2.5.so
3c5a61c000-3c5a61d000 RW-P 0001c000 08:02 1405202/lib64/ld-2.5.so
3c5a800000-3c5a94d000 R-XP 00000000 1405203/lib64/libc-2.5.so
3c5a94d000-3c5ab4d000 --- P 0014d000 08:02 1405203/lib64/libc-2.5.so
3c5ab4d000-3c5ab51000 r -- p 0014d000 08:02 1405203/lib64/libc-2.5.so
3c5ab51000-3c5ab52000 RW-P 00151000 08:02 1405203/lib64/libc-2.5.so
3c5ab52000-3c5ab57000 RW-P 3c5ab52000 00:00 0
2ad9f31db000-2ad9f31dc000 RW-P 2ad9f31db000 00:00 0
2ad9f31ed000-2ad9f31ef000 RW-P 2ad9f31ed000 00:00 0
2ad9f31ef000-2ada730ef000 RW-s 00000000 60686353/home/MongoDB/shard200/fs_media_recommend.8
2ada730ef000-2ada730f0000 RW-P 2ada730ef000 00:00 0
7fffd6e4f000-7fffd6e64000 RW-P 7ffffea000 0 [Stack]
Ffffffffff600000-ffffffffffe00000 --- P 00000000 00:00 0 [vdso]

This is the virtual address space corresponding to the process. We can clearly see that the virtual space corresponds to the ing between files in my code. What is the size?

16968:./test9
0000000000400000 4 k r-X --/root/test9
0000000000600000 4 k rw ---/root/test9
2017003c5a400000 112 k r-X --/lib64/ld-2.5.so
2017003c5a61b000 4 k r ----/lib64/ld-2.5.so
2017003c5a61c000 4 k rw ---/lib64/ld-2.5.so
2017003c5a800000 1332 k r-X --/lib64/libc-2.5.so
2017003c5a94d000 2048 K -----/lib64/libc-2.5.so
2017003c5ab4d000 16 k r ----/lib64/libc-2.5.so
2017003c5ab51000 4 k rw ---/lib64/libc-2.5.so
2017003c5ab52000 20 k RW --- [anon]
20172ad9f31db000 4 k rw --- [anon]
20172ad9f31ed000 8 k rw --- [anon]
20172ad9f31ef000 2096128 k rw-S-/home/MongoDB/shard200/fs_media_recommend.8
20172ada730ef000 4 k rw --- [anon]
20177fffd6e4f000 84 k RW --- [Stack]
Ffffffffff600000 8192 K ----- [anon]
Total 2107968 K

Clearly, the size is the size mapped in my code. You can learn more about the virtual space.

[Root @ web-dev-146 ~] # Cat, Proc, 16968, and status
Name: test9
State: S (sleeping)
Sleepavg: 98%
Tgid: 16968
PID: 16968
Ppid: 9017
Tracerpid: 0
UID: 0 0 0 0
GID: 0 0 0 0
Fdsize: 256.
Groups: 0 1 2 3 4 6 10
Vmpeak: 2099776 KB
Vmsize: 2099776 KB
Vmlck: 0 KB
Vmhwm: 39440 KB
Vmrss: 39440 KB
Vmdata: 36 KB
Vmstk: 84 KB
Vmexe: 4 kb
Vmlib: 1444 KB
Vmpte: 116 KB
Stabrk: 1ede2000 KB
BRK: 1ede2000 KB
Stastk: 7fffd6e63850 KB
Threads: 1
Sigq: 1/36864
Sigpnd: 0000000000000000
Shdpnd: 0000000000000000
Sigblk: 0000000000000000
Sigign: 0000000000000000
Sigcgt: 0000000000000000
Capinh: 0000000000000000
Capprm: 00000000 fffffeff
Capeff: 00000000 fffffeff
Cpus_allowed: 00000000,00000000, 00000000,00000000, 00000000,00000000, 00000000,000 0000f
Mems_allowed: Invalid bandwidth, 00000001

We can see that the virtual space is 2099776kb, which is consistent with the data corresponding to the top process,

The above example shows that the virt virtual memory does not actually occupy the size of the physical memory and the latter disk space. This size is determined by the fact that the process has placed a ing table in the physical memory, no process is assigned. There is always another ing data structure, that is

When a page about the virtual address space of a process is loaded, a series of vm_area_struct will be automatically generated. Each vm_area_struct describes a part of the process, such as executing code and data. Linux virtual storage management supports most Standard Virtual Memory operations, such as reading, disabling, sharing, and page missing. Once the vm_area_struct structure is generated, you can perform virtual memory operations by pointing to the vm_operation_struct pointer in the structure.

3 shows the relationship between data structures managed by virtual storage.

Figure 3 Data Structure Relationship of virtual storage management

This structure references the online one. It is inevitable that the actual memory space is not too large. It only records the header and tail information in the virtual space allocation content, so you will know the size of the occupied virtual space.

We mentioned that virt is placed in the SWAp partition. Therefore, when the process is initialized, we find that virt and swap have the same size and are both virtual,

However, when we read the content of the ing file, the page is interrupted, and the content of the ing file is loaded into the real physical memory, which is the experiment we did above, we will find that the physical memory RES is constantly increasing. However, as some content has been imported into the physical memory, the operating system will release the swap disk space.

PID user PR Ni virt res shr s % CPU % mem time + swap code data command
9193 root 15 0 3974 M 3.0g 640 s 0.0 19.0 28:50. 33 933 M 328 3.9g redis-Server

At this time, we found that virt = swap + Res is very easy to understand. Since your physical memory has stored data, of course, I need to leave unnecessary redundant space for other purposes.

First, I will add what I think.