Linux Memory detailed

--linux Memory Explanation

-----------------2014/05/24

Linux's memory is not as intuitive as windows, and this article is going to cover Linux's memory in detail.

See here the memory information for the Linux command free and vmstat output.

Mem Line used represents the memory that has been allocated, and free represents the memory that has not been allocated, and GKFX represents shared memory.

Calculation method: Cache statistics line Free=mem (free) +mem (buffers) +mem (cached)

Shared memory

For shared memory, which is primarily used to share data between different processes in a UNIX environment, is a method of interprocess communication, and the general application does not request the use of shared memory, nor does the author verify the effect of shared memory on the above equation.

For example, Oracle for Linux platforms is a multi-process program, and the SGA and PGA are all data regions shared and communicated by each process, and all can see a large amount of shared memory allocated. While MySQL is multi-threaded, the system level behaves as a process and does not require shared memory.

What is the difference between Linux cache and buffer?

Access to files and access to the device under Linux is usually increased by the cache, which is the default behavior of the system. The cache needs to consume our memory, although this memory can be released voluntarily through commands such as Echo 3>/proc/sys/vm/drop_caches. But sometimes we still need to understand who is consuming our memory.

Let's start by understanding the memory usage:

[[email protected] ~]# free       total used       free     shared    buffers     cachedmem:      24676836     626568   24050268          0      30884     508312-/+ buffers/cache:      87372   24589464Swap:      8385760

Page cache and buffer cache have always been two more confusing concepts, and there are a lot of people on the internet arguing and guessing what the difference is between the two caches, and finally there is no unified and positive
It is true that the concept of page cache and buffer cache has bothered me during my time at work, but the two concepts are actually very clear when analyzed. If you can understand the nature of these two caches, then we may be more handy when analyzing IO issues. The page cache is actually for the file system, the file cache, and the data at the file level is cached to page cache. The logical layer of the file needs to be mapped to the actual physical disk, and this mapping is done by the file system. When the page cache's data needs to be refreshed, the data in the page cache is given to buffer cache, but this processing is simple after the 2.6 kernel, with no real cache operation.

Buffer cache is a caching of disk blocks that, in the absence of a filesystem, caches data directly into the buffer cache, for example, the file system's metadata is cached in buffer cache.

simply put, the page cache is used to cache the file data, and the buffer cache caches the disk data. in the case of a file system, the data is cached to the page cache, if the disk is read and written directly using a tool such as
DD, then the data is cached to buffer cache to add that each device in the filesystem layer will be assigned a def _blk_ops file operation method, this is the method of operation of the device, there will be a radix tree under the inode of each device, this radix tree will place the page of cached data. The page number will be displayed in the buffer column of the top program. If the device makes a file system, it generates an inode, which allocates operations such as Ext3_ops, which is the method of the file system, where there is also a radix tree under the inode, which caches the page of the file. The number of cached pages is counted in the cache column of the top program. As can be seen from the above analysis, the buffer cache and page cache in the 2.6 kernel are consistent in processing, but there is a conceptual difference between the page cache Cache,buffer for the file is the cache for the disk block data, that's all.

with great Systemtap, we can use the Stap script to understand who is consuming our cache:

#这个命令行用来调查谁在加数据入page_cache

[Email protected] ~]# stap-e ' probe Vfs.add_to_page_cache {printf ("dev=%d, devname=%s, ino=%d, index=%d, nrpages=%d/n", Dev, devname, Ino, Index, nrpages)} ' ... dev=2, devname=n/a, ino=0, index=2975, nrpages=1777dev=2, devname=n/a, Ino=0, Ind ex=3399, nrpages=2594dev=2, devname=n/a, ino=0, index=3034, nrpages=1778dev=2, devname=n/a, ino=0, index=3618, nrpages= 2595dev=2, devname=n/a, ino=0, index=1694, nrpages=106dev=2, devname=n/a, ino=0, index=1703, nrpages=107dev=2, devname= N/A, ino=0, index=1810, nrpages=210dev=2, devname=n/a, ino=0, index=1812, nrpages=211 ...

At this time we copy a large file:

[[email protected] ~]$ CP huge_foo.file bar# This time we can see the contents of the file is added to the cache: ... dev=8388614, Devname=sda6, ino= 2399271, index=39393, nrpages=39393dev=8388614, Devname=sda6, ino=2399271, index=39394, nrpages=39394dev=8388614, Devname=sda6, ino=2399271, index=39395, nrpages=39395dev=8388614, Devname=sda6, ino=2399271, index=39396, nrpages= 39396dev=8388614, Devname=sda6, ino=2399271, index=39397, nrpages=39397dev=8388614, Devname=sda6, ino=2399271, index= 39398, nrpages=39398dev=8388614, Devname=sda6, ino=2399271, index=39399, nrpages=39399dev=8388614, Devname=sda6, ino= 2399271, index=39400, nrpages=39400dev=8388614, Devname=sda6, ino=2399271, index=39401, nrpages=39401dev=8388614, Devname=sda6, ino=2399271, index=39402, nrpages=39402dev=8388614, Devname=sda6, ino=2399271, index=39403, nrpages= 39403dev=8388614, Devname=sda6, ino=2399271, index=39404, nrpages=39404dev=8388614, Devname=sda6, ino=2399271, index= 39405, nrpages=39405dev=8388614, Devname=sda6, ino=2399271, index=39406, nrpages=39406dev=8388614, Devname=sda6, ino=2399271, index=39407, nrpages=39407dev=8388614, Devname=sda6, ino=2399271, index= 39408, nrpages=39408dev=8388614, Devname=sda6, ino=2399271, index=39409, nrpages=39409dev=8388614, Devname=sda6, ino= 2399271, index=39410, nrpages=39410dev=8388614, Devname=sda6, ino=2399271, index=39411, nrpages=39411 ...