Linux memory allocation

In the Linux memory allocation mechanism, the first use of physical memory, when the physical memory is idle (enough), does not release its memory consumption, even if the memory-hogging program has been shut down, the program occupied by the memory used for cache use, for open programs, or read just accessed data will be faster.




A Let's take a look at an example of memory usage:
[Email protected] ~]$ free-m
Total used free shared buffers Cached
mem:72433 67075 5357 0 558 62221
-/+ buffers/cache:4295 68138
swap:72096 91 72004
The above results show the used of 67075M, but (-/+ Buffers/cache) minus buffers and cache results can be seen, so the current process actually consumes memory is 4296M.
It can be understood that in the memory allocation mechanism of Linux, the first use of physical memory, when the physical memory is idle (enough), does not release its memory consumption, even if the memory-hogging program has been shut down, the program occupies the memory used for cache use, for open programs, Or reading the data that has just been accessed will be faster.
As the above example: the use of 72433M of memory, 67075M is occupied, but the buuffer and cached parts as a cache, you can use hit rate to improve the efficiency of use, and this part of the cache is ready to release according to the instructions, we can think that this part of the memory is not actually used, You can also think of it as free.
So look at the memory currently being used by the process is used-(Buffers+cache), it can also be considered that if swap is not heavily used, mem is sufficient, only the mem is actually occupied by the current process (without the buffers and cache), will only be used for swap.
Two Impact of swap configuration on performance
Allocating too much swap space wastes disk space, and there is too little swap space for the system to get an error. If the system is running out of physical memory, the system will run slowly, but still be able to run, and if the swap space is exhausted, the system will have an error. For example, the Web server can derive multiple service processes (or threads) depending on the number of requests, and if the swap space is exhausted, the service process cannot start, and the "application is out of memory" error usually occurs, causing the deadlock of the service process to be severe. Therefore, the allocation of swap space is very important.

Typically, the swap space should be greater than or equal to the size of the physical memory, the minimum should not be less than 64M, usually the size of the swap space should be 2-2 of physical memory. 5 times times. However, depending on the application, there should be different configurations: if it is a small desktop system, you only need a small swap space, while the large server system requires different sizes of swap space depending on the situation. In particular, the database server and Web server, with the increase in traffic, the demand for swap space will also increase, generally for the physical memory below 4G, configuration twice times more than swap,4g configuration 1 time times.
In addition, the number of swap partitions has a significant impact on performance. Because swap operations are disk IO operations, if there are multiple swap zones, the swap space is allocated in a rotating manner to all swaps, which greatly balances the IO load and speeds up swap. If there is only one swap area, all switching operations will make the swap area very busy, leaving the system most of the time in a waiting state and inefficient. With the performance monitoring tool you will find that the CPU is not very busy at this time, but the system is slow. This shows that the bottleneck on the IO, relying on the speed of the CPU can not solve the problem.
Three. Linux Memory mechanism
Linux supports virtual memory (mmemory), virtual memory refers to the use of disk as RAM expansion, so that the size of the available memory increases accordingly. The kernel writes the contents of a memory block that is temporarily unused to the hard disk, so that this memory can be used for other purposes. When the original content needs to be used, they are re-read into memory. These operations are completely transparent to the user; Programs running under Linux just see a lot of memory available and don't notice that part of them is residing on the hard drive from time to time. Of course, reading and writing hard drives is much slower than using real memory directly (thousands of times times slower), so the program will not be as fast as it has been running in memory. The portion of the hard disk used as virtual memory is called swap space.
In general, the pages in the swap space are first swapped into memory, and if there is not enough physical memory at this time to accommodate them, they will be swapped out (into other swap spaces). If there is not enough virtual memory to accommodate all of these pages, Linux will fluctuate and not be normal, but after a long time Linux will recover, but the system is not available at this time.
Sometimes, although there is a lot of free memory, there is still a lot of swap space being used. This can happen, for example, if there is a need to exchange at some point, but later a large process that takes up a lot of physical memory ends and frees up memory. The data that is exchanged is not automatically swapped into memory unless it is needed. At this point, the physical memory remains idle for a period of time. There's nothing to worry about, but it doesn't matter what it is.
Many operating systems use the method of virtual memory. Because they only need to swap space at run time to resolve that swap space is not being used at the same moment, the rest is a waste, in addition to the swap space of the operating system that is currently running. So it would be more efficient to have them share a swap space.
Note: If a few people use the system at the same time, they will consume memory. However, if two people run a program at the same time, the total amount of memory consumption is not doubled, because there is only one copy of the code page and the shared library.

Linux systems often use swap space at all times to keep as much free physical memory as possible. Even if there is nothing to do with memory, Linux will swap out the memory pages that are temporarily unused. This avoids the time required to wait for an exchange: When the disk is idle, it can be exchanged in advance. Swap space can be spread over several hard disks. This can improve performance for the speed of the associated disk and the access mode to the disk.

Disk reads and writes are slow compared to accessing physical memory. Also, it is common to read the same portion of the disk multiple times in the corresponding shorter time. For example, someone might read an e-mail message First, then read the message into the editor for a reply, and then copy the message to the folder again, so that the mail program reads it again. Or consider how many times the LS command will be used in a system with many users. By reading the information from disk only once and saving it in memory, you can speed up all other reads, in addition to the first read. This is called disk buffering, and the memory that is used for this purpose is called a cache buffer (buffer cache). However, because memory is a limited and insufficient resource, high-speed buffering is unlikely to do much (it cannot contain all the data that is needed). When the buffer is filled with data, the data that is not used for the longest time is discarded to free up memory space for new data.

The disk buffering technique is equally valid for write disk operations. On the one hand, the data that is written to the disk is often read out quickly (for example, the original code file is saved to a file and is read by the compiler), so it is a good idea to put the data that is being written into the buffer. On the other hand, the program can speed up the operation by putting data into the buffer instead of writing it to disk immediately. Later, the write operation can be completed in the background without delaying the execution of the program.
Most operating systems have high-speed buffering (although they may be called differently), but not all of them adhere to the above principles. Some are direct writes (Write-through): Data is immediately written to disk (and of course, the data is also put into the cache). If the write operation is done later, then the cache is called background write (Write-back). Background writing is more efficient than direct writing, but it's also easy to make mistakes: if a machine crashes or suddenly loses power, the data changed in the buffer is lost. If the data that is still not written contains important, thin information, it may even mean that the file system, if any, is incomplete.
For the above reasons, there are a lot of log file system, the data in the buffer after modification, the file system will also be recorded modification information, so that even if the system power down, the system restarts will first recover data from the log records, to ensure that the data is not lost. Of course these questions are no longer covered by this article.
For these reasons, never turn off the power supply before using the appropriate shutdown procedure, the Sync command empties (flushes) buffer, that is, forcing all data that is not written to be written to disk, to make sure that all writes have been completed. In a traditional UNIX system, a program called update runs in the background and does a sync operation every 30 seconds, so there's usually no need to manually use the Sync command. Linux also has a daemon, Bdflush, which performs more frequent but not full sync operations to avoid the sudden freeze of disk I/O operations caused by sync at times.
In Linux, Bdflush is initiated by update. There is usually no reason to worry about this, but if for some reason the bdflush process dies, the kernel warns you that you are going to start it manually (/sbin/update).
The cache is not actually a buffer file, but a buffer block, which is the smallest unit of disk I/O operations (in Linux, they are usually 1KB). In this way, the thin data of directories, super blocks, other file systems, and disk data for non-file systems can be buffered. The effect of buffering is largely determined by its size. If the buffer is too small, it is useless. It can only hold a bit of data, so when reused, all buffered data is emptied. The actual size depends on the frequency of data read and write, and how often the same data is accessed. Only by means of experiments can we know.
If the cache has a fixed size, then the cache is too big to be good, because it causes the idle memory to be too small to make the swap operation (which is also slow). For the most efficient use of real memory, Linux automatically uses all the free memory as a buffer, and when the program needs more memory, it also automatically reduces the size of the buffer.
This is generally the general mechanism of Linux memory, the real Linux memory operating mechanism is far more complex than this.

Http://bbs.chinaunix.net/thread-3609448-1-1.html

Linux memory allocation