Introduction to memory management features of Linux

This chapter describes the features of Linux memory management, namely, virtual memory and disk buffering. Describes the purpose, working principle, and other things of memory management to be taken into account by the system administrator.

What is virtual memory?

Linux supports Virtual Memory. Virtual Memory refers to the use of disks as Ram extensions, so that the available memory size increases accordingly. The kernel will write the content of memory blocks that are not currently used to the hard disk, so that this memory can be used for other purposes. When raw content needs to be used, they are re-read into the memory. These operations are completely transparent to users. Programs Running in Linux only see a large amount of memory for use, and do not notice that some of them reside on the hard disk from time to time. Of course, reading and writing hard disks is much slower than using real memory directly (several thousand times slower), so the program will not run as fast as it has been running in the memory. The hard disk used as the virtual memory is called the swap space (SWAP)
Space ).

Linux can use a regular file or an independent partition in the file system as a swap space. Swap partition is faster, but it is easy to change the size of the swap file (you do not need to re-partition the entire hard disk, and you can install anything from the temporary partition ). When you know how much swap space you need, you should use swap partitions, but if you are not sure, you can first use a swap file and then use the system for a while, you can feel the size of the swap space. At this time, you can create a swap partition when you are sure of its size.

You should know that Linux allows several swap partitions and/or swap files at the same time. This means that if you only occasionally need another swap space, you can set an additional swap file at that time instead of allocating it all the time.

Operating System terminology Note: Computer Science often exchanges [swapping] (write the entire process to the swap space) and page scheduling [paging] (at a certain time point, only several thousand bytes of fixed size are written to the swap space. Page scheduling is usually more effective, which is also the practice of Linux, but the traditional Linux term refers to the exchange.

Create a swap space

A swap file is a common file. It is not special to the kernel. It is related to the kernel because it cannot have holes, and it is prepared using mkswap. Moreover, it must reside on a local hard disk and cannot reside in a file system loaded through NFS for implementation reasons.

Holes are important. Swap files keep disk space, so that the kernel can quickly swap out pages without assigning disk sectors to files. The kernel only uses any sector that has been allocated to the swap file. Because a hole in the file means that there is no disk sector allocation (the corresponding part of the hole to the file), this type of file with holes cannot be used for the kernel.

A good way to create a non-hole swap file is by using the following command:

$ dd if=/dev/zero of=/extra-swap bs=1024 count=1024 \

The above/extra-swap is the name of the swap file, and the size is given by the value after COUNT =. The memory pages written by the kernel are 4 kilobytes in size. If the size is not a multiple of 4, the last several kilobytes will not be used.

There is nothing special about a swap partition. You can create a partition like other partitions. The only difference is that it is used as an original partition, that is, it does not include any file system. Marking swap partitions as type 82 (Linux swap partitions) is a good idea; this will make the list of partitions clearer, though not necessarily for the kernel.

After creating an swap file or a swap partition, you must write a signature at the beginning of it. This signature includes some management information used by the kernel. The command \ cmd {mkswap} is used as follows:

$ mkswap /extra-swap 1024Setting up swapspace, size = 1044480 bytes

Note that the swap space is not used yet: it already exists, but the kernel does not use it as the virtual memory. You must be very careful when using mkswap because it does not check whether this file or partition is used by others. You can easily use mkswap to overwrite important files and partitions! Fortunately, you only need to use mkswap when installing the system.

Linux memory management program limits each swap space to a maximum of about 127 MB (due to various technical reasons, the actual limit size is (4096-10) * 8*4096 = 133890048 $ bytes, or 127.6875 MB ). However, you can use up to 16 swap spaces at the same time, with a total capacity of almost 2 GB.

Use of swap space

An initialized swap space is put into use using the command Swapon. This command tells the kernel that the swap space can be used. The path to the swap space is given as a parameter. Therefore, the command for starting to use the swap on a temporary swap file is as follows:

$ swapon /extra-swap

You can automatically use the swap space by adding it to the/etc/fstab file.

/dev/hda8 none swap sw 0 0/swapfile none swap sw 0 0

To start the description file, run the Swapon-a command to start all swap spaces listed in/etc/fstab. Therefore, the Swapon command is generally only used when an additional swap space is required.

You can use the free command to monitor the usage of the swap space. It will show how much swap space is used.

total used free shared buffersSwap: 32452 6684 25768

The first output line (MEM :) shows the usage of the physical memory. The total column does not show the memory used by the kernel, which is usually close to 1 MB. Used column shows the total used memory (the second row does not include the buffer ). The free column shows all unused idle memory. The shared column shows the size of memory shared by several processes. The more shared memory, the better the situation. Cache column (Buffer
Column) shows the size of the current disk cache. The cached column shows the size of the used cache.

The last line (SWAP :) shows the information corresponding to the swap space. If the value of this row is zero, your swap space is not hit.

You can also use the top command to obtain the same information, or use the file/proc/meminfo In the proc file system. It is usually difficult to obtain the usage of the specified swap space.

You can run the swapoff command to remove a swap space. This is usually not necessary, except for temporary swap space. In general, pages in the swap space are first swapped into the memory; if there is not enough physical memory to hold them, they will be swapped out (to other SWAP spaces ). If you do not have enough virtual memory to accommodate all these pages, Linux will fluctuate and become abnormal. However, after a long period of time, Linux will recover, but the system is no longer available. Before moving a swap space, you should check (for example, use free) whether there is enough free memory.

All swap spaces automatically used by Swapon-A can be moved using the swapoff-a command. For this command, refer to the/etc/fstab file to determine what to move. Any swap space manually configured and used will always be available.

Sometimes, although there is a lot of idle memory, there is still a lot of swap space in use. This may happen. For example, if there is a need for swap at a certain moment, but then a large process that occupies a lot of physical memory ends and releases the memory. The exchanged data is not automatically exchanged into the memory unless necessary. In this case, the physical memory remains idle for a period of time. There is nothing to worry about, but we can feel relieved when we know what it is.

Many operating systems use virtual memory. Because they only need swap space at run time, that is, they will never use swap space at the same time. Therefore, in addition to the swap space of the currently running operating system, others are a waste. So it will be more efficient for them to share a swap space. This is possible, but you need some knowledge. The howto tip documents contain some suggestions on how to implement this practice.

Some people will tell you that you need to allocate swap space by twice the physical memory capacity, but this is wrong. The following is a suitable practice:

. Estimate your total memory requirements. This is the maximum memory capacity you need at a certain time point, that is, the total memory required for all programs you want to run at the same time point. You can do this by running all programs at the same time.

For example, if you want to run X, you will allocate about 8 MB of memory to it, and GCC needs several megabytes (some files require an unusually large amount of memory, up to dozens of megabytes, but usually about 4 MB should be enough), and so on. The kernel itself needs about 1 MB of bytes, a common shell, and other tools may need several hundred kilobytes (that is, the total size is 1 MB ). There is no need for accurate calculation, and the crude rate estimation is enough, but you must consider the worst case.

Note that if several people use this system at the same time, they will all consume memory. However, if two people run a program at the same time, the total memory consumption is not doubled, because only one copy of the code page and shared library exists.

The free and PS commands are helpful for estimating the required memory capacity.

The estimation in step 1 is relaxed. This is because the estimation of the amount of memory occupied by the program is usually inaccurate, because you may forget a few programs you want to run, and, make sure you have extra space to use, just in case. This takes several megabytes. (It is better to allocate more swap space than to allocate less swap space, but it does not need to be too much so that the entire hard disk is used, because swap space is a waste of space. See the following section to increase swap space .) Similarly, because the processing value is better, you can increase the capacity value to an integer MB.

Based on the above calculation, you will know the total amount of memory you will need. Therefore, in order to allocate swap space, you only need to subtract the actual physical memory capacity from the total memory required, and you will know how much swap space you need. (In Some UNIX versions, you also need to allocate space for the physical memory image, so the total amount calculated in step 2 is the capacity of the swap space you need, instead of performing the subtraction operation .)

If the calculated swap space is much larger than your physical memory (more than twice), you usually need to buy some memory. Otherwise, the system performance will be very low.

It is a good idea to have a few swap spaces, even if the calculation points out that you do not need one. The swap space is often used in Linux to keep as much free physical memory as possible. Even if there is no need for memory, Linux will swap out memory pages temporarily unused. This avoids the time required to wait for the swap: When the disk is idle, the swap can be done in advance.

Swap space can be dispersed on several hard disks. This improves the performance of related disks and access modes. You may want to experiment with several schemes, but you need to realize that these experiments are often very difficult. Don't trust one solution, because this is not always the case.

High-Speed Buffer

Compared with accessing (real) memory, disk [3] reads and writes are very slow. In addition, it is common to read the same part of the disk multiple times in a relatively short period of time. For example, someone may first read an e-mail message, then read the message into the editor for a reply, and then copy the message to the folder, make the email program read it again. Or consider how many times the LS command will be used in a system with many users. By reading information from the disk only once and saving it in the memory, you can speed up all other reads except for the first read. This is called Disk buffer (Disk
Buffer cache ).

Unfortunately, because memory is a limited and inadequate resource, high-speed buffering cannot do much (it cannot accommodate all the data that needs to be used ). When the buffer is full of data, the data that is not used for the longest time will be discarded to free up memory space for new data.

The disk Buffer technology is equally effective for disk write operations. On the one hand, data written to the disk is often quickly read (for example, the original code file is saved to a file and read by the compiler ), therefore, it is a good idea to buffer the data to be written. On the other hand, programs can speed up operation by placing data in a buffer instead of writing it to a disk immediately. Later, write operations can be completed in the background without delaying program execution.

Most operating systems have high-speed buffering (although different), but not all follow the above principles. Write-through: data is immediately written to the disk (of course, data is also cached ). If the write operation is performed later, the cache is called write-back ). Background writing is more effective than direct writing, but it is also prone to errors: if the machine crashes, or suddenly loses power, or the data written by the floppy disk in the buffer is removed from the drive before being written to the floppy disk, the changed data in the buffer is lost. If the data that has not been written contains important notes, this may even mean that the file system (if any) is incomplete.

For the above reason, do not turn off the power before using the appropriate Shutdown Process (see chapter 6), do not detach the floppy disk from the drive before detaching (if loaded, do not remove a floppy disk before any program that is using a floppy disk instructs you to complete the floppy disk operation and the floppy disk lights are off. The sync command flushes buffer forces all unwritten data to the disk to determine that all write operations have been completed. In traditional Unix systems, a program called update runs on the background and performs the sync operation every 30 seconds. Therefore, you do not need to use the sync command manually. In Linux, there is another background program, bdflush, which runs more frequently but is not a full synchronization operation, to avoid the disk freezing caused by frequent disk I/O operations.

In Linux, bdflush is started by update. There is usually no reason to worry about this, but if the bdflush process is dead for some reason, the kernel will warn you about it, and you have to start it manually (/sbin/update ).

Cache is not actually a buffer for files, but a buffer block. It is the smallest unit for disk I/O operations (in Linux, they are usually 1 kb ). In this way, the directory, the super block, the note data of other file systems, and the disk data of non-file systems can be buffered.

The buffer effect is mainly determined by its size. If the buffer size is too small, it is useless: it can only hold a little data, so when it is reused, all buffered data will be dumped. The actual size depends on the data read/write frequency and the frequency at which the same data is accessed. Only the experiment method can be used.

If the cache has a fixed size, the cache is too large or not, because it will make the idle memory too small and lead to swap operations (this is also slow ). To make the most effective use of the actual memory, Linux automatically uses all idle memory as the high-speed buffer. When the program requires more memory, it will automatically reduce the buffer size.

In Linux, you do not need to do anything for caching. It is completely automatic. You don't have to worry about shutting down and removing a floppy disk in addition to the steps mentioned above.