Data write mechanism of Linux system--deferred write

We all know that before Linux shuts down a command to run is the sync, this command is synchronous, so why run this? And before the data changes we've seen, why run this command? To answer this question, let's talk about the Linux implementation mechanism in this area.

First we start with the buffer and the cache, such as:

Both buffer and cache can be translated into caches, but what is the difference?

Cache

The purpose is for data reuse, to a certain extent, to solve the efficiency of reading, which is used to store the data often used, and not every time to go to the disk read, if the operation of the data is not used, it will go to disk search. This can be agreed to a certain degree of fast and slow equipment (such as: CPU and hard disk), in addition to the cache also has a swap-out mechanism, is the original often used, now not commonly used to clear out, so that there is room for the latest access to the data, in order to reach the next visit directly from the cache read.

Buffer

In order to improve the efficiency of writing such as disk, but also in order to coordinate the speed of the device, to avoid causing the data to write to the disk queue caused congestion, because the kernel submits the data to the write queue does not matter, it must wait until there is a return value. Therefore, the role of buffer is to write the data (the write () function in Linux) into buffer, and then back in the background to the other mechanism, the buffer data submitted to the queue, the final complete synchronization to disk process.

We know that the user-initiated program will run in the in-memory user space, when the data is in memory, if we need to save the data at this time, the system is actually called a write () function, and then call Sync () or Fsync () function (for any program that wants to write data to disk, the process is the same, with some exceptions).

By the way, that's why some people say Linux consumes memory more than Windows does.

User space: the area where the general process is located, the user initiates, and the code for this area does not have direct access to the hardware

kernel space: operating system area, access to hardware

When the write () function is called, once the function returns a normal value, we may think that the data has been written to the disk, but in fact, the operating system in the implementation of the disk file Io, in order to ensure the efficiency of the IO, in memory using a dedicated address space, which is called the kernel space, And in the kernel space there will be a data buffer for the IO (this buffer is buffered), the function of the write () function is to write the data into the kernel space of the IO buffer.

The IO buffer of the kernel space also has a certain size, and when the buffer is not full or there is no synchronization cycle, the data passed by the write () function is continuously written to the buffer, and when the buffer is full or a synchronization period, the contents of the buffer are committed to the output queue. When the data arrives at the queue first, the real disk IO operation begins, and the data is written to disk (although it is written to disk, but the real action is not the movement but the copy, and the IO buffer of the kernel space frees up the space occupied by the data after the copy is completed). This approach is called deferred write.

So there is a problem, when the write () function is called does not mean that the data is really saved to the disk, but there will be an illusion, that is, when you request the file again, you can display the last time you update the content, in fact, this content is not read from the disk, Instead, it reads from the buffer of the user space. Then the problem mentioned above, if the data in the kernel space in the IO buffer, and at this time the operating system failure, power outages and other anomalies will result in loss of data.

To address data loss issues, the UNIX system provides three functions for sync, Fsync, and Fdatasync.

Function	Function
Sync	The function returns 0 for success, which is responsible for pushing the modified contents of all the IO buffers in the kernel space to the input queue, and then returning, which does not wait for all disk IO operations to complete. So even if the sync function is called, it is not a successful save to disk.
Fsync	The function returns 0 for success, unlike sync, which only takes effect on a single file of the specified file descriptor, forces all modified data connected to the file to be transferred to disk, waits for disk IO to complete, and then returns. When the function returns 0 o'clock, it does not really represent a successful save to disk. The database calls Fsync () after the write () is called.
Fdatasync	It is similar to Fsync, which affects only the file Data section and does not involve data attributes such as inode information. So it requires less write-disk operations than Fsync.

If you look at the contents of the above, you should understand why you should run the sync command before shutting down.

Data write mechanism of Linux system--deferred write