Io_ Buffering and non-buffering of Linux system programming

The following is a log-like code, the message content of the communication and the content of the environment parameters, is to create a file, using standard IO fopen, fprintf for output records. But in the debugging, at first I was dumbfounded, the file was created successfully, but real-time view there is no data record. After half a day of fear and repeated troubleshooting, found to be the standard IO buffer mechanism put together, ashamed ah ...

Code Transfer from http://blog.csdn.net/mr_chenping/article/details/9166937

Here is an example program that simulates my project program:

#include <stdio.h> #include <stdlib.h> #include <string.h> #include <errno.h> #include <        Fcntl.h>int Main () {file* fp=null;        const char *filename_1= "Test_fprintf.log";        const char *filename_2= "Test_write.log";        int FD;        fp = fopen (Filename_1, "WB");                if (fp = = NULL) {printf ("Open%s failed,%s\n", Filename_1, Strerror (errno));        return-1;        }//setbuf (FP, NULL);        Set NULL for standard IO Auto Assign//set _IONBF to not buffer io//setvbuf (FP, NULL, _IONBF, 0); FD = open (filename_2, o_wronly| O_creat|        O_EXCL, 0666);                if (FD < 0) {printf ("Open%s failed,%s\n", Filename_2, Strerror (errno));        return-1;                } while (1) {fprintf (FP, "Test fprintf.\n");                fprintf (FP, "-------Test fprintf.\n");                fprintf (FP, "=======test fprintf.\n");        Can be refreshed to force full-buffer data to be passed to the kernel buffer zone        There is a kernel complete write disk operation//fflush (FP);                Write (FD, "Test open.\n", sizeof ("Test open.\n"));                Write (FD, "--------test open.\n", sizeof ("--------Test open.\n"));                Write (FD, "--------test open.\n", sizeof ("--------Test open.\n"));        Sleep (1);                                                 } return 0;}

Run the above example program in the background, and then view the two log files in real time, you will find that the Testfrpintf.log file is always empty at the beginning, while Testwrite.log is constantly having data written in the following states:

I was wondering why the file would be empty. It can be seen that standard IO buffers 4096Bytes of data, and the actual disk writes are made when so much data is reached, while the system call write is written directly without buffering.

　　The standard IO Library provides buffering to minimize the number of read and write calls, reducing the time to execute IO, which provides three types of buffering:

1. Fully buffered. After filling the standard IO buffer 4096Bytes (the buffer is full) for the actual IO operation (through the write system call, the data is passed to the kernel buffer zone, the final kernel writes the data to the disk), the disk file is usually the full buffer, the above example is the use of buffering.
2. Row buffers. The actual IO operation is encountered in the input and output (the buffer is full) (through the write system call, the data is passed to the kernel cache, the final kernel writes the data to the disk), and when a terminal is involved, a row buffer is usually used. The most frequently used printf function is a row buffer, so there is no sense of buffering.
3. With no buffering. The standard IO library does not buffer characters for storage. The standard error stream stderr is usually not buffered.

ISO C requires the following buffering characteristics:

• They are fully buffered when and only if the standard input and standard outputs do not involve interactive devices.
• Standard error is never a full buffer.

Many systems use the following types of buffering by default:

• Standard error is not buffered.
• In the case of other flows involving end devices, they are buffered; otherwise they are fully buffered.

Of course, for standard IO streams, we can also change the buffer type, or refresh it directly. The following two functions are available in ISO C to change the buffer type:

void Setbuf (FILE *fp, char *buf); BUF is null, which means to close the buffered int setvbuf (FILE *fp, char *buf, int mode, size_t size); Successful return 0, error returns non 0 value

The mode parameter in the SETVBUF function can be: _iobuf full buffer, _iolbf row buffer, _IONBF without buffering, if BUF is null, the standard IO Library will automatically allocate the appropriate length (constant bufsiz) buffer for the stream. In general, the length of the buffer should be selected by the system and the buffer automatically allocated so that the standard IO library will automatically release the buffer when the stream is closed.

To force flushing a stream, use the function:

int fflush (FILE *fp); Successful return 0, error return EOF

I use this function in the project to solve the depressed.

Fflush (NULL); Flush all output streams

　　Add a little bit of knowledge:

The read () and write () system calls do not directly initiate disk requests when manipulating disk files, but instead replicate data between the user space buffer and the kernel buffer cache. For example, the following call transfers 3 bytes of data from user-space memory to a buffer in kernel space.

Write (FD, "abc", 3);

Write () returns randomly. At a later point, the kernel writes (flushes) the data in its buffer to the disk. (therefore, it can be said that system calls are not synchronized with disk operations)

In the same vein, for input, the kernel reads data from the disk and stores it in the kernel buffer. The read () call reads the data from the buffer until the data in the buffer is read, and the kernel reads the next paragraph of the file into the buffer cache.

This design makes read () and write () fast and does not require a wait (slow) disk operation. At the same time, this design is extremely efficient because it reduces the number of disk transfers that the kernel has to perform. (pre-read and full write)

　　Two words:

1.read () and write () are responsible for replicating data in the user space buffer and the kernel cache cache.

2. The kernel is responsible for reading data from disk to the kernel cache (pre-read), and when the kernel buffer area is full, write to disk (full write).

To summarize:

　　Top-down, the first is to pass the user data through the Stdio library to the stdio buffer (typically 4096Bytes, or the standard IO can be automatically assigned), the buffer is located in the user-state memory area. When the buffer is full (the row buffer encounters ' \ n ', full buffer full 4096Bytes), the Stdio library calls the Write () system call and passes the data to the kernel buffer zone (in the kernel memory area). Eventually, the kernel initiates disk operations and passes the data to disk.

Use Fflush () to force a flush of the stdio buffer (via write () calls) to pass data to the kernel buffer zone.

The Fsync () SYN () system call will flush the buffered data and all metadata related to the open file descriptor fd to disk.

The first thing to understand is the concept of non-buffering: the so-called non-buffering, does not mean that the kernel does not provide buffering, but only a simple system call, not a function library call. In-system check disk read and write will provide a block buffer, when writing data with the Write function, call the system call directly, write data to block buffer to queue, when the block buffer reaches a certain amount, the data will be written to disk. Therefore, the so-called unbuffered I/O means that the process does not provide buffering capabilities. Each time a write or read function is called, a direct system call is called.
While buffered I/O is a process to improve the input and output stream, provides a stream buffer, when using the Fwrite function network disk write data, the data is written to the stream buffer, when a certain condition, such as a stream buffer full, or refresh the stream buffer, this time will send the data once to the kernel to provide the block buffer, It is then written to disk by block Buffering.
As a result, buffered I/O will have fewer calls to the system than non-buffered I/O when it writes the same amount of data to the disk.

Finally, take a picture to summarize.

Io_ Buffering and non-buffering of Linux system programming