Linux Pipeline detailed

Original link: http://blog.csdn.net/qq_38646470/article/details/79564392

#符号表示
|and piping special image.
# Effect:
Pipeline is a very important way of communication in Linux, is to connect the output of one program directly to the input of another program, often said pipe refers to the nameless pipe, the nameless pipe can only be used in the relationship between the process, which is its most important difference with the famous pipe.
A well-known pipe called Named pipe or FIFO (first-in, in-out) can be created with function Mkfifo ().
# Implementation Mechanism
In Linux, pipelines are a very frequent communication mechanism. In essence, a pipeline is also a file, but it is different from the general file, the pipeline can overcome the use of files to communicate the two problems, in particular, the performance is:

1. Limits the size of the pipe. In fact, a pipeline is a fixed-size buffer. In Linux, the buffer size is 1 pages, or 4K bytes, so that it does not grow as much as the file does. Using a single fixed buffer can also cause problems, such as when writing a pipeline, which can become full when this happens, then the write () call to the pipeline will be blocked by default, waiting for some data to be read so that there is enough space for write () to call.
   2. The read process may also work faster than the write process. When all current process data has been read, the pipeline becomes empty. When this happens, a subsequent read () call will be blocked by default, waiting for some data to be written, which resolves the issue where the read () call returns the end of the file.

Note: Reading data from a pipe is a one-time operation, once the data is read, it is discarded from the pipeline, freeing up space to write more data.

Structure of the pipe

In Linux, the implementation of pipelines does not use a dedicated data structure, but instead uses the file structure of the filesystem and the index node inode of the VFS. By pointing two file structures to the same temporary VFS index node, the VFS index node points to a physical page.
#管道的读写
Pipeline implementation of the source code in FS/PIPE.C, there are many functions in the pipe.c, of which two functions are more important, namely the pipe reading function pipe_read () and the pipe write function Pipe_wrtie (). The pipe write function writes the data by copying the bytes to the physical memory pointed to by the VFS index node, while the pipe read function reads the data by copying the bytes in the physical memory. Of course, the kernel must use a mechanism to synchronize access to the pipe, and for this reason the kernel uses locks, waits for queues, and signals.
When the write process writes to the pipeline, it takes advantage of the standard library function write (), and the file structure of the file can be found by the system based on the document descriptor passed by the library function. The file structure specifies the address of the function (that is, the write function) that is used for the write operation, and the kernel calls the function to complete the write operation. Before the Write function writes data to memory, it must first check the information in the VFS index node and meet the following criteria for actual memory copy work:
1. There is enough space in the memory to hold all the data to be written;
2. The memory is not locked by the read program.
If the above conditions are met, the Write function first locks the memory and then copies the data from the address space of the write process to memory. Otherwise, the write process sleeps in the wait queue of the VFS index node, and then the kernel calls the scheduler, and the scheduler chooses other processes to run. The write process is actually in an interruptible wait state, and the write process will receive a signal when there is enough space in the memory to hold the write data, or when the memory is unlocked, the read process wakes the write process. When the data is written to memory, the memory is unlocked, and all the read processes that hibernate on the index node are awakened.
The read process of the pipeline is similar to the write process. However, the process can return an error message immediately when no data or memory is locked, rather than blocking the process, depending on the open mode of the file or pipeline. Conversely, a process can hibernate the wait queue in the index node waiting for the write process to write to the data. When all processes have completed the pipeline operation, the index node of the pipeline is discarded, and the shared data page is freed.
Because the implementation of the pipeline involves many file operations, it is not difficult to understand when the reader learns the code in the pipe.c after learning about the file system.
The Linux pipeline has a limit on the size of the previous write operation that was blocked. The kernel-level buffers that are used specifically for each pipe are exactly 4096 bytes. Write operations that exceed 4K at a time will be blocked unless the reader empties the pipeline. This is actually not a limitation, because the read and write operations are implemented in different threads.

Resources:
What is a Linux pipeline

Linux Pipeline detailed