Linux Learning--pipelines

This article briefly describes the pipeline in the operating system, and mainly addresses the following two issues:

1, the internal implementation of the pipeline

2, the capacity of the pipeline?

A pipeline is a way of communicating between different processes in the operating system.

Depending on the relationship between the processes of communication, pipelines are divided into anonymous and non-anonymous pipelines

Where anonymous pipes can only be used to communicate between processes that have a "blood relationship", and named pipes are used for any two-process communication

In addition, the pipeline is unidirectional, so that 2 pipelines can achieve two-way communication between processes

The principle of pipeline implementation:

We know that the data between processes is private, even the parent-child process, so, to allow 2 processes to share a certain data, we can create a file under the specified path, and then one of the processes will transfer the data to this file, and another process reads the file information, can achieve communication between processes, of course, considering the efficiency, it is usually impossible to take place on disk.

In addition, because of some mechanisms, pipelines provide "streaming" services, how much data to write one at a time, how much data to read at one time, do not need strict rules

Under Linux, the implementation of the pipeline does not use a dedicated data structure, but instead uses the file structure of the filesystem and the index node inode of the VFS.

The two file structure points to the same temporary VFS index node, and the VFS index node points to a physical page.

Such as:

650) this.width=650; "title=" 1.gif "alt=" Wkiol1d3p0ijrgycaabvcghd_1u664.gif "src=" http://s4.51cto.com/wyfs02/M01/ 83/96/wkiol1d3p0ijrgycaabvcghd_1u664.gif "/>

There are two file data structures in it, but they define the file operation routine address to be different, one of which is the routine address to write data to the pipeline, and the other is to read the data D routine address from the pipeline.

In this way, the user program system call is still the usual file operation, but the kernel uses this abstract mechanism to realize the special operation of the pipeline.

Inode node Information structure struct inode {...    struct pipe_inode_info  *i_pipe ;...  };//number of pipeline buffers #define pipe_buffers  (16)//Pipe buffer object Structure struct pipe_buffer {     struct page *page; //descriptor address of the Pipe buffer page box     unsigned int  offset, len; //the current position of valid data in the page box, and the length of the valid data     struct pipe_buf_operations * ops; //Pipeline Buffer Method table address};//pipeline information structure struct pipe_inode_info {    wait_queue_head_t  wait; //Pipeline Wait Queue     unsigned int nrbufs, curbuf; // The number of buffers containing the data to be read and the index of the first buffer containing the data to be read     struct pipe_buffer bufs[pipe_buffers]; // Pipe buffer Descriptor Array     struct page *tmp_page; //cache area Page box pointer      unsigned int start;  //current pipeline buffer read Location     unsigned int readers;  //read process flag, or number   &NBsp; unsigned int writers; //Write process flags, or numbering     unsigned int  waiting_writers; //the number of write processes in the waiting queue for sleep     unsigned int r_counter; // Similar to readers, but when the process waiting to write to the FIFO is using     unsigned int w_counter; //similar to writers, However, when waiting for a process to write to the FIFO, use     struct fasync_struct *fasync_readers; //for asynchronous i/through the signal o Notify     struct fasync_struct *fasync_writers; //for asynchronous I/O notifications through the signal};

As for the VFS objects mentioned above, in later versions of linux2.6, these objects are organized into PIPFS special file systems to speed up their processing

Capacity of the pipe:

If the pipeline is full, then the write end will not continue to write the data to the pipeline.

So how big is the pipeline capacity?

In practice, there are two concepts that are critical to understanding pipelines. One is the pipe capacity. The other is the atomic nature of the pipe operation. Pipe capacity is limited. If the pipeline is full, the write operation is blocked in blocking mode, and the failure is returned in a non-blocking manner. Different systems have different pipe capacity limits. The application module should not rely on specific capacity constraints, the correct design is: Once the data arrives process should consume data as soon as possible, avoid the writing process to block for a long time. Starting with the Linux 2.6.11 version, the pipeline capacity is 65536 bytes. POSIX 1-2001 Specifies that atomic operations are to be written to the pipe with data less than pipe_buf bytes long, and that data written longer than Pipe_buf bytes is not atomic. Linux on the Pipe_buf is 4096 bytes, more detailed description: 1, blocking mode, N<=PIPE_BUF (n is the number of bytes written, the same below): The write operation is atomic operation, if the PIPE space is not enough to block. 2, non-blocking mode, N<=pipe_buf: Write operation is atomic operation, if the PIPE space is insufficient, then failure, errno set to Eagain. 3, blocking mode, N>pipe_buf: Write operation is not an atomic operation, the data written may be in the cross-arrangement with other process writes, the write operation is blocked until all the data is finished. 4, non-blocking mode, N>pipe_buf: Write operation is not atomic operation, if the PIPE space is insufficient, then failure, errno set to Eagain. The data written may be interleaved with data written by other processes. The actual write may be less than n (partial write), and the caller should check the length of the write actual write.

Three concepts: 1, page buffer size: 4K2, total buffer size: 64K1, <4k data is sent immediately, to page 2, >4k data, will be divided into multiple pages of data, sent in batches. function write either blocks or succeeds (copy all data to kernel buffer, there is no case of copy only part of data), abnormal swap back-1

Reference:

Http://www.cppblog.com/aaxron/archive/2014/03/24/206312.aspx

(Testing of Linux pipeline capacity)

Http://blog.sina.com.cn/s/blog_629b701e0100zrk3.html

(Implementation mechanism of Linux pipelines)

Linux Learning--pipelines