Inter-process communication in Linux: pipelines and famous Pipelines

Pipelines and famous Pipelines

In this series, the author outlines several main methods for inter-process communication in Linux. Among them, pipelines and named pipelines are one of the earliest inter-process communication mechanisms. pipelines can be used for inter-process communication with kinship. Famous pipelines overcome the restrictions that pipelines do not have names, in addition to the functions of pipelines, it also allows communication between unrelated processes. Recognizing the read and write rules of pipelines and named pipelines is the key to applying them in programs. This article discusses in detail the communication mechanism of pipelines and named pipelines, the instance is used to verify its read/write rules by program. This helps to enhance the reader's perceptual knowledge of the read/write rules, and also provides application examples.

1. Pipeline overview and related API applications

1.1 key concepts related to pipelines

As one of the first Unix IPC formats supported by Linux, pipelines have the following features:

• The pipeline is half-duplex, and data can only flow in one direction. when both parties need to communicate, two pipelines need to be established;
• It can only be used between parent and child processes or sibling processes (kinship processes );
• An independent file system is formed: An MPs queue is a file for processes at both ends of the MPs queue, but it is not a common file. It does not belong to a file system, but a self-built portal, A file system exists only in memory.
• Data Reading and Writing: The content written by a process to the pipeline is read by the process at the other end of the pipeline. The written content is added at the end of the MPs queue buffer, and data is read from the buffer header each time.

1.2 create an MPS queue:

#include <unistd.h>
int pipe(int fd[2])

The two ends of the pipeline created by this function are in the middle of a process, which does not make much sense in actual application. Therefore, after a process creates a pipeline by pipe (), it generally fork another sub-process, then, the communication between parent and child processes is realized through the pipeline (so it is not difficult to launch. As long as there is a kinship between the two processes, the kinship here refers to a common ancestor, you can use pipelines for communication ).

1.3 MPs queue read/write rules:

The describe characters FD [0] and FD [1] can be used to describe the two ends of the MPs queue respectively. Note that the two ends of the MPs queue are fixed with tasks. That is, one end can only be used for reading, represented by the descriptive word FD [0], called the pipeline Reading end; the other end can only be used for writing, represented by the descriptive word FD [1, it is called the pipe write end. An error occurs if you try to read data from the write end of the pipeline or write data to the read end of the pipeline. Generally, file I/O functions can be used in pipelines, such as close, read, and write.

Read data from the MPs queue:

• If the write end of the pipeline does not exist, it is deemed that the data has been read to the end, and the number of read bytes returned by the READ function is 0;
• When the write end of the pipeline exists, if the number of bytes requested is greater than pipe_buf, the number of existing data bytes in the pipeline is returned. If the number of bytes requested is not greater than pipe_buf, return to the existing data bytes in the MPs Queue (in this case, the data volume in the MPs queue is smaller than the requested data volume), or return the number of bytes of the request (in this case, the data volume in the MPs queue is not smaller than the requested data volume ). Note: (pipe_buf is defined in include/Linux/limits. H. Different kernel versions may vary. Posix.1 requires that pipe_buf be at least 512 bytes, and Red Hat 7.2 is 4096 ).

Read rule verification for pipelines:

/************** * Readtest. C * **************/ # Include <unistd. h> # Include <sys/types. h> # Include <errno. h> Main () { Int pipe_fd [2]; Pid_t PID; Char r_buf [100]; Char w_buf [4]; Char * p_wbuf; Int r_num; Int cmd; Memset (r_buf, 0, sizeof (r_buf )); Memset (w_buf, 0, sizeof (r_buf )); P_wbuf = w_buf; If (pipe (pipe_fd) <0) { Printf ("pipe create error/N "); Return-1; } If (pid = fork () = 0) { Printf ("/N "); Close (pipe_fd [1]); Sleep (3); // make sure the parent process closes the write end R_num = read (pipe_fd [0], r_buf, 100 ); Printf ("read num is % d the data read from the pipe is % d/N", r_num, atoi (r_buf )); Close (pipe_fd [0]); Exit (); } Else if (pid> 0) { Close (pipe_fd [0]); // read Strcpy (w_buf, "111 "); If (write (pipe_fd [1], w_buf, 4 )! =-1) Printf ("parent write over/N "); Close (pipe_fd [1]); // write Printf ("parent close FD [1] Over/N "); Sleep (10 ); } } /*************************************** *********** * Program output result: * Parent write over * Parent close FD [1] Over * Read num is 4 the data read from the pipe is 111 * Additional conclusions: * After the pipeline write end is closed, the written data will remain there until it is read. **************************************** ************/

Write Data to the MPs queue:

• When writing data to an MPS queue, Linux does not guarantee the atomicity of writing data. As soon as the MPs queue buffer has an idle area, the write process tries to write data to the MPs queue. If the read process does not read data from the buffer in the pipeline, the write operation will be blocked.
  Note: It is meaningful to write data to the MPs queue only when the read end of the MPs queue exists. Otherwise, the process that writes data to the pipeline receives the sifpipe signal from the kernel. The application can process the signal or ignore it (the default action is to terminate the application ).

Verification of write rules for pipelines 1: dependency of the write end on the read end

# Include <unistd. h> # Include <sys/types. h> Main () { Int pipe_fd [2]; Pid_t PID; Char r_buf [4]; Char * w_buf; Int writenum; Int cmd; Memset (r_buf, 0, sizeof (r_buf )); If (pipe (pipe_fd) <0) { Printf ("pipe create error/N "); Return-1; } If (pid = fork () = 0) { Close (pipe_fd [0]); Close (pipe_fd [1]); Sleep (10 ); Exit (); } Else if (pid> 0) { Sleep (1); // wait until the sub-process completes the operation to close the read end Close (pipe_fd [0]); // write W_buf = "111 "; If (writenum = write (pipe_fd [1], w_buf, 4) =-1) Printf ("write to pipe error/N "); Else Printf ("the bytes write to pipe is % d/N", writenum ); Close (pipe_fd [1]); } }

The output result is: broken pipe because the pipeline and the read ends of all its fork () products have been closed. If the read end is retained in the parent process, that is, after writing pipe, the read end of the parent process is shut down and pipe is written normally. You can verify this conclusion by yourself. Therefore, when writing data to the MPs queue, there should be at least one process, in which the read end of the MPs queue is not closed; otherwise, the above error will occur (pipeline split, the process receives the sigpipe signal. The default action is Process Termination)

Verification of write rules for pipelines 2: Linux does not guarantee atomic verification of write Pipelines

# Include <unistd. h> # Include <sys/types. h> # Include <errno. h> Main (INT argc, char ** argv) { Int pipe_fd [2]; Pid_t PID; Char r_buf [4096]; Char w_buf [4096*2]; Int writenum; Int rnum; Memset (r_buf, 0, sizeof (r_buf )); If (pipe (pipe_fd) <0) { Printf ("pipe create error/N "); Return-1; } If (pid = fork () = 0) { Close (pipe_fd [1]); While (1) { Sleep (1 ); Rnum = read (pipe_fd [0], r_buf, 1000 ); Printf ("Child: readnum is % d/N", rnum ); } Close (pipe_fd [0]); Exit (); } Else if (pid> 0) { Close (pipe_fd [0]); // write Memset (r_buf, 0, sizeof (r_buf )); If (writenum = write (pipe_fd [1], w_buf, 1024) =-1) Printf ("write to pipe error/N "); Else Printf ("the bytes write to pipe is % d/N", writenum ); Writenum = write (maid [1], w_buf, 4096 ); Close (pipe_fd [1]); } } Output result: The Bytes write to pipeline 1000 The Bytes write to pipe 1000 // note that this row output shows the Non-atomicity of write The Bytes write to pipeline 1000 The Bytes write to pipeline 1000 The Bytes write to pipeline 1000 The Bytes write to pipe 120 // note that this row output shows the Non-atomicity of write The Bytes write to pipe 0 The Bytes write to pipe 0 ......

Conclusion:

When the number of writes is less than 4096, the write is non-atomic!
If you change the number of bytes written to the parent process to 5000 twice, you can easily draw the following conclusion:
When the amount of data written to the MPs queue exceeds 4096 bytes, the idle space in the buffer zone is written into the data (completed) until all data is written. If no process reads data, the data is blocked.

1.4 MPs queue application example:

Instance 1: used for Shell

A pipeline can be used to redirect input and output. It directs the output of one command to the input of another command. For example, after you type who │ WC-l in a shell program (such as Bourne shell or C shell, the corresponding shell program will create the pipes between the WHO and WC processes and the two processes. Consider the following command line:

For $ kill-l running results, see Appendix 1.

$ Kill-L | grep sigrtmin running result:

30) SIGPWR 31) SIGSYS 32) SIGRTMIN 33) SIGRTMIN+1 34) SIGRTMIN+2 35) SIGRTMIN+3 36) SIGRTMIN+4 37) SIGRTMIN+5 38) SIGRTMIN+6 39) SIGRTMIN+7 40) SIGRTMIN+8 41) SIGRTMIN+9 42) SIGRTMIN+10 43) SIGRTMIN+11 44) SIGRTMIN+12 45) SIGRTMIN+13 46) SIGRTMIN+14 47) SIGRTMIN+15 48) SIGRTMAX-15 49) SIGRTMAX-14

Example 2: used for Kinship-related inter-process communication

The following example shows the specific application of the pipeline. The parent process sends some commands to the child process through the pipeline. The child process parses the commands and processes the commands accordingly.

# Include <unistd. h> # Include <sys/types. h> Main () { Int pipe_fd [2]; Pid_t PID; Char r_buf [4]; Char ** w_buf [256]; Int childexit = 0; Int I; Int cmd; Memset (r_buf, 0, sizeof (r_buf )); If (pipe (pipe_fd) <0) { Printf ("pipe create error/N "); Return-1; } If (pid = fork () = 0) // Sub-process: parses the commands obtained from the pipeline and processes them accordingly. { Printf ("/N "); Close (pipe_fd [1]); Sleep (2 ); While (! Childexit) { Read (pipe_fd [0], r_buf, 4 ); Cmd = atoi (r_buf ); If (cmd = 0) { Printf ("Child: receive command from parent over/n now child process exit/N "); Childexit = 1; } Else if (handle_cmd (CMD )! = 0) Return; Sleep (1 ); } Close (pipe_fd [0]); Exit (); } Else if (pid> 0) // Parent: send commands to child { Close (pipe_fd [0]); W_buf [0] = "003 "; W_buf [1] = "005 "; W_buf [2] = "777 "; W_buf [3] = "000 "; For (I = 0; I <4; I ++) Write (pipe_fd [1], w_buf [I], 4 ); Close (pipe_fd [1]); } } // The following is the sub-process command processing function (specific to the Application ): Int handle_cmd (INT cmd) { If (CMD <0) | (CMD> 256 )) // Suppose child only support 256 commands { Printf ("Child: Invalid command/N "); Return-1; } Printf ("child: the CMD from parent is % d/N", CMD ); Return 0; }

1.5 limitations of Pipelines

The main limitations of MPs queue are as follows:

• Only unidirectional data streams are supported;
• It can only be used between unrelated processes;
• No name;
• The buffer of the MPs queue is limited (the MPs queue is in the memory and a page size is allocated to the buffer when the MPs queue is created );
• The pipeline transmits a non-formatted byte stream, which requires that the reader and writer of the pipeline have to specify the data format in advance, for example, how many bytes are counted as a message (or command, or record) and so on;

2. Overview of famous pipelines and related API applications

2.1 key concepts related to famous Pipelines

A major restriction of a pipeline application is that it has no name, so it can only be used for Kinship-related inter-process communication. After a famous Pipeline (named pipe or FIFO) is proposed, this restriction is overcome. FIFO is different from pipelines in that it provides a path name associated with it and exists in the file system as a FIFO file. In this way, the process does not have a kinship with the FIFO creation process, as long as the path can be accessed, it can communicate with each other through FIFO (the process that can access this path and the process that creates the FIFO). Therefore, data can be exchanged through non-FIFO processes. It is worth noting that the first in first out (FIFO) is strictly followed, and data is always returned from the beginning for the read of the pipeline and the first in first out (FIFO, write to them to add the data to the end. They do not support file location operations such as lseek.

2.2 creation of famous Pipelines

#include <sys/types.h> #include <sys/stat.h> int mkfifo(const char * pathname, mode_t mode)

The first parameter of the function is a common path name, that is, the name of the first FIFO after the function is created. The second parameter is the same as the mode parameter in the open () function for opening a common file. If the first parameter of mkfifo is an existing path name, The eexist error is returned. Therefore, the typical Call code first checks whether the error is returned. if the error is returned, you only need to call the function to open the FIFO. General file I/O functions can be used for FIFO, such as close, read, write, and so on.

2.3 opening rules for famous Pipelines

A famous Pipeline has one more open operation than the pipeline: open.

FIFO open rules:

If a read-only FIFO is enabled for the current open operation, if a corresponding process has enabled this FIFO for writing, the current open operation will be successful; otherwise, it may be blocked until a corresponding process opens the FIFO for writing (the blocking flag is set for the current open operation); or, it will be returned successfully (the blocking flag is not set for the current open operation ).

If the current enable operation is to enable the FIFO for writing, if a corresponding process has opened the FIFO for reading, the current enable operation will be successful; otherwise, it may be blocked until a corresponding process opens the FIFO for reading (the blocking flag is set for the current open operation); or, an enxio error is returned (the blocking flag is not set for the current open operation ).

For more information about how to verify an open rule, see appendix 2.

2.4 read/write rules for famous Pipelines

Read data from FIFO:

Convention: if a process blocks the access to the FIFO to read data from the FIFO, the read operation in the process is a read operation with a blocking flag.

• If a process writes to enable FIFO and there is no data in the current FIFO, the read operation with the blocking flag is always blocked. -1 is returned if no blocking mark is set for the read operation. The current errno value is eagain, prompting you to try again later.
• For read operations with the blocking flag configured, there are two reasons for blocking: data in the current FIFO, but other processes are reading the data. In addition, there is no data in the FIFO. The reason for blocking is that there are new data writes in the FIFO, regardless of the size of the data written to the message or the amount of data requested by the read operation.
• The read blocking mark only applies to the first read operation of the process. If there are multiple read operation sequences in the process, after the first read operation is awakened and the read operation is completed, other read operations to be executed will not be blocked, even if there is no data in the FIFO when the read operation is executed (at this time, the read operation returns 0 ).
• If no process write is enabled, the read operation with the blocking flag set will be blocked.

NOTE: If data exists in the FIFO, the read operation with the blocking flag is not blocked because the number of bytes in the FIFO is smaller than the number of bytes requested to read, the read operation returns the existing data volume in the FIFO.

Write Data to FIFO:

Convention: if a process blocks the opening of the FIFO to write data to the FIFO, the write operation in the process is a write operation with a blocking flag.

Write operations with blocking flag configured:

• When the data volume to be written is not greater than pipe_buf, Linux ensures the atomicity of writing. If the idle buffer of the pipeline is insufficient to accommodate the number of bytes to be written, the system goes to sleep until the buffer can accommodate the number of bytes to be written.
• When the data volume to be written is greater than pipe_buf, Linux will no longer guarantee the atomicity of writing. As soon as the FIFO buffer has an idle area, the write process will attempt to write data to the pipeline, and the write operation will return after writing all the data written by the request.

For write operations without blocking flag set:

• When the data volume to be written is greater than pipe_buf, Linux will no longer guarantee the atomicity of writing. After the buffer is fully written into all the FIFO idle buffers, the write operation returns.
• When the data volume to be written is not greater than pipe_buf, Linux ensures the atomicity of writing. If the current FIFO idle buffer can accommodate the number of bytes written by the request, the result is returned successfully. If the current FIFO idle buffer cannot accommodate the number of bytes written by the request, the eagain error is returned, remind me to write it later;

Verify the FIFO read/write rules:

The following two read/write programs are provided for FIFO. You can verify the read/write rules by adjusting a few places in the program or using the command line parameters of the program.

Procedure 1: Write a FIFO Program

# Include <sys/types. h> # Include <sys/STAT. h> # Include <errno. h> # Include <fcntl. h> # Define kerbero_server "/tmp/kerberoserver" Main (INT argc, char ** argv) // The parameter is the number of bytes to be written. { Int FD; Char w_buf [4096*2]; Int real_wnum; Memset (w_buf, * 2 ); If (mkfifo (primary o_server, o_creat | o_excl) <0) & (errno! = Eexist )) Printf ("cannot create external oserver/N "); If (FD =-1) If (errno = enxio) Printf ("Open error; no reading process/N "); FD = open (fifo_server, o_wronly | o_nonblock, 0 ); // Set non-blocking flag // FD = open (fifo_server, o_wronly, 0 ); // Set the blocking flag Real_wnum = write (FD, w_buf, 2048 ); If (real_wnum =-1) { If (errno = eagain) Printf ("write to FIFO error; try later/N "); } Else Printf ("Real write num is % d/N", real_wnum ); Real_wnum = write (FD, w_buf, 5000 ); // 5000 is used to test the non-atomicity when the number of written bytes exceeds 4096 // Real_wnum = write (FD, w_buf, 4096 ); // 4096 is used to test the atomicity of writing bytes not greater than 4096 If (real_wnum =-1) If (errno = eagain) Printf ("try later/N "); } Program 2: Test the write FIFO rule with program 1. The first command line parameter is the number of bytes read from the FIFO request. # Include <sys/types. h> # Include <sys/STAT. h> # Include <errno. h> # Include <fcntl. h> # Define kerbero_server "/tmp/kerberoserver" Main (INT argc, char ** argv) { Char r_buf [4096*2]; Int FD; Int r_size; Int ret_size; R_size = atoi (argv [1]); Printf ("requred real read bytes % d/N", r_size ); Memset (r_buf, 0, sizeof (r_buf )); FD = open (fifo_server, o_rdonly | o_nonblock, 0 ); // FD = open (fifo_server, o_rdonly, 0 ); // Compile the read program into two different versions: the blocking version and the non-blocking version. If (FD =-1) { Printf ("Open % s for read error/N "); Exit (); } While (1) { Memset (r_buf, 0, sizeof (r_buf )); Ret_size = read (FD, r_buf, r_size ); If (ret_size =-1) If (errno = eagain) Printf ("no data avlaible/N "); Printf ("Real read bytes % d/N", ret_size ); Sleep (1 ); } Pause (); Unlink (kerbero_server ); }

Application description:

Compile the read program into two different versions:

• Blocked read version: Br
• And non-blocking read version NBR

Compile the write program into two or four versions:

• Non-blocking and the number of bytes written by the request exceeds pipe_buf version: nbwg
• Non-blocking and the number of bytes written by the request is no greater than pipe_buf version: Version NBW
• Blocked and the number of bytes written by the request exceeds pipe_buf. Version: BWG
• The number of bytes that are blocked and requested to write is no greater than pipe_buf. Version: bw

The following describes how to use BR, NBR, and w to replace blocking and non-blocking reads in the corresponding program.

Verify the blocking write operation:

1. When the data volume written by the request is greater than pipe_buf, the non-Atomicity is as follows:
   • NBR 1000
   • BWG
2. When the data volume written by the request is not greater than pipe_buf, the atomicity is as follows:
   • NBR 1000
   • BW

Verify non-blocking write operations:

1. When the data volume written by the request is greater than pipe_buf, the non-Atomicity is as follows:
   • NBR 1000
   • Nbwg
2. The data size written by the request must not be greater than the atomicity of pipe_buf:
   • NBR 1000
   • NBW

When the number of bytes written to a request exceeds 4096, the write Atomicity is not guaranteed regardless of whether the blocking mark is set. But there are essential differences between the two:

For blocked write operations, the write operation will wait until all data is written in the idle area of the FIFO, and the data written by the request will eventually be written into the FIFO;

Non-blocking write is returned (the actual number of bytes written) after the Free Zone of the first-in-first-out is fully written. Therefore, some data cannot be written.

The verification of read operations is relatively simple and will not be discussed.

2.5 famous Pipeline application example

After the corresponding read/write rules are verified, the application instance does not seem necessary.

Summary:

Pipelines are often used in two ways: (1) pipelines are often used in Shell (as input redirection). In this way, the creation of pipelines is transparent to users; (2) It is used for Kinship-related inter-process communication. you can create your own pipelines and complete read/write operations.

FIFO can be said to be the promotion of pipelines, which overcomes the pipe's no-name restrictions, so that unrelated processes can also use the first-in-first-out communication mechanism for communication.

The data in the pipeline and FIFO is a byte stream, and the application must determine the specific transmission "protocol" in advance to transmit messages of specific significance.

To flexibly use pipelines and FIFO, understanding their read/write rules is critical.

Appendix 1: The running result of kill-L shows all signals supported by the current system:

1) SIGHUP    2) SIGINT    3) SIGQUIT    4) SIGILL 5) SIGTRAP    6) SIGABRT    7) SIGBUS    8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR 31) SIGSYS 32) SIGRTMIN 33) SIGRTMIN+1 34) SIGRTMIN+2 35) SIGRTMIN+3 36) SIGRTMIN+4 37) SIGRTMIN+5 38) SIGRTMIN+6 39) SIGRTMIN+7 40) SIGRTMIN+8 41) SIGRTMIN+9 42) SIGRTMIN+10 43) SIGRTMIN+11 44) SIGRTMIN+12 45) SIGRTMIN+13 46) SIGRTMIN+14 47) SIGRTMIN+15 48) SIGRTMAX-15 49) SIGRTMAX-14 50) SIGRTMAX-13 51) SIGRTMAX-12 52) SIGRTMAX-11 53) SIGRTMAX-10 54) SIGRTMAX-9 55) SIGRTMAX-8 56) SIGRTMAX-7 57) SIGRTMAX-6 58) SIGRTMAX-5 59) SIGRTMAX-4 60) SIGRTMAX-3 61) SIGRTMAX-2 62) SIGRTMAX-1 63) SIGRTMAX

In addition to describing the application of pipelines, we will discuss these signals in the next topic.

Appendix 2: Verify the FIFO open rules (mainly to verify the dependency of write open on read open)

#include <sys/types.h> #include <sys/stat.h> #include <errno.h> #include <fcntl.h> #define FIFO_SERVER "/tmp/fifoserver" int handle_client(char*); main(int argc,char** argv) { int r_rd; int w_fd; pid_t pid; if((mkfifo(FIFO_SERVER,O_CREAT|O_EXCL)<0)&&(errno!=EEXIST))     printf("cannot create fifoserver/n"); handle_client(FIFO_SERVER); } int handle_client(char* arg) { int ret; ret=w_open(arg); switch(ret) { case 0: { printf("open %s error/n",arg); printf("no process has the fifo open for reading/n"); return -1; } case -1: {     printf("something wrong with open the fifo except for ENXIO");     return -1; } case 1: {     printf("open server ok/n");     return 1; } default: {     printf("w_no_r return ????/n");     return 0; } }    unlink(FIFO_SERVER); } int w_open(char*arg) //0    open error for no reading //-1 open error for other reasons //1    open ok { if(open(arg,O_WRONLY|O_NONBLOCK,0)==-1) { if(errno==ENXIO)     {      return 0;     }     else     return -1; } return 1; }

References:

1. Second volume of UNIX Network Programming: inter-process communication, Author: W. Richard Steven S, Translator: Yang jizhang, Tsinghua University Press. The rich examples and Analysis of inter-process communication between Unix have great inspiration for program development in the Linux environment.
2. Linux kernel source code Scene Analysis (top and bottom), Mao decao, Hu Ximing, Zhejiang University Press, the best reference materials to verify a conclusion and idea;
3. Advanced Programming in UNIX environments; Author: W. Richard Steven s; Translator: Yu jinyuan; Mechanical Industry Press. It has rich programming instances and key functions along with the development of UNIX.
4. Http://www.linux.org.tw/CLDP/gb/Secure-Programs-HOWTO/x346.html point Linux sigaction implementation basis, Linux source code ../kernel/signal. C more explained the problem;
5. Pipe manual, the most direct and reliable reference
6. FIFO manual, the most direct and reliable reference

About the author

Zheng yanxing, male, is now pursuing a doctorate degree in network direction from the computer College of the National Defense University. You can contact him by email mlinux@163.com.