Linux Environment interprocess Communication (i): Pipelines and famous pipelines

in this series, the author outlines several main means of inter-process communication between Linux. Pipelines and well-known pipelines are among the earliest inter-process communication mechanisms, pipelines can be used for communication between affinity processes, and well-known pipelines overcome the limitations of pipe without name, so that, in addition to having the functionality of a pipeline, it allows for inter-process communication without affinity. Recognizing the rules of reading and writing of pipelines and famous pipelines is the key to apply them in the program, this paper, on the basis of discussing the communication mechanism of pipelines and famous pipelines in detail, validates the reading and writing rules by examples, which is helpful to enhance readers ' perceptual knowledge of reading and writing rules, and also provides application examples.

1, pipeline overview and related API application

1.1 Key concepts related to piping

The pipeline is one of the original UNIX IPC forms supported by Linux and has the following features:

• The pipe is half-duplex, the data can only flow in one direction, when two sides need to communicate, need to establish two pipelines;
• Can only be used between parent-child processes or sibling processes (affinity processes);
• Separate form a separate file system: A pipe is a file for the process at both ends of the pipe, but it is not an ordinary file, it does not belong to a file system, but is a separate file system, and only exists in memory.
• Read and write data: a process that writes to the pipeline is read out by the process at the other end of the pipeline. The content that is written is added to the end of the pipe buffer every time, and the data is read from the head of the buffer each time.

1.2 Creation of pipelines:

#include <unistd.h>int pipe (int fd[2])

The two ends of the pipeline created by the function are in the middle of a process, which is not significant in practice, so a process typically fork a sub-process after it is created by pipe () and then communicate between parent and child processes through a pipeline (so it is not difficult to roll out, as long as there are affinity in two processes, The kinship here refers to having a common ancestor, which can be communicated by means of pipelines.

1.3 Read and write rules for pipelines:

The two ends of the pipe can be described by the description character Fd[0] and fd[1], and it is important to note that the ends of the pipe are fixed on the task. That is, one end can only be used for reading, represented by the description word fd[0], which is called the pipe reading end, and the other end can only be used for writing, by the description word fd[1] to be said to be the pipe write end. If you attempt to read data from the pipe write end, or write data to the pipe read end, it will cause an error to occur. I/O functions for general files can be used for pipelines such as close, read, write, and so on.

To read data from the pipeline:

• If the write end of the pipeline does not exist, it is considered that the end of the data has been read, the Read function returns the number of read bytes is 0;
• When the write side of the pipeline exists, if the number of bytes requested is greater than pipe_buf, the number of existing data bytes in the pipeline is returned, and if the requested number of bytes is not greater than pipe_buf, the number of existing data bytes in the pipeline is returned (in this case, the amount of data in the pipeline is less than the requested amount of data) or return the requested number of bytes (in this case, the amount of data in the pipeline is not less than the requested amount of data). Note: (Pipe_buf is defined in Include/linux/limits.h, different kernel versions may vary.) POSIX.1 requires a minimum of 512 bytes for pipe_buf, and 4096 in Red Hat 7.2).

About read rule validation for pipelines:

/************** * readtest.c * **************/#include <unistd.h> #include <sys/types.h> #include < Errno.h>main () {int pipe_fd[2];p id_t pid;char r_buf[100];char w_buf[4];char* p_wbuf;int r_num;int cmd;memset (R_buf, 0,sizeof (R_BUF)) memset (w_buf,0,sizeof (r_buf));p _wbuf=w_buf;if (pipe (PIPE_FD) <0) {printf ("Pipe Create error\n") ; return-1;} if ((Pid=fork ()) ==0) {printf ("\ n"); Close (pipe_fd[1]); sleep (3);//Make sure the parent process closes the write-side r_num=read (pipe_fd[0],r_buf,100);p rintf ("Read Num is%d of the data read from the pipe is%d\n", R_num,atoi (R_BUF)); Close (pipe_fd[0]); exit ();} else if (pid>0) {close (pipe_fd[0]),//readstrcpy (W_buf, "111"), if (write (pipe_fd[1],w_buf,4)!=-1) printf ("Parent Write over\n "); Close (pipe_fd[1]);//writeprintf (" Parent close fd[1] over\n "); sleep (10);}} /************************************************** * Program Output Result: * Parent write over * parent close fd[1] over * Read NUM is 4 The data read from the pipe is 111 * Additional conclusion: * After the pipe write end is closed, the data written will persist until it is read out. ****************************************************/

To write data to the pipeline:

• When writing data to a pipeline, Linux does not guarantee the atomicity of writes, and the pipeline buffer has an idle area, and the write process attempts to write data to the pipeline. If the read process does not read the data in the pipeline buffer, the write operation will be blocked.
  Note: Writing data to a pipeline is meaningful only if the read side of the pipeline exists. Otherwise, the process that writes data to the pipeline receives the sifpipe signal from the kernel, which the application can process or ignore (the default action is the application termination).

Validation of write rules for pipelines 1: write-side dependence on read-side existence

#include <unistd.h> #include <sys/types.h>main () {int pipe_fd[2];p id_t pid;char r_buf[4];char* w_buf;int Writenum;int Cmd;memset (r_buf,0,sizeof (R_BUF)), if (pipe (PIPE_FD) <0) {printf ("Pipe Create error\n"); return-1;} if ((Pid=fork ()) ==0) {close (pipe_fd[0]); Close (pipe_fd[1]); sleep (); exit ();} else if (pid>0) {sleep (1); Wait for the child process to finish closing the read operation Close (Pipe_fd[0]),//writew_buf= "111", if ((Writenum=write (pipe_fd[1],w_buf,4)) ==-1) printf ("Write To pipe error\n "); elseprintf (" The bytes write to pipe is%d \ n ", writenum); Close (pipe_fd[1]);}}

The output is: broken pipe, the reason is that the pipe and all of its fork () product read end has been closed. If you keep the read end in the parent process, that is, after you finish writing the pipe, and then close the read end of the parent process, the pipe is also written, and the reader can verify the conclusion himself. Therefore, when writing data to the pipeline, at least one process should exist, where the pipe read end is not closed, otherwise the above error (pipeline fracture, process received a sigpipe signal, the default action is process termination)

Validation of write rules for pipelines 2:linux does not guarantee atomic verification of write pipelines

#include <unistd.h> #include <sys/types.h> #include <errno.h>main (int argc,char**argv) {int pipe_fd[ 2];p id_t pid;char r_buf[4096];char w_buf[4096*2];int writenum;int rnum;memset (r_buf,0,sizeof (R_BUF)); if (Pipe (pipe_ FD) {<0) {printf ("Pipe Create error\n"); return-1;} if ((Pid=fork ()) ==0) {close (pipe_fd[1]), while (1) {sleep (1), Rnum=read (pipe_fd[0],r_buf,1000);p rintf ("Child:readnum Is%d\n ", rnum);} Close (pipe_fd[0]); exit ();} else if (pid>0) {close (pipe_fd[0]),//writememset (r_buf,0,sizeof (R_BUF)), if (Writenum=write (Pipe_fd[1],w_buf, ) ==-1) printf ("Write to Pipe error\n"), elseprintf ("The bytes write to pipe is%d \ n", writenum); Writenum=write (Pipe_f d[1],w_buf,4096); Close (pipe_fd[1]);}} Output: The bytes write to pipe 1000the bytes write to pipe 1000//Note that this line of output illustrates the non-atomicity of the Write, the bytes write to pipe 1000the bytes WR Ite to pipe 1000the bytes write to pipe 1000the bytes write to pipe 120//Note that this line of output illustrates the non-atomicity of writes the bytes write to pipe 0the b Ytes write to Pipe 0 ...

Conclusion:

Write non-atomic when the number of writes is less than 4096!
If you change the number of two write bytes in the parent process to 5000, it is easy to draw the following conclusions:
When the amount of data written to the pipeline is greater than 4096 bytes, the free space of the buffer is written to the data (completion) until all data is written, and if no process reads the data, it is blocked.

1.4 Examples of piping applications:

Example one: for Shell

The pipeline can be used for input-output redirection, which directs the output of one command directly to the input of another command. For example, when you type who│wc-l in a shell program (Bourne shell or C shell, and so on), the shell program creates the WHO and the WC two processes and the pipelines between the two processes. Consider the following command line:

The results of $kill-l operation are shown in attached one.

$kill-L | The results of grep sigrtmin run as follows:

SIGPWR31) SIGSYS32) SIGRTMIN33) sigrtmin+134) sigrtmin+235) sigrtmin+336) sigrtmin+437) sigrtmin+538) sigrtmin+639) sigrtmin+740) sigrtmin+841) sigrtmin+942) sigrtmin+1043) sigrtmin+1144) sigrtmin+1245) SIGRTMIN+1346) sigrtmin+1548) SIGRTMAX-1549) SIGRTMAX-14

example Two: For inter-process communication with affinity

The following example gives the concrete application of the pipeline, the parent process sends some commands to the child process through the pipeline, the child process parses the command, and the corresponding processing according to the command.

#include <unistd.h> #include <sys/types.h>main () {int pipe_fd[2];p id_t pid;char r_buf[4];char** w_buf[256] int Childexit=0;int i;int Cmd;memset (r_buf,0,sizeof (R_BUF)), if (pipe (PIPE_FD) <0) {printf ("Pipe Create error\n"); return-1;} if ((Pid=fork ()) ==0)//Sub-process: Parses the command obtained from the pipeline and handles the corresponding {printf ("\ n"), close (pipe_fd[1]), Sleep (2), while (!childexit) {Read ( pipe_fd[0],r_buf,4); Cmd=atoi (R_BUF), if (cmd==0) {printf ("child:receive command from parent over\n now child process exit \ n "); childexit=1;} else if (handle_cmd (cmd)!=0) return;sleep (1);} Close (pipe_fd[0]); exit ();} else if (pid>0)//parent:send commands to Child{close (pipe_fd[0]); w_buf[0]= "003"; w_buf[1]= "005"; w_buf[2]= "777"; w_ Buf[3]= "for" (i=0;i<4;i++) write (pipe_fd[1],w_buf[i],4); close (pipe_fd[1]);}} The following is the command handler for the child process (application specific): int handle_cmd (int cmd) {if (cmd<0) | | (cmd>256)) Suppose child commands{printf ("Child:invalid command \ n"); return-1;} printf ("Child:the cmd from the parent is%d\n", cmd); return 0;}

1.5 Limitations of pipelines

The main limitations of the pipeline are reflected in its characteristics:

• Only one-way data streams are supported;
• Can only be used between processes that have affinity;
• No Name;
• The buffer of the pipeline is finite (piping is present in memory and is allocated a page size for the buffer when the pipeline is created);
• The pipeline transmits the unformatted byte stream, which requires that the reader and writer of the pipeline must agree the format of the data beforehand, such as how many bytes count as a message (or command, or record), etc.

Back to top of page

2. Well-known pipeline overview and related API applications

2.1 Key concepts related to famous pipelines

A significant limitation of pipeline applications is that it has no name, so it can only be used for inter-process communication with affinity, which is overcome when a named pipe (named pipe or FIFO) is presented. A FIFO differs from a pipe in that it provides a path name associated with it, which exists in the file system as a FIFO file. Thus, even processes that do not have affinity to the FIFO creation process, as long as they can access the path, can communicate with each other through the FIFO (the process that accesses the path and the creation process of the FIFO), so that processes that are not related to FIFO can also exchange data. It is important to note that FIFO strictly adheres to first-in, FIFO, which reads from the beginning of the pipeline, and is always returning data from the start, and writes the data to the end. They do not support file location operations such as Lseek ().

2.2 Creation of famous pipelines

#include <sys/types.h> #include <sys/stat.h>int mkfifo (const char * pathname, mode_t mode)

The first parameter of the function is a normal pathname, which is the name of the FIFO after creation. The second parameter is the same as the mode parameter in the open () function that opens the normal file. If the first parameter of Mkfifo is an already existing pathname, the eexist error is returned, so the typical calling code first checks to see if the error is returned, and if it does return the error, simply call the function that opened the FIFO. I/O functions for general files can be used in FIFO, such as close, read, write, and so on.

2.3 Open rules for famous pipelines

A famous pipe is more than a pipe. Open operation: Opening.

Open rules for FIFO:

If the current open operation is open FIFO for read, the current open operation will return successfully if the process has already opened the FIFO for write, otherwise, it may block until a corresponding process is written to open the FIFO (the current open operation has a blocking flag set); Successful return (the current open operation does not have a blocking flag set).

If the current open operation is to open a FIFO for write, the current open operation will return successfully if the corresponding process has opened the FIFO for reading, otherwise, it may block until the corresponding process has opened the FIFO for reading (the blocking flag is set for the current open operation); Returns a Enxio error (the current open operation does not have a blocking flag set).

For validation of open rules see attached 2.

2.4 Read and write rules for famous pipelines

To read data from the FIFO:

Convention: If a process blocks the open FIFO in order to read data from the FIFO, the read operation in the process is called a read operation with the blocking flag set.

• If there is a process write open FIFO and there is no data in the current FIFO, it will be blocked for read operations that have a blocking flag set. For a read operation that does not have a blocking flag set, 1 is returned, and the current errno value is Eagain, reminding you to try again later.
• For read operations that have a blocking flag set, there are two reasons for blocking: There is data in the current FIFO, but other processes are reading the data, and there is no data in the FIFO. The reason for blocking is that there are new data writes in the FIFO, regardless of the size of the volume of data written, and regardless of the amount of data requested by the read operation.
• Read Open blocking flag only applies to the first read operation of this process, if there is more than one read sequence in the process, after the first read operation is awakened and the read operation is completed, the other read operation will no longer block, even when performing a read operation, no data in the FIFO is the same (at this point, the read operation returns 0).
• If there is no process write open FIFO, read operations that have the blocking flag set are blocked.

Note: If there is data in the FIFO, the read operation that sets the blocking flag does not block because the number of bytes in the FIFO is less than the number of bytes requested, and the read operation returns the amount of data that is already in the FIFO.

To write data to the FIFO:

Convention: If a process blocks the open FIFO in order to write data to the FIFO, then the write operation in the process is called a write operation that sets the blocking flag.

For write operations that have a blocking flag set:

• When the amount of data to be written is not greater than PIPE_BUF, Linux guarantees the atomicity of the write. If the pipe free buffer is not sufficient to hold the number of bytes to write at this time, it goes to sleep until a one-time write is started when the number of bytes to be written is accommodated in the buffer.
• When the amount of data to be written is greater than Pipe_buf, Linux will no longer guarantee the atomicity of writes. Once the FIFO buffer has an idle area, the write process attempts to write data to the pipeline, and the write is returned after all the requested data has been written.

For write operations that do not have a blocking flag set:

• When the amount of data to be written is greater than Pipe_buf, Linux will no longer guarantee the atomicity of writes. After all FIFO idle buffers have been written, the write operation returns.
• When the amount of data to be written is not greater than PIPE_BUF, Linux guarantees the atomicity of the write. If the current FIFO free buffer can hold the number of bytes requested to be written, it returns successfully after completion, and if the current FIFO free buffer does not hold the number of bytes requested to write, the Eagain error is returned and the reminder is later written;

Validation of FIFO read-write rules:

The following provides two read-write programs to the FIFO, a few places in the appropriate tuning program or the program's command line parameters can be used to verify the various FIFO read-write rules.

Program 1: Write the FIFO program

#include <sys/types.h> #include <sys/stat.h> #include <errno.h> #include <fcntl.h> #define FIFO _server "/tmp/fifoserver" main (int argc,char** argv)//parameter is the number of bytes that will be written {int Fd;char w_buf[4096*2];int real_wnum;memset (w_ BUF,0,4096*2); if (Mkfifo (fifo_server,o_creat| O_EXCL) <0) && (errno!=eexist)) printf ("Cannot create fifoserver\n"), if (fd==-1) if (Errno==enxio) printf (" Open error; No reading process\n "); Fd=open (fifo_server,o_wronly| o_nonblock,0);//Set non-blocking flag//fd=open (fifo_server,o_wronly,0);//Set Blocking flag real_wnum=write (fd,w_buf,2048); if (real_wnum== -1) {if (Errno==eagain) printf ("Write to FIFO error; Try later\n ");} else printf ("Real write num is%d\n", real_wnum); Real_wnum=write (fd,w_buf,5000);//5000 used to test non-atomicity of write bytes greater than 4096//real_ Wnum=write (fd,w_buf,4096);//4096 is used to test for atomic if (Real_wnum==-1) if (errno==eagain) printf ("Try later\n") when the write byte is not greater than 4096;}

Program 2: Test write FIFO rules with program 11, the first command line parameter is the number of bytes that are requested to be read from the FIFO

#include <sys/types.h> #include <sys/stat.h> #include <errno.h> #include <fcntl.h> #define FIFO _server "/tmp/fifoserver" main (int argc,char** argv) {char r_buf[4096*2];int fd;int r_size;int ret_size ; R_size=atoi (argv[1]);p rintf ("requred Real Read bytes%d\n", r_size); memset (R_buf,0,sizeof (R_BUF)); Fd=open (fifo_ server,o_rdonly| o_nonblock,0);//fd=open (fifo_server,o_rdonly,0);//Here you can compile the read program into two different versions: blocked version and non-blocking version if (fd==-1) {printf ("Open%s for Read error\n "); exit ();} while (1) {memset (r_buf,0,sizeof (R_BUF)); Ret_size=read (fd,r_buf,r_size); if (ret_size==-1) if (errno==eagain) printf ( "No Data avlaible\n");p rintf ("real Read bytes%d\n", ret_size); sleep (1);} Pause (); unlink (fifo_server);}

Program Application Notes:

Compile the read program into two different versions:

• Blocked Read version: BR
• and non-blocking read version NBR

Compile the write program into two four versions:

• Non-blocking and request write bytes greater than PIPE_BUF version: NBWG
• Non-blocking and the number of bytes requested is not greater than Pipe_buf version: Version NbW
• The number of bytes blocked and the requested write is greater than PIPE_BUF version: BWG
• The number of bytes blocked and requested to write is not greater than PIPE_BUF version: Version bw

The following will use BR, NBR, W instead of blocking read, non-blocking read in the corresponding program

To verify the blocking write operation:

1. Non-atomicity when the amount of data requested to be written is greater than Pipe_buf:
   • NBR 1000
   • Bwg
2. When the amount of data written by the request is not greater than the atomicity of Pipe_buf:
   • NBR 1000
   • Bw

To verify a non-blocking write operation:

1. Non-atomicity when the amount of data requested to be written is greater than Pipe_buf:
   • NBR 1000
   • Nbwg
2. The amount of data written by the request is not greater than the atomicity of Pipe_buf:
   • NBR 1000
   • NbW

Regardless of whether the write-open blocking flag is set, the write atomicity is not guaranteed when the number of bytes requested to write is greater than 4096. But the two are essentially different:

In the case of blocking writes, the write operation waits until all data is written, and the requested data is eventually written to the FIFO after the idle area of the FIFO is written.

Instead of blocking writes, it returns (the number of bytes actually written) after the idle area of the FIFO is written, so some of the data is ultimately not able to be written.

Validation of read operations is relatively straightforward and is no longer discussed.

2.5 Examples of well-known piping applications

Once the corresponding read-write rules have been verified, the application instance seems to be unnecessary.

Back to top of page

Summary:

Pipelines are commonly used in two areas: (1) Pipelines are often used in the shell (as input input redirects), in which case the pipeline is created transparently to the user, (2) is used for inter-process communication with affinity, the user creates the pipeline himself, and reads and writes.

FIFO can be said to be the promotion of pipelines, overcome the pipe No name restrictions, so that the non-affinity process can also be used in first-out communication mechanism for communication.

Pipelines and FIFO data are byte streams, and applications must identify specific transport "protocols" in advance, using messages that propagate a particular meaning.

To flexibly apply pipelines and FIFO, it is critical to understand their read and write rules.

The operating results of the attached 1:KILL-L show all signals supported by the current system:

1) SIGHUP 2) SIGINT 3) Sigquit 4) SIGILL5) SIGTRAP 6) SIGABRT 7) Sigbus 8) SIGFPE9) SIGKILL10) SIGUSR111) SIGSEGV12 R213) SIGPIPE14) SIGALRM15) SIGTERM17) SIGCHLD18) SIGCONT19) SIGSTOP20) SIGTSTP21) SIGTTIN22) SIGTTOU23) SIGURG24) SIGXCP U25) SIGXFSZ26) SIGVTALRM27) SIGPROF28) SIGWINCH29) SIGIO30) SIGPWR31) SIGSYS32) SIGRTMIN33) sigrtmin+134) sigrtmin+235) sigrtmin+336) sigrtmin+437) sigrtmin+538) sigrtmin+639) sigrtmin+740) sigrtmin+841) sigrtmin+942) sigrtmin+1144) sigrtmin+1245) sigrtmin+1346) sigrtmin+1447) sigrtmin+1548) SIGRTMAX-1549) SIGRTMAX-1351) SIGRTMAX-1252) SIGRTMAX-1153) SIGRTMAX-1054) SIGRTMAX-955) SIGRTMAX-856) SIGRTMAX-757) SIGRTMAX-559) SIGRTMAX-460) SIGRTMAX-361) SIGRTMAX-262) SIGRTMAX-163

In addition to being used here to illustrate piping applications, the following topics will be discussed in the classification of these signals.

Attachment 2: Validation of FIFO open rules (Primary verify write open dependency on read open)

#include <sys/types.h> #include <sys/stat.h> #include <errno.h> #include <fcntl.h> #define FIFO _server "/tmp/fifoserver" int handle_client (char*), main (int argc,char** argv) {int r_rd;int w_fd;pid_t pid;if ((Mkfifo ( Fifo_server,o_creat| O_EXCL) <0) && (errno!=eexist)) printf ("Cannot create fifoserver\n"); Handle_client (fifo_server);} int handle_client (char* arg) {int ret;ret=w_open (ARG); switch (ret) {case 0:{printf ("Open%s error\n", Arg);p rintf ("No Process have the FIFO open for reading\n "); return-1;} case-1:{printf ("Something Wrong with Open, FIFO except for Enxio"); return-1;} Case 1:{printf ("Open Server ok\n"); return 1;} default:{printf ("W_no_r return----\ n"); return 0;}} Unlink (fifo_server);} int W_open (CHAR*ARG)//0 Open error for no reading//-1 open error for other REASONS//1 open ok{if (open (arg,o_wronl y| o_nonblock,0) ==-1) {if (Errno==enxio) {return 0;} elsereturn-1;} return 1;}

References

• UNIX Network Programming Volume II: Interprocess communication, W.richard Stevens, translator: Yang Jizhang, Tsinghua University Press. The rich examples and analysis of UNIX inter-process communication are very instructive to the development of program in Linux environment.
  
  
• Linux Kernel source code scenario analysis (top, bottom), Maudeca, Hu Himing, Zhejiang University Press, when it is necessary to verify a conclusion, the best reference material;
  
  
• UNIX environment Advanced Programming, W.richard Stevens, translator: Yu Jinyuan, Machinery Industry press. Has rich programming examples, and the key function accompanies the development process of UNIX.
  
  
• Http://www.linux.org.tw/CLDP/gb/Secure-Programs-HOWTO/x346.html the realization base of Linux under Sigaction, Linux source code. /KERNEL/SIGNAL.C a more descriptive question;
  
  
• Pipe Handbook, the most direct and reliable reference
  
  
• FIFO Handbook, the most direct and reliable reference material
  

Read (31) | Comments (0) | Forwards (0) |0

Previous: Linux kernel source Analysis--Kernel boot (5) Image kernel boot (rest_init function) (Linux-3.0 ARMv7)

Next post: Multithreaded programming

Related Popular articles

• Common Linux Service ports
• Xmanager 2.0 for Linux configuration
• "Rootfs build" BusyBox httpd ...
• OpenWrt in Luci study notes
• What is a shell?

• Linux DHCP Peizhi ROC
• Soft links to Unix files
• What does this command mean, I'm new ...
• What does sed-e "/grep/d" mean ...
• Who can help me solve Linux 2.6 10 ...

Leave something to the owner! ~~ Comment on the hot topic

Linux Environment interprocess Communication (i): Pipelines and famous pipelines