Inter-process communication for Linux-pipelines

Inter-process communication for Linux-pipelines

版权声明：本文章内容在非商业使用前提下可无需授权任意转载、发布。转载、发布请务必注明作者和其微博、公众号地址，以便读者询问问题和甄误反馈，共同进步。微博ID：orroz公众号：Linux系统技术

Objective

Pipelines are the oldest inter-process communication mode in a UNIX environment. This article mainly explains how to use pipelines on Linux environments. Reading this article can help you solve the following problems:

1. What are pipelines and why should there be pipes?
2. How to classify pipelines?
3. What is the implementation of the pipeline?
4. How big is the pipe?
5. Can the size of the pipe be adjusted? How to adjust?

What is a pipe?

Pipe, English for pipe. This is a very important concept that we will introduce when we are learning the Linux command line. Its inventor was Douglas Macelroy, who was also the inventor of the early Shell on Unix. After he invented the shell, found that the system operation to execute the command, there is often a need to the output of one program to another program to process, this operation can use the input and output redirection plus files, such as:

[[email protected] pipe]$ ls  -l /etc/ > etc.txt[[email protected] pipe]$ wc -l etc.txt 183 etc.txt

But it seems too troublesome. Therefore, the concept of piping came into being. Currently in any shell, you can use the ' | ' Connecting two commands, the shell connects the input and output of the front and back two processes to a single pipe to achieve the purpose of interprocess communication:

[[email protected] pipe]$ ls -l /etc/ | wc -l183

Comparing the two methods, we can also understand that the pipeline is essentially a file, the previous process opens the file in writing, and the subsequent process opens in read mode. So the front is written and read, so the communication is realized. In fact, piping is designed to follow Unix's "Everything is file" design principle, which is essentially a file. The Linux system directly implements the pipeline as a file system, using VFS to provide an operational interface to the application.

Although the implementation is in the form of a file, the pipeline itself does not occupy disk or other external storage space. In the implementation of Linux, it occupies a memory space. So, the pipeline on Linux is a memory buffer that operates as a file.

Classification and use of pipelines

There are two types of pipelines on Linux:

1. Anonymous pipeline
2. Named pipes

These two pipelines are also known as famous or nameless pipes. The most common form of anonymous pipelines is the "|" that we use most often in shell operations. It is characterized only by the use of a parent-child process, which must open a pipeline file before the child process is generated, and then fork to produce the child process, which is the process by copying the process address space of the parent process to obtain the same pipe file descriptor, to achieve the purpose of using the same pipeline communication. In addition to the parent-child process, no one knows the descriptor of the pipe file, so the information in this pipeline cannot be passed to other processes. This guarantees the security of the transmitted data and, of course, reduces the commonality of the pipeline, and the system also provides a named pipe.

We can use the Mkfifo or Mknod command to create a named pipe, which is no different from creating a file:

[[email protected] pipe]$ mkfifo pipe[[email protected] pipe]$ ls -l pipe prw-r--r-- 1 zorro zorro 0 Jul 14 10:44 pipe

You can see that the file types created are special and P types. Indicates that this is a pipeline file. With this pipeline file, the system has a global name for a pipeline, so any two unrelated processes can communicate through this pipeline file. For example we now have a process to write this pipeline file:

At this point the write operation is blocked because no one at the other end of the pipeline reads. This is the default behavior of the kernel for pipe file definitions. At this point, if a process reads this pipeline, then the blocking of this write operation will be lifted:

[[email protected] pipe]$ cat pipe xxxxxxxxxxxxxx

As you can see, when we finish this file, the echo command on the other end returns. This is the named pipe.

Linux systems, both for named Pipes and anonymous pipelines, use the same file system as the operating behavior of the underlying file system called Pipefs. You can find in the/etc/proc/filesystems file whether your system supports this file system:

[[email protected] pipe]$ cat /proc/filesystems |grep pipefsnodev   pipefs

After observing how to use the pipeline in the command line, let's look at how to use the pipeline in system programming.

PIPE

We can call anonymous pipes and Named pipes, respectively, pipe and FIFO. This is mainly because in system programming, the system call that creates the anonymous pipe is pipe (), and the function that creates the named pipe is Mkfifo (). You can also create a FIFO by using the Mknod () system call and specifying a file type of S_ififo.

Using the pipe () system call, you can create an anonymous pipe, which is a prototype of the system call:

#include <unistd.h>int pipe(int pipefd[2]);

This method will create two file descriptors, which can be referenced using the PIPEFD array to refer to these two descriptors for file manipulation. PIPEFD[0] is read-open as a read descriptor of the pipeline. PIPEFD[1] is write-open as a write descriptor for the pipeline. Data written from the pipe write end is cached by the kernel until someone reads it from the other end. Let's look at how to use the pipeline in a process, although this example doesn't make sense:

[[email protected] pipe]$ cat pipe.c#include <stdlib.h>#include <stdio.h>#include <unistd.h>#include <string.h>#define STRING "hello world!"int main(){    int pipefd[2];    char buf[BUFSIZ];    if (pipe(pipefd) == -1) {        perror("pipe()");        exit(1);    }    if (write(pipefd[1], STRING, strlen(STRING)) < 0) {        perror("write()");        exit(1);    }    if (read(pipefd[0], buf, BUFSIZ) < 0) {        perror("write()");        exit(1);    }    printf("%s\n", buf);    exit(0);}

The program creates a pipeline, writes a string to the pipeline, reads it from the pipe, and prints it on the standard output. Use a diagram to illustrate the status of this program:

A process sends itself messages to itself this is certainly not called interprocess communication, so in practice we do not use pipelines in a single process. After the pipe is created, the process often has to fork to produce the child process, as shown:

[[email protected] pipe]$ cat pipe_parent_child.c#include <stdlib.h> #include <stdio.h> #include < unistd.h> #include <string.h> #include <sys/types.h> #include <sys/wait.h> #define string "Hello world! "    int main () {int pipefd[2];    pid_t pid;    Char Buf[bufsiz];        if (pipe (PIPEFD) = =-1) {perror ("pipe ()");    Exit (1);    } PID = fork ();        if (PID = =-1) {perror ("fork ()");    Exit (1);        } if (pid = = 0) {/* This is the child. */printf ("Child pid is:%d\n", Getpid ());            if (read (pipefd[0], buf, Bufsiz) < 0) {perror ("write ()");        Exit (1);        } printf ("%s\n", buf);        Bzero (buf, Bufsiz);        snprintf (buf, Bufsiz, "Message from Child:my pid is:%d", getpid ());            if (write (pipefd[1], buf, strlen (buf)) < 0) {perror ("write ()");        Exit (1);   }} else {/* This is parent */printf ("Parent pid is:%d\n", Getpid ());     snprintf (buf, Bufsiz, "Message from Parent:my pid is:%d", getpid ());            if (write (pipefd[1], buf, strlen (buf)) < 0) {perror ("write ()");        Exit (1);        } sleep (1);        Bzero (buf, Bufsiz);            if (read (pipefd[0], buf, Bufsiz) < 0) {perror ("write ()");        Exit (1);        } printf ("%s\n", buf);    Wait (NULL); } exit (0);}

The parent process sends a message to the child process, the child process receives the message after it is received, and then sends a message to the parent process, and the parent process then prints the message received from the child process. Program execution Effect:

[[email protected] pipe]$ ./pipe_parent_child Parent pid is: 8309Child pid is: 8310Message from parent: My pid is: 8309Message from child: My pid is: 8310

From this program we can see that the pipeline can actually implement a half-duplex communication mechanism. A parent-child process that uses the same pipeline can send messages to each other. We can also see some of the features of pipeline reading and writing, namely:

In the case where there is no data in the pipeline, the read operation on the pipe is blocked until there is data in the pipe. When the amount of data written at a time does not exceed the capacity of the pipeline, the write operation of the pipe is generally not blocked, the data will be written directly to the pipeline buffer.

If you think this article is good, please sweep the code to donate to the poor Zorro.

Of course, the write operation will not be blocked in all cases. Here we have to first understand the core implementation of the pipeline. As mentioned above, the pipeline is actually a kernel-controlled memory buffer, since it is a buffer, there is a capacity limit. We put the pipeline up to a maximum amount of data that can be cached is called pipesize. When the kernel is processing the pipeline data, the underlying also calls a method like read and write to copy the data, the kernel operation can be manipulated each time the amount of data is limited, the general operation length is a page, that is, the default is 4k bytes. We call Pipebuf the amount of data that can be manipulated each time. In the POSIX standard, there is a length limit for pipebuf, which requires a minimum length of not less than 512 bytes. The PIPEBUF function is that when the kernel is processing a pipeline, the operation is atomic if the data length of each read-write operation is less than PIPEBUF. The effect of pipesize is that a write operation greater than its length is blocked until the data in the current pipeline is read.

Before the Linux 2.6.11, Pipesize and Pipebuf were actually the same. After that, Linux re-implemented a pipeline cache and implemented a different concept with the PIPEBUF implementation of the write operation, forming a default length of 65536 bytes of pipesize, and pipebuf only affects the atomicity of the associated read and write operations. After the Linux 2.6.35, F_GETPIPE_SZ and F_SETPIPE_SZ operations were implemented in the FCNTL system invocation method to view the current pipeline capacity and set the pipeline capacity. The maximum capacity capacity of a pipe can be set at/proc/sys/fs/pipe-max-size.

#define BUFSIZE 65536......ret = fcntl(pipefd[1], F_GETPIPE_SZ);if (ret < 0) {    perror("fcntl()");    exit(1);}printf("PIPESIZE: %d\n", ret);ret = fcntl(pipefd[1], F_SETPIPE_SZ, BUFSIZE);if (ret < 0) {    perror("fcntl()");    exit(1);}......

The effect of Pipebuf and pipesize on pipeline operations is due to behavior changes because the pipe descriptor is set to non-blocking mode, where n is the amount of data to write:

O_nonblock off, n <= pipe_buf:

n bytes of write operations are atomic, and the write system call may be blocked because the pipe capacity (pipesize) does not have enough space to hold the N-byte length.

O_nonblock Open, n <= pipe_buf:

If there is enough space to hold the N-byte length, the write call returns to success immediately and writes the data. There is not enough space to return an error immediately, and errno is set to Eagain.

O_nonblock off, n > Pipe_buf:

Write operations on n bytes are not guaranteed to be atomic, meaning that the data for this write operation may intersect with data from other processes writing the pipeline. Write operations are blocked when the pipe capacity is less than the length of the data to be written.

O_nonblock open, n > Pipe_buf:

If the pipe space is full. The write call returns an error and errno is set to Eagain. If it is not full, it may write from 1 to n bytes in length, depending on the remaining space length of the current pipeline, and the data may intersect with data from other processes.

The above is something to be aware of when using a half-duplex pipe, because in this case, there may be multiple processes at both ends of the pipeline for read-write processing. If you add threads, things can get more complicated. In fact, we do not recommend this for use when using pipelines. Pipeline recommended use is its single-mode: that is, only two process communication, one process only write the pipeline, the other process read-only pipeline. Implemented as:

[[email protected] pipe]$ cat pipe_parent_child2.c#include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <string.h> #include <sys/types.h> #include <sys/wait.h> #define STRING " Hello world! "    int main () {int pipefd[2];    pid_t pid;    Char Buf[bufsiz];        if (pipe (PIPEFD) = =-1) {perror ("pipe ()");    Exit (1);    } PID = fork ();        if (PID = =-1) {perror ("fork ()");    Exit (1);        } if (pid = = 0) {/* This is the child. */Close (pipefd[1]);        printf ("Child pid is:%d\n", Getpid ());            if (read (pipefd[0], buf, Bufsiz) < 0) {perror ("write ()");        Exit (1);    } printf ("%s\n", buf);        } else {/* This is parent */close (pipefd[0]);        printf ("Parent pid is:%d\n", Getpid ());        snprintf (buf, Bufsiz, "Message from Parent:my pid is:%d", getpid ());       if (write (pipefd[1], buf, strlen (buf)) < 0) {perror ("write ()");     Exit (1);    } wait (NULL); } exit (0);}

This program is actually simpler than the previous one, and the parent process closes the read end of the pipeline and writes only the pipeline. The child process closes the write end of the pipeline, and the read-only pipeline. The opening effect of the entire pipeline is finally shown as follows:

At this time two processes only with the pipeline to achieve a single-work communication, and this state does not have to consider multiple processes at the same time the pipeline write data intersection problem, this is the most classic way to open the pipeline, but also the recommended way to use the pipeline. In addition, as a programmer, even if we understand the implementation of the Linux pipeline, our code can not rely on its characteristics, so when processing the pipeline when the cross-judgment or to determine whether the error check or check, so that the code can be more robust.

Fifo

The implementation of a named pipe in the bottom line is exactly the same as the anonymous pipe, except that the named pipe has a globally visible file name for someone else open to use. There are two ways to create a named pipe file in a program, one is to use the MKFIFO function. The other is to use the Mknod system call, the example is as follows:

[[email protected] pipe]$ cat mymkfifo.c#include <stdio.h>#include <sys/types.h>#include <sys/stat.h>#include <stdlib.h>int main(int argc, char *argv[]){    if (argc != 2) {        fprintf(stderr, "Argument error!\n");        exit(1);    }/*    if (mkfifo(argv[1], 0600) < 0) {        perror("mkfifo()");        exit(1);    }*/    if (mknod(argv[1], 0600|S_IFIFO, 0) < 0) {        perror("mknod()");        exit(1);    }    exit(0);}

We use the first parameter as the path to the file that was created. Once created, other processes can be used using the open (), read (), write () standard file operations, and so on. All the rest of the operation is similar to anonymous pipe usage. It is important to note that, whether named or anonymous, its file description does not have the concept of an offset, so it cannot be adjusted with Lseek.

Nameless Pipe Demo (communication in a process):

 #include <unistd.h> #include <sys/types.h> #include <errno.h> #include <stdio.h> #include <string.h > #include <stdlib.h>int main () {int pipe_fd[2];p id_t pid;char buf_r[100];char* p_wbuf;int r_num;memset (Buf_r, 0,sizeof (Buf_r));/* Create Pipe */if (pipe (PIPE_FD) <0) {printf ("Pipe Create error\n"); return-1;} /* Create Child process */if ((Pid=fork ()) ==0)//Sub-process execution sequence {printf ("\ n"), close (pipe_fd[1]),//Sub-process first closes the write-end of the pipeline pipe_fd[0] is the read end of the pipe sleep (2);/* Let the parent process run first so that the parent process writes the child process before the content reads if you do not sleep, you will not be able to read the data */if ((R_num=read (pipe_fd[0],buf_r,100)) >0) {printf ("%d numbers read from The pipe is%s\n ", r_num,buf_r);}  Close (pipe_fd[0]); exit (0); }else if (pid>0)//child process after sleep, the parent process executes the sequence {close (pipe_fd[0]);//The parent process first closes the read-end of the pipe if (write (pipe_fd[1], "Hello", 5)!=-1) printf (" Parent write1 hello!\n "), if (write (pipe_fd[1]," pipe ", 5)!=-1) printf (" Parent write2 pipe!\n "); Close (pipe_fd[1]); Waitpid (pid,null,0); /* Wait for the child process to end */exit (0);} return 0;} 

Well-known pipeline demo (can be used for any two process communication):

Write

#include <sys/types.h> #include <sys/stat.h> #include <errno.h> #include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #define FIFO_SERVER "/tmp/myfifo" int main (int argc, char** argv) {int Fd;char w_buf[100];int nwrite;/*o_rdonly read-only open. O_wronly only write Open. O_RDWR read, write open. O_append is added to the end of the file each time it is written. O_creat create this file if it does not exist. When using this selection, it is necessary to describe the third parameter mode, which describes the access permission of the new file to the throne. O_EXCL if O_creat is specified and the file already exists, an error occurs. This tests whether a file exists and if it does not, it becomes an atomic operation to create the file. O_trunc If this file exists and is open for read-only or write-only success, the length is truncated to 0. O_noctty if p a t h n a m e refers to an end device, then this equipment is not assigned as the control terminal of this process. O_nonblock if p a t h n a m e refers to an F I F O, a block special file, or a character special file, this option sets the non-blocking mode for this open operation and subsequent I/O operations for this file.  The O_sync makes every w r i T e wait until the physical I/O operation is complete. *//* Create a well-known pipe */    if (Mkfifo (fifo_server,o_creat| o_excl| O_RDWR) <0) && (errno!=eexist)         printf ("Cannot create fifoserver\n");     /* Open pipe */    fd=open (fifo_server,o_rdwr| o_nonblock,0);    if (fd==-1)    &nbsp {        perror ("open");        exit (1);    }        /* Entry detection */    if (argc==1)     {         printf ("Please send something\n");        exit (-1);     }    strcpy (w_buf,argv[1]);        /* writing data to pipe */     if ((Nwrite=write (fd,w_buf,100)) ==-1)     {         if (errno==eagain)             printf ("The FIFO has not been read yet. Please try later\n ");    }    else         printf (" Write% S to the fifo\n ", W_buf);    close (FD); Close pipe     return 0;}  

/;}

Read

#include <sys/types.h> #include <sys/stat.h> #include <errno.h> #include <fcntl.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #define FIFO "/tmp/myfifo" int main (int argc,char** argv) {char buf_r[100];int  fd;int  nread;printf ("Preparing for Reading bytes...\n"), Memset (Buf_r,0,sizeof ( Buf_r));/* Open pipe o_rdonly| O_nonblock:*/fd=open (fifo,o_rdonly| o_nonblock,0), if (fd==-1) {perror ("open"); exit (1);} while (1) {memset (buf_r,0,sizeof (buf_r)), if ((Nread=read (fd,buf_r,100)) ==-1)//Read {if (errno==eagain) printf ("No data Yet\n ");} printf ("Read%s from fifo\n", buf_r);//The content read is empty until you run write. After the write is run, the content read is: Write content. Write the data, after reading the pipeline will become empty sleep (1);}   The next three sentences will not be run, but will not affect the performance of the program when the program is executed in the above dead loop when it receives the signal, it will end immediately without executing the following three words. These will be mentioned in the signal processing in the back, now do not understand the relationship, this problem left you to learn the signal processing to solve. Close (FD); Close the pipe pause (); /* Pause, wait for signal */unlink (FIFO); Delete file}

Other topics on plumbing, such as the use of Popen and Pclose, have been made clear in the relevant chapters of the Advanced Programming for UNIX environment. If you want to learn to supplement this knowledge, see this

Inter-process communication for Linux-pipelines