Use pipelines for programming in Linux

Use pipelines for programming in Linux

Http://www.lupaworld.com/tutorial-view-aid-10366.html

 

Introduction

In this article, we will learn about the GNU/Linux pipeline. Although the pipeline model is very old, it is still a very useful inter-process communication mechanism. We will learn what a semi-bidirectional pipeline is and what a famous pipeline is. They all provide a FIFO (first-in-first-out) queuing model to allow inter-process communication.

 

MPs Queue Model

A Visualized pipeline is described as a unidirectional connector between two entities. For example, let's take a look at the following GNU/Linux Command:

Ls-1 | WC-l

This command creates two processes, one associated with LS-l and the other associated with WC-L. Then it connects the two processes (11.1) by setting the standard input of the second process to the standard output of the first process ). The result is that the number of files in the current subdirectory is calculated.

Our command sets a pipeline between two GNU/Linux commands. After the command LS is executed, its output is used as the input of the second command WC (word count. This is a one-way pipeline-Communication occurs in one direction. The connection between the two commands is completed by GNU/Linux. We can also do this in the application (we will prove it later ).

 

Anonymous pipeline and named Pipeline

An anonymous pipeline or one-way pipeline provides a method for a process to communicate with a sub-process (anonymous type. This is because there is no way to find an anonymous process in the operating system. Its most common usage is to create an anonymous pipeline in the parent process, and then pass the pipeline to its child process, and then they can communicate. NOTE: If two-way communication is required, we should consider using the socket (sockets) API.

Another type of MPs queue is the famous MPs queue. The function of a famous pipeline is similar to that of an anonymous pipeline. The difference is that it can exist in a file system and can be found by all processes. This means that processes without kinship can use it for communication.

In the following sections, we will learn famous and anonymous pipelines at the same time. We will make a quick tour of the pipeline, and then we will learn in detail the pipeline API and the GNU/Linux system-level commands that support pipeline programming.

 

Whirlwind tour

Let's start our whirlwind tour with a simple pipeline programming model example. In this simple example, we create a pipeline in a process and write a message to it. The next step is to read the previously written message from this pipeline, then display it.

Listing 11.1: A simple pipeline example

1: # include

2: # include

3: # include

4:

5: # define max_line 80

6: # define pipe_stdin 0

7: # define pipe_stdout 1

8:

9: int main ()

10 :{

11: const char * string = {"a sample message ."};

12: int ret, mypipe [2];

13: Char buffer [max_line + 1];

14:

15:/* Create the pipe */

16: ret = pipe (mypipe );

17:

18: If (ret = 0 ){

19:

20:/* write the message into the pipe */

21: Write (mypipe [pipe_stdout], String, strlen (string ));

22:

23:/* read the message from the pipe */

24: ret = read (mypipe [pipe_stdin], buffer, max_line );

25:

26:/* null terminate the string */

27: Buffer [RET] = 0;

28:

29: printf ("% s/n", buffer );

30:

31 :}

32:

33: Return 0;

34 :}

In listing 11.1, we used the pipe function in line 16 to create our pipeline. We passed in an integer array with two elements representing our pipeline to the pipe function. The pipeline is defined as a pair of separate file descriptors-one input and one output. We can write data from one end of the pipeline and read data from the other end of the pipeline. If the pipeline is successfully created, the API function pipe returns 0. Based on the returned results, mypipe will include two new file descriptors that represent the input of the pipeline (mypipe [1]) and the output of the pipeline (mypipe [0. (PS: In file descriptor indicates that the file or device associated with it is equivalent to an input device; out file descriptor indicates that the file or device associated with it is equivalent to an output device .)

In line 21, we use the function write to write our messages to the pipeline. We specify the stdout (standard output) Descriptor (from the application perspective, rather than the pipeline ). The pipeline now contains our messages, which can be read by the function read in 24 rows. From the perspective of the application, we use the stdin (standard input) descriptor to read data from the pipeline. The function read saves the messages read from the pipeline to the variable buffer. To make the buffer truly a string, we add a 0 (null) after it, and then we can use the printf function to display it in 29 rows.

Although this example is interesting, we can use any number of mechanisms to execute communication between ourselves. Next we will learn more complex examples of communication between processes (whether related or unrelated.

 

Detailed Review

Although the vast majority of pipeline models are function pipe, there are two other functions that should be discussed on their availability Based on pipeline programming. Table 11.1 lists the functions that we will discuss in detail in this chapter:

 

API function usage

Pipe creates an anonymous Pipeline

DUP creates a copy of a file descriptor

Mkfifo creates a famous pipe (first-in-first-out)

 

At the same time, we will also learn some other functions that can be used for pipeline communication, especially those that can use pipeline for communication.

Note: We must remember that a pipeline is just a file descriptor, so any function that can operate on the file descriptor can use the pipeline. This includes select, read, write, fcntl, and freopen, but not just these functions.

Pipe

The API function pipe creates a new pipeline represented by an array containing two file descriptors. The pipe function is prototype as follows:

# Include

Int pipe (int fds [2]);

If the pipe function is successful, it returns 0; otherwise, it returns-1, and errno is set as appropriate. If the returned result is successful, the array FDS (which is passed by reference) will have two active file descriptors. The first element of the array is a file descriptor that can be read by the application. The second element is a file descriptor that can be written by the application.

 

Now let's look at a more complex example of the next application pipeline in a multi-process application. In this application, we will create an pipeline in row 14, and then create a new process (child process) in the parent process of the program using the fork function in Row 16 ). In the sub-process, we try to read from the input file descriptor of our pipeline (18 rows), but if there is nothing to read, we will suspend this sub-process. After the read operation, we terminate the string through a null (that is, let an array character into a string) and print the string we read. The parent process simply writes a test string to the pipeline by using the output (write) file descriptor (the second element of the array in the pipeline structure), and then waits for the child process to exit using the wait function.

Note: apart from the fact that our child processes inherit the file descriptors created by the parent process using the pipe function and use them to communicate with another child process or parent process, there is no more eye-catching about this program. Recall: Once the fork function is completed, our processes are independent (except that the child process inherits the features of the parent process, such as the pipeline descriptor ). Since the memory is separated, the pipe method provides an interesting model for inter-process communication.

Listing 11.2 illustrates the pipeline model between two processes.

1: # include

2: # include

3: # include

4: # include

5:

6: # define max_line 80

7:

8: int main ()

9 :{

10: int thepipe [2], RET;

11: Char Buf [max_line + 1];

12: const char * testbuf = {"A test string ."};

13:

14: If (pipe (thepipe) = 0 ){

15:

16: If (Fork () = 0 ){

17:

18: ret = read (thepipe [0], Buf, max_line );

19: Buf [RET] = 0;

20: printf ("child read % s/n", Buf );

21:

22 :}

23: else {

24: ret = write (thepipe [1], testbuf, strlen (testbuf ));

25: ret = wait (null );

26:

27 :}

28:

29 :}

30:

31: Return 0;

32 :}

Note: In these simple programs, we did not discuss how to close the pipeline because the resources associated with the pipeline will be automatically released once a process ends. Even so, calling the function close to close the descriptor of the pipeline is a very good programming practice, such:

Ret = pipe (mypipe );

...

Close (mypipe [0]);

Close (mypipe [1]);

If the write end of the MPs queue is closed and a process wants to read data from the MPs queue, a 0 value is returned. This means that the MPs queue is no longer in use and should be closed. If the read end of the MPs queue is closed and a process wants to write data to the MPs queue, a signal is generated. This signal (which will be discussed in chapter 12th-Introduction to socket programming) is called sigpipe. Applications that want to write data to pipelines usually contain a signal handle that captures such a situation.

 

 

DUP and dup2

The DUP and dup2 functions are very useful for copying a file descriptor. They are often used to redirect stdin, stdout, or stderr of a process. The DUP and dup2 functions are prototype as follows:

# Include

Int DUP (INT oldfd );

Int dup2 (INT oldfd, int targetfd );

The DUP function allows us to copy a descriptor. We pass an existing descriptor to the function, and then it returns a new descriptor identical to the previous one. This means that the two descriptors share the same structure. For example, we use a file descriptor to execute an lseek (search in the file), and the current position of the file (internal pointer of the file) is the same in the second. The use of the function DUP is shown in the following code snippet:

Int fd1, fd2;

...

Fd2 = DUP (fd1 );

Note: creating a descriptor before fork calls produces the same effect as calling the function DUP. The sub-process receives a replication descriptor, just as it imitates the DUP call.

The function dup2 is similar to dup, but it allows the caller to specify the ID of an active descriptor target descriptor. After the function dup2 returns successfully, the new target descriptor copies the first Descriptor (targetfd = oldfd ). Let's take a look at the following code snippet to illustrate dup2 usage:

Int oldfd;

Oldfd = open ("app_log", (o_rdwr | o_create), 0644 );

Dup2 (oldfd, 1 );

Close (oldfd );

In this example, we open a new file named "app_log" and receive a file descriptor named fd1. We call dup2 by using oldfd and 1, so we replace file descriptor 1 (stdout standard output) with oldfd (our new file ). Any data written to stdout with standard output will be written into a file called "app_log. Note that we disable the oldfd after copying it. However, this does not close the newly opened file because file descriptor 1 is now -- we can think so.

 

 

Now let's learn a more complex example. Let's look back at the previous section in this chapter to make the LS-L output a WC-l input. We now use a c Application to explore how to allow this example (listing 11.3 ).

At the beginning of listing 11.3, we created our pipeline in line 9 and then created the child process of the program in line 13-16 and then created the parent process in line 20-23. Stdout (standard output) descriptor is disabled in row 13. Here, the sub-process provides the LS-l function and won't write it into stdout. On the contrary, it is written into our pipeline (DUP is used for redirection ). In line 14, we used dup2 to redirect stdout to our pipeline (PFDS [1. Once such an operation is completed, we close the input of our pipeline (because it will never be used ). Finally, we use the execlp function to replace the image of our sub-process with the image of the command LS-L. Once this command is executed, any output will be transmitted to the input.

Now let's take a look at the receiver of the pipeline. The parent process acts as such a role and follows a similar pattern. We first disable stdin (standard input) in 20 rows because we will not receive any data from it. Next, we will use the function dup2 (line 21) to make stdin the output end of the pipeline. This is done by making the file descriptor 0 (generally stdin) function the same as PFDS [0. We disabled the stdout standard output end of the MPs Queue (PFDS [1]) because it is not used here (22 rows ). Finally, execlp is used to run the WC-1 (23 rows) command that treats the content in the pipeline as its input ).

Listing 11.3: assembly line command C program

1: # include

2: # include

3: # include

4:

5: int main ()

6 :{

7: int PFDS [2];

8:

9: If (pipe (PFDS) = 0 ){

10:

11: If (Fork () = 0 ){

12:

13: Close (1 );

14: dup2 (PFDS [1], 1 );

15: Close (PFDS [0]);

16: execlp ("ls", "ls", "-1", null );

17:

18:} else {

19:

20: Close (0 );

21: dup2 (PFDS [0], 0 );

22: Close (PFDS [1]);

23: execlp ("WC", "WC", "-l", null );

24:

25 :}

26:

27 :}

28:

29: Return 0;

30 :}

In this program, 01:10 is important and worth writing down-this is how our sub-process redirects its own output to the input of the pipeline, the parent process redirects its own input to the output of the pipeline-a very useful technique worth remembering.

Mkfifo

The mkfifo function is used to create a file (also known as a pipe) that provides the FIFO function in the file system ). So far, we have discussed anonymous pipelines. They are dedicated to communication between a parent process and its Child processes. Famous pipelines are visible in the file system and can be used by any process. The mkfifo function is prototype as follows:

 

# Include

Int mkfifo (const char * pathname, mode_t mode );

The command mkfifo requires two parameters. The first (pathname) is a special file to be created in the file system. The second mode specifies the read/write permissions of the FIFO. If the command succeeds, mkfifo will return 0, and-1 will be returned if the command fails (and errno will be written as appropriate ). Let's take a look at the next example of creating a FIFO using the mkfifo function:

Int ret;

...

Ret = mkfifo ("/tmp/cmd_pipe", s_ififo | 0666 );

If (ret = 0 ){

// Named Pipe successfully created

} Else {

// Failed to create Named Pipe

}

In this example, we use the pipeline _pipe file to create a FIFO (famous Pipeline) in the sub-directory/tmp ). Next, we open the file for reading and writing, and through it we can communicate. Once we open a famous Pipeline, we can use a typical I/O command to read and write. For example, the following is a code segment that uses fgets to read from the pipeline:

PFP = fopen ("/tmp/pai_pipe", "R ");

...

Ret = fgets (buffer, max_line, PFP );

We can use the following code segment to write the pipeline for the above Code:

PFP = fopen ("/tmp/pai_pipe", "W + );

...

Ret = fprintf (PFP, "here's a test string! /N ");

The interesting thing about famous pipelines is that they work in a place that is seen as a collection point. We will discuss this in the discussion of the system command mkfifo. A reader cannot open a famous Pipeline unless a writer actively opens the other end of the pipeline. The reader will be blocked when calling the OPEN function until a writer appears. Despite such a restriction, the famous pipeline is still a useful mechanism for inter-process communication.

System commands

Let's take a look at a system command related to the MPs pipeline model. The command mkfifo is the same as the API function mkfifo, allowing us to create a famous pipeline on the command line.

Mkfifo

Command mkfifo is one of the two methods for creating a famous pipe (Special FIFO file) on the command line. The command mkfifo is generally used as follows:

Mkfifo [Options] Name

Here, options-M indicates the mode (permission), and name indicates the name of the famous pipeline to be created (including the path if needed ). If no permission is specified, the default value is 0644. The following is an example. It creates a famous Pipeline named pipeline _pipe in the directory/tmp:

$ Mkfifo/tmp/cmd_pipe

We can simply adjust the option by using option-M. The following is an example of setting the permission to 0644 (but we must first Delete the original ONE ):

$ RM pai_pipe

$ Mkfifo-M 0644/tmp/pai_pipe

Once the permission is created, we can use this pipeline to communicate on the command line. On another terminal, we use the echo command to write to the famous Pipeline pai_pipe:

$ Echo HI> pai_pipe

When this command ends, our reader will be awakened and ended (the sequence of the reader commands is clearly completed below ):

$ Cat pai_pipe

Hi

$

The preceding example shows that a famous pipeline can be used not only in C programs but also in scripts.

You can also use the mknod command to create a famous Pipeline (the special file that follows it can be many other types of files ). Run the mknod command to create a famous Pipeline;

$ Mknod pai_pipe P

Here, the famous Pipeline pipeline _ pipe is created in the current subdirectory (P is the type of the famous pipeline ).

Summary

In this chapter, we made a whirlwind tour of famous and anonymous pipelines. We reviewed the program and command line methods for creating pipelines, and also reviewed the typical I/O mechanisms for communication using them. We also reviewed how to use the DUP and dup2 functions for I/O redirection. Although pipelines are very useful, these commands or functions are still very useful in a specific scenario (no matter where a file descriptor is used, such as a socket or file ).

Pipeline programming API

# Include

Int pipe (INT filedes [2]);

Int DUP (INT oldfd );

Int dup2 (INT oldfd, int targetfd );

Int mkfifo (const char * pathname, mode_t mode );

Source: 51cto