Shell Pipeline Redirection Basics Tutorial

Pipelines exist to address inter-process communication issues, allowing data to be passed between two processes, passing the output data of one process to another process as its input data

1.8.1 Anonymous Pipe "|"

A pipe symbol, like a pipe, passes the data from the pipeline to the pipe outlet.

Pipelines exist to address inter-process communication problems by allowing data to be passed between two processes, passing the output data of one process to another as its input data. The left side of the pipeline is the data giver and the right side of the pipeline is the data receiver.

For example echo "abcd" | passwd --stdin username , the output "ABCD" of the process Echo is represented as the input data of the process passwd.

Basic pipe symbols and their usage are easy to understand. The question now is, for ps aux | grep "ssh" , why is the grep process appearing in the results?

[Root@xuexi ~]# ps aux | grep sshroot    1211 0.0 0.1 82544 3600?    Ss  Jul26  0:00/usr/sbin/sshd-droot   25236 0.0 0.2 145552 5524?    Ss  05:28  0:00 sshd:root@pts/0root   25720 0.1 0.2 145416 5524?    Ss  06:15  0:00 sshd:root@pts/1root   25770 0.0 0.0 112648  948 pts/1  s+  06:15  0:00 grep--color=auto SSH

According to the general idea, the first implementation of PS, the output will be output data to grep, when grep has not run and PS has been run, why can also be counted to the grep process information? The reason is that the pipeline implements inter-process communication, there are intersections between the two processes, the process information is collected after the PS process is run, grep has started and is waiting to receive the data state, and when PS collects any data, the output is put into memory and filtered by the pipeline to grep.

The essence of the pipeline is data transfer, and the output data on the left side of the pipeline is stored in memory and read by the process on the right. If the memory is not sufficient to fully store the output data, the process on the left side of the pipeline waits until the right side of the pipeline pulls the data from the part of the pipe so that the process on the left side of the pipeline continues to output, and the process on the right of the pipeline starts immediately after the process on the left side of the pipeline is started, Waits for data to be passed by the receive pipeline.

In other words, the process of the left and right side of the pipeline runs almost without sequencing.

Then PS aux | How does grep "SSH" avoid the process of grep itself appearing in the results? There are two methods:

Method One: PS aux | grep "SSH" | Grep-v "grep"

Method Two: PS aux | grep "Ss[h]"

[Root@xuexi ~]# ps aux | grep ss[h]root    1211 0.0 0.1 82544 3600?    Ss  Jul26  0:00/usr/sbin/sshd-droot   25236 0.0 0.2 145552 5524?    Ss  05:28  0:00 sshd:root@pts/0root   25720 0.0 0.2 145416 5524?    Ss  06:15  0:00 SSHD:ROOT@PTS/1

Method one is to apply the "-V" attribute of grep, and the second is to apply the characteristics of regular expressions.

In the process of using anonymous pipes, it is possible to find that the processes on both sides of the pipeline belong to one process group, that is, the data on the left of the pipeline can only be passed to the process on the right side of the pipeline, and no other process can read the data. But in addition to anonymous pipes, there are named Pipes, where the named Pipes store data from one process into a single pipe file (FIFO), and other processes can read the pipeline file to read the data in it, meaning that the data reader is no longer restricted. For Named Pipes, refer to the Linux/unix operating system kernel or programming class books, which are generally described in detail.

1.8.2 Redirection

1.8.2.1 Redirection Basics

The most common file descriptors for standard input (stdin), standard output (stdout), and standard error output (stderr) are 0, 1, and 2, where 0, 1, and 2 are also considered to be their numeric designators. For the output information, can be considered to print the information on the screen, and did not give the error is the standard output, give the error is the standard error output, of course, this indicates biased, but easy to understand. You can also customize your own descriptors for advanced redirection, which may be described in a future article.

Standard input =/dev/stdin = Code 0 = < or << symbol.

Standard output =/dev/stdout = Code 1 = > or >> symbol.

Standard error output =/dev/stderr = Code 2 = Use 2> or 2>> symbol.

<, >, 2> implementation is the overlay function,>>, 2>> implementation is the added functionality, but << is not an append function, but rather represents here to generate the document (here document), This is explained in the following cat and redirect mates. In addition, there is <<<, which represents the string here, also shown below.

Sometimes, using "-" also means/dev/stdin. Such as:

[Root@xuexi ~]# Cat/etc/fstab | Cat-

Common 2>&1 and &> symbols in scripts that both redirect stdout and stderr to the same place redirect all output content. such as the most common "&>/dev/null".

Dropping the stdout or stderr to/dev/null means discarding the output information, and in turn, redirecting/dev/null to a file means emptying the file.

[Root@xuexi ~]# cat/dev/null > ab.sh

In addition to this, there are several ways to quickly empty the file

[Root@xuexi ~]# > Ab.sh[root@xuexi ~]#: > ab.sh       # or "True >ab.sh", in fact they are all equivalent to ">ab.sh" [Root@xuexi ~]# Echo ' > Ab.sh[root@xuexi ~]# truncate-s 0 ab.sh  # Truncate command for shrinking and extending file sizes [Root@xuexi ~]# dd If=/dev/null of=ab.sh

Finally, the most important point: in the statement with the redirect symbol, the file is truncated before the command executes. So if you are editing a file and redirecting the results of the edits back to this file, an exception will occur because there will be no suitable content for editing after truncation. A simple example is as follows:

[Root@xuexi ~]# head a.log > A.log

Sometimes it is more dangerous to use the ">" overlay output directly. You can use SET-C to set the output redirection file to not overwrite if it already exists. Use set +c to cancel the set-c effect. If you still want to force overrides when you set up set-c, you can use ">|" Instead of ">" to redirect the output. Similarly, error output has this feature.

[Root@xuexi tmp]# set-c[root@xuexi tmp]# cat Flip >ttt.txt-bash:ttt.txt:cannot Overwrite existing File[root@xuexi TM p]# Cat Flip >| Ttt.txt[root@xuexi tmp]# Set +c

1.8.2.2 Cat and redirect mates

With cat use, you can enter content into a file by branch.

[Root@xuexi tmp]# cat <<eof>log.txt  # covers the way input to log.txt> this is stdin character> EOF

You can also use the following method.

[Root@xuexi tmp]# cat >log1.txt <<eof > This are stdin character first!> EOF

On the one hand, the EOF section must use "<<eof", which represents the here document, after which the input is entered as a document to Cat. Since it is the document, there must be a document terminator that marks the end of the document, and the Terminator uses the character after the here document, for example, EOF. Instead of using EOF, the same is true with other characters, but the Terminator of document must be changed as well. Such as:

[Root@xuexi ~]# cat <<abcx> 123> 345> abcx123345

On the other hand, >log1.txt indicates that the contents of the document are overwritten in the Log1.txt file, and if it is to be appended, >>log1.txt is used. So, append the following way:

[Root@xuexi tmp]# cat >>log1.txt <<eof > This are stdin character first!> EOF

Or

[Root@xuexi tmp]# Cat <<eof>>log1.txt > This is stdin character first!> EOF

1.8.2.3 Tee Dual Orientation

You can use tee dual orientation. In general, redirection either enters information into a file or prints it to the screen, but it is cumbersome to output to the screen and output to a file. The idea can be realized with the dual-orientation feature of tee.

Tee [-A] file

Option Description:

-A: The default is to overwrite the output to a file and use this option to change to the append behavior.

File: In addition to the output to standard output, it is also output to file. If file is "-", then it is entered again into standard output.

For example, the following code, the file contents of a beginning to save all to B.log, while the copy to the back of the cat, using this cat and save the content to the X.log. Where "-" stands for the front stdin.

[Root@xuexi tmp]# Cat A * | Tee B.log | Cat->x.log

You can also print directly to the screen:

[Root@xuexi tmp]# Cat A * | Tee B.log | Cat

Tee is saved to a file by default using overrides, and can be appended to a file using the-a option. Such as:

[Root@xuexi tmp]# Cat A * | Tee-a B.log | Cat

You can now use cat and redirect to create a file or write content to a file while displaying a copy on the screen.

[Root@xuexi tmp]# Cat <<eof | Tee ttt.txt> x y> z 1> EOFX yz 1

1.8.2.4 << and <<<

,<< and <<< are special redirect symbols in bash. << says here document,<<< that this is the here string.

Here document has been explained above, for here string, the <<< string as the input data.

For example:

passwd--stdin User <<< Password_value

Equivalent to:

echo Password_value | passwd--stdin User