Shell pipeline and execution sequence analysis

1. Basic Concepts
A, I/O redirection is usually related to FD, and shell FD is usually 10, that is, 0 ~ 9;
B. There are three commonly used FD types: 0 (stdin, standard input), 1 (stdout, standard output), and 2 (stderr, standard error output ), the default value is related to keyboard, monitor, and monitor;
C. Use <to change the reading data channel (stdin) so that it can read from the specified file;
D. Use> to change the output data channel (stdout, stderr) and output it to the specified file;
E and 0 are the default values of <, so <and 0 <are the same. Similarly,> and 1> are the same;
F. In IO redirection, The stdout and stderr pipelines are prepared before reading data from stdin;
G. Pipeline "|" (pipe line): stdout of the previous command receives stdin of the next command;
H. The tee Command copies stdout to the file without affecting the original I/O;
I. bash (ksh) Command Execution Process: Analysis command-variable evaluate-command substitution (''and $ ()-redirect-Wildcard expand-confirm path-Execute Command;
J. () Place the command group in sub-shell for execution, also known as nested sub-shell. It has a very important feature: Inherit the Standard input, output, and, and error plus any other open file descriptors.
K. exec command: It is often used to replace the current shell and restart a shell. In other words, there is no starting shell. When you use this command, any existing environment will be cleared ,. When exec operates on file descriptors, it is only in this case that exec will not overwrite your current shell environment.

2. Basic IO
Cmd> file: redirects stdout to the file.
Cmd> file: redirects stdout to the file (append)
Cmd 1> fiel redirects stdout to file
Cmd> file 2> & 1 redirect stdout and stderr together to the file.
Cmd 2> file: redirects stderr to the file.
Cmd 2> file redirects stderr to the file (append)
Cmd> file 2> & 1 redirects stderr and stderr together to the file.
Cmd <file> file2 cmd command uses file as stdin and file2 as stdout
Cat <> file: open the file in read/write mode.
Cmd <file cmd command uses file as stdin
Cmd <delimiter Here document, read from stdin until the delimiter is encountered

3. Advanced IO
> & N use the system to call dup (2) copy the file descriptor n and use the result as the standard output.
<& N standard input copied from file descriptor n
<&-Disable Standard input (keyboard)
> &-Disable Standard output
N <&-indicates that n is disabled
N> &-indicates that output n is disabled.
All the above forms can lead to a number. In this case, the created file descriptor is specified by this number rather than the default 0 or 1. For example:
... 2> run a command in file and direct the error output (file descriptor 2) to file.
... 2> & 1 run a command and merge its standard output and output. (Strictly speaking, file descriptor 2 is created by copying file descriptor 1, but the effect is usually that two streams are merged .)

2> & 1 Description: 2> & 1, that is, FD2 = FD1. Here, it does not mean that the value of FD2 is equal to the value of FD1, because> is used to change the data channel sent, which is generally described: run stderr to stdout.
However, when using a form similar to cmd 1> & 3, the principle is the same, but often different from 2> & 1 and 1> & 2 are usually used for merging.

Note: The cmd n> & n and exec n> & n of common cmd commands are different.

Exec 0 exec 1> outfilename # open the file outfilename as stdout
Exec 2> errfilename # open the file errfilename as stderr
Exec 0 <&-# disable FD0
Exec 1> &-# disable FD1
Exec 5> &-# disable FD5

Q:
If FD0, FD1, and FD2 are disabled, what are the consequences?
What is the difference between restoring FD0, FD1, and FD2 and disabling FD0, FD1, and FD2? What are the codes?
FD3 ~ Enabled ~ FD9, after we use it, do you think it is to close them or restore them?

The following is a prompt (the example is from CU ):
Exec 6> & 2 2> ver # FD2 (originally sent to monitor) directed to the file ver
Command> dev/null & # discard FD1 (stdout)
Exec 2> & 6 # restore FD2

4. Simple Example (where yes.txtexists and No. TXT does not exist)
A, stdout, and stderr are both sent to egrep through the pipeline:
(Ls yes.txt 2> & 1; ls no.txt 2> & 1) 2> & 1 | egrep \ *> file
(Ls yes.txt; ls no.txt) 2> & 1 | egrep \ *> file

###
This example shows how to understand the command execution sequence and pipeline "|"
Before executing the command, you must perform redirection and connect the stdout of the nested sub-shell to the stdin of the egrep command.
The two commands in () can be considered as one command. Its stdout (FD1) uses "|" as the stdin of egrep, plus 2> & 1, the initial stdout and stderr are both sent to the pipeline "|.
###

B. Nothing is sent to egrep through a pipeline, and all is sent to monitor.
(Ls yes.txt 2> & 1; ls no.txt 2> & 1)> & 2 | egrep \ *> file
Although FD2 is transferred to FD1 in (), when> & 2 is encountered outside (), all results are sent to monitor.

5. Intermediate examples (here, the file you exists, and the files no and wu do not exist)
R2007 brother: http://bbs.chinaunix.net/forum/viewtopic.php? T = 221848 & show_type = new & sid = cf30398c911e0d2b16313c6922123f67

Condition: stderr sends the message to egrep through the pipeline, and the correct message is still sent to monitor (unchanged)

Exec 4> & 1; (ls you no 2> & 1 1 1> & 4 4> &-; ls wu 2> & 1 1> & 4 4 4> &-) | egrep \ *> file; exec 4> &-
Or
Exec 4> & 1; (ls you no; ls wu) 2> & 1 1> & 4 4> &-| egrep \ *> file; exec 4> &-

R2007 has already posted a detailed description on it. If two conditions are added:
(1) concurrent operation of cmd1 and cmd2 is required;
(2) Assign the return value of cmd1 to the variable ss.

Then:
Exec 3> & 1; exec 4> & 1
Ss =$ (ls you no 2> & 1 1> & 3 3> &-; echo $? > & 4) | egrep \ *> file) 4> & 1)
Exec 3> &-; exec 4> &-

Note:
Exec 3> & 1; 4> & 1
### Create FD3 to restore FD1 from the following ls Statement (sub-shell) to FD1, that is, output to monitor, you can regard FD3 as the initial FD1 hard disk backup (that is, output to monitor );
### Create FD4 and use it to save the return value of ls (echo $ ?), You can think of FD4 as a store for computing "echo $?" during the test ?" Draft paper;

(Ls you no 2> & 1 1> & 3 3> &-; echo $? > & 4)
### Remember the sub-shell and pipeline mentioned above. This command First inherits from FD0, FD1, FD2, FD3, and FD4, which are located before the pipeline. Therefore, before running the command, the FD1 of the sub-shell is connected to the pipeline "|.
However, the condition is that stderr is sent to egrep through the pipeline, and stdout is still output to monitor.
Therefore, through 2> & 1, The FD1 pipeline of the sub-shell is "sent" to "FD2, so the stderr in the sub-shell is sent to the pipeline" | ";
Then, through 1> & 3, the previous "hard disk backup" is restored to the FD1 of the sub-shell, so the FD1 in the sub-shell is sent to the monitor.
3> &-, and close 3;
Run echo $? , The output value should have been sent to the pipeline, and the output is sent to "draft paper" FD4 through> & 4, and left for backup.

(Ls you no 2> & 1 1> & 3 3> &-; echo $? > & 4) | egrep \ *> file)
Therefore, stderr sends egrep through the pipeline, and stdout sends it to monitor. However, FD4 still exists. Where does it go?
$ (Ls you no 2> & 1 1> & 3 3> &-; echo $? > & 4) | egrep \ *> file) 4> & 1)
The last 4> & 1 means to redirect FD4 to FD1. However, because it is output in $ (), its value is assigned to the variable ss.

Disable FD3 and FD4 in the last line.

6. Advanced examples
Lightspeed Moderator: Shell classic problems [I/O redirection] (http://bbs.chinaunix.net/forum/viewtopic.php? T = 452079 & show_type = new)
[Q] for command cmd1, cmd2, cmd3, IPv4, how to use one-way pipelines to complete the following functions:
1. Parallel Execution of all commands
2. stdin is not required for cmd1 and cmd2.
3. stdout of cmd1 and cmd2 is directed to stdin of cmd3.
4. stderr of cmd1 and cmd2 is directed to stdin of limit 4.
5. stdout of cmd3 is directed to file a and stderr is directed to the screen.
6. stdout of IPv4 is directed to file B, and stderr is directed to the screen.
7. The return code of cmd1 is assigned to the variable s.
8. Temporary files cannot be used.

Solution:
Exec 3> & 1; exec 4> & 1
My_value = $(cmd1 1> & 3; echo $? > & 4) | cmd2) 3> & 1 | cmd3> a 2> & 3) 2> & 1 | 1_4> B) 4> & 1)
Exec 3> &-; exec 4> &-

Explanation:
Exec 3> & 1; exec 4> & 1
Set up FD3, restore cmd1 to FD1, and recover cmd3 to fd2;
Create FD4 and save "echo $ ?" The "draft" of the output value"

The first pair of parentheses goes to the first pipe: (cmd1 1> & 3; echo $? > & 4) |
Cmd1 itself does not have stdin, and its stdout was originally sent to the first pipeline. Due to the role of 1> & 3, its stdout was sent to FD3;> & 4 actually sends the return code of cmd1 to fd4. The stderr of cmd1 is waiting for processing by default. Finally, nothing was sent to the pipeline;

The second pair of brackets goes to the second pipe: (cmd1 1> & 3; echo $? > & 4) | cmd2) 3> & 1 |
Because the stdout of cmd1 In the first bracket is sent to FD3, the left end of the MPs queue does not have any input, and cmd2 does not have stdin. The stdout of cmd2 is the default value;
The second pair of brackets shows a command, and all of its stdout is sent to the second pipe "|". Meanwhile, due to the role of 3> & 1, in the past, stdout of cmd1 was sent to FD3 and together with stdout of cmd2. Therefore, both stdout of cmd1 and cmd2 were sent to the second pipe "| ". The stderr of cmd1 and cmd2 is still waiting for the next step by default;

The third to the third pipe: (cmd1 1> & 3; echo $? > & 4) | cmd2) 3> & 1 | cmd3> a 2> & 3) 2> & 1 |
Cmd3> a 2> & 3: After cmd3 receives and processes stdin from the pipeline, its stdout is sent to file a, and its stderr is sent to FD3. Because FD3 inherits FD1, its stderr is actually sent to monitor. If there is no "2> & 3", The stderr of cmd3 will interfere with the error output of cmd1 and cmd2, so it is required;
Think of the third bracket as a command, and stdout is sent to the pipeline "|". Because 2> & 1, stderr is also sent to the pipeline. However, because stdout of cmd1 and cmd2 has been sent to cmd3 for processing, while stdout of cmd3 is output to file a, and stderr of cmd3 is also sent to monitor, only stderr of cmd1 and cmd2 is actually sent to the pipeline.

When 4> B: When 4 receives the stdin from the MPs queue, its stdout is directed to file B, and stderr is output to monitor by default.

The fourth pair of parentheses :( (cmd1 1> & 3; echo $? > & 4) | cmd2) 3> & 1 | cmd3> a 2> & 3) 2> & 1 | 1_4> B) 4> & 1
The FD1 and FD2 commands in the four pairs of brackets are all processed, but there is "echo $? > & 4 "not handled. "4> & 1" is used to "bind FD4 to stdout". However, because stdout of other commands has been processed, only $? .
Because $ () creates an pipeline and the input end is a () command, $? Is assigned to the variable my_value.

The last line is to disable FD3 and FD4.
In addition, restore the stdout: exec 1> & 2 redirected or closed, and restore the stderr: exec 2> & 1. If both stdout and stderr are disabled and the original FD is not saved, you can use: exec 1>/dev/tty to restore.

++ ++
I tried to answer the following questions. If any errors occur, please correct them!

7. In an Interactive shell, execute I/O redirection with exec.
1) Can Stdin and stderr be directed to a file? What are the results?
A. in interactive shell, stdin can be directed to a file. Execute: exec 0. The result is that every row in the in file will be automatically executed, and an exit command will be added at the end to exit (or return to the normal shell ).
For example, in file content: $ more in
Date
Read lsp
Echo hahha
Echo "this is $ lsp"

Execute the command at the prompt: $ exec 0 $ date
Tue Jan 18 18:29:07 HKT 2005
$ Read lsp # The "hahaha" of the "echo hahha" sentence under it has been read into the variable lsp.
$ Echo "this is $ lsp"
This is echo hahha
$ Exit

B. In interactive shell, stderr can be directed to a file. Execute: exec 2> err
Result: The command prompt PS is blocked, and the entered command is also blocked. However, the result of command execution will be displayed back to the screen if it is stdout. If it is stderr, it will not be displayed back to the screen. The command prompt, command, and stderr are saved to the file err. For example:
$ Exec 2> err
Err in out # execute the ls command
Tue Jan 18 18:55:58 HKT 2005 # execute the date command, and then execute "ls nofile". The nofile does not exist.
$ # Execute the exit command

Now let's check the err file:
$ More err
[Lsp @ ii lsp] $ ls
[Lsp @ ii lsp] $ date
[Lsp @ ii lsp] $ ls nofile
Ls: nofile: No such file or directory
[Lsp @ ii lsp] $ exit
Exit

C. In interactive shell, stdout can be directed to a file. This makes us commonly used. Let's not talk about it. It is to direct the wrong Output content to the file. Correct output content is not affected.

2) can Stdin and Stderr be disabled? What are the results?
In the interactive shell, if stdin is disabled, for example, exec 0 <&-, the result is to exit (or return to the normal shell ).
In the interactive shell, if stderr is disabled, for example, exec 2> &-, and the status is the same as stderr, It is not saved.
In the interactive shell, If you disable stdodigest, for example, exec 1> &-, you only need to run the commands sent to monitor with stdout or stderr content, such as ls and date, an error will be reported: "ls: write error: Bad file descriptor ". Others such as cd, mkdir ,...... These commands are not affected.

3) If stdin, stdout, and stderr are redirected or disabled, but the original FD is not saved, can it be restored to the default state?
* ** If stdin is disabled, it will cause exit, and it cannot be restored.
* ** If one of stdout and stderr is redirected or disabled, it can be restored because they are sent to monitor by default (but I do not know if there will be any other impact ). For example, restore the redirected or closed stdout: exec 1> & 2, restore the redirected or closed stderr: exec 2> & 1.
* ** If both stdout and stderr are disabled and the original FD is not saved, you can use: exec 1>/dev/tty to restore the data.

++
The following is a reference to r2007's reply! Thank you!
++

8. cmd> a 2> a and cmd> a 2> & 1 why is it different?
Cmd> a 2> a: Both stdout and stderr are directly sent to file a. File a is opened twice, which causes stdout and stderr to overwrite each other.
Cmd> a 2> & 1: stdout is directly sent to file a. stderr is sent to file a After inheriting the FD1 pipeline. File a is opened only once, that is, it is opened by FD1.

Their differences are:
Cmd> a 2> a is equivalent to two pipelines, FD1 and FD2, that compete with each other to use file;
Cmd> a 2> & 1 only uses one pipeline FD1, but it already includes stdout and stderr.
In terms of IO efficiency, cmd> a 2> & 1 is more efficient.