Shell piping and execution sequence analysis _linux shell

1. Basic Concepts
A, I/O redirection is usually related to FD, Shell FD is usually 10, namely 0~9;
b, commonly used FD has 3, for 0 (stdin, standard input), 1 (stdout, standard output), 2 (stderr, standard error Output), the default and keyboard, monitor, monitor related;
C, the use of < to change the read Data channel (stdin), from the designated file read;
D, use > To change the sent out data channel (stdout, stderr), so that the output to the designated file;
E, 0 is the default value of <, so < is the same as 0<;,> is the same as 1>;
F, in IO Redirect, stdout and stderr pipeline will be ready before the data from stdin read;
G, Pipeline "|" (Pipe line): The stdout of the previous command receives the stdin of the next command;
H, tee command is without affecting the original I/O, the stdout copies of a copy to the file;
I, Bash (Ksh) procedure to execute a command: Parse command-variable Evaluation-command substitution (' and $ ())-redirect-wildcard expansion-Determine path-execute command;
J, () put command group Sub-shell to execute, also known as nested Sub-shell, it has a very important feature is: Inherit the parent shell standard input, output, and error plus an Y other Open file descriptors.
K, EXEC command: Often used to replace the current shell and restart a shell, in other words, did not start the child shell. Any existing environment will be purged when this command is used. When exec operates on a file descriptor, it is only then that EXEC does not overwrite your current shell environment.

2. Basic IO
cmd > File redirect stdout to file
cmd >> file redirect stdout to file (append)
CMD 1> fiel redirect stdout to file
cmd > File 2>&1 redirect stdout and stderr to file
cmd 2> file redirect stderr to file
cmd 2>> file redirect stderr to file (append)
cmd >> file 2>&1 redirect stderr and stderr to file
The cmd < file >file2 cmd command takes the file file as stdin to file2 files as stdout
Cat <>file Open File in read-write mode
cmd < file cmd command with the file file as stdin
CMD << delimiter here document, read from stdin until you encounter the delimiter delimiter

3, Advanced IO
>&n uses system call DUP (2) to copy the file descriptor N and use the result as standard output
<&n standard input Copy from file descriptor n
<&-Turn off standard input (keyboard)
>&-Turn off standard output
n<&-indicates that n-number input is closed
n>&-indicates that n-number output is closed
All of these forms can lead to a number, where the file descriptor is specified by this number instead of the default 0 or 1. Such as:
... 2>file runs a command and directs the error output (file descriptor 2) to file.
... 2>&1 runs a command and merges its standard output and output. (Strictly speaking, a file descriptor of 2 is created by copying a file descriptor, but the effect is usually to merge two streams.) )

2>&1 Description: 2>&1 is also fd2=fd1, this is not to say that FD2 value is equal to FD1 value, because > is to change the sent out of the data channel, the popular saying is: The stderr and to stdout.
But using a form like the cmd 1>&3, the principle is the same, but often different from the 2>&1 and 1>&2 commonly used for merging.

Note: The General cmd command cmd n>&n and exec n>&n are different.

EXEC 0 EXEC 1>outfilename # Open file Outfilename as stdout
EXEC 2>errfilename # Open file Errfilename as stderr
EXEC 0<&-# Close FD0
EXEC 1>&-# Close FD1
EXEC 5>&-# Close FD5

Ask:
If FD0, FD1, FD2 are closed, what are the consequences?
What is the difference between resuming FD0, FD1, FD2 and closing FD0, FD1, and FD2? What is the code, respectively?
Open the FD3~FD9, after we run out, do you think they are closed or restored?

The following are hints (examples from CU):
exec 6>&2 2>ver # FD2 (which was originally sent to monitor) is directed to the file ver
Command >>dev/null & #丢弃FD1 (stdout)
EXEC 2>&6 # Restore FD2

4, simple example (where yes.txt exist, No.txt does not exist)
A, stdout and stderr were sent to Egrep by pipeline:
(LS yes.txt 2>&1;ls no.txt 2>&1) 2>&1|egrep \* >file
(LS yes.txt;ls no.txt) 2>&1|egrep \* >file

###
This example is to let everyone: understand the order of command execution and the pipe "|"
Before the command is executed, the process of redirection is done first, and the stdout of the nested Sub-shell to the stdin of the Egrep command.
Nested Sub-shell, the two commands in () can be considered as a command. Its stdout (FD1) is egrep by "|" as stdin, plus 2>&1, the initial stdout and stderr are sent to the pipe "|".
###

B, nothing was piped to egrep, all sent to monitor.
(LS yes.txt 2>&1;ls no.txt 2>&1) >&2|egrep \* >file
Although in () inside will FD2 transfer to FD1, but in () outside, encounter >&2, all the results are sent to monitor.

5, the middle order example (where you this file is present, No and Wu these two files do not exist)
Brother R2007: http://bbs.chinaunix.net/forum/viewtopic.php?t=221848&show_type=new&sid= cf30398c911e0d2b16313c6922123f67

Condition: StdErr to Egrep via pipeline, the correct message is still given to monitor (invariant)

EXEC 4>&1; (ls you no 2>&1 1>&4 4>&-;ls Wu 2>&1 1>&4 4>&-) |egrep \* >fi Le;exec 4>&-
Or
EXEC 4>&1; (ls you no;ls wu) 2>&1 1>&4 4>&-|egrep \* >file;exec 4>&-

Brother R2007 has been posted in detail, if two conditions are added:
(1) Require CMD1 and CMD2 to run in parallel;
(2) Assign the return value of the CMD1 to the variable SS.

is the following:
EXEC 3>&1;exec 4>&1
ss=$ ((ls you no 2>&1 1>&3 3>&-;echo $ >&4) |egrep \*)
EXEC 3>&-;exec 4>&-

Description
EXEC 3>&1;4>&1
### establishes FD3, which is used to restore the FD1 in the following LS statement (sub shell) to the normal FD1, that is, to monitor, you can see FD3 as the most initial FD1 of the hard disk backup (that is, output to monitor);
### establishes the FD4 to be used to hold the return value of LS (echo $), and you can see FD4 as your test for storing the computed "echo $". of Grass manuscript;

(ls you no 2>&1 1>&3 3>&-;echo $ >&4)
### you remember the son Shell and the pipe that you mentioned earlier. This command will first inherit FD0, FD1, FD2, FD3, FD4, which is located in front of the pipe, so the Shell's own FD1 and piping "|" are first placed before running the command. Connected.
But our condition is that stderr through the pipeline to the egrep,stdout still output to monitor.
So through the 2>&1, first to the shell of the FD1 pipe "to" FD2, so the stderr in the child shell sent to the pipeline "|" ；
Again through 1>&3, the previous "hard disk backup" to the son Shell FD1, so the FD1 in the child shell into monitor.
Again through 3>&-, will 3 close;
Then run echo $? , the output value should be sent to the pipeline, through the >&4, the output sent to the "draft paper" FD4, left to spare.

(ls you no 2>&1 1>&3 3>&-;echo $ >&4) |egrep \*)
So, stderr through the pipeline to Egrep, stdout to monitor, but, and FD4, where it sent?
$ ((ls you no 2>&1 1>&3 3>&-;echo $ >&4) |egrep \*)
The final 4>&1 is to redirect the FD4 to the FD1. But because its output is in $ (), its value is assigned to the variable SS.

The last line closes FD3, FD4.

6. High-order examples
Lightspeed Moderator Big: Shell classic problem [I/O redirection] (http://bbs.chinaunix.net/forum/viewtopic.php?t=452079&show_type=new)
[Q] for orders cmd1, CMD2, Cmd3, CMD4. How to use a one-way pipeline to complete the following functions:
1. All commands run in parallel
2. Cmd1 and cmd2 do not need stdin
3. StdOut of CMD1 and Cmd2 directed to Cmd3 's stdin
4. StdErr of Cmd1 and Cmd2 directed to CMD4 's stdin
5. Cmd3 stdout directed to file A, stderr directed to the screen
6. CMD4 stdout directed to File B, stderr directed to the screen
7. CMD1 's return code is assigned to the variable s
8. Temporary documents cannot be used

Workaround:
EXEC 3>&1; EXEC 4>&1
My_value=$ (((cmd1 1>&3; echo $ >&4) | cmd2) 3>&1 | cmd3 >a 2>&3) 2>&1 | Cmd4 & GT;B) 4>&1)
EXEC 3>&-; EXEC 4>&-

Explain:
EXEC 3>&1; EXEC 4>&1
The establishment of FD3, to restore its FD1 to cmd1 and to restore its FD2 for cmd3;
Set up FD4, save "echo $?" "Draft paper" for output values

First pair of parentheses to the first pipe: (Cmd1 1>&3 echo $ >&4) |
CMD1 itself does not have stdin, its stdout was to be sent to the first pipeline, because of the role of 1>&3, its stdout was sent to FD3, and the role of >&4 is actually the CMD1 after the operation of the return code sent to FD4. The CMD1 stderr defaults to wait for the next processing. Finally, there is no way to send anything to the pipe;

Second pair of parentheses to the second pipe: ((cmd1 1>&3 echo $ >&4) | cmd2) 3>&1|
Because the stdout of the CMD1 in the first bracket is sent to the FD3, the pipe left without any input, cmd2 thus there is no stdin. The stdout of CMD2 is the default;
The second pair of parentheses is seen as a command, and all of its stdout are sent to the second pipe "|" At the same time, due to the role of 3>&1, the original cmd1 of the stdout in FD3 and CMD2 to join together, so stdout and cmd1 cmd2 are sent to the second pipeline "|". and CMD1, cmd2 stderr still default waiting for the next step to deal with;

Third pair of parentheses to the third pipeline: (((Cmd1 1>&3 echo $ >&4) | Cmd2 3>&1 | cmd3 >a) 2>&3
CMD3 >a 2>&3:cmd3 receives the stdin from the pipe, its stdout to file A, whose stderr is sent to FD3, because FD3 inherits FD1, and in fact its stderr is sent to monitor. If there is no "2>&3", then the cmd3 stderr will interfere with CMD1 and CMD2 error output, so it is necessary;
The third bracket is fully regarded as an order, its stdout sent to the pipe "|", as 2>&1, so stderr also sent to the pipeline. However, since the CMD1, CMD2 stdout has been sent to the CMD3 processing, and cmd3 stdout output to file A,cmd3 is also sent to monitor stderr, so in fact only cmd1 and CMD2 sent to the pipeline.

CMD4 &GT;B:CMD4 receives processing stdin from the pipe, its stdout is directed to the file B,stderr the default output to monitor.

Four pairs of parentheses: (((cmd1 1>&3; echo $ >&4) | cmd2) 3>&1 | cmd3 >a 2>&3) 2>&1| cmd4 >b ) 4>&1
The FD1 and FD2 of all the commands in the four-pair brackets were processed, but there was "echo $?" >&4 "not handled. "4>&1" is the role of "will FD4 and to stdout", but because the stdout of the other orders are processed, in fact, only $? The value.
And since $ () will create a pipe, the input end is () within the command, so $? Value is assigned to the variable my_value.

The last line is to close FD3 and FD4.
Another: Restores the redirected or closed stdout:exec 1>&2, restores the redirected or closed stderr:exec 2>&1. If stdout and stderr are all closed, and do not save the original FD, you can use: Exec 1>/dev/tty recovery.

++++++++++++++++++++++++++++++++++++++++++++
I try to answer the following questions. If there is a mistake, I also ask you to correct the predecessors!

7. In an interactive (Interactive) shell, I/O redirection is performed with exec.
1. Stdin, stderr can be directed to the file? What's the result?
A, in an interactive shell, you can direct stdin to a file. Executive: EXEC 0 The result is that each row in the file will be executed automatically, and an exit command will be executed at the end, causing the exit (or retreat back to the normal shell).
As in file contents: $ More in
Date
Read LSP
Echo Hahha
echo "This is $LSP"

Execute command at prompt: $ exec 0 $ date
Tue 18:29:07 HKT 2005
$ read LSP # the "hahaha" of "echo Hahha", which is supposed to be the following, has been read into the variable LSP.
$ echo "This is $LSP"
This is echo Hahha
$ exit

b, in an interactive shell, you can direct stderr to a file. Executive: EXEC 2>err
The result is that the command prompt PS is masked and the command entered is blocked. But the result of the command execution, if it is stdout, will echo back to the screen, and if it is stderr, it will not be displayed back to the screen. The command prompt, commands, and stderr are saved to the file err. Such as:
$ exec 2>err
Err in Out # Execute LS command
Tue 18:55:58 HKT 2005 # Executes the date command, and then executes the "LS nofile", nofile This file does not exist
$ # Execute Exit command

Now let's look at the Err file:
$ More Err
[Lsp@ii lsp]$ ls
[Lsp@ii lsp]$ Date
[Lsp@ii lsp]$ ls nofile
Ls:nofile:No such file or directory
[Lsp@ii lsp]$ exit
Exit

C, in an interactive shell, you can direct stdout to a file. This is what we use so often. I will not say it. is to direct the wrong output to a file. The correct output content is not affected.

2). Stdin, can Stderr be closed? What's the result?
In an interactive shell, if stdin is turned off, such as exec 0<&-, the result is an exit (or retreat back to the normal shell).
In an interactive shell, if stderr is turned off, such as exec 2>&-, the state is directed to a file with stderr, the only difference is that it is not saved.
In an interactive shell, if you turn off Stdoutr, such as exec 1>&-, any command that has stdout or stderr content sent to monitor, such as LS or date, will have an error: "Ls:write Error:bad File descriptor ". Other such as CD, MkDir 、...... This type of command is not affected.

3. If stdin, stdout, stderr is redirected or closed, but does not save the original FD, can you revert it to the default state?
If stdin is turned off, because it causes an exit, it must not be restored.
If you redirect or close stdout and stderr one of them, you can recover because they are sent to monitor by default (but do not know if there are any other effects). Restore redirected or Closed stderr:exec 2>&1, such as Stdout:exec 1>&2 that are redirected or closed.
If stdout and stderr are all closed, and do not save the original FD, you can use: Exec 1>/dev/tty recovery.

+++++++++++++++++++
The following reference to the reply of brother r2007! Thanks to you!
+++++++++++++++++++

8, cmd >a 2>a and cmd >a 2>&1 why different?
CMD >a 2>a:stdout and stderr are sent directly to file A, a file is opened two times, resulting in stdout and stderr covering each other.
CMD >a 2>&1:stdout sent directly to file A, stderr is inherited FD1 pipeline, and then sent to file a. A file is opened only once, that is, FD1 it open.

The difference between them is:
CMD >a 2>a equivalent to the use of FD1, FD2 two competing with each other to use file a pipeline;
And the cmd >a 2>&1 only uses one pipeline FD1, but already includes stdout and stderr.
In terms of IO efficiency, the efficiency of the CMD >a 2>&1 is higher.