Shell knowledge point supplement (2)-redirection /;,&&, |/pipeline command/grep/sort/uniq/WC/Tee/TR/COL/join/paste/expand/split /-

1. Redirection

1. Standard input (stdin): code 0, use <or <; (rewrite or append)
2. standard output (stdout): the code is 1, use> or>;
3. Standard Error output (stderr): Code 2, Use 2> or 2>;

Eg:

Yee @ LOONG :~ $ Ifconfig> 1.txt
Yee @ LOONG :~ $ Vim 1.txt
Yee @ LOONG :~ $ LS-Al 2.txt> 1.txt
Ls: unable to access 2.txt: no file or directory
Yee @ LOONG :~ $ LS-Al 2.txt 2> 1.txt
Yee @ LOONG :~ $ Vim 1.txt
Yee @ LOONG :~ $ LS-Al 1.txt 2.txt 2> 1.txt
-RW-r -- 1 yee 922 10-18 1.txt

• 1>: outputs the correct data to the specified place.
• 2>: outputs error data to a specified location.

If I only need the correct data, what if I do not need the wrong information? Well, the/dev/null garbage bin is very important at this time! What is/dev/null? Basically, it is like a "black hole" waste bin function! When anything you enter is directed to this virtual waste bin device, "it will disappear out of thin air ~~』, This is useful! For example, in the above example, we can do this to discard the error message!
[Dmtsai @ Linux ~] $ Find/home-name testing> list_right 2>/dev/null
The error message will "disappear !』 What if I want to write all the data to the same file? At this time, special writing is required. Please note the following!
[Dmtsai @ Linux ~] $ Find/home-name testing> List 2> List <= incorrect syntax
[Dmtsai @ Linux ~] $ Find/home-name testing> List 2> & 1 <= Correct syntax
Please pay special attention to this! You must use 2> & 1 to write data to the same file at the same time.

<What is it !? Haha! In the simplest statement, it means reading data that needs to be input by the keyboard through the file. For example, we can use Cat to input some data on the keyboard and then write it into a file. For example:
[Root @ Linux ~] # Cat> catfile
Testing
Cat file test
<= Click [CTRL] + D here to end the input!
At this time, a catfile file will be generated, and the file content is just entered. Can I replace keyboard input with other files? Yes! Do this!
[Root @ Linux ~] # Cat> catfile <somefile
I can edit somefile first, and then output the data to the catfile command described above! Can this be understood? After being able to understand <, it would be a nightmare to come back again <this two consecutive symbols smaller ~ It indicates the ending input character! For example, "I want to use Cat to directly output the input message to the catfile, and when the EOF is input, the input ends.", I can do this:
[Root @ Linux ~] # Cat> catfile <EOF
> This is a test testing
> OK now stop
> EOF <= enter this item, hey! It's over now!
Have you seen it? With the <control character on the right side, we can terminate an input without entering [crtl] + D! This is very helpful for programming! Well, why do we need to use command output for redirection? This problem will surely affect your attention, if you have never written a script! Let's talk about it!
• When the screen output information is important and we need to save it;
• When the program in the background execution does not want him to interfere with the normal output results of the screen;
• The execution results of some routine system commands (such as files written in/etc/crontab;
• We already know some possible error messages when executing commands, so we want to discard them with "2>/dev/null;
• When error messages and correct messages need to be output separately.

2. Command Execution judgment basis:;, & |

Use semicolons (;) in the middle of the instruction to separate the instruction. As a result, after the instruction before the semicolon is executed, the subsequent instruction will be executed immediately. This is really convenient ~ Let's look at it from another perspective. If I want to create an archive under a directory, that is, if the directory exists, I will create this archive. If it does not exist, forget it ~ If the directory exists, you can use some bash-provided declarative functions. But here we assume that I do not know the command, but I know that I can use ls to determine whether the directory exists. That is to say, I can use ls directoryname to determine whether a file exists and use touch to create a file. These two commands are related. How can I write them? Haha! Available
& Let's do it!

[Root @ Linux ~] # Ls/tmp & Touch/tmp/testingagin
Do you remember we talked about this strange variable in the Variable Section "$? ? If there is no error message in the command execution result, $? will be returned? = 0. If there is an error, the return value will not be 0! Through this judgment, we can also use & to determine that the execution result of the current command is correct (for example, when only standard output is available), we can then execute subsequent commands, otherwise, it will be skipped! Therefore, if ls/tmp is correct, touch/tmp/testingagin will be executed! In case:
[Root @ Linux ~] # Ls/vbird & Touch/vbird/test
This is because the/vbird directory cannot exist in my system! Therefore, if you execute ls/vbird, an error will be returned, so the subsequent touch/vbird/test will naturally not act! Do you understand? From another perspective, if I want to create a file if it does not exist, I will skip this step? Very easy ~ You can do this:
[Root @ Linux ~] # Ls/tmp/vbirding | touch/tmp/vbirding

Since the command is executed one by one, if you really want to use judgment, then the order of & | cannot be wrong ~ In general, there can be a maximum of three values in the limit type, that is:
Command1 & command2 | command3
In addition, the Order will not change, because in general, command2 and command3 will place commands that can certainly be successfully executed.

Eg:

Yee @ LOONG :~ $ LS-Al 1.txt & Echo "1" | echo "2"
-RW-r -- 1 yee 93 10-18 09:13 1.txt
1
Yee @ LOONG :~ $ LS-Al 3.txt & Echo "1" | echo "2"
Ls: unable to access 3.txt: no file or directory
2
Yee @ LOONG :~ $

3. Pipeline command (PIPE)

This pipeline command'| 』It can only process the correct information sent through the previous command, that is, the standard output (stdout) information. It does not directly process stdandard errors.

Cut
Isn't cut a "cut? That's right! This command can "cut" a piece of message to him ~ The processed message is in line! Let's talk about it below:
[Root @ Linux ~] # Cut-d' delimiter '-F Fields
[Root @ Linux ~] # Cut-C character range
Parameters:
-D: followed by separator characters. Used with-F;
-F: Splits a piece of information into segments based on the-D separator, and uses-F to get the meaning of the segments;
-C: extracts a fixed character range in units of characters (characters;
Example:
Example 1: retrieve the PATH variable. I want to find the third path.
[Root @ Linux ~] # Echo $ path
/Bin:/usr/bin:/sbin:/usr/local/bin:/usr/x11r6/bin:/usr/games:
[Root @ Linux ~] # Echo $ PATH | cut-d': '-F 5
# Hey hey! In this way, the directory name/usr/local/bin will appear!
# Because we use: As the separator, the fifth is/usr/local/bin!
# What if I want to list 3rd and 5th ?, This is the case:
[Root @ Linux ~] # Echo $ PATH | cut-d': '-F 3,5
Example 2: Get all strings after 12th characters in the Output Message of the Export
[Root @ Linux ~] # Export
Declare-x histsize = "1000"
Declare-x inputrc = "/etc/inputrc"
Declare-x kdedir = "/usr"
Declare-x lang = "zh_tw.big5"
...... Other omissions ......
[Root @ Linux ~] # Export | cut-C 12-
Histsize = "1000"
Inputrc = "/etc/inputrc"
Kdedir = "/usr"
Lang = "zh_tw.big5"
...... Other omissions ......
# Do you know what's going on? -C can be used to process formatted output data!
# We can also specify a value range, for example, the characters 12th-20 are cut-C 12-20!
Example 3: Use last to show the login information for this month, leaving only the user name
[Root @ Linux ~] # Last
Vbird tty1 192.168.1.28 Mon Aug 15)
Vbird tty1 192.168.1.28 Mon Aug 15)
[Root @ Linux ~] # Last | cut-D ''-F 1
# You can use last to obtain the user information for logging on to the host in the last month,
# We can use the space character interval to retrieve the first information, that is, the user account!
# However, because vbird tty1 contains several spaces, not only one space
# Tty1 cannot be cut-d'-F, 1, 2, and so on! The output result is not what we want.
This cut is really useful! However, to be honest, there are not many opportunities to use cut unless you are often analyzing log files! Okay! The main purpose of cut is to "break down the data in the same row !』, It is most often used when analyzing some data or text data! This is because sometimes we take some characters as split parameters and then cut the data to get the data we need. I also use this function very often! Especially when analyzing log files! However, cut may be difficult to process data with multiple spaces.

Grep

The cut is to extract a certain part of the message we want, while grep is to analyze a line of information. If there is any information we need, we will take this line out ~ The simple syntax is as follows:
[Root @ Linux ~] # Grep [-acinv] 'search string 'filename'
Parameters:
-A: searches binary files for data using text files.
-C: calculates the number of times the 'search string' is found.
-I: Case sensitivity is ignored, so the case sensitivity is the same.
-N: returns the row number by the way.
-V: reverse selection, that is, the line without the 'search string' content is displayed!
Example:
Example 1: extract the line with the root in the last step;
[Root @ Linux ~] # Last | grep 'root'
Example 2: In contrast to the example, retrieve it if there is no root!
[Root @ Linux ~] # Last | grep-V 'root'
Example 3: In the last output message, if there is a root, it is taken out, and only the first column is taken.
[Root @ Linux ~] # Last | grep 'root' | cut-d'-F1
# After removing the root user, you can use the Cut Processing of the previous command to get the first column!
Grep is a great command! He supports too many syntaxes ~ In regular notation, there is a lot of data that can be processed.

Sort
Sort is an interesting command. It can help us sort data and sort data based on different data types! For example, numbers and text are sorted differently. In addition, the sorted characters are related to the encoding of the language family. Therefore, we recommend that you use lc_all = C to unify the language family and better sort the data.
[Root @ Linux ~] # Sort [-fbmnrtuk]
Parameters:
-F: Case sensitivity differences are ignored. For example, a and a are considered to be the same encoding;
-B: Ignore the leading space character;
-M: sort by month name, such as Jan and Dec;
-N: Use "numbers only" for sorting (sorted by text type by default );
-R: reverse sorting;
-U: uniq. Only one row is displayed in the same data;
-T: delimiter. The default Delimiter is the tab key;
-K: the meaning of sorting by field,
Example:
Example 1: individual accounts are recorded under/etc/passwd. Sort accounts.
[Root @ Linux ~] # Cat/etc/passwd | sort
ADM: X: 3: 4: ADM:/var/adm:/sbin/nologin
Apache: X: 48: 48: Apache:/var/www:/sbin/nologin
Bin: X: 1: 1: Bin:/bin:/sbin/nologin
Daemon: X: 2: 2: daemon:/sbin/nologin
# I omit a lot of output ~ According to the data above, sort is the default "sort by the first" data,
# And the default type is "text! So from A to the end!
Example 2: The content of/etc/passwd is separated by:. What should I do if I want to sort it in the third column?
[Root @ Linux ~] # Cat/etc/passwd | sort-t': '-K 3
Root: X: 0: 0: Root:/root:/bin/bash
Iiimd: X: 100: 101: iiimf server:/usr/lib/iiim:/sbin/nologin
Uucp: X: 10: 14: uucp:/var/spool/uucp:/sbin/nologin
Operator: X: 11: 0: Operator:/root:/sbin/nologin
Bin: X: 1: 1: Bin:/bin:/sbin/nologin
Games: X: 12: 100: games:/usr/games:/sbin/nologin
# Do you see the output part of the special font? How can they be arranged like this? Haha! That's right ~
# If you sort data in the text type, it would have been like this, and you want to sort data using numbers:
# Cat/etc/passwd | sort-t': '-K 3-N
# That's all! Use that-N to tell sort to order by number!
Example 3: Use last to retrieve and sort the output data by account only
[Root @ Linux ~] # Last | cut-D ''-F1 | sort
Sort is also a commonly used command! Because we often need to compare some information! Let's take the second example above! Today, suppose you have many accounts and you want to know the largest user ID!

• Uniq

If I have finished sorting and want to list only one duplicate information, what can I do?
[Root @ Linux ~] # Uniq [-ic]
Parameters:
-I: case-insensitive characters are ignored;
-C: Count
Example:
Example 1: Use last to list accounts. Only the account column is retrieved. After sorting, only one account is retrieved;
[Root @ Linux ~] # Last | cut-D ''-F1 | sort | uniq
Example 2: What if I want to know the total number of Logon times for each user?
[Root @ Linux ~] # Last | cut-D ''-F1 | sort | uniq-C
This command is used to delete duplicate rows and only display one. For example, you need to know who is logged on to your host this month, regardless of the number of Logon times, use the example above. (1) list all the data first; (2) Separate the names; (3) sort and (4) show only one! This command is used to reduce the number of duplicate items. Therefore, you need to combine the sorted files for processing.

• WC

If I want to know how many words are in the/etc/man. config file? How many rows? How many characters can be used? In fact, we can use the WC command to achieve this! He can help us calculate the overall data of the output message!
[Root @ Linux ~] # WC [-LWM]
Parameters:
-L: only list rows;
-W: only the number of words listed (one English word );
-M: The number of characters;
Example:
Example 1: How many related words, lines, and characters are in/etc/man. config?
[Root @ Linux ~] # Cat/etc/man. config | WC
138 709 4506
# The three output numbers are "lines, words, and characters 』
Example 2: I know that using last to output the login user, but the last two lines of last are not the account content,
How can I obtain the total number of users logging on to the system this month using a single command string?
[Root @ Linux ~] # Last | grep [A-Za-Z] | grep-V 'wtmp' | WC-l
# Since the blank line and the wtmp line are output at the bottom of the last line, I use
# When grep extracts non-blank rows and removes the wtmp row, you can understand the number of rows to be calculated.

Bidirectional redirection: Tee

[Root @ Linux ~] # Tee [-A] File
Parameters:
-A: add data to the file in the append mode!
Example:
[Root @ Linux ~] # Last | tee last. List | cut-d ""-F1
# This example allows us to save the last output to the last. list file;
[Root @ Linux ~] # Ls-L/home | tee ~ /Homefile | more
# This example stores the LS data ~ /Homefile, and the screen also has an output message!
[Root @ Linux ~] # Ls-L/| tee-~ /Homefile | more
# Note: The files after tee will be overwritten. Therefore, we need to add-
# This parameter can accumulate messages.

• Tr

Tr can be used to delete text in a message or replace text messages!
[Root @ Linux ~] # Tr [-Ds] set1...
Parameters:
-D: Delete the set1 string in the message;
-S: replace repeated characters!
Example:
Example 1: Convert all lowercase characters in the last Output Message to uppercase characters:
[Root @ Linux ~] # Last | tr' [A-Z] ''[A-Z]''
Example 2: delete the colon (:) In the/etc/passwd Output Message
[Root @ Linux ~] # Cat/etc/passwd | tr-d ':'
Example 3: delete the DOS archive's broken line character ^ m:
[Root @ Linux ~] # Cat/home/test/dostxt | tr-d' \ R'> dostxt-nom
# That/R refers to the DOS Broken Line Character

• Col

[Root @ Linux ~] # Col [-x]
Parameters:
-X: Convert the tab key to an equal space key.
Example:
[Root @ Linux ~] # Cat-A/etc/man. config <= at this time, many ^ I symbols are displayed, that is, Tab
[Root @ Linux ~] # Cat/etc/man. config | col-x | cat-A | more
# Hey hey! In this way, the [Tab] button will be replaced with the Space key, and the output will be more beautiful!

Join

Join can see the meaning of the word surface (join/join). It is processing data between two archives, and it is mainly processing the two archives, add the row with the same data together. The following simple example is used to describe:
[Root @ Linux ~] # Join [-ti12] file1 file2
Parameters:
-T: by default, join separates data with space characters and compares the data of the "first field,
If the two files are the same, the two data records are merged into one row, and the first field is placed in the first one!
-I: Case sensitivity differences are ignored;
-1: This is the number 1, which indicates the meaning of "the field used for analyzing the first file;
-2: indicates the meaning of "which field should be used to analyze the second File.
Example:
Example 1: Use the root identity to integrate/etc/passwd and/etc/shadow data into a column
[Root @ Linux ~] # Join-t': '/etc/passwd/etc/shadow
Bin: X: 1: 1: Bin:/bin:/sbin/nologin: *: 12959: 0: 99999: 7 :::
Daemon: X: 2: 2: daemon:/sbin/nologin: *: 12959: 0: 99999: 7 :::
ADM: X: 3: 4: ADM:/var/adm:/sbin/nologin: *: 12959: 0: 99999: 7 :::
# Because of the permissions of/etc/shadow, it must be root! And/etc/passwd
# Fields are separated with/etc/shadow. Therefore, you must use the-t': 'standard field to separate characters.
# In addition, because both/etc/shadow and/etc/passwd use the first field as the account name,
# You can paste the data from the same row together!
# In addition, take a closer look at the/etc/shadow content and the/etc/passwd content. You will find that,
# Both of them start with an account, and the output data above contains a special font, which indicates
# Content of the Second file. In the content section of the second archive
# The first file is the same, so of course it is omitted, so it becomes the output above.
Example 2: we know that the fourth field of/etc/passwd is GID, which is recorded in
The third field in/etc/group. How can I integrate the two files?
[Root @ Linux ~] # Join-t': '-1 4/etc/passwd-2 3/etc/group
0: Root: X: 0: Root:/root:/bin/Bash: Root: X:
1: Bin: X: 1: Bin:/bin:/sbin/nologin: Bin: X: Root, bin, daemon
2: daemon: X: 2: daemon:/sbin/nologin: daemon: X: Root, bin, daemon
4: ADM: X: 3: ADM:/var/adm:/sbin/nologin: ADM: X: Root, ADM, daemon
# This example is more obvious! The first line of the original/etc/passwd should be:
# Root: X: 0: 0: Root:/root:/bin/bash
# The content of the first line of/etc/group should be:
# Root: X: 0:
# I will remove the third column of the First archive and the third column of the second archive, and place it at the front of the output,
# Add the remaining data to him!

Paste

Paste directly "paste the two rows together and separate them with the [Tab] Key! Simple usage:
[Root @ Linux ~] # Paste [-D] file1 file2
Parameters:
-D: It can be followed by separator characters. The default Delimiter is separated by [Tab!
-: If the file part is written as-, it indicates the meaning of the data from standard input.
Example:
Example 1: paste the same line of/etc/passwd and/etc/shadow together.
[Root @ Linux ~] # Paste/etc/passwd/etc/shadow
Bin: X: 1: 1: Bin:/bin:/sbin/nologin bin: *: 12959: 0: 99999: 7 :::
Daemon: X: 2: 2: daemon:/sbin/nologin daemon: *: 12959: 0: 99999: 7 :::
ADM: X: 3: 4: ADM:/var/adm:/sbin/nologin ADM: *: 12959: 0: 99999: 7 :::
# Note! The same row is separated by a [Tab] key!
Example 2: Read/etc/group (use Cat) and paste it with the example! And only the first three rows are retrieved.
[Root @ Linux ~] # Cat/etc/group | paste/etc/passwd/etc/shadow-| head-N 3
# This example focuses on the use! That often represents stdin!

• Expand

This is to convert the [Tab] key into a space key:
[Root @ Linux ~] # Expand [-T] File
Parameters:
-T: numbers can be followed. Generally, one tab can be replaced by eight space bars.
You can also define the number of characters that a [Tab] button represents!
Example:
Example 1: extract the first manpath in/etc/man. config, and only take the first three rows;
[Root @ Linux ~] # Grep '^ manpath'/etc/man. config | head-N 3
Manpath/usr/man
Manpath/usr/share/man
Manpath/usr/local/man
# The flag at the beginning of the line is ^. We will leave it for the next section! Concept first!
Example 2: Inherit. If I want to list all the symbols? (Using CAT)
[Root @ Linux ~] # Grep '^ manpath'/etc/man. config | head-N 3 | cat-
Manpath ^ I/usr/man $
Manpath ^ I/usr/share/man $
Manpath ^ I/usr/local/man $
# Are there any differences? Yes ~ [Tab] keys can be displayed as ^ I by CAT-.
Example 3: If I set the [Tab] key to 6 characters?
[Root @ Linux ~] # Grep '^ manpath'/etc/man. config | head-N 3 | \
> Expand-T 6-| cat-
Manpath/usr/man $
Manpath/usr/share/man $
Manpath/usr/local/man $
123456123456123456 .....
# Take a closer look at the preceding Numerical Description. Because I use six characters to represent the length of a [Tab,
# Man.../usr is separated by 12 (two [tabs]) characters! If the tab is changed to 9,
# The situation is different! It is hard to understand here ~ You can set a few more numbers to view them!
Expand is also fun ~ The [Tab] is automatically converted into a space key ~ Therefore, in the preceding example, the character ^ I cannot be found when cat-A is used ~ In addition, the biggest function of [Tab] is to arrange the format neatly! After we convert it to a space key, the Space key will also increase the size according to our own definition ~ So, instead of a ^ I, it will be changed to eight blank spaces.

Split command: Split

If your archive is too large and some portable devices cannot be copied, find the split file! He can help you split a large file into small files by its size or number of lines! Fast and effective!
[Root @ Linux ~] # Split [-BL] File prefix
Parameters:
-B: the size of the file to be split. The unit can be added, for example, B, K, and M;
-L: split by the number of rows.
Example:
Example 1: My/etc/termcap has more than seven hundred kb. If you want to divide it into 300 kb files?
[Root @ Linux ~] # Cd/tmp; split-B 300 k/etc/termcap
[Root @ Linux TMP] # ls-l termcap *
-RW-r -- 1 Root 307200 August 17 00:25 termcapaa
-RW-r -- 1 Root 307200 August 17 00:25 termcapab
-RW-r -- 1 Root 184848 August 17 00:25 termcapac
# The file name can be retrieved at will! As long as we write the leading text, the archive will
# Xxxaa, xxxab, xxxac and other methods to create small files!
Example 2: how to combine the above three small files into one file named termcapback
[Root @ Linux TMP] # Cat termcap *> termcapback
# Is it easy? Just use data stream to redirect! Simple!
Example 3: Use the LS-Al/output to record every 10 rows into a file.
[Root @ Linux TMP] # ls-Al/| split-L 10-lsroot
# The focus is on that! Generally, if stdout/stdin is required, but there is no file,
# Some are just-, then that-will be treated as stdin or stdout ~

Usage of minus signs

Pipeline commands are very important in Bash's continuous processing program! In addition, Log File analysis is also a very important part, so please pay special attention to it! In addition, in pipeline commands, stdout of the previous command is often used as the stdin. Some commands need to use the file name (such as tar) for processing, this stdin and stdout can be replaced by minus signs "-". For example:
[Root @ Linux ~] # Tar-CVF-/home | tar-xvf-
The above example is: "I pack the files in/home, but the packaged data is not recorded in the file, but transmitted to stdout. After going through the pipeline, send tar-CVF-/home to tar-xvf -』. The next one is to use the stdout of the previous command. Therefore, we do not need to use file!

From laruence's private kitchen