Shell sed filter details

Shell sed filter explanation 1. Sed introduction sed is an online editor that processes a line of content at a time. During processing, the currently processed rows are stored in the temporary buffer, called the pattern space. Then, the sed command is used to process the content in the buffer, send the buffer content to the screen. Next, process the next row, and repeat until the end of the file. The file content is not changed unless you use the redirection storage output. Sed is mainly used to automatically edit one or more files, simplify repeated operations on files, and write conversion programs. The following describes the Gnu Sed 3.02 version. 2. the address can be used to locate the row you want to edit. The address is composed of numbers, the two rows separated by commas indicate the range of rows starting and ending with the two rows (including the rows indicated by the number of rows ). For example, 1 and 3 indicate 1, 2, 3 rows, and the dollar sign ($) indicates the last row. The range can be determined by data, regular expressions, or a combination of the two. 3. the Sed command calls the sed command in two forms: * sed [options] 'command' file (s) * sed [options]-f scriptfile file (s) a \ adds a line of text behind the current line. B lable branch to the marked place in the script. If the branch does not exist, it is routed to the end of the script. C \ use new text to change the text of this line. D. Delete the row from the position of the template block (Pattern space. D. Delete the first line of the template block. I \ insert text on the current row. H. Copy the content of the template block to the buffer in the memory. H append the content of the template block to the buffer g in the memory to obtain the content of the memory buffer and replace the text in the current template block. G to obtain the content of the memory buffer and append it to the end of the block Text of the current template. L The list cannot print the character list. N. Read the next input line and use the next command to process the new line instead of the first command. N append the next input row to the template block and embed a new line between them to change the number of the current row. P prints the row of the template block. P (uppercase) prints the first line of the template block. Q: Exit Sed. R file reads rows from the file. The t labelif branch starts from the last line. Once the condition is met or the T and t commands are executed, the Branch is routed to the command with a label or to the end of the script. T label indicates the branch with an error. Starting from the last line, once an error occurs or the T and t commands are executed, the Branch is routed to the command with a label or to the end of the script. W file write and append the template block to the end of the file. W file write and append the first line of the template block to the end of the file .! Indicates that the subsequent commands will work on all unselected lines. S/re/string Replace the regular expression re with string. = Print the current row number. # Extend the annotation to the next line break. The replacement mark * g indicates full replacement in the row. * P indicates printing rows. * W indicates writing rows into a file. * X indicates the text in the SWAp template block and the text in the buffer. * Y indicates to translate a character into another character (but not used in regular expressions) (uppercase/lowercase letters can be converted, but tr is more convenient). 4. option-e command, -- expression = command allows multiple edits. -H, -- help: print the help and display the address of the bug list. -N, -- quiet, -- silent cancel the default output. -F, -- filer = script-file guide sed script file name. -V, -- version: print the version and copyright information. 5. The starting point of the metadatabase Character Set ^ anchor is as follows:/^ sed/matches all rows starting with sed. $ The End Of The Anchor row is as follows:/sed $/matches all rows ending with sed .. Match a non-linefeed character, for example,/s. d/match s, followed by any character, and then d. * Match zero or multiple characters, such as:, love, which is ** love **. \ <Specifies the start of a word, for example, "/\ <love. \> Anchor specifies the end of a word, such as/love \>/matches a row containing a word ending with love. X \ {m \} repeated characters x, m times, such as:/0 \ {5 \}/matched rows containing 5 o. The characters x \ {m, \} are repeated at least m times, for example,/o \ {5, \}/matches rows with at least 5 o. The characters x \ {m, n \} must be repeated at least m times, not more than n times, such as:/o \ {5, 10 \}/matching rows of 5--10 o. 6. delete an instance: d command * $ sed '2d 'example ----- Delete the second line of the example file. * $ Sed '2, $ d' example ----- delete all rows from the second row to the end of the example file. * $ Sed '$ d' example ----- Delete the last row of the example file. * $ Sed '/test/'d example ----- delete all rows containing test in the example file. Replace: s command * $ sed's/test/mytest/G' example ----- replace test with mytest in the entire line. If no g tag exists, only the first matched test in each row is replaced with mytest. * $ Sed-n's/^ test/mytest/P' example ----- (-n) option and the p Flag are used together to print only the replaced rows. That is to say, if the test at the beginning of a row is replaced with mytest, print it. * $ Sed's/^ 192.168.0.1/& localhost/'example ----- & symbol represents the part found in the replacement string. All rows starting with 192.168.0.1 are replaced with their own localhost and changed to 192.168.0.1localhost. * $ Sed-n's/\ (love \) able/\ 1rs/P' example ----- love is marked as 1, and all loveable will be replaced with lovers, the replaced line is printed out. * $ Sed's #10 #100 # g'example ----- whatever the character, followed by the s command is considered as a new separator. Therefore, "#" is a separator here, replaces the default "/" separator. Replace all 10 values with 100 values. The range of selected rows: comma * $ sed-n'/test/,/check/P' example ----- all rows within the range specified by the template test and check are printed. * $ Sed-n'5,/^ test/P' example ----- print all rows starting from the fifth line to the first line containing the start of test. * $ Sed '/test/,/check/s/$/sed test/'example ----- for the rows between the template test and west, the end of each row is replaced by the string sed test. Multi-Point Editing: e command * $ sed-e '1, 5d '-e's/test/check/'example ----- (-e) option allows multiple commands to be executed in the same line. As shown in the example, the First Command deletes lines 1 to 5, and the second command replaces test with check. The command execution order has an impact on the result. If both commands are replacement commands, the first replacement command will affect the result of the second replacement command. * $ Sed -- expression ='s/test/check/'-- expression ='/love/d' example ----- a better command than-e is -- expression. It can assign values to sed expressions. Read from a file: the content in the r command * $ sed '/test/r file 'example ----- file is read and displayed after the line Matching test. If multiple lines match, the file content is displayed under all matched rows. Write file: w command * $ sed-n'/test/w file' example ----- in example, all rows containing test are written into file. APPEND command: a command * $ sed '/^ test/a \ ---> this is a example 'example <----- 'this is a example' is appended to the end of the row starting with test, sed requires that command a be followed by a backslash. Insert: I command $ sed '/test/I \ new line ------------------------- 'example if test is matched, insert the text following the backslash to the front of the matching line. Next: n command * $ sed '/test/{n; s/aa/bb/;}' example ----- if test is matched, move it to the next line of the matching line, replace the aa of this row with bb, print the row, and continue. Deformation: The y command * $ sed '1, 10y/abcde/ABCDE/'example ----- converts all abcde in line 1-10 to uppercase. Note that this command cannot be used for regular expression metacharacters. Exit: q command * $ sed '10q' example ----- exit sed after printing the 10th rows. Keep and get: h command and G command * $ sed-e '/test/H'-e' $ G example ----- when sed processes files, each row is saved in a temporary buffer called the mode space. Unless the row is deleted or the output is canceled, all processed rows are printed on the screen. The mode space is cleared, and a new row is saved for processing. In this example, the row Matching test is found and saved to the mode space. The h Command copies the row and saves it to a special buffer zone called the guaranteed cache. The second statement means that when the last line is reached, the G command extracts the row that maintains the buffer and places it back in the mode space, and append it to the end of the row that already exists in the mode space. In this example, It is appended to the last row. Simply put, any row containing test is copied and appended to the end of the file. Keep and swap: h command and x command * $ sed-e '/test/H'-e'/check/X' example ----- swap mode space and keep the content of the buffer. That is, to swap the rows containing test and check. 7. The Sed script is a sed command list. When Sed is started, the file name of the script is guided by the-f option. Sed is very picky about the commands entered in the script. There cannot be any blank or text at the end of the command. If there are multiple commands in one line, separate them with semicolons. Comments rows starting with # And cannot span rows. =, You can easily implement many basic sed functions. You can use tr as the (extremely) Simplified variant of sed: it can replace another character with one character, or it can completely remove some characters. You can also use it to remove duplicate characters. This is what all tr can do. Tr is used to convert characters from standard input through replacement or deletion. Tr is mainly used to delete control characters in a file or convert characters. When using tr, You need to convert two strings: String 1 for query and string 2 for various conversions. When tr is just executed, the characters in string 1 are mapped to the characters in string 2, and the conversion operation starts. The tr command with the most common options is in the format of tr-c-d-s ["string1_to_translate_from"] ["string2_to_translate_to"] <input-file here: -c: replace this character set with the supplement set of Character Set 1. the character set must be ASCII. -D: delete all input characters in string 1. -S deletes all recurring character sequences and retains only the first string. The duplicate strings are compressed into one string. Input-file is the conversion file name. Although other formats can be used for input, this format is the most commonly used. 2. When the character range specifies the content of string 1 or string 2, only single character or string range or list can be used. A string consisting of characters in a-z. [A-Z] A string consisting of characters within the A-Z. [0-9] numeric string. \ Octal is a three-digit octal number, which corresponds to valid ASCII characters. [O * n] indicates the repeated occurrence of the character O for a specified number of times. Therefore, [O * 2] matches the OO string. Different expressions of specific control characters in tr stenographer meaning octal character \ a Ctrl-G ringtone \ 007 \ B Ctrl-H Escape Character \ 010 \ f Ctrl-L go to line feed \ 014 \ n Ctrl-J newline \ 012 \ r Ctrl-M press enter \ 015 \ t Ctrl-I tab key \ 011 \ v Ctrl-X \ 0303133 application example (1 remove repeated lowercase characters tr in oops.txt -s "[a-z]" <oops.txt> result.txt (2) delete the empty line tr-s "[\ 012]" <plan.txt or tr-s ["\ n"] <plan.txt (3) sometimes you need to delete ^ M in the file, newline tr-s "[\ 015]" "[\ n]" <file or tr-s "[\ r]" "[\ n]" <file (4) uppercase to lowercase cat a.txt | tr "[a-z]" "[A-Z]"> B .txt (5) delete day of a specified character for a week Process Table. The task is to delete all numbers from it and only retain the date. The date can be in uppercase or lowercase format. Therefore, you must specify two character ranges: [a-z] and [A-Z]. command tr-cs "[a-z] [A-Z]" "[\ 012 *]" to include all lines of the file not in [a-z] or [A-Z] (all Greece letter) put the string in string 1 and convert it to a new line. -S indicates that all new lines are compressed, and-c indicates that all letters are retained. The original file is as follows, followed by the tr command: tr-cs "[a-z] [A-Z]" "[\ 012 *]" <diary.txt (6) the first function of conversion control character tr is to convert control characters, especially when downloading files from dos to UNIX, especially when you forget to set the ftp option for line feed conversion by carriage return. Cat-v filename displays control characters. Cat-v stat.txt box aa ^ 12 ^ M apple bbas ^ 23 ^ M ^ Z conjecture '^' is the tab key. Each line ends with Ctrl-M and the end of the file is Ctrl-Z. The following is the modification method. Use the-s option to view the ASCII table. ^ The octal code of ^ is 136, ^ M is 015, the tab key is 011, and ^ Z is 032. The final function will be completed as follows. Replace ^ With the tab key. The command is "\ 136" "[\ 011 *]". Redirects the result to the temporary working file stat. tmptr-s "[\ 136]" [\ 011 *] "<stat.txt> stat. tmp replaces ^ M at the end of each row with a new line, and removes ^ Z with \ n. The input must come from the temporary working file stat. tmp. Tr-s "[\ 015] [\ 032]" "\ n" <stat. tmp: delete all the tab keys with spaces. Run the tr-s Command [\ 011] "[\ 040 *]" <input. file (7) replace all the colons in the passwd file with the tab key to increase readability of tr-s "[:] "" [\ 011] "</etc/passwd or tr-s" [:] "" [\ t] "</etc/passwd (8) make the PATH readable. If echo $ PATH or echo $ LD_LIBRARY_PATH and other similar commands are used to display the PATH information, we will see a lot of paths connected with colons, the tr command can convert these colons into carriage return, so that these paths are quite readable echo $ PATH | tr ":" "\ n" (9) all these commands can be used in vi! Remember to add the line range and exclamation point (!) You want to process before the tr command (!), For example, 1, $! Tr-d '\ t' (dollar sign indicates the last line ). (10) In addition, when someone sends you a text file created on Mac OS or DOS/Windows, you will find tr very useful. If you do not save the file as a UNIX line break to indicate the end of the line format, you need to convert the file to the UNIX format. Otherwise, some command utilities will not process the files correctly. Mac OS ends with a carriage return character (\ r) at the end of the line. Many text processing tools process such files as one line. To solve this problem, use the following tips: Mac-> UNIX: tr "\ r" "\ n" <macfile> unixfileUNIX-> Mac: tr "\ n" "\ r" <unixfile> macfileMicrosoft DOS/Windows conventions, each line of text ends with a carriage return character (\ r) followed by a line break (\ n. To solve this problem, run the following command: DOS-> UNIX: tr-d "\ r" <dosfile> unixfileUNIX-> DOS: in this case, awk is required, because tr cannot insert two characters to replace one character. The awk command to be used is awk '{print $0 "\ r"}' <unixfile> dosfile Note: Both can be done with sed.