Regular expressions, find, grep, awk, sed

1. Regular expressions
(1) Regular expressions are generally used to describe the special use of text patterns, consisting of ordinary characters (such as A-Z character) and special characters (called metacharacters, such as/, *,?, etc.).
(2) Basic meta character set and its meaning
^: matches only the beginning of the line. If ^a matches a line starting with a abc,a2e,a12,aaa,......
$: Matches only the end of the line. If ^a matches a line ending with a bca,12a,aaa,.......
*: Matches 0 or more of this single character. if (a) * match null, A,AA,AAA,....
[]: Matches only [] inside characters. Can be a single character or a sequence of characters, using "," to separate the different strings inside to match. You can also use-to represent the range of character sequences within [], such as [1-5] representation [12345]
\: Used only to mask the special meaning of a meta-character. such as \*,\ ', \ ", \|,\+,\^,\. such as
.: (dot) matches only any single character.
pattern\{n\}: Used only to match the number of occurrences of the preceding pattern. N is the number of times. such as a\{2\} matches AA.
Pattern\{n,\}: The meaning is the same as above, but the number of times is at least N. If a\{2,\} matches aa,aaa,aaaa,.....
Pattern\{n,m\}: The meaning is the same as above, but the number is between N and M. Like a\{2,4\} matches aa,aaa,aaaa of three
(3) Illustrate:
^$: Match blank line
^.$: Matches a line that contains one character
\*\.pas: Matches all characters or files ending with *.pas
[0123456789] or [0-9]: Suppose to match any number
[A-z]: any lowercase letter
[A-za-z]: Any uppercase and lowercase letter
[S,s]: Match Case S
[0-9]\{3\}\. [0-9]\{3\}\. [0-9]\{3\}\. [0-9]\{3\}: A string that matches the IP address [0-9]\{3\} three 0-9; \. : Match point (note that the dot here is a special character, so use "\" to block its meaning)
2.find Introduction (1) Find commands with certain signature files that traverse the current directory or even the entire file system to view certain files or directories, which are typically executed in the background when traversing large file systems.
(2) General form of the Find command
Find Pathname-options [-print-exec-ok]
The directory path that the-pathname:find command looks for. As with "." To represent the current directory, with/to represent the system root directory
The-print:find command outputs the matched file to the standard output
The-exec:find command executes the shell command given by the parameter to the matching file, the corresponding command form is
' Command ' {} \; (Note the spaces between {} and \)
The-ok and-exec function the same, except that the shell command given by the parameter is executed in a more secure mode, and a prompt is given before each command to let the user determine whether to execute.
Options are available in the following ways:
-name: Find files by file name
-perm: Find files according to file permissions
-user: Find files According to the owner of the file
-group: Find files according to the group to which the files belong
-mtime-n +n The file changes time to find the file,-n means that the file change time is now less than n days, +n indicates that the file change time is now N days ago. The Find command also has the-atime and-ctime options, but they are all similar to the-mtime option.
-size N[c] finds files with a file length of n blocks, with C indicating the length of the file in bytes.
-nogroup finds a file that does not have a valid owning group, that is, the group to which the file belongs does not exist in/etc/groups
-newer file1!file2 Find change time than file File1 new but older file than file File2
-depth first find the specified directory has no matching file, if no then find in the subdirectory
-type find a file of a certain type, such as
B: Block device files
D: Catalogue
E: Character device file
p; Piping file
L: Symbolic Link file
F: Normal file
(3) Example of Find command
Find-name "*.txt"-print find the txt end file and output it to the screen
Find/cmd ". Sh"-print find all SH files in/cmd directory and output
Find. -perm 755-print Find the file with permission 755 under the current directory and output
Find ' pwd '-user root-print look up the current directory under the main root of the file, and output
Find./-group sunwill-print look for files in the current directory that are owned by the main Sunwill
Find/var-mtime-5-print Find all files in the/var directory with a change time of 5 days
Find/var-mtime +5-print Find all files in the/var directory with a change time of 5 days ago
Find/var-newer "Myfile1"! -newer "Myfile2"-print find the/var directory under myfile1 New, but older than myfile2 all files.
Find/var-type D-print Find all directories under the/var directory
Find/var-type L-print Find all the symbolic link files under the/var directory.
Find. -size +1000000c-print find files larger than 1000000 bytes in the current directory
Find/-name "Con.file"-depth-print Look for "Con.file" in the root directory, if none is found in its subdirectories
Find. -type f-exec ls-l {} \; Find out if there are normal files in the current directory, and if so, execute ls-l
(4) Xargs command
When a matching file is processed using the-EXEC option of the Find command, the Find command passes all matching files to exec. Unfortunately, some systems have a limit on the length of the command that can be passed to exec, so that the find command runs for a few minutes, even if an overflow error occurs. The error message is usually "parameter column too Long" or "parameter column overflow". This is where xargs is used, especially with the Find command, where exec initiates multiple processes, and Xargs will be multiple, with only one
Find./-perm-7-print | Xargs chmod o-w Find a file with permission 7 and pass it to chmod processing
3.grep Introduction (1) The general format of grep is grep [options] basic Regular expression [file]
String parameters are best used in double quotes, one to prevent being misunderstood as a shell command, and two to find strings that can be used to look up multiple words
-C: Outputs only the count of matching rows
-I: Case insensitive (only for single characters)
-H: Do not display file names when querying multiple files
-H: Show only file names
-L: Only file names with matching characters are output when querying multiple files
-N: Show only matching rows and their line numbers
-S: does not display error messages that do not exist or have no matching text.
-V: Displays all lines that do not contain matching text.
(2) Illustrate:
grep ^[^210] myfile matches rows in myfile that start with non-2, 1, 0
grep "[5-8][6-9][0-3]" myfile matches the first bit of 5|6|7|8 in MyFile and the second bit 6|7|8| 9, the third digit is a line of three characters for 0|1|2|3
grep "4\{2,4\}" myfile matches rows with 44,444 or 4444 in myfile
grep "\?" MyFile matches a line with any character in myfile
(3) grep command class name
[[: Upper:]] means [a-z]
[[: Alnum:]] means [0-9a-za-z]
[[: Lower:]] means [a-z]
[[: Space:]] Indicates a space or TAB key
[[:d Igit:]] means [0-9]
[[: Alpha:]] means [a-za-z]
such as: grep "5[[:d igit:]][[:d igit:]]" myfile match myfile contains 5 lines that start with two digits.
4.awk Introduction
You can browse and extract information from a file or string based on a specified rule, which is a self-explanatory language.
(1) awk command-line mode awk [-F filed-spearator] ' command ' input-files
awk script: All awk commands insert a file and make the awk program executable, and then use the awk command interpreter as the first line of the script to invoke it by typing the script name. awk scripts are made up of various operations and patterns.
The mode section determines when an action statement triggers and triggers an event. (Begin,end)
Action to process the data and place it in {} to indicate (print)
(2) Separators, fields, and records
When Awk executes, its browse domain is marked as $1,$2,... $n. This method becomes the domain identifier. $ full for all domains.
(3) Illustrate:
awk ' {print $} ' test.txt |tee test.out all rows in the output test.txt represent all domains
Awk-f: ' {print $} test.txt |tee Test.out ' Ibid. Just the delimiter for ":"
awk ' BEGIN {print ' ipdate\n "}{print" \ T "$4} end{print" End-of-report "} ' test.txt
Print "End-of-report" Intermediate print body information at the end of printing "ipdate" at the beginning, such as a total of three messages, then the output is as follows:
Ipdate
1 First
2 Second
3 Third
End-of-report
(4) Match operator ~ match,!~ mismatch
Cat test.txt |awk ' $0~/210.34.0.13/' matches rows 210.34.0.13 in Test.txt
awk ' $0!~/210.34.0.13 ' test.txt matches rows that are not 210.34.0.13 in Test.txt
awk ' {if ($1== "210.34.0.13") print $} ' Test.txt matches the row of the first field in Test.txt in 210.34.0.13.
5.sed introduction SED does not deal with initialization files, it operates just a copy, and then all changes are output to the screen if not redirected to a file.
Sed is a very important text filtering tool, using a single line of commands or using a pipeline with grep and awk together. is a non-interactive text stream editor.
(1) Three ways to call SED
Use the SED command line format as: sed [options] sed command input file
Use the SED script file format: sed[options]-F SED script file input file
sed script file [options] input file
-whether using a shell command line or a script file, sed accepts input from standard input if no input file is specified, typically a keyboard or redirection result.
(2) The options of the SED command are as follows
-N: Do not print
-C: The next command is to edit the command
-F: If you are calling the SED script file
(3) How SED queries text in a file
--Use line number, can be a simple number, or a range of line numbers
--Using regular expressions
(4) How to read the text
x x is a line number
X, y indicates line numbers range from X. to Y
/pattern/query for rows that contain patterns
/pattern/pattern/querying rows that contain two patterns
Pattern/,x query for rows containing patterns on a given line number
x,/pattern/matching rows by line number and pattern query
x,y! Query does not contain rows with the specified line number x and Y
(5) Basic sed edit command
P Print matching lines
D Delete a matching row
= Show file line number
A\ Append new text information after locating line numbers
I\ inserting new text information after locating line numbers
C\ to replace positioned text with new text
s replaces the pattern with the replacement mode
R read files from another file
W write text to a file
Q When the first pattern match is complete, eject or exit immediately
L Display control characters equivalent to eight forbidden ASCII codes
{} The command group executed on the anchor row
N reads the next line of text from another file and attaches it to the next line
G Paste the pattern 2 into the/pattern n/
Y transfer character
(6) Illustrate:
Sed-n ' 2p ' test.txt prints the second line of information (note:-N is not printing mismatched information, if not added-N, all information of the file is printed instead of matching information)
Sed-n ' 1,4p ' test.txt print the first line to line fourth information
Sed-n '/los/p ' test.txt mode matches los and prints out
Sed-n ' 2,/los/p ' Test.txt starts from the second line. Know match first Los
Sed-n '/^$/p ' test.txt match blank line
Sed-n-E '/^$/p '-e '/^$/= ' test.txt print blank lines and line numbers
Sed-n '/good/a\morning ' test.txt append morning to the matched good
Sed-n '/good/i\morning ' test.txt inserted in front of the matched good morning
Sed-n '/good/c\morning ' Test.txt replaces the matched good to morning
Sed ' 1,2d ' test.txt delete lines 1th and 2
Sed ' s/good/good morning/g ' test.txt match good and replace with goodmorning
Send ' s/good/& hello/p ' test.txt match to good just add hello after it
Send ' S/good/hello &/p ' test.txt match to good just precede it with Hello
6. Merging and splitting (Sort,uniq,join,cut,paste,split) (1) Sot command
Sort [options] files many different fields are sorted in different column order
-C test file is sorted
-m merge two sorted files
-U Delete all the same rows
-o stores the output file name of the sort result
-T domain delimiter, starting with a non-whitespace or tab order
+N:N is a column number, use this column number to start sorting
-n Specifies that the sort is a numeric classification item on a field
-R comparison Inversion
Sort-c test.txt test files are categorized
Sort-u test.txt Sort and merge the same rows
Sort-r Test.txt in reverse order
Sort-t "/" +2 test.txt separated by "/", the second field starts to classify
(2) Uniq command
uniq [Options] files Remove or disallow duplicate rows from a text file
-U displays only non-repeating rows
-D displays only duplicate rows, with each repeating row showing only one row
-C Print the number of occurrences of each repeating row
-f:n is a number, the first n fields are ignored
Uniq-f 2 Test.txt ignores the first 2 domains
(3) Join command
Join [options] file1 file2 used to connect lines from two categorical text files
-an,n is a number that displays unmatched rows from file n when connected
-onm, connection domain, n is file number, M is domain number
-jnm,n is the file number, M is the domain number, and the other domain is used as the connection domain
-T, the domain delimiter. The field delimiter used to set a non-space or TAB key.
(4) Split command
Split-output_file_size Intput_filename Output_filename
Used to split large files into small files.
-B N, the size of each split file n
-c N, up to n bytes per split file line
-l N, number of rows per split file
-N, same-l n
Split-10 Test.txt splits test.txt into 10 rows of small files
(5) Cut command
Cut-c n1-n2 filename Displays the text from the beginning N1 to N2 for each line.

Cut-c 3-5 Test.txt shows the 3rd to 5th characters of each line in a test.txt

Regular expressions, find, grep, awk, sed