Common techniques for Linux text processing

1. Find File Lookup

Find txt and PDF files:

Find. (-name "*.txt"-o-name "*.pdf")-print

Regular way to find. txt and PDF:

Find. -regex ". * (. txt|. PDF) $ "

-iregex: Ignoring case-sensitive regular

Negative arguments to find all non-txt text:

Find. ! -name "*.txt"-print

Specify the search depth to print out the current directory file (depth is 1):

Find. -maxdepth 1-type F

Custom Search

• Search by Type

Find. -type d-print//List all directories only

-type f File/L symbolic link/d catalogue

Find supports file retrieval types that can differentiate between common files and symbolic links, directories, and so on, but binary and text files cannot be distinguished directly by the type of find;

The file command can check the specific type of files (binary or text):

$file redis-cli # binary file redis-cli:elf 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared Lib s), for Gnu/linux 2.6.9, not stripped$file redis.pid # text file, Redis.pid:ASCII text

Therefore, you can use the following command combination to find all binaries in the local directory:

LS-LRT | awk ' {print $9} ' |xargs file|grep elf| awk ' {print '} ' |tr-d ': '

• 
  

• -atime access Time (in days, minutes units is-amin, similar to the following)

• -mtime modification Time (content modified)

• -ctime Change Time (metadata or permission changes)

• Search by Time

• 
  

All files accessed on the 7th day of the latest:

Find. -atime 7-type F-print

All files that have been visited in the last 7 days:

Find. -atime-7-type F-print

Search all files that were visited 7 days ago:

Find. -atime +7 Type F-print

• Search by Size:

W-word k M G looking for files larger than 2k:

Find. -type f-size +2k

Search by permissions:

Find. -type f-perm 644-print//Find all files with executable permissions

Search by User:

Find. -type f-user weber-print//Find the files owned by the user Weber

Follow-up action found after

• Delete

Delete all SWP files in the current directory:

Find. -type f-name "*.SWP"-delete

Another syntax:

Find. Type F-name "*.SWP" | Xargs RM

• Perform actions (powerful exec)

Change ownership in the current directory to Weber:

Find. -type f-user root-exec chown Weber {};

Note: {} is a special string, for each matching file, {} will be replaced with the corresponding file name;

Copy all the found files to another directory:

Find. -type f-mtime +10-name "*.txt"-exec cp {} old;

• Combine multiple commands

If you need to execute multiple commands later, you can write multiple commands into a single script. The script can then be executed when the-exec is invoked:

-exec./commands.sh {};

Delimiter of the-print

Default use of ' ' as the delimiter for the file;

-print0 uses ' ' as a delimiter for the file, so you can search for files that contain spaces;

2. grep Text Search

grep match_patten File//default access matching line

Common parameters

• -O only outputs matching lines of text VS -v outputs no matching lines of text

• 
  

• The number of times the text is contained in the-C statistic file

• Grep-c "text" filename

• -N Prints matching line numbers

• -I ignore case when searching

• -L print File name only

Recursive search of text in a multilevel directory (the programmer searches for code favorites):

grep "Class". -r-n

Match multiple modes:

Grep-e "Class"-E "vitural" file

grep outputs a file name (-Z) with 0 as the trailing character:

grep "Test" file*-lz| xargs-0 RM

Integrated application: Finds all SQL lookups with a where condition in the log:

Cat log.* | TR A-Z | grep "from" | grep "WHERE" > B

Find Chinese Example: Project catalog in utf-8 format and gb2312 format two kinds of files, to find the word is Chinese;

1. Find its UTF-8 encoding and gb2312 encoding are e4b8ade69687 and D6D0CEC4 respectively.

2. Inquire:

   GREP:GREP-RNP "\xe4\xb8\xad\xe6\x96\x87|\xd6\xd0\xce\xc4" * can

Chinese character coding query: http://bm.kdd.cc/

3. Xargs command line parameter conversion

Xargs can convert input data into command-line arguments for a particular command, so that it can be combined with a number of commands. Like grep, for example, find;-Convert multi-line output to single-line output

Cat file.txt| Xargs

n is the delimiter between multiple lines of text

• Convert a single line to multiple lines of output

Cat Single.txt | Xargs-n 3

-N: Specify the number of fields to display per row

Xargs parameter Description

• -D defines delimiters (the delimiter is n for multiple lines by default)

• -n Specifies that the output is multiple lines

• -I {} Specifies the replacement string, which is replaced when the xargs extension is used, when multiple arguments are required for the command to be executed

• -0: Specify 0 for Input delimiter

Example:

Cat File.txt | Xargs-i {}./command.sh-p {} -1# number of counting lines find source_dir/-type f-name "*.cpp"-print0 |xargs-0 Wc-l#redis store data via string, pass Over set storage index, all values need to be queried by index:./REDIS-CLI Smembers $ | awk ' {print '} ' |xargs-i {}./redis-cli get {}

4. Sort sorting

Field description

• -N Sort by number vs-d in dictionary order

• -R Reverse Order

• -k n Specifies sorting by nth column

Example:

SORT-NRK 1 DATA.TXTSORT-BD Data//ignore leading whitespace characters such as spaces

5. Uniq Eliminate Duplicate rows

• Eliminate duplicate rows

Sort Unsort.txt | Uniq

• Count the number of times each line appears in a file

Sort Unsort.txt | Uniq-c

• Find duplicate rows

Sort Unsort.txt | Uniq-d

You can specify the duplicates that need to be compared in each row:-S start position-W comparison character number

6. Convert with TR

• General usage

echo 12345 | Tr ' 0-9 ' 9876543210 '//Add decryption conversion, replace the corresponding character cat text| TR ' '//tab to space

• TR Delete character

Cat File | Tr-d ' 0-9 '//Delete all numbers

-C Seeking complement set

Cat File | Tr-c ' 0-9 '//Get all the numbers in the file cat file | Tr-d-C ' 0-9 '//Delete non-digital data

• TR compression characters

Tr-s the repeating characters that appear in compressed text; most commonly used to compress extra spaces:

Cat File | Tr-s "

• Character class

• Various character classes are available in TR:

• 
  

• Alnum: Letters and Numbers

• Alpha: Letters

• Digit: Digital

• Space: white space characters

• Lower: lowercase

• Upper: Uppercase

• Cntrl: Controlling (non-printable) characters

• Print: Printable characters

How to use: TR [: Class:] [: Class:]

TR ' [: Lower:] ' [: Upper:] '

7. Cut split text by column

• Intercept the 2nd and 4th columns of a file

cut-f2,4 filename

• Go to all columns except column 3rd of the file

CUT-F3--complement filename

• -D Specify delimiter

Cat-f2-d ";" FileName

• 
  

• N-nth field to end

• -M 1th Field M

• N-m N to M Fields

• Range of cut and take

• 
  

• 
  

• -B in bytes

• -C in Characters

• -F in fields (using delimiters)

• Cut-to-take units

• 
  

Example:

Cut-c1-5 File//print first to 5th character cut-c-2 file//print first 2 characters

5th to 7th columns of truncated text

$echo String | Cut-c5-7

8. Paste concatenation of text by column

Stitch two text together by column;

Cat File112cat file2colinbookpaste file1 file21 colin2 Book

The default delimiter is a tab character, which can be specified with the-D delimiter:

Paste File1 file2-d "," 1,colin2,book

9. Tools for WC statistics lines and characters

$WC-L File//statistic number of rows $wc-w file//Statistics number of words $wc-c file//statistic characters

Sed text replacement weapon

• First place replacement

Sed ' s/text/replace_text/' file//replace the first matching text of each line

• Global substitution

Sed ' s/text/replace_text/g ' file

After the default substitution, output the replaced content, if you need to replace the original file directly, use-I:

Sed-i ' s/text/repalce_text/g ' file

• To remove a blank line

Sed '/^$/d ' file

• Variable conversions

The matched string is referenced by the tag &.

echo this is en example | Sed ' s/w+/[&]/g ' $>[this] [is] [en] [Example]

• SUBSTRING matching tag

The first matching parenthesis content is referenced using the tag one

Sed ' S/hello ([0-9])//'

• Double quotation mark Evaluation

Sed is usually quoted as a single quotation mark, or double quotation marks are used, and double quotation marks are used to evaluate an expression:

Sed ' s/$var/hlloe/'

When using double quotes, we can specify variables in the SED style and in the replacement string;

Eg:p=pattenr=replacedecho "line con a patten" | Sed "s/$p/$r/g" $>line con a replaced

• Other examples

String insertion character: Converts each line of content in the text (ABCDEF) to Abc/def:

Sed ' s/^. {3}/&//g ' file

One. awk Data Flow processing tool

• AWK script Structure

awk ' begin{statements} statements2 end{statements} '

• Working style

1. Execute the statement block in begin;

2. Read a line from the file or stdin, and then execute the STATEMENTS2, repeating the process until the file is fully read;

3. Execute the end statement block;

Print printing when moving forward

• When you use print without parameters, the forward

Echo-e "Line1line2" | awk ' Begin{print ' "start"} {print} end{print "END"} '

• When print is separated by commas, the parameters are bounded by spaces;

echo | awk ' {var1 = ' v1 '; var2 = "V2"; var3= "V3"; print var1, var2, var3;} ' $>V1 V2 v3

• Use the-stitching method ("" as the stitching character);

echo | awk ' {var1 = ' v1 '; var2 = "V2"; var3= "V3"; print var1 "-" var2 "-" VAR3;} ' $>v1-v2-v3

Special variable: NR NF $ $ $

NR: Indicates the number of records, in the course of the implementation of the forward number;

NF: Indicates the number of fields, the total number of fields that should go forward during the execution;

$: This variable contains the text content of the current line during execution;

$: The text content of the first field;

$: The text content of the second field;

Echo-e "Line1 F2 F3 line2 line 3" | awk ' {print NR ': ' $ '-' $ '-' $ '

• Print the second and third fields of each row

awk ' {print $, $ $} ' file

• Number of rows in the statistics file

awk ' END {print NR} ' file

• Accumulate the first field of each row

Echo-e "1 2 3 4" | awk ' begin{num = 0;p rint "BEGIN";} {sum + = $;} END {print "= ="; Print sum} '

Passing external variables

Var=1000echo | awk ' {print Vara} ' vara= $var # input from Stdinawk ' {print Vara} ' vara= $var file # input from files

To filter the rows that awk handles with a style

awk ' NR < 5 ' #行号小于5awk ' nr==1,nr==4 {print} ' file #行号等于1和4的打印出来awk '/linux/' #包含linux文本的行 (can be specified with regular expressions, super powerful) awk '!/linu x/' #不包含linux文本的行

Set delimiter

Use-F to set the delimiter (the default is a space):

Awk-f: ' {print $NF} '/etc/passwd

Read command output

Using Getline, the output of the external shell command is read into the variable cmdout:

echo | awk ' {"grep root/etc/passwd" | getline cmdout; print Cmdout} '

Using loops in awk

for (i=0;i<10;i++) {print $i;} For (i in array) {print array[i];}

Eg: The following string, which prints out the time string:

2015_04_02 20:20:08:mysqli Connect failed, please check connect info$echo ' 2015_04_02 20:20:08:mysqli connect failed, pl Ease Check Connect info ' |awk-f ': ' {for (i=1;i<=;i++) printf ("%s:", $i)} ' >2015_04_02 20:20:08: # This way the last colon is printed out $ Echo ' 2015_04_02 20:20:08:mysqli connect failed, please check connect info ' |awk-f ': ' {print $ ': ' $ $ ': ' $ $;} ' >2015_04_02 20:20:08 # This way to meet the demand

And if you need to print out the rest of the section (the time section and the following text are printed separately):

$echo ' 2015_04_02 20:20:08:mysqli connect failed, please check connect info ' |awk-f ': ' {print $ ': ' $ $ ': $ $; Print $4;} ' >2015_04_02 20:20:08>mysqli Connect failed, please check connect info

Print lines in reverse order: (Implementation of the TAC command):

Seq 9| awk ' {LIFO[NR] = $ LNO=NR} end{for (; lno>-1;lno--) {print Lifo[lno];}} '

Awk combines grep to find the specified service and then kill it

ps-fe| grep MSV8 | Grep-v Mforward | awk ' {print $} ' | Xargs kill-9;

AWK implements head, tail commands

• Head

awk ' Nr<=10{print} ' filename

• Tail

awk ' {buffer[nr%10] = $;} End{for (i=0;i<11;i++) {print buffer[i%10]}} ' filename

Print the specified column

• Awk method implementation

LS-LRT | awk ' {print $6} '

• Cut Mode implementation

LS-LRT | Cut-f6

Print the specified text area

• Determine line number

Seq 100| awk ' Nr==4,nr==6{print} '

• Determine text

Print text between Start_pattern and End_pattern:

awk '/start_pattern/,/end_pattern/' filename

Example:

SEQ 100 | awk '/13/,/15/' cat/etc/passwd| awk '/mai.*mail/,/news.*news/'

awk common built-in functions

Index (string,search_string): Returns the position search_string appears in the string

Sub (regex,replacement_str,string): Replace the first content of the regular match with the REPLACEMENT_STR;

Match (regex,string): Checks if the regular expression matches the string;

Length (String): Returns the string length

echo | awk ' {"grep root/etc/passwd" | getline cmdout; print length (cmdout)} '

printf, similar to the C language, formats the output:

Seq 10 | awk ' {printf '->%4s ', ' $ '

12. Iterate over the lines, words, and characters in the file

1. Iterate through each line in the file

• While Loop method

While read Line;doecho $line;d One < file.txt to sub shell:cat file.txt | (While read line;do echo $line;d one)

• Awk method

Cat file.txt| awk ' {print} '

2. Iterate through each word in a row

For word in $line;d oecho $word;d One

3. Iterate through each of the characters

${string:start_pos:num_of_chars}: Extracts a character from a string; (bash text slices)

${#word}: Returns the length of a variable word

For ((i=0;i<${#word};i++)) Doecho ${word:i:1);d One

Display files in ASCII characters:

$od-C filename

Common techniques for Linux text processing