A collection of basic text tools for Shell programming basics

A Shell introduction: 1 description

The shell is a program written in C that is a bridge for users to use Linux. The shell is both a command language and a programming language.
A Shell is an application that provides an interface through which users access the service of the operating system kernel.
Ken Thompson's SH is the first Unix shell,windows Explorer to be a typical graphical interface Shell.
Shell Environment
Shell and Java, PHP programming, as long as there is a text editor can write code and a script to explain the execution of script interpreter can be
There are many types of shells
Bourne Shell (/ur/bin/sh or/bin/sh)
Nourne again Shell (/bin/bash)
C Shell (/USR/BIN/CSH)
K Shell (/usr/bin/ksh)
Shell for Root (/sbin/sh)
Our current focus is on bash, the Bourne again shell.

2 Introduction to Scripts 1 What is a script

Files that record commands according to certain logical relationships

2 How to create a shell script

A vinx.sh
In general, the end of the script is ". Sh" which is not a system specification, is an industry norm
B
The #!/bin/bash script header, which is the interpreter used by the script, is also the default script to run when the child shell opened, you can also use this way, you can use the relative path/usr/bin/env bash to find X program through Env and execute him, This avoids the path error of the script execution program
C
The contents of the script are the logical relationships between commands and command execution

3 How scripts are executed:

1 chmod +x then use./Script name execution
2 execution via SH script name

4 Script Writing specifications
Write in the VIM/ETC/VIMRC

Press F4 to show the following results, so you can write briefly

Two command sets 1 diff command

Function: Find out the difference of text in two files and show it
Format: diff [Options] file 1 file 2
Options:
-U: Put different text in two files together, and can output to another file
-R: Recursively compares all files in the directory (must be used when comparing directories)
Instance:

The comparison results in 3 cases.
1 a the first file needs to increase the output content to be the same as the second file
2 c modifications, which contain additions and modifications, or individually modified
3 d Delete, the first file is removed differently and the second file is the same
4 < indicates missing ASHG on right
5 >i1234 means missing i1234 on the left
Redirect to File

Resolution: @@ 代表 a certain range
"-" on behalf of the first file
"+" on behalf of the second file

2 Patch Patching

1 See if the installation
[[email protected] ~]# RPM-QA patch
Patch-2.7.1-8.el7.x86_64
2 How to use
Format:
patch [Options] Raw files < patch files
-B: Back up the original file

source files end With. orig

3 Cut Instruction

Function: Cut is the shell tool that outputs the selected portions of each line of a file to the standard output stream, and is one of the commonly used order file processing commands, which is a processing object with each behavior
Format:
Cut [Options] [file]
Main parameters:
-B: Fetch in bytes
-C: Acquired in characters
-D: Specifies the delimiter instead of the default tab
-F: Specifies the position of the item after the option is split
-N: This option is used with-B,-nb, meaning that multiple-byte segmentation is not necessary

Because Chinese characters are two bytes in a word, they cannot be taken out of one byte.
Practical application:
1 Crawl network card IP address

Description-D "" means that a space is the delimiter, and-F means that its IP address is the tenth column relative to the split position.
2 Getting the disk usage in the root directory

4 General summary of sort and uniq1

The sort command sorts the rows in the files specified by the file parameter and writes the results to standard output. If the file parameter specifies multiple files, the sort command connects the files and sorts them as a file.

2 Sort uses

-N means sort by first column

-R Flashback Arrangement

-U Remove Duplicate numbers

-O output file Redirect to own
-t Specifies the separator
-k Specifies the column to sort

3 Uniq for de-weight and combine sort

-U display non-repeating rows
-D Displays duplicate rows
-C each row displays and counts the number of repetitions

6 TEST Usage 1 General statement

Test can be used for testing expressions, including: string comparisons, arithmetic comparisons, file existence, attributes, types, and so on. For example, determine whether the file is empty, whether the file exists, whether it is a directory, whether the variable is greater than 5, whether the string equals "Longshuai", whether the string is empty, and so on. In the shell, almost all judgments are implemented using test.
Where [] is exactly equivalent to test, it's just a different notation. The double bracket [[]] is basically equivalent to [], it supports more conditional expressions, and also allows the use of logical operators "&&", "| |", "!" in double brackets. and "()", but these use single brackets can be achieved, but write more than a few single brackets. The single-bracket [] function that cannot be implemented is a regular expression match, and [[]] can be implemented. Therefore, you do not need to consider using double-brackets in cases where you can use single-square brackets.
and [] the same [-e/mnt] must have a space before and after. Otherwise it will be an error

2 conditional expressions

Whether the-e file exists (exist)
The-f file exists and is a normal file
The-D file exists and is a block device. Its access is random access
The-c file is present and is a character device, and its access is sequential access
-B file is present and directory
Whether the-C file exists and is a character device
-s file exists and is a socket
Whether the-p file exists and is a pipe file FIFO
-l file exists and is a linked file
Basic case where Yes is executed when the condition is set and no is executed when the condition is not established

3 File property check.

-R file exists and is currently user-readable
If the-W file file exists and the current user can write
The-x file file exists and the current user can execute
If the-u file is present, it is set suid
The-G file exists and is set Sgid
The-K file exists and the paste bit is set
If the-s file is present and larger than 0 bytes, that is, detects if the file is a blank file
Whether the-n file is present and whether it was modify since the last read

Comparison of 42 files;

File1-nt File2 judge whether File1 is newer than file2
File1-ot File2 judge whether File1 is older than file2
File1-ef file2 judge whether File1 and File2 are the same file, the main meaning is to determine whether the two files all point to the same inode on the same partition

5 even integers, support positive negative numbers, but do not support decimals,

Int1-eq Int2 two values equal (equal)
Int1-ne Int2 Two values are not equal
INT1-GT Int2 N1 greater than N2
Int1-lt Int2 N2 greater than N1
Int1-ge Int2 N1 is greater than or equal to N2
Int1-le Int2 N1 less than N2

6 determinant String

-Z string to determine whether the string is empty, or True if string is null
-N string to determine if the string is empty, or false if it is empty
String1=string2 String1 is the same as string2, the same returns true, "= =" and "=" are equivalent, but "=" portability is better
String1 = = string2
STR1!=str2 Str1 Not equal to STR2 is true
Str1> str2 Str1 greater than str2 is true
str1< str2 Str1 Less than str2 is true

7 Logical operators

-A or && a && b if a is executed correctly, the second command is executed
-O or | | a| | b If a does not execute correctly, the second command is executed
！ To reverse an expression

7 GREP1 General statement

grep (Global search Regular expression (RE) and print out of the line, full search of regular expressions and print out rows) is a powerful text search tool that uses regular expressions to search for text. and print out the matching lines.
The grep family of Unix includes grep, Egrep, and Fgrep. Egrep and Fgrep commands are only a small difference from grep. Egrep is the extension of grep, which supports more re metacharacters, and fgrep is fixed grep or fast grep, which regards all the letters as words, that is, the metacharacters in the regular expression represents the literal meaning back to itself, no longer special. Linux uses the GNU version of grep. It is more powerful and can use the Egrep and FGREP functions with the-G,-e,-f command line options.
The pattern is a filter that is written by regular expressions, or by characters and basic text characters
Regular expression: A pattern written by a class of special character sets, in which some characters do not represent the literal meaning of a character, but rather a function of control or a wildcard.
are divided into two categories:
1 Basic Regular expression (BRE): grep
2 Extended Regular Expression (RER): Egrep grep-e
When you talk about grep, you have to say the basic regular expression, because grep works according to the basic regular expression.
Match any single character: use double brackets to denote weak references (meaning that if there is a variable in the middle of the parentheses, the variable can be executed), a strong reference is represented by a single parenthesis, indicating that each character is matched directly.

2 Common functions

Options
-C outputs only the count of matching rows.
-I is case-insensitive (only for single-character).
-H does not display a file name when querying multiple files.
-L Only output file names that contain matching characters when querying multiple files.
-N Displays matching lines and line numbers.
-S does not display error messages that do not exist or have no matching text.
-V Displays all lines that do not contain matching text.

grep ' \<root ' file

grep ' ^root ' file

grep ' root\> ' file

grep ' root$ ' file

The difference is that $,^ end and start match are the end and beginning of the line, while \< and \> match the beginning and end of the word

3 Extended grep

* 0 or more times
? 0 or 1 times
+ one or more times
{n} appears n times
{m,n} appears m-n times
' (XY) ' {n} xy appears n times
X.*y begins with X y end

Egrep ' x|y ' file. Matches the row where x or Y is located

8 SED1 General statement

The Sed:stream Editor Text editor operates as a line when manipulating text, so it is also called a line compiler.
SED can delete (delete), alter (change), add (append), insert (insert), close, swap, or read data from other files into a text > piece, or replace (substuite) their strings, or transformations ( Tranfer) where the letters and so on. For example, the continuous blank line in a file is deleted into a line, the "local" string is replaced with "remote", "T" letter to "T", the 10th row of data and the 11th data is combined.
In sum, when the SED reads a line of data into the pattern space by the standard input, the SED executes the editing of the data in the pattern space one by one in accordance with the editing instructions of the SED script, and then sends the result in the pattern space to the standard output. Then read the next line of data. Perform the above actions until all the data is read > done.
Summary, remember:
(1) Sed always handles input in line
(2) The SED does not deal with the original file but the copy of the original file.

2 sed common usage

A\? Add one or more rows after the current line. Multiple rows, except for the last row, need "\" to continue at the end of each line

C\? Replace the text in the current line with the new text after this symbol. Multiple rows, except for the last row, need "\" to continue at the end of each line


I\? insert text before the current line. Multiple rows, except for the last row, need "\" to continue at the end of each line
? d? Delete rows

H. Copy the contents of the pattern space to the staging buffer
? H. Append the contents of the pattern space to the staging buffer
G. Copy the contents of the staging buffer to the schema space, overwriting the original content
? G. Append the contents of the staging buffer to the schema space, appended to the original content
? l? list nonprinting characters
? p? Print line


N? read the next input line and start processing it from the next command instead of the first command
Q. End or exit SED
Read the input line from the file
?! Apply commands to all rows except the selected row
Replace one string with another
G. Global substitution in a row
Write the selected line to a file
? x-Swap the contents of the staging buffer with the pattern space
Y? Replace a character with another character (you cannot use the Y command on a regular expression) \
? options? features
?-e multiple edits, which are used when multiple SED commands are applied to an input line
?-n? canceling the default output
?-F? Specify the file name of the SED script

3 Sed Advanced Usage

1 sed-e ' s/bin/aaa/g '-e ' s/bbb/ccc/g ' file name
2 sed ' s/bin/aaa/g;s/bbb/ccc/g ' file name
3 Vim Rule
s/aaa/bbb/g
S/bbb/ccc/g
Sed-f Rule File
4 sed ' s/#//g ' file replace # with space

5 sed ' s////g ' file will/replace with space
6 sed ' [email protected]/@@g ' will/replace with space

7 sed ' 2s/:/##/g ' will be the second line of all: Replace # #

8 sed ' 2,3s/:/##/g ' file to replace the second and third lines: #

9 sed '/root/,/ftp/s/:/##/g ' passwd

Ten sed ' 1s/:/##/g ' file matches change all

Each sed ' 1s/:/##/' file match changes the first occurrence

Sed-n ' $p ' file displays last line

Sed ' G ' file to increase line spacing

Sed ' $! G ' file, do not add a blank line to the last column

9 AWK1 General statement:

AWK is a programming language for scanning and processing text and data under Linux/unix, and data can come from standard input/file/pipe
The version of the awk compiler that is commonly used in Linux is Mawk,gawk, with Redhat as the representative of Gawk. The use of Ubuntu as a representative is Mawk
Gawk enables the implementation of open source code for the AWK interpreter of GNU Project
Principle:
AWK scans the file row by line, from the first line to the last line, looking for rows that match a particular pattern, and doing the appropriate work on those lines
AWK basic structure includes pattern matching and processing procedures
Pattern {Action}
Tips:
When awk reads each row of the file's contents, it compares the row to the given pattern, and if the match executes the process, otherwise does nothing to the row
If no processing script is specified, the matching rows are displayed to the standard output, and the default processing action is the print line,
If no pattern match is specified, all data is matched by default
Awk has two special patterns: Begin and end, which are placed before any data is read and executed after all data reads are complete

2 AWK Basic Syntax format

1 format
Gawk [x]-F program-file [-] File ...
Note: X represents awk support for Postix and GNU two options,-F followed by script file, file indicates the document name to be processed
2 Gwak Support Options
-F FS
--field-separator FS
Specify a delimiter with FS as the input line (the default delimiter is a space or tab)
-V Var=val
Before the process, set a variable var to Val
-F Program-file
Read the awk instruction from the script file to replace the output processing script in the command arguments
-W Compst
-W Traditional
--compat
--traditional
Running the AWK,GNU extension option in compatibility mode is ignored
-W copyleft
Output short GNU Copyright information

3 AWK Program Structure

mode {Action Instruction}
The pattern can be:
Ebgin
END
An expression
An expression, an expression
The action needs to be caused by {}.

4 Built-in variables

The following is an awk built-in variable
ARGC number of command line arguments
FileName The name of the current input document
FNR the current record number of the current input document, especially if there are multiple inputs
Document is useful
The current record number of the NR input stream if there are multiple files, sort directly up
NF number of fields currently logged record string per line
FS Field Separator
OSF output field separator, default is space
ORS output record separator, default to line break
RS input Record separator, default to line break

1 record fields: NF The default delimiter is a space, where the number of columns for each line is delimited by a space

2 record fields; NR number of lines used for statistical text files

This is the end usage

3 filename: Used to display the current file name, if not in end, how many rows, how many filenames are displayed

5 Field delimiter:

-F

Specify the separator: FS
Note: If you use FS to change delimiters, you need to define FS at begin, because you need to change the field delimiter before reading the first line
Specify multiple field separators

Before is or the relationship, all processing
Specify a separator and replace the default delimiter
-ofs and begin usage

6 Expressions and operators

Expressions consist of variables, constants, functions, regular expressions, operators, variables in awk by string variables and numeric variables, and if variables defined in awk are not initialized, the initial value is an empty string or 0
Plus +
Reduction
By
Remove
Residual%
Power Operation ^
Self-add + +
Self-reduction--
Add Value + =
Assign value after subtraction-=
Assign value after multiplying *=
Assigning values after dividing/=
Greater than >
Less than <
Greater than or equal to >=
Less than or equal to <=
equals =
Not equal to! =
&& and
|| Or

Statistical blank Lines

A collection of basic text tools for Shell programming basics