Linux--shell Script Programming 2

Debugging Methods for Shell scripts

• -N: Read the command in the script but not execute it to check for syntax errors in the script
• -V: Executes scripts while printing executed script commands to standard error output
• -X: Provides trace execution information and prints each command and result in sequence

There are three ways to use these options, one to provide parameters at the command line:

$ sh-x./script.sh

The second is to provide parameters at the beginning of the script:

#! /bin/sh-x

The third method is to enable or disable parameters in the script with the SET command:

#! /bin/shif [-Z "$"]; Then  set-x    # enable x parameter  echo "Error:insufficient Args."  Exit 1  Set +x    # disable x parameter fi

Note: This allows you to track and debug only one segment of the script.

Regular expressions

Character class:

Examples of character     meanings                          .   Match any one of the characters          ABC. can match ABCD, ABC9, etc. []  matches any one of the characters in parentheses  [abc]d can match AD, BD, or cd-to   denote a range of characters within [] brackets    [0-9a-fa-f] You can match a hexadecimal number ^   at the beginning of the [] parenthesis, and match any character except the one in parentheses  [^xy] to match any character except XY, so [^xy]1 can match A1, B1 but does not match X1, Some of the named character classes predefined by the Y1grep tool: [[: Alpha:]] matches one letter, [[:d Igit:]] matches a number

Quantity qualifier:

Example of character    meaning                             ?   The unit immediately preceding it should match 0 or one time    [0-9]?\.[ 0-9] Match 0.0, 2.3,. 5, etc. in the regular expression is a special character, so you need to escape it, take the literal value +   immediately before the cell should match one or more    [a-za-z0-9_.-][email protected ][a-za-z0-9_.-]+\. [a-za-z0-9_.-]+ Match Email address * The   unit immediately preceding it should match 0 or more times    [0-9][0-9]* matches at least one digit, equivalent to [0-9]+,[a-za-z_]+[a-za-z_0-9]* Matches the C language identifier {n} the cell immediately preceding it should exactly match n times       [1-9][0-9]{2} matches an integer from 100 to 999 {N,} The  cell immediately preceding it should match at least N times     [1-9][0-9]{2,} Matches three-bit or more (with three-bit) integers {, m} The  cells immediately preceding it should match up to M times     [0-9]{,1} equivalent to [0-9]?{ N,M} The unit immediately preceding it should match at least N times, up to M times   [0-9]{1,3}\.[ 0-9]{1,3}\. [0-9] {1,3}\. [0-9] {1,3} matches IP address

Position Qualifier

                Example of character meaning ^ matches the position of the beginning of the line  ^content match the position at the end of the content$ match row at the beginning of a line  $ match at the end of a line; ^$ matches a blank line \<  match the position   at the beginning of the word \<th matches ... this, but does not match Ethernet, tenth\>  matches the end of the word   p\> match leap ..., but does not match the parent, sleepy\b  Matches the position of the beginning or end of a word     \bat\b match ... at ..., but does not match the  position of the first and the end of a non-word match for cat, Atexit, batch\b   \bat\b match battery, but does not match ... Attend, hat ...　　　　　　　　

Other special characters

Character  meaning    example \    escape character, ordinary character escapes for special characters, special character escapes for ordinary characters    < written \< denotes the position of the beginning of the word, special characters. Write \. and \ write \ \ to match as plain characters () to   enclose part of a regular expression as a unit, you can use the Quantity qualifier    ([0-9]{1,3}\.) for the entire cell. {3} [0-9] {1,3} matching IP address |    Joins two sub-expressions, representing or the relationship      N (o|either) matches no or neither

The above is the extended specification for grep regular expressions, and the basic specification also has these syntaxes, just characters ? +{}| () should be interpreted as ordinary characters , to indicate that the above special meaning needs to be added \ escaped . If you use grep instead of egrep and do not add the-e parameter, you should write the regular expression in accordance with the basic specification.

grep

grep is looking for a row that contains a pattern, not a row that exactly matches a pattern.

1. Role

The grep command in a Linux system is a powerful text search tool that uses regular expressions to search for text and print matching lines. The grep full name is global Regular expression Print, which represents the globally regular expression version, and its use rights are for all users.

The grep family includes grep, Egrep, and Fgrep. Egrep and Fgrep commands are only a small difference from grep. Egrep is the extension of grep, which supports more re metacharacters, and fgrep is fixed grep or fast grep, which regards all the letters as words, that is, the metacharacters in the regular expression represents the literal meaning back to itself, no longer special. Linux uses the GNU version of grep. It is more powerful and can use the Egrep and FGREP functions with the-G,-e,-f command line options.

2. Format

grep [Options]

3. Main parameters

• -C: Outputs only the count of matching rows.
• -I: Case insensitive.
• -H: The file name is not displayed when querying multiple files.
• -L: Only file names that contain matching characters are output when querying multiple files.
• -N: Displays matching lines and line numbers.
• -S: does not display error messages that do not exist or have no matching text.
• -V: Displays all lines that do not contain matching text.
• --color=auto: You can add a color display to the keywords you find.

4. Example

The $ grep ' test ' d* shows all the lines in the file that begin with D that contain test. The $ grep ' test ' AA bb cc Displays the line in the aa,bb,cc file that matches the test. The $ grep ' [a-z]\{5\} ' AA displays all lines that contain a string of at least 5 consecutive lowercase characters per string. $ grep ' w\ (es\) t.*\1′aa\1 represents a match to a character in () if West is matched, es is stored in memory, labeled 1, and then searched for any character (. *) followed by another ES (\1), which is found to display the line. If you use Egrep or GREP-E, you do not have "\" number to escape, directly written as ' W (es) t.*\1′ on it.

Find

When running a very resource-intensive find command, many people tend to put it in the background because it can take a long time to traverse a large file system (this refers to a file system with more than 30G bytes).

1. Find command format:

Find Pathname-options [-print-exec-ok ...]

2. Parameters:

• The directory path that the Pathname:find command looks for. For example, use. To represent the current directory, and/to represent the system root, recursively lookup.
• The-print:find command outputs the matched file to standard output.
• The-exec:find command executes the shell command given by the parameter to the matching file. The corresponding command is in the form of ' command ' {} \;, note the space between {} and \;
• -ok: The same as-exec, except that the shell command given by the parameter is executed in a more secure mode, prompting the user to determine whether to execute before executing each command.

3. Options:

• -name finds files by file name.
• -perm to find files according to file permissions.
• -prune Use this option to have the Find command not be found in the currently specified directory, and if you use the-depth option at the same time,-prune will be ignored by the Find command.
• -user Search for files according to the owner of the file.
• -group finds files according to the group to which the files belong.
• -nogroup finds a file that does not have a valid owning group, that is, the group to which the file belongs does not exist in/etc/groups.
• -nouser finds a file without a valid owner, that is, the owner of the file does not exist in the/etc/passwd.
• -newer file1! File2 look for a file that changes time than the file File1 new but older than the file file2.
• -type find a file of a certain type, such as:
  • B-block device files.
  • D-Directory.
  • C-character device file.
  • P-Pipeline file.
  • L-Symbolic link file.
  • F-Normal file.
• -size N:[c] finds files with a file length of n blocks, with C indicating the length of the file in bytes.
• -depth when looking for a file, first find the file in the current directory, and then look in its subdirectories.
• -fstype find files located in a type of file system that are typically found in the configuration file/etc/fstab, which contains information about the file system in the system.
• -mount does not cross the file system mount point when locating files.
• -follow if the find command encounters a symbolic link file, it tracks to the file that the link points to.
• -amin N Find the last n minutes of files accessed in the system
• -atime N Find the last n*24 hour Access file in the system
• -cmin n Find files in the last n minutes of the system changed file status
• -ctime n Find files that have changed file status in the last n*24 hours of the system
• -mmin n Find files that have changed file data in the last N minutes of the system
• -mtime n Find files that have changed file data for the last n*24 hours in the system

4. Use exec or OK to execute shell commands

When using find, just write the desired action in a file, you can use the exec to match the find lookup, it is convenient for some operating systems only allow the-EXEC option to execute commands such as LS or ls-l. Most users use this option to find old files and delete them. It is recommended that you take a look at the LS command before you actually execute the RM command to delete files, confirming that they are the files you want to delete.

The EXEC option is followed by the command or script that you want to execute, followed by a pair of {}, a space and a \, and finally a semicolon. In order to use the EXEC option, you must use the Print option at the same time. If you verify the Find command, you will see that the command outputs only the relative path and file name from the current path.

Example:

Find.  -type f-exec ls-l {} \;# Find all the normal files in the current directory, open a sub-process to execute the ls-l command (put all the found files into the buffer, and then execute the command together) find Logs-type f-mtime +5-exec rm {} \;# find the files in the/logs directory to find the change time before 5th and delete them find. -name "*.conf"  -mtime +5-ok rm {} \;# Find all file names in the current directory to. conf end, change time over 5th, interactive reminders. # < RM .... /conf/httpd.conf >? Press the y key to delete the file and press N to not delete it. Find/etc-name "passwd*"-exec grep "root" {} \;# the find command first matches all files named "passwd*" and then executes the grep command to see if there is a root user in these files.

5. Detailed options:

A.name Options

The file name option is the most common option for the Find command, either used alone or in conjunction with other options.

You can use a file name pattern to match files, remembering to enclose the filename pattern in quotation marks.

No matter what the current path is, if you want to find the file name in your root $home that matches *.txt, use ~ as the ' pathname ' parameter, and the tilde ~ represents your $home directory.

To find all the ' *.txt ' files in the current directory and subdirectories, you can use:

$ find. -name "*.txt"-print

You want the current directory and subdirectories to find file names that begin with an uppercase letter, which can be used:

$ find. -name "[a-z]*"-print

To find files with the file name beginning with host in the/etc directory, you can use:

$ find/etc-name "host*"-print

To find files in the $home directory, you can use:

$ find ~-name "*"-print or find. -print

To get the system running at a high load, start looking for all the files from the root directory:

$ find/-name "*"-print

If you want to find the file name in the current directory with two lowercase letters, followed by two digits, and finally the. txt file, the following command will be able to return files such as the file named Ax37.txt:

$find. -name "[A-z][a-z][0-9][0-9].txt"-print

B. Using the PERM option

Follow the file permission mode with the-perm option to find files by file permission mode. It is best to use the octal permission notation.

For example, in the current directory to find file permission bit 755 file, that is, the file owner can read, write, execute, other users can read, execute files, can be used:

$ find. -perm 755-print

There is also a way of expression: in front of the octal number to add a bar-, the expression is matched, such as 007 is equivalent to 777,-006 equivalent to 666

# ls-l# Find. -perm 006# Find. -perm-006-perm mode: The file license exactly conforms to Mode-perm/mode: The file permission part conforms to the mode,6 representative reads and writes, as long as has 2 one in the line-perm-mode: The file license fully conforms to mode, must have the specified all permission, Too much, such as Find permissions 006:007,777 ... also OK

C. Ignore a directory

If you want to ignore a directory when you're looking for a file, because you know that directory doesn't have the file you're looking for, you can use the-prune option to indicate which directories you want to ignore. Be careful when using the-prune option, because if you use the-depth option at the same time, the-prune option is ignored by the Find command.

If you want to find the file under the/apps directory but do not want to find it in the/apps/bin directory, you can use:

$ find/apps-path "/apps/bin"-prune-o-print

Avoid multiple folders

\ ( -path/home/itcast/f1-o-path/home/itcast/f2 \)-prune-o-print

Note (before the \, note (after the space.)

D. Using the user and Nouser options

Find files by file owner, such as in the $home directory to find files belonging to the main itcast file, you can use:

$ find ~-user Itcast-print

Look for files in the/etc directory that belong to the main UUCP:

$ find/etc-user Uucp-print

In order to find files that are already deleted from the master account, you can use the-nouser option. This will enable you to find files that are not valid accounts in the/etc/passwd file. When using the-nouser option, you do not have to give the user name; the Find command can do the work for you.

For example, to find all such files in the/home directory, you can use:

$ find/home-nouser-print

E. Using the group and Nogroup options

Just like the user and Nouser options, the Find command has the same options for the group of users that the file belongs to, in order to find files belonging to the Itcast user group under the/apps directory, you can use:

$ find/apps-group Itcast-print

To find all files that do not have a valid group of users, you can use the Nogroup option. The following find command looks for such files from the root directory of the file system

$ find/-nogroup-print

F. Find files by change time or access time, etc.

You can use the Mtime,atime or CTime option if you want to find the file by changing the time. If the system suddenly does not have free space, it is possible that the length of a file grows rapidly during this period, you can use the Mtime option to find such a file.

Use a minus sign-to limit the time to change the file within the current n days, and use the Plus + to limit the change time before the current n days of the file.

To find files that change within 5th of the system root directory, you can use:

$ find/-mtime-5-print

In order to find files that change time before 3rd in the/var/adm directory, you can use:

$ find/var/adm-mtime +3-print

G. Finding new or older files than a file

You can use the-newer option if you want to find all files that have changed time than one file but older than the other. The general form of it is:

Newest_file_name! Oldest_file_name one of them,! is a logical non-symbol.

H. Using the Type option

To find all the directories in/etc directory, you can use:

$ find/etc-type D-print

To find all types of files except directories in the current directory, you can use:

$ find. ! -type D-print

In the/etc directory to find all the symbolic link files, you can use

$ find/etc-type L-print

H. Using the SIZE option

files can be searched by file length, and the length of the file referred to here can be measured either in blocks or in bytes. The length of the measured file in bytes is expressed as n C, and the length of the block measurement file is only represented by a number.

When looking up files by file length, this is generally the size of the file in bytes, and it is easier to convert by using blocks to measure the file system. In the current directory, look for files with a file length greater than 1 m bytes :

$ find. -size +1000000c-print

Look for files with a file length of exactly 100 bytes in the/home/apache directory:

$ find/home/apache-size 100c-print

Look for files that are longer than 10 blocks in the current directory (a block equals 512 bytes):

$ find. -size +10-print

I. Using the depth option

When you use the Find command, you may want to match all the files and find them in the subdirectory. Use the depth option to enable the Find command to do so. One reason for this is that when you use the Find command to back up the file system to tape, you want to back up all the files first, and then back up the files in the subdirectories.

In the following example, the Find command starts at the root of the file system and looks for a file named Con.file.

It will first match all the files and then go to the subdirectory to find them.

$ find/-name "CON. FILE "-depth-print

J. Using the Mount option

You can use the Mount option of the Find command to find a file in the current file system (without entering another file system).

From the current directory, look for files in the file system with the file name ending in XC:

$ find. -name "*. XC "-mount-print

6, Xargs

When a matching file is processed using the-EXEC option of the Find command, the Find command passes all matching files to exec execution. However, some systems have a limit on the length of the command that can be passed to exec so that an overflow error occurs after the Find command runs for a few minutes. The error message is usually "parameter column too Long" or "parameter column overflow". This is where the Xargs command is used, especially with the Find command.

The find command passes the matched file to the Xargs command, and the Xargs command takes only a subset of the files at a time instead of all, unlike the-exec option. This allows it to first process a portion of the file that was first fetched, then the next batch, and so on.

In some systems, the use of the-EXEC option initiates a corresponding process for processing each matching file, not all of the matching files are executed once as parameters, so that in some cases there will be too many processes and degraded system performance, so the efficiency is not high;

With the Xargs command, there is only one process. In addition, when using the Xargs command, whether to get all the parameters at once or to get the parameters in batches, and the number of parameters to get each time will be determined according to the command's options and the corresponding tunable parameters in the system kernel.

Take a look at how the Xargs command is used with the Find command, and give some examples.

The following example finds every normal file in the system, and then uses the Xargs command to test what type of file they belong to

#find. -type F-print | Xargs file

In the current directory, look for files with read, write, and execute permissions for all users, and reclaim the appropriate write permissions:

# ls-l# Find. -perm-7-print | Xargs chmod o-w# ls-l

Use the grep command to search all common files for the word hello:

# Find. -type F-print | Xargs grep "Hello"

Use the grep command to search for the word hello in all normal files in the current directory:

# Find. -name \*-type F-print | Xargs grep "Hello"

Note that in the example above, \ is used to cancel the special meaning of the * in the shell in the Find command.

The Find command, with exec and Xargs, allows the user to execute almost all commands against the matching file.

Sed

Sed means the stream editor, which is commonly used as a filter in shell scripts and makefile, which introduces the output of the previous program to the input of SED, which is converted to another format output through a series of editing commands. Sed and vi both originate from the early Unix Ed tools, so many SED commands are the same as the last command of VI.

The basic format of the SED command line is

SED option ' script ' file1 file2 ... sed option-f scriptfile file1 file2 ...

Option meaning:

--version            Displays the SED version. --help               Displays the help document. -N,--quiet,--silent  silent output, by default, the SED program automatically prints the contents of the pattern space after all script instructions have been executed, and these options can mask automatic printing. The-e script            allows multiple script directives to be executed. -F Script-file,--file=script-file   reading script instructions from a file is great for writing an automated scripting program! -I,--in-place        directly modify the source file, after the script instructions processed content will be output to the source file (the source file is modified) use caution! -L N,--line-length=n This option specifies the length of the governor that the L instruction can output, and the l instruction is used to output nonprinting characters. --posix             disables the GNU SED extension feature. -R,--regexp-extended  use extended Regular expression-s in script directives,--separate      by default, SED takes multiple filenames specified by the command line as a long contiguous input stream. The GNU SED allows them to be treated as separate files, so that regular expressions do not cross-file matches. -U,--unbuffered    the minimum cache input and output.

The above is only the option function description of the SED program itself, as for the specific script instructions (that is, the operation of the contents of the file) we will describe in detail, here is a brief introduction of several script instructions as an example of the SED program.

A,append        Append i,insert        insert d,delete        Delete s,substitution  replace

such as: $ SED "2a itcast"./testfile add "Itcast" after the second line of the output testfile content.

$ sed "2,5d" testfile

The files processed by SED can be either redirected from a standard input, or when command-line arguments are passed in, command-line arguments can pass in multiple files at once, and SED is processed sequentially. The edit command for SED can be passed directly to the command line parameter, or it can be written as a script file and then specified with the-f parameter, the format of the edit command is

/pattern/action

Where pattern is a regular expression, action is an edit operation. The SED program reads the pending file one line at a time, and if a line matches the pattern, the action is executed, and if a command has no pattern and only action, the action will be used for each row of the file to be processed.

Commonly used SED commands

/pattern/p  prints lines that match the pattern/pattern/d Delete rows matching the pattern  /pattern/s/pattern1/pattern2/   find rows that match the pattern, Replace the first string that matches pattern1 in the row with pattern2/pattern/s/pattern1/pattern2/g to  find the line that matches the pattern, and replace all strings that match pattern1 in the row with pattern2

Using the P command, it is important to note that SED outputs the contents of the pending file along with the processing results to the standard output, so the P command indicates that the line that matches the pattern is printed in addition to the contents of the file. For example, a file testfile content is

123abc456

Print lines that contain ABC

$ sed '/abc/p ' testfile123abcabc456

To output only the processing results, you should add the-n option, which is equivalent to the grep command

$ Sed-n '/abc/p ' testfileabc

Using the D command does not require the-n parameter, such as deleting a line containing ABC

$ sed '/abc/d ' testfile123456

Note that the SED command does not modify the original file, and the Delete command only indicates that some rows do not print output, rather than deleting from the original file.

When using the Find and Replace command, you can copy the string that matches pattern1 to the pattern2, for example:

$ sed ' s/bc/-&-/' testfile123a-bc-456pattern2 in & represents the string in the current line of the original file that matches the PATTERN1

Another example:

$ sed ' s/\ ([0-9]\) \ ([0-9]\)/-\1-~\2~/' testfile-1-~2~3abc-4-~5~6

The \1 in pattern2 represents the content that matches the first () parenthesis of pattern1, and \2 represents the content that matches the second () parenthesis of pattern1. SED uses the basic regular expression specification by default, and if the-r option is specified with the extended specification, then the () parentheses do not have to be escaped.

$ sed  ' s/yes/no/;s/static/dhcp/'  ./testfile Note: Use semicolons to separate instructions. $ Sed-e ' s/yes/no/'-e ' s/static/dhcp/' testfile Note: Use the-e option.

If the content of Testfile is

Now remove all HTML tags so that the output is
Hello Worldwelcome to the world of regexp!
How do you do it? If you use the following command
$ sed ' s/<.*>//g ' testfile
The result is two blank lines that filter out all the characters. This is because the quantifier in the regular expression matches the string as long as possible, which is called greedy (greedy). For example, when the SED is dealing with the first line,,<.*> matches not or such tags, but
Such a whole line, because this line starts with <, and the middle is a number of arbitrary characters, and the end is >. So how does this order change? Leave the students to think about practice.
Awk
SED deals with documents in an Act unit, where awk is stronger than sed in that it can be processed not only as a unit but also as an entity. The default line delimiter for awk is newline, the default column delimiter is a contiguous space and tab, but the row and column separators can be customized, such as there are several fields for each row of the/etc/passwd file, and the column delimiter for awk can be redefined by separating the fields: and process this file as a unit. Awk is actually a very complex scripting language, and there are branching and looping constructs like the C language, but the basic usage is similar to SED, and the basic form of the awk command line is:
awk option ' script ' file1 file2 ... awk option-f scriptfile file1 file2 ...
Like sed, awk handles files that can be redirected either by standard input or when command-line arguments are passed in, and the edit command can be passed directly to the command-line arguments, or you can specify a script file with the-f parameter, with the format of the edit command:
/pattern/{actions}condition{actions}
Like SED, pattern is a regular expression, and actions are a series of actions. The AWK program reads the pending file one line at a time, if a line matches the pattern, or if the condition condition is met, the actions are performed if an awk command has only the Actions section, and the actions are applied to each row of the file to be processed. For example, the contents of a file testfile indicate the stock quantity of a store:
ProductA  30ProductB  76ProductC  55
Print the second column of each row:
$ Awk ' {print $;} ' testfile307655
The automatic variable, $ $, is the first column, the second column, and so on, similar to the location parameter of the shell script, and $ A represents the entire current row. For example, if the stock of a product is below 75, an order is required at the end of the line:
$ Awk ' $2<75 {printf "%s\t%s\n", $ A, "REORDER";} $2>=75 {print $;} ' testfileproducta reorderproductb<    C5/>76PRODUCTC    REORDER
It can be seen that awk also has printf functions that are very similar to the C language. The condition part of the awk command can also be two special condition-begin and end, and for each pending file, the actions after the begin are executed once before the entire file is processed. The actions after end are executed once after the entire file has been processed.
The awk command can use variables like the C language (but do not need to define variables), such as counting empty rows in a file
$ Awk '/^ *$/{x=x+1;} END {print x;} ' testfile
Just like the shell environment variables, some awk variables are predefined and have special meanings:
awk common built-in variables
FileName  of the current input file, which is a read-only NR  line number for the current line, is read-only, and R represents the  number of columns owned by the current row RECORDNF, which is read-only, and F represents the column delimiter in the FIELDOFS output format. The default is the column of the space fs  input file, the default is a continuous space and tabors output format line delimiter, the default is the newline character Rs  input file line delimiter, the default is a newline character
For example, a list of user accounts in the print system
$ Awk ' BEGIN {fs= ': '} {print $;} '/etc/passwd
Linux--shell Script Programming 2