Shell-awk Advanced Applications

Awk
I. Introduction of AWK
scripting language
Variable..... Process Control array functions

       use awk for Data statistics
       1 Number of users in the statistical system, number of built-in users  
       2 statistics Site Server traffic for the day    access_log
        3 Send a reminder message to the top 10 mailbox accounts with the largest mailbox usage per week
       4 to output the top 10 process names of the current system for 10 minutes with the highest CPU utilization
      
      in shell scripts
            use awk to get the data to the shell program for processing.
       Check the usage of the current system root partition every 5 minutes, the free space of the root partition is small          sends alarm information to all terminals of the current system at 10G        
------------------------------------- ---------------------------------
awk Syntax format    (rpm-q   gawk)
awk       ' actions '     file list  
awk   options     ' conditional actions '     file list

Command | awk option ' conditional action '

---------------------------------------------------------------------
awk execution process?
All rows in the file are processed line by row in the behavior Processing unit, and the next line is processed automatically after the output of the current row until all rows have been processed.

----------------------------------------------------------------------
Options?
-F "Symbol" specifies the split symbol for the column awk uses a space or \tab as the delimiter for the column by default
----------------------------------------------------------------------
Use of the awk variable (built-in variable custom variable call shell)
1. Built-in variables

The entire line of text currently read in
FNR Save the Ordinal (line number) of the current processing line within the original text
NR record number of entries currently read (number of lines)
FileName of the file currently being processed by filename
NF records the number of fields in the current row (number of columns) field $n Specifies the nth field of the separator, such as $ $, $ $ for the 1th and 3rd columns, respectively
FS Save or SET field separator space \tab

awk ' {print FNR} ' a.txt
awk ' {print Nr,fnr} ' a.txt
awk ' {print FILENAME} ' a.txt b.txt
head/etc/passwd | Awk-f ":" ' {print $} '
head/etc/passwd | Awk-f ":" ' {print $1,$3} '

awk ' {print $} ' a.txt
awk ' {print ' abc '} ' A.txt

df-h | grep/boot$ | awk ' {print '} '

echo 9G | Awk-f "[. G] "' {print $} '
echo 3.5G | Awk-f "[. G] "' {print $} '

2 Custom variables
Variable name = value
awk ' {i=101;name= ' Lucy ';p rint i,name} ' a.txt
awk ' {i=101;name= ' Lucy ';p rint "I,name"} ' A000.txt

3 awk invokes shell variables
Name1=lucy Name2=tom
Panax Notoginseng echo $name 1 $name 2
Awk-v x= $name 1-v y= $name 2 ' {print x, y} ' a.txt

[Email protected] ~]# x=203
[[email protected] ~]# awk ' {print ' $x '} ' a.txt
203
203
203
[Email protected] ~]#

[[email protected] ~]# awk-v y= $x ' {print y} ' a.txt call variable when-V *=$*
203
203
203
[Email protected] ~]#

ENVIRON invoke shell environment variable, format: environ["variable name"]
Awk-v y= $HOSTNAME ' {print y} ' a.txt
Echo $USER $HOSTNAME
awk ' {print environ[' USER ']} ' a.txt
awk ' {print environ[' HOSTNAME "]} ' a.txt
------------------------------------------------------------------
Invoking the shell custom variable when executing the SED command
Sed-n ' 1,3p ' a.txt
X=1 y=3
echo $x $y
Sed-n ' $x ', ' $y ' P ' a.txt

----------------------------------------------------------------------
What is the order of awk processing data?
awk ' A.txt
awk ' {} ' A.txt
awk ' begin{} {} ' A.txt
awk ' {} end{} ' A.txt
awk ' begin{} {} end{} ' A.txt

begin{} pre-line processing
Do the initialization of the data definition
Awk-f ":" ' {print $} ' a.txt
"begin{fs=": "}{print $ ' a.txt"

{} row processing
Commands executed on each line multiple commands used; interval

end{} line post-processing
After all the lines in the file have been processed, the command executed
Summary line output for the data

awk ' {print $0}end{print $} ' a.txt
End{print awk '/etc/passwd
Useradd Bob
End{print awk '/etc/passwd
A.txt awk ' {print fnr,$0} '
Fnr}{print awk ' begin{print fnr}end{print FNR} ' a.txt

The number of rows for awk ' end{print ' file ' filename ' is ' NR ' line '} '/etc/passwd

awk ' begin{fs= ': ";p rint" user\tuid\thome "}{print" \ T ", $" t ", $6}end{print" sum lines is "NR} ' a.txt

Awk-f ":" ' {print $ \ t ', $ "t", $6}end{print "sum lines is" NR} ' a.txt | Sed ' 1iuser uid home '

-------------------------------------------------------------------
How is the awk condition represented?
awk ' {print $} ' a.txt
awk ' condition {print $} ' a.txt

Numerical comparison
NUM1 symbol num2
= = equals! = does not equal
> greater than >= greater than or equal to

< less than <= less than or equal to
975 awk ' {print $} ' a.txt
976 awk ' fnr==5{print $} ' a.txt
977 awk ' nr==5{print $} ' a.txt
978 awk ' 1==5{print $} ' a.txt
979 awk ' 1==1{print $} ' a.txt
981 awk ' Fnr<=3{print $} ' a.txt
982 awk ' Fnr==3{print $} ' a.txt
983 awk ' Fnr!=5{print $} ' a.txt

Awk-f ":" ' $3==500{print $1,$3} '/etc/passwd

string comparison
"str1" symbol "STR2"
= = equals! = does not equal

Awk-f ":" ' $1== "rsync" {print $} '/etc/passwd
987 awk-f ":" ' $1== ' root ' {print '} '/etc/passwd
988 awk-f ":" ' $1!= ' root ' {print '} '/etc/passwd

--------------------------------------------------------------------
Regular expressions
Value symbol/Regular expression/

Match ~
No match! ~

awk ' $0~/r/{print $} '/etc/passwd
Awk-f ":" ' $1~/^...$/{print $ '/etc/passwd
Awk-f ":" ' $1!~/^...$/{print $ '/etc/passwd
------------------------------------------------------------------
Logical comparison (multiple conditions)
Logic and && multiple conditions set up simultaneously
Logical OR | | Multiple conditions have one condition can be
Logical Non! Take counter

PLJ Bob Sync

Awk-f ":" ' $1== "PLJ" | |  $1== "Bob" | | $1== "Sync" {print '} '/etc/passwd

Awk-f ":" ' $1== "rsync" && $3<500 {print $1,$3} '/etc/passwd

---------------------------------------------------------------------

Operator
+   -   *   /   %
++
--
+=   -=  *=  /=

Even lines in the output file
awk ' Fnr%2==0{print fnr,$0} ' a.txt

Awk-f ":" ' $3<500{i++} $3>=500{y++}end{print i,y} '/etc/passwd

Head-1/etc/passwd | awk ' {i+=3}end{print i} '
Head-3/etc/passwd | awk ' {i+=3}end{print i} '
---------------------------------------------------------------------
Total number of columns in the statistics text
Reg.txt
AAA FF CCC 3
AAA FF CCC www 4
AAA ff CCC xxx YYY CCCCC 6
: Wq

awk ' begin{i=0} {I+=NF} end{print i} ' reg.txt
---------------------------------------------------------------------
SEQ 200 | awk ' begin{i=0} $0%3==0 &&
$0%13==0{print $0;i++} end{print i} '
--------------------------------------------------------------------
awk Process Control
If branch structure
Single Branch if (conditional expression) {command executed when condition is set}
146 awk-f ":" ' {if ($3==0) print $} '/etc/passwd
147 awk-f ":" ' {if ($3==0) print $0;print FNR} '/etc/passwd
148 awk-f ":" ' {if ($3==0) {print $0;print FNR}} '/etc/passwd
149 awk-f ":" ' {if ($3==0) {print $0,fnr}} '/etc/passwd
Awk-f ":" ' {if ($3==0) {print fnr,$0}} '/etc/passwd

Awk-f ":" ' $3==0{print $0,fnr} '/etc/passwd

Dual-Branch if (conditional expression) {command executed when the condition is established}else{}

Awk-f ":" ' {if ($3==0) {print Fnr,$0}else{print $1,$6}} '/etc/passwd

Multi-Branch
if (condition 1) {edit directive 1}else if (condition) {edit directive 2}: ..
else{edit Instruction N}

Awk-f ":" {if ($3==0) {print $ "is Admin user"}else if ($3>=1&&$3<500) {print $1,$3 "inside user"}els E{print $1,$3 "Outside User"} '/etc/passwd

---------------------------------------------------------------------
Loop structure while do. While for

while (condition) {loop body}

awk ' begin{i=1;while (i<=3) {print i;i++}} '

awk ' {i=1;while (i<=3) {print i;i++}} ' A.txt

awk ' {i=1;while (i<=3) {print $0;i++}} ' A.txt

Awk-f ":"
' {i=1; while (I<=NF) {
if ($i ~/root/) {j + +}; i++}} end{print J} '
/etc/passwd
-------------------------------------------------------------------
do{Cycle body}while (condition) first to perform the cyclic body after judging the condition

awk ' begin{do{print I;i++}while (i<=3)} '

[[email protected] ~]# awk ' begin{do{i++;p rint i}while (i>=3)} '
1
[[email protected] ~]# awk ' Begin{while (i>=3) {i++;p rint i}} '
[Email protected] ~]#

----------------------------------------------------------------------
For loop
for (initial value; condition; step) {Loop body}
awk ' begin{for (i=1;i<=5;i++) {print i}} '

----------------------------------------------------------------------
Circular structure Control statements
Break ends the execution of its own loop structure
Continue ends the loop of its own loop structure and begins the next cycle

awk ' begin{for (i=1;i<=5;i++) {print I;break}} '

195 awk ' begin{for (i=1;i<=5;i++) {print I;break}} '
196 awk ' begin{for (i=1;i<=5;i++) {Break;print i}} '
197 awk ' begin{for (i=1;i<=5;i++) {if (i==3) {break};p rint i}} '
198 awk ' begin{for (i=1;i<=5;i++) {if (i==3) {Countinu};p rint i}} '
199 awk ' begin{for (i=1;i<=5;i++) {if (i==3) {continue};p rint i}} '

Control Awk's handling of file rows
Next skips the current line and reads the next line of text to begin processing
Exit end text read in, transfer to end{} execution

If no end{} exits the awk processing operation directly

1003 head/etc/passwd > A.txt
1004 awk ' {exit} ' A.txt
1005 awk ' Fnr==3{exit}{print fnr,$0} ' a.txt
1006 awk ' Fnr==3{next}{print fnr,$0} ' a.txt
1007 awk ' {next}{print fnr,$0} ' a.txt
1008 awk ' {print fnr,$0}{next} ' a.txt

--------------------------------------------------------------
[email protected] ~]# cat C.txt
Eth0 Link encap:ethernet HWaddr 00:0c:29:b6:48:a6
Encap:ethernet HWaddr 00:0c:29:b6:48:a6
[Email protected] ~]#
awk ' Nf<2{next} {print $} ' C.txt

----------------------------------------------------------------
awk Array
Defining arrays
Array name [subscript]= Value
Array name ["Subscript"]= value
Array name [subscript]= "Value"

awk ' begin{ip[1]= ' 192.168.1.1 ", ip[2]=101;ip[" Age "]=29} '

Call array element
Array name [subscript]

Output array elements
Print array name [subscript]

awk ' begin{ip[1]= ' 192.168.1.1 ", ip[2]=101;ip[" age "]=29;print ip[" Age "],ip[2],ip[1]} '

awk ' begin{for (i=1;i<=5;i++) {ip[i]= "172.16.0." i}; for (x=1;x<=5;x++) {print ip[x]}} '

-------------------------------------------------------------------

awk ' begin{stugrp["S1"]= "Tom" stugrp["S2"]= "Lucy" stugrp["S3"]= "Bob"; print stugrp["S3"]} '

Iterate through the loop structure of an array element
For (variable in array name) {Loop body}

awk ' begin{stugrp["S1"]= "Tom" stugrp["S2"]= "Lucy" stugrp["S3"]= "Bob" stugrp["S4"]= "Jerry"; For (PLJ in stugrp) {print PLJ, STUGRP[PLJ]}} '

Head-3/etc/passwd | Awk-f ":" ' {usergrp[$3]=$1}end{for (x in usergrp) {print x, usergrp[x]}} '

Write script getupwd-awk.sh to achieve the following requirements:

1) Find a local user using bash as the login shell
2) List the shadow password records for these users

3) Save to Getupwd.log on a per-line "user name-password record"

Count the number of occurrences of each user name in the A.txt file
[email protected] ~]# cat A.txt
Root
Bin
Daemon
Root
Bin
Lucy
Lucy
Lucy
Daemon
Root
Bin
Daemon
Root
Bin
Daemon
[Email protected] ~]#
---------------------------------------------------------------------
Shell Foundation (Bash feature flow control variable flow-controlled function execution script)
Expect awk sed grep array function variable substitution

Shell-awk Advanced Applications