Shell (awk) array use case

1. Take the IP address of the ifconfig eth0

[Email protected] ~]$ ifconfig eth0

Eth0 Link encap:ethernet HWaddr 00:0c:29:68:47:71

inet addr:192.168.48.144 bcast:192.168.48.255 mask:255.255.255.0

Inet6 ADDR:FE80::20C:29FF:FE68:4771/64 Scope:link

Up broadcast RUNNING multicast mtu:1500 metric:1

RX packets:48161 errors:0 dropped:0 overruns:0 frame:0

TX packets:25022 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:36048837 (34.3 MiB) TX bytes:2202106 (2.1 MiB)

[[email protected] ~]$ ifconfig eth0|awk-f ' [:]+ ' Nr==2{print nr,$4} '

2 192.168.48.144

2. Count Apache log list IP access requests ranking

[email protected] ~]$ cat 2.txt

10.0.0.3--[21/mar/2015-07:50:17+0800]*get/http/1.1*200 19 *-*

10.0.0.3--[21/mar/2015-07:50:17+0800]*get/http/1.1*200 19 *-*

10.0.0.5--[21/mar/2015-07:50:17+0800]*get/http/1.1*200 19 *-*

10.0.0.3--[21/mar/2015-07:50:17+0800]*get/http/1.1*200 19 *-*

10.0.0.3--[21/mar/2015-07:50:17+0800]*get/http/1.1*200 19 *-*

10.0.0.6--[21/mar/2015-07:50:17+0800]*get/http/1.1*200 19 *-*

10.0.0.3--[21/mar/2015-07:50:17+0800]*get/http/1.1*200 19 *-*

10.0.0.3--[21/mar/2015-07:50:17+0800]*get/http/1.1*200 19 *-*

[[email protected] ~]$ awk ' {print $} ' 2.txt |sort|uniq-c

6 10.0.0.3

1 10.0.0.5

1 10.0.0.6

[[email protected] ~]$ awk ' {array[$1]++}end{for (k in array) {print k,array[k];} ' 2.txt

10.0.0.3 6

10.0.0.5 1

10.0.0.6 1

3. Count the number of domain name visits

[email protected] ~]$ cat 1.txt

Http://www.baidu.com/index.html

Http://www.163.com/1.html

Http://www.cnblogs.com/index.html

Http://www.baidu.com/2.html

Http://www.163.com/index.html

Http://www.qq.com/index.html

Http://www.baidu.com/3.html

Http://www.163.com/2.html

Http://www.baidu.com/2.html

[[email protected] ~]$ awk ' {split ($0,array, "/+"); Key=array[2];count[key]++}end{for (KK in count) {print Kk,count[kk]}} ' 1.txt

Www.qq.com 1

Www.cnblogs.com 1

Www.baidu.com 4

Www.163.com 3

[[email protected] ~]$ awk-f "[/]+" ' {array[$2]++}end{for (i in array) {print i,array[i]}} ' 1.txt

Www.qq.com 1

Www.cnblogs.com 1

Www.baidu.com 4

Www.163.com 3

4. Calculate the total wage and average salary of each person

[[email protected] ~]$ awk ' {array[$2]+=$3;count[$2]++}end{for (key in count) {print Key,array[key] "K", Array[key]/count [Key]}} ' 3.txt

Kuqi 20k 10

Jeacen 14k 7

Yingsui 27k 13.5

Xiaofen 16k 8

Wodi 23k 11.5

Yideng 21k 10.5

5. deduplication for local IP and remote IP and count repetitions

[email protected] ~]$ cat 4.txt

Proto recv-q send-q Local addree Foreign addree State

TCP 0 0 0.0.0.0:80 0.0.0.0:* LISTEN

TCP 0 0 115.29.49.213:80 117.136.27.254:13779 syn_recv

TCP 0 0 115.29.49.213:80 113.97.117.157:1847 syn_recv

TCP 0 0 115.29.49.213:80 117.136.40.20:19594 syn_recv

TCP 0 0 115.29.49.213:80 117.136.40.20:19595 syn_recv

TCP 0 0 115.29.49.213:80 121.236.219.69:45363 syn_recv

TCP 0 0 0.0.0.0:21 0.0.0.0:* LISTEN

TCP 0 0 0.0.0.0:22 0.0.0.0:* LISTEN

UNIX 3 [] syream CONNECTED 11183664/tmp/mysql. SOCK

UNIX 3 [] syream CONNECTED 11183646/tmp/mysql. SOCK

UNIX 3 [] syream CONNECTED 11183665/tmp/mysql. SOCK

UNIX 3 [] syream CONNECTED 11183668/tmp/mysql. SOCK

UNIX 3 [] syream CONNECTED 11183654/tmp/mysql. SOCK

UNIX 3 [] syream CONNECTED 11183655/tmp/mysql. SOCK

UNIX 3 [] syream CONNECTED 11183668/tmp/mysql. SOCK

UNIX 3 [] syream CONNECTED 11183676/tmp/mysql. SOCK

UNIX 3 [] syream CONNECTED 11183672/tmp/mysql. SOCK

[[email protected] ~]$ awk-f ' [:]+ '/^tcp/{array[$4 ' "$6]++}end{for (key in array) {print Key,array[key]}} ' 4.txt

115.29.49.213 117.136.40.20 2

0.0.0.0 0.0.0.0 3

115.29.49.213 113.97.117.157 1

115.29.49.213 121.236.219.69 1

115.29.49.213 117.136.27.254 1

6. Statistic source IP port and destination IP simultaneously go to heavy

[email protected] ~]$ cat 5.txt

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (TCP 10.0.0.131:1227->210.192.121.172:80 on IXP7) [0,0]

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (TCP 10.0.0.131:1227->210.192.121.172:80 on IXP7) [0,0]

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (TCP 10.0.0.131:1227->210.192.121.172:80 on IXP7) [0,0]

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (TCP 10.0.0.131:1227->210.192.121.172:80 on IXP7) [0,0]

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (TCP 10.0.0.131:1227->210.192.121.172:80 on IXP7) [0,0]

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (TCP 10.0.0.43:54963->203.81.19.92:80 on PPP6) [0,0]

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (UDP 10.0.0.19:1441->121.14.96.233:80 on ppp0) [0,0]

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (UDP 172.16.1.103:57318->211.147.6.3:80 on PPP2) [0,0]

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (TCP 172.16.1.203:4372->61.135.163.86:80 on IXP7) [0,0]

Dec 2 01:17:42 10.0.0.0 RV016 rgfw-out:accept (TCP 10.0.0.131:1227->210.192.121.172:80 on IXP7) [0,0]

[Email protected] ~]$ vim 5.sh

{

Split ($9,array, ":|->")

SIP=ARRAY[1]

SPORT=ARRAY[2]

MIP=ARRAY[3]

if (! ( (SIP,SPORT,MIP) in tee)) {

TEE[SIP,SPORT,MIP] = 1

}

}

end{

For (key in tee)

Print key

}

[Email protected] ~]$ awk-f 5.sh 5.txt

172.16.1.103 57318 211.147.6.3

10.0.0.19 1441 121.14.96.233

172.16.1.203 4372 61.135.163.86

10.0.0.43 54963 203.81.19.92

10.0.0.131 1227 210.192.121.172

The optimization effect is as follows

Start at 5.sh add begin { printf("%-16s %-6s %-16s\n","SIP","SPORT","MIP")}

[Email protected] ~]$ awk-f 5.sh 5.txt

SIP SPORT MIP

172.16.1.10357318211.147.6.3

10.0.0.191441121.14.96.233

172.16.1.203437261.135.163.86

10.0.0.4354963203.81.19.92

10.0.0.1311227210.192.121.172

Shell (awk) array use case