Enterprise shell script Analysis and cutting Apache log combat

One, analysis Apache log

1, there is a document SHELL.SH, the contents are as follows:

[Root@test3root] #catshell. sh

Http://www.baidu.com/index.html

Http://www.google.com/index.html

Http://www.baidu.com/get.html

Http://www.baidu.com/set.html

Http://www.google.com/index.html

Http://www.yahoo.com.cn/put.html

It is now required to intercept the domain name in the file, to count the number of occurrences of the duplicate domain name, and then to sort by number of times, after which the results are as follows:

3www.baidu.com

2www.google.com

1www.yahoo.com.

Sort compares each row of the file as a unit, comparing it to each other, from the first character backwards, sequentially by the ASCII code value, and finally outputting them in ascending order, Uniq is to remove the same row next to each other and keep only one row

[Root@test3 ~]# awk-f "/" ' {print $} ' shell.sh |sort |uniq-c

3 www.baidu.com

2 www.google.com

1 www.yahoo.com.cn

2, in the Apachelog to find the most visited 10 IP

The/usr/local/apache2/logs/access_log format is as follows

192.168.46.1-CHEN[21/SEP/2013:14:04:48+0800] "get/phpmyadmin/themes/pmahomme/img/tab_hover_bg.pnghttp/1.1" 200502

[Root@test3 ~]# awk ' {print $} '/usr/local/apache2/logs/access_log |sort|uniq-c|head-n 10

7 127.0.0.1

228 192.168.46.1

3. In the Apache log to find the most visited the number of minutes

The/usr/local/apache2/logs/access_log format is as follows

192.168.46.1-CHEN[21/SEP/2013:14:04:48+0800] "get/phpmyadmin/themes/pmahomme/img/tab_hover_bg.pnghttp/1.1" 200502

[Root@test3 ~]# awk ' {print $} '/usr/local/apache2/logs/access_log|cut-c 14-18 |sort|uniq-c|sort-nr|head

33 13:55

30 13:35

19 13:22

15 13:54

15 13:45

15 13:38

15 13:36

13 13:04

10 12:59

9 13:18

4. Find the most visited pages in the Apache log

The/usr/local/apache2/logs/access_log format is as follows

192.168.46.1-CHEN[21/SEP/2013:14:04:48+0800] "get/phpmyadmin/themes/pmahomme/img/tab_hover_bg.pnghttp/1.1" 200502

[Root@test3 ~]# awk ' {print $} '/usr/local/apache2/logs/access_log |sort|uniq-c|sort-nr|head

46/

44/phpmyadmin/

10/phpmyadmin/js/jquery/jquery-1.6.2.js?ts=1359376847

9/phpmyadmin/js/update-location.js?ts=1359376847

9/phpmyadmin/js/jquery/jquery-ui-1.8.16.custom.js?ts=1359376847

9/phpmyadmin/js/jquery/jquery.qtip-1.0.0-rc3.js?ts=1359376847

9/phpmyadmin/js/functions.js?ts=1359376847

8/phpmyadmin/js/cross_framing_protection.js?ts=1359376847

7/phpmyadmin/themes/pmahomme/jquery/jquery-ui-1.8.16.custom.css

7/phpmyadmin/themes/pmahomme/img/sprites.png

5, in the Apache log to find the most visited (load the heaviest) in several time periods (in minutes), and then look at these times which IP access the most?

The/usr/local/apache2/logs/access_log format is as follows

192.168.46.1-CHEN[21/SEP/2013:14:04:48+0800] "get/phpmyadmin/themes/pmahomme/img/tab_hover_bg.pnghttp/1.1" 200502

The following is the amount of access to the time period

[Root@test3 ~]# awk ' {print $} '/usr/local/apache2/logs/access_log |cut-c 9-18 |uniq-c|sort-nr|head

33 2013:13:55

30 2013:13:35

19 2013:13:22

15 2013:13:54

15 2013:13:45

15 2013:13:38

15 2013:13:36

10 2013:12:59

9 2013:13:18

9 2013:13:16

6,apache-related system operations

1, view the Apache process:

PS aux | grep httpd | Grep-v grep | Wc-l

2, view TCP connections for port 80:

Netstat-tan | grep "established" | grep ": 80" | Wc-l

3, through the log to view the number of IP connections, filter repeat:

Cat Access_log | grep "19/may/2011" | awk ' {print $} ' | Sort | uniq-c | Sort-nr

4, the IP connection of the day the highest IP is doing something (the original spider):

Cat Access_log | grep "19/may/2011:00" | grep "61.135.166.230" | awk ' {print $} ' | Sort | uniq-c | Sort-nr | Head-n 10

5, the same day access page row Top 10 URL:

Cat Access_log | grep "19/may/2010:00" | awk ' {print $} ' | Sort | uniq-c | Sort-nr | Head-n 10

6, with tcpdump Sniff 80-port access to see who's the tallest

Tcpdump-i ETH0-TNN DST Port 80-c 1000 | Awk-f "." ' {print $. ' $ "." $ "." $} ' | Sort | uniq-c | Sort-nr

Then from the log to see what the IP is doing:

Cat Access_log | grep 220.181.38.183| awk ' {print '/t ' $} ' | Sort | uniq-c | Sort-nr | Less

7, view the number of IP connections for a time period:

grep "2006:0[7-8]" Www20110519.log | awk ' {print $} ' | Sort | uniq-c| Sort-nr | Wc-l

8, the maximum number of connections in the current Web server 20 IP addresses:

Netstat-ntu |awk ' {print $} ' |sort | uniq-c| Sort-n-R | Head-n 20

9, view the top 10 most visited IP in the log

Cat Access_log |cut-d '-F 1 |sort |uniq-c | Sort-nr | awk ' {print $} ' | Head-n |less

10, view the log more than 100 times the IP

Cat Access_log |cut-d '-F 1 |sort |uniq-c | awk ' {if ($ >) print $} ' |sort-nr |less

11, view the most recently accessed files

Cat Access_log |tail-10000|awk ' {print $} ' |sort|uniq-c|sort-nr|less

12, view pages that are accessed more than 100 times in the log

Cat Access_log | Cut-d '-F 7 | Sort |uniq-c | awk ' {if ($ >) print $} ' | Less

13, list files with a transmission time of more than 30 seconds

Cat Access_log|awk ' ($NF >) {print $} ' |sort-n|uniq-c|sort-nr|head-20

14, list the most time-consuming pages (more than 60 seconds) and the number of corresponding pages

Cat Access_log |awk ' ($NF > && $7~//.php/) {print $} ' |sort-n|uniq-c|sort-nr|head-100