Use shell analysis Nginx log Baidu Web page Spider List page visit situation

#!/bin/bash#desc: this scripts for baidunews-spider#date:2014.02.25#testd in  centos 5.9 x86_64#saved in /usr/local/bin/baidu-web.sh#written by [email  Protected] www.zjyxh.comdt= ' date -d  "Yesterday"  +%m%d ' if [ $1x != x  ] ;then  if [ -e $1 ] ;then     grep  -i  "baiduspider/2.0"  $1 > baiduspider-${dt}.txt     num= ' Cat baiduspider-${dt}.txt|wc -l '      echo  ' Baiduspider number  is ${num},file is baidu-${dt}.txt "     cat baiduspider-${dt }.txt|awk  ' {print $7} ' |sort |uniq -c|sort -r > ' ls ${1}|cut -c  1-10 '-${dt}.txt     echo  "$1 was done"     else        echo  "$1 not exsist!"   fielse     echo  "Usage: $0 file_path" fi

This time with the shell analysis Baidu Web spider with Baidu News spider A method, nothing is to change the key words from Baiduspider-news to baiduspider/2.0.

This article is from the "Tri Xiaohui blog" blog, make sure to keep this source http://coralzd.blog.51cto.com/90341/1590956

Use shell analysis Nginx log Baidu Web page Spider List page visit situation