Linux/nginx How to view search engine spider crawler behavior _linux Shell

Summary

SEO optimization of the first step is to let spider spiders often come to your site to patronize, the following Linux commands can let you know the spider crawling situation clearly. Below we analyze for Nginx server, log file directory:/usr/local/nginx/logs/access.log,access.log This file should be the last day of the log situation, first please look at the log size, if very large ( More than 50MB) it is recommended that you do not use these commands for analysis, as these commands consume the CPU or are updated and executed on the analysis machine to avoid affecting the speed of the Web site.

Linux shell command

1. Baidu Spider crawls the number of times

Cat Access.log | grep Baiduspider | Wc

The leftmost value shows the number of crawls.

2. Baidu Spider's detailed record (Ctrl C can terminate)

Cat Access.log | grep Baiduspider

You can also use the following command:

Cat Access.log | grep Baiduspider | Tail-n 10
Cat Access.log | grep Baiduspider | Head-n 10

Just look at the last 10 or the top 10, which will tell you the time and date of the log file's start record.

3. Baidu Spider Crawl home detail records

Cat Access.log | grep Baiduspider | grep "Get/http"

Baidu Spider seems to home very much love every hour to patronize, and Google and Yahoo spider more like inside page.

4. Baidu Spider sex record time point distribution

Cat Access.log | grep "Baiduspider" | awk ' {print $} '

5. Baidu Spider Crawl page by number descending sequence table

Cat Access.log | grep "Baiduspider" | awk ' {print $} ' | Sort | uniq-c | Sort-r

The text of the baiduspider changed into Googlebot can view Google's data, in view of the specificity of the mainland, we should be more concerned about Baidu log.

Attached: (mediapartners-google) Google adsense Spider's detailed crawling record

Cat Access.log | grep mediapartners

What is Mediapartners-google? Google AdSense ads can be related to the content, because each contains AdSense ads were visited, soon there is a mediapartners-google spider came to this page, so after a few minutes and then refresh can show relevance ads, really awesome ah!

Ps:linux under Nginx How to enable Web log, view spider crawler

The default path is the one you specified at installation time.

If you use an installation package such as Lnmp

You can do it under the shell.

Whereisnginx

After finding the appropriate path

In Nginx under the Conf folder look at the configuration file, log file if recorded

There are paths in the config file ~