A way to view the visiting records of search engine spiders in the Web log

Absrtact: For many of the novice to do the Web site, have not been systematically taught by the network technology and build station knowledge to learn, do the site is by self-study, encounter problems in the forum post questions, not understand the site optimization, for more basic operations through the site log

For many novice Web sites, have not been systematically taught by the network technology and build station knowledge to learn, do the site is by self-study, encounter problems in the forum post questions, not understand the site optimization, for more basic operations-through the Web site log to see the spider visit the situation do not know where to look, how to see. Two days ago to see a lot of people post questions, reply to the answer is more refined, not specific, the questioner is still foggy, now I have to use their own web site to operate the system once, submitted to you for reference, there is wrong place, please criticize.

1, open the FTP login software, I use the FLASHFXP, login space ftp

Login to FTP, you will find a Wwwlogs folder under the root directory, (some weblog, note: Different server space generated log file directory name is not the same, for reference only, the general folder contains log characters is the Journal folder).

2, open the Wwwlogs folder, there are some date format features as the filename of the. gz suffix end of the file, these are the log files we need to download to the local.

3, download to the computer desktop, decompression open, inside is a notepad format file, open the file, see the following image of the code, I downloaded the March 7 file.

4, Analysis code

Above Figure 1 is Baidu Spider's IP address;

2 is the spider visit Date time (March 6, 2012 1:21 22 seconds), March 7 log files are recorded from the early morning of March 6, to March 7 1:11 39 seconds of the entire time period of the N multiple visit records);

3 is Baidu Spider Baiduspider

4 is the address of the webpage visited by my website;

5 is Sogou Sogou Spider's visit, also can see the time and visited the webpage.

If it's a simple view, you can search for Baiduspider in Notepad, and if you want to analyze it accurately, you can use some specialized analysis software. Analysis of what time period Baidu Spider to the most frequent, then we will update our website content in this time period, it is easy to be included in Baidu.

Through the analysis of Spider visit records, you can understand the general situation of the site, and no longer for Baidu does not release the page or not included in the problem and distress.

Spider visit Normal, can be exact, search engine on your station is very friendly, adhere to update their website, there will be a good collection.

Note: Some shared IP space may not support logging, for independent IP virtual host to provide daily log download, but there is no log function can refer to the use of spiders crawling plug-ins some methods to analyze.