Web site Log Analysis research and analysis of the site's SEO problems

From doing SEO start the boss has been to instill in me: "To do scientific SEO, regardless of whether the flow is up or down to know the source, and to understand these at the same time to determine the direction of Web site traffic must rely on the log analysis, rather than by the sense of what factors use traffic changes. So from the beginning from the data, for the site data real monitoring, from the data to judge the site's problems, and then make corresponding changes. This is one of the reasons that our website has been steadily developing from a straight down flow to a rally up to a rally.

The recent web site traffic has been rising very fast, but the site of a part of the crawl volume is plummeting, but the index is still increasing the flow of the increase in the constant. And that's why, at the end of August, there was a drop in crawl volume, but has been confused by the flow of the front eyes did not do a data analysis of the crawl drop, but also because of doing some optimization and ignore the crawl volume of this piece, until today to this piece of data to do a preliminary analysis, Here is a simple analysis of ideas and results to share to everyone:

First of all, in order to know exactly which day the crawl volume has changed a lot, and there is a more obvious contrast to each month, so first of all, the next log from June to September, 17th from the daily crawl amount of the following figure:

　　

According to this chart, the amount of grab in June is still normal, and at the end of July there is a noticeable increase in the crawl, but after August 10 began to move down and never return to the original peak. At the same time around September 8 began to appear a straight downward trend, from this diagram can be seen, the site on July 26 began to do the optimization of the crawl, at the same time 10 if it is normal fluctuations then the 11th number 12th site should have been adjusted, and this adjustment is to affect the spider on the site crawl. The site began to crawl up from September 4, while number 8th began to fall, and these two time points are still to be analyzed.

Next, according to the previous analysis of the site's convention first analysis of the next site response time trend, as shown in the following figure:

　　

　From the above two charts contrast analysis:

First of all, the response time curve of September is very large, and the amount of grab fluctuates very much, and the wave of the curve is the opposite. The 4th Start response time is falling, while the site's crawl volume is going up, until 8th, the response time has fallen to its lowest point. The crawl volume also rose to the highest level from 9th to begin response time rise, crawl volume also began to decline, and has been down, and then there are rally, although the 13th start site response time and decline, But the amount of fetching has been falling.

From the face of the September analysis can be seen, the September spider on the site to crawl the most obvious problem is the response time of the site, this period of time with the increase in traffic, the pressure of the server is also rising, resulting in the site's response to a lot of decline and impact on the Web site crawl. But the 13th start crawl volume is still falling, there may be other reasons in addition to the site response speed. So the next August data graphs are analyzed:

For August, the number of grab 10th to reach the highest point after the catch began to fall, and has never returned to that high, according to the principle of daily web site regularly updated mass data, the crawl amount should be up, but it is falling. In contrast, the following figure found that the response time of August 26th was very smooth and relatively fast. From this comparison it can be explained that the August crawl volume fluctuations and the response speed of the site has no relationship.

So then the next spider in August to crawl the site back to the status code: the following figure:

　　

As can be seen from the above figure, spider in August on the site's crawl process appeared 2% of the 500 status code. According to statistics, at this time 2% of the data are all from August 11 to 13th. In other words, this three-day site has been a serious site internal error, the server can not complete the request.

Therefore, the following changes to the site before and after August 10 are analyzed: Through the analysis of the site code (because of the technology, the site code changes time is easy to find), found that on August 11, the site added a function, and this feature takes three of tables, and in the number of ignore a link, Therefore, the site features updated three days after the creation of a lot of error pages until three days after the discovery to get rid of.

Combining the above two analyses, we can see that the recent decrease in crawl volume is mainly two reasons:

1, a large number of visits to the Web site to reduce response speed.

2, the website internal error causes the spider to produce a new unfavorable judgment to the website.

Of course, this is only a preliminary analysis of the two conclusions, the log will continue to analyze because in the subsequent analysis also found that the recent Web site generated a lot of junk links, and the source of these links may also be a problem with the program. All problems are slowly surfacing from the analysis of IIS logs. The reason is found, the solution has been very simple, the corresponding problem to deal with it.

Copyright statement: Please respect the original content of the author, if you want to reprint, please specify the original address of the article, author/Sina Weibo: Chushen

Original address: http://blog.sina.com.cn/s/blog_74db7fd701018arl.html