New station does not include do not ignore the robots.txt file

Today, I accidentally saw a webmaster friend, talking about why he was not included in the new station, from the cause of the problem to solve the problem of a series of work. Feel more typical, many people may encounter, now posted out to share to everyone.

problem Reason:

The webmaster in the new station soon after, the site has a major internal bug (loophole). The webmaster first used robots.txt files to screen the search engine's crawl, and then to the site BNG repair.

Problem Handling:

When the site's bug fixes completed, the user again to the robots.txt file was modified to allow all search engines to crawl. On the surface to see this step can wait for the search engine to crawl content, but one weeks past the site a little reaction has not, view the Web site log, even spiders have been to the wood. So simply delete the robots.txt file of the website.

problem principle:

Through Baidu Webmaster tool detection, found the original reason is hidden in robots.txt file, when he found that the site has BNG, he banned the search engine crawl, and spiders found this site is forbidden to crawl, will default a time period not to climb. Baidu Webmaster Tools show that the site robots.txt file update time to stay in the ban search engine crawling that day, after all did not update. This also proves that the spider is crawling to a page that is forbidden to crawl, it is not crawling, and by default the next time to crawl in the interval, This can explain a lot of webmaster why in the lifting of the robots.txt file limit spiders crawl, also want a paragraph will be back to collect new content reason, but previously Baidu did not out webmaster tools We do not know just.

Solution:

Know the principle, then the solution is much easier. Please take a look at the following picture:

First to open Baidu Webmaster Tools-robots.txt Tools, in the update can be, the new station does not include must not ignore robots.txt files.

This article Copyright: SEO Academy www.lyylx.com reprint Please specify!