A closer look at the actual effect of robots.txt rules

Absrtact: October 20, I made a large-scale adjustment to the blog, just like to treat the growing trees in general pruning shear leaves, in order to have a better growth trend. The robots.txt was fully utilized by me. Now a work week is about to pass, robots.txt document

On the October 20, I made a big adjustment to the blog, just like to treat the trees in need of growing, pruning and cutting leaves, in order to have a better growth trend. The robots.txt was fully utilized by me. Now a work week is about to pass, robots.txt file rules are correct, whether it has been in effect? Baidu Google and other search engine response to the robots.txt rules for index adjustment? As a webmaster I need to study deeply, in order to dynamically grasp the status of blog.

The survey found that Google's response to the robots.txt faster, the third day in the Web site management tools found signs. Baidu performance is not satisfactory, said that Baidu does not recognize robots.txt rules that is nonsense, but the reaction cycle so long, it will inevitably leave a slow reaction to suspicion.

Look at Google's response to robots.txt rules

Of the adjustments made on 20th, there were two rules I deleted later. Open my blog's robots.txt, and 20th to adjust to write down the contrast, know the changes. The reason for this adjustment is that, as in the 20th wording, the next day I discovered that the sitemaps of three selected addresses in Webmaster Tools--Blocked by robots.txt file rules--is not necessary. The screenshot was not found, the following three selected can look at:

Sitemap Site Map Submitted

Responding to robots.txt rules, Google stopped crawling over more than 2000 connection addresses. That more than 500 can not find the address, because some time ago deleted article tags tags sequelae. Here is a screenshot:

More than 2000 connection addresses are restricted by robots.txt rules

Rummaging through every page, no problems found-except for some/?p= short connection that people love dearly, all perfect flawless. Strictly speaking, should be the robots.txt rule does not have the question, Gu ge executes the robots.txt rule. Google query "site:***.com inurl:?p" only found incomplete 14 (no title or summary). These addresses will be cleared in the near future.

See Baidu's response to robots.txt rules

20th has robots.txt file rules, what is the situation?

The rules of 20th, the above figure is not known to cross, or I was dazzled? I checked IIS log records, Baidu after 20th has several times downloaded robot.txt files, the server returned is 200 success status code. No wonder Baidu does not recruit you to see the webmaster.

Baidu "Dear stationmaster, I am your father" high attitude, whether should change?