Analysis of the response of major search engines to robots.txt files

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

Have not seen a specific robots.txt file on the implementation of the various search engines, recently, I specifically for the robots.txt file for the major search engines did an experiment, see them this is the clear agreement, the implementation of the actual implementation is what the situation. Below I on the major search engine on the robots.txt file response test data, for you to explain in detail, for everyone's reference.

First of all this site in Baidu, Google,yahoo, Sogou, search, Bing, Youdao included are normal, and then I directly in the robots.txt file set to prohibit all search engine spiders to visit and crawl included this site. Since the date the file was put on the day, the fastest response is Google, which is also in my expectation, Google no matter what things to do, processing finished to achieve their desired effect of reaction speed is a first-class ideal. Even when doing Google rankings, their operating speed and expected Googlre ranking effect of the reaction speed synchronization, the most satisfying mo too. This is also a lot of people do not understand, why do optimization sometimes takes a lot of time, because the different search engines on the information crawling response is very different, so sometimes, you have done everything. You still have to wait for search engine to crawl, search engine crawl only then begin to analyze the quality of your information, quality Gao will increase to the website weight up.

So in the shortest possible time to make the most efficient and ideal SEO results, should have the right decision-making ability and efficient implementation capacity, from the time of the optimization of the site, you do everything is to the site can play an effect, these things to do, on the site can play much role in the process of optimization, In response to the different phenomena, can be timely to make accurate processing, at all times to the entire site to optimize the operation of the risk assessment to control (because the search engine has a lot of traps waiting for you to jump).

Then said other search engine on the robots.txt file response, Google only spent 4 days, deleted the entire domain name of the Web site, followed by Tencent Soso after 6 days also deleted the entire domain name of the Web site, then Bing, Sogou 7 days to delete the domain name included. So far NetEase Youdao, Baidu did not have the slightest action, in Baidu only a few keywords ranking a little back, but did not delete any domain name pages included.

The data from this experimental analysis, I hope that you can play a reference role in the operation of such a similar robots.txt problem in the future. At present, in view of Baidu and the Youdao has not changed the slightest, so need to continue to observe a period of time, such as late have the results, and then do a detailed robots.txt Baidu feature article description, I hope you continue to pay attention to. This article Crystal Maple Leaf SEO Blog original, retain the Copyright http://www.seocnz.com/seo/reprint please keep the link.