Actual Combat analysis: Modify the robots file Baidu Google's response to the site

To do the site has been done for such a long time, webmaster can encounter things have met, the most common is nothing but the site is down right, the site snapshot does not update the main keyword rankings decline, and the number of outside chain to reduce, and so on, these problems are often due to the initial preparation of the site is not ready on the line results, Lead to the late replacement of the site plate or frequent changes to other spiders often crawling files caused by today's small series and we discuss the changes in the Web site after the impact of the website and some of the search engine response.

After the first modification of the robots file Baidu Baidu Google response

Snapshot updated, found that is 4 days a cycle, the daily flow is very large, but the user in my Shield/user/file, spiders can not crawl, the largest flow is the user center, I tried to change the robots file, so that the spider can access the file. Google previously included 120,000, remember a very profound time is to modify the robots.txt file, Google was included already more than 120,000, the next day nothing to continue to more than 1000 per day to capture the speed, and then see Baidu this aspect, also nothing. But the unfortunate thing happened ... Snapshot update once again site: The domain name found, Google included 60,000 (the original is more than 120,000), Baidu this aspect is also changed, the original 35,600 to update the snapshot of the 49,000 after the collection, was modified to put the robots file to block Group Buy page (that is called), At that time, I do not know what is the cause of Google included a large reduction, Baidu included a large increase, but continue to test, only to find the cause of death ...

Two times to modify a robots file angered spiders, Google direct K station

The beginning of the station is not built for a long time always like to modify this change that, the home is moving up, my station security robots are also I "decorated" is extremely rich, things happen at the end of the year, the robots file shielding Baidu Google search outside the spider (at that time Sogou Rogue grabbed to achieve the site traffic is not enough), The third day after the modification, from Google included this way down the Tao Bao Canton station until the day of 60,000 to 1200, see the heart was cool a cut, so far Google has not crawled the site, included in the 1000 hovering.

Small knitting also has a second to collect the blog station is also because of the modification of the robots file, Google Direct K station, at that time the station to modify the file too frequent, resulting in counterproductive effect, Baidu also lasted a week did not update the snapshot, spiders are extremely sensitive to the robots file, this is the new station can not touch the optimization of mine area.

At the beginning of the construction station should be a comprehensive set of files to prevent late changes

Later modifications to the robots are an unfriendly performance of the major search engines, in particular, Google, with the engine of humanity, search engine is also a "temper", when your site produced a habit, your site a little change is likely to cause the search engine aversion, like the modified robots, The modification of the robots not only has an impact on the content, but also the weight of the site can not be ignored. Modify the robots will cause a snapshot of the site does not update, included reduced, the number of chain down, the site's main keywords down even the top hundreds of can not find the keyword ranking.

Spiders to your station is to look at the root of the Web site is the first file, the spider's time is also very urgent, such as Google's global search engine companies to go to thousands of sites every day to crawl the content of the Web site, frequently modify the file is to modify your website and the agreement between the spider, for The general spider will follow the robots protocol, according to the rules you set up to crawl the allowed Web page files and content.

After a few months of small series of actual combat results I suggest: robots are in the beginning of the site should be set up, do not include a lot of sites or weights up when you want to modify to modify, unless you do not care from the search engine to those traffic.

Open Shop Save money net Ye Weiqing original article http://www.yeweiqing.com/this article Copyright Ye Weiqing @ Open Shop Save money Net all, welcome reprint, reprint please indicate author and source. Thank you