Sina Blog update robot file "shielding door" misunderstanding elimination

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

A few days ago there is an article that because many AdSense spam information to do outside the chain, resulting in spam blog flooded Sina, so Sina blog began shielding Baidu Spider, netizens found that most of Sina blog snapshots in Baidu has not been updated, by looking at the Sina file (http://blog.sina.om.cn/robots.txt) found its content:

#####################################################

# SINA BLOG prohibit search engine included configuration file

# File: ~/robots.txt

# write: Wolf

# Date: 2005-03-24

#####################################################

#开放百度的搜索引擎的User-agent Code, * represents all ###########

User-agent:baiduspider

#限制不能搜索的目录, Disallow: Open all directories for empty

Disallow:/admin/

Disallow:/include/

Disallow:/html/

Disallow:/queue/

Disallow:/config/

#限制的搜索引擎的User-agent Code, * represents all ###########

User: *

#临时限制对所有目录的禁止抓取

Disallow:/

Because the document has a sentence: User-agent:baiduspider so many webmaster, Sina has completely shielded Baidu spider, in the future rely on the Sina blog to do outside the chain of methods has not been done.

After seeing the news, I felt a few doubts:

1, Sina as a world-renowned Chinese portal site, its data processing capacity, the management of illegal content is enough to cope with the current spam information, and shielding Baidu Spiders will lead to a large reduction in traffic, which is inconsistent with Sina's development strategy.

2, the date of the document is: 2005-03-24, seems to be Sina blog just online when the document was established, if the recent Sina network management has modified the robots, the date should be changed to a new date.

3, familiar with the writing of the friends know that the robots.txt file is only set to prohibit Baidu spider access to the background folder, and no other restrictions.

Based on the above doubts, I feel that the point of view in the article there is a mistake, things should not be so bad, sure enough, on August 18, netizens found Sina blog quietly replaced the robots.txt file, the content should read:

#开放百度的搜索引擎的User-agent Code

User-agent:baiduspider

#限制不能搜索的目录, Disallow: Open all directories for empty

Disallow:/admin/

Disallow:/include/

Disallow:/html/

Disallow:/queue/

Disallow:/config/

#开放bing. com Search Engine user code

User-agent:msnbot

#限制不能搜索的目录, Disallow: Open all directories for empty

Disallow:/admin/

Disallow:/include/

Disallow:/html/

Disallow:/queue/

Disallow:/config/

User-agent:bing

#限制不能搜索的目录, Disallow: Open all directories for empty

Disallow:/admin/

Disallow:/include/

Disallow:/html/

Disallow:/queue/

Disallow:/config/

#限制的搜索引擎的User-agent Code, * represents all ###########

User: *

#临时限制对所有目录的禁止抓取

Disallow:/

#限制不能搜索的目录, Disallow: Open all directories for empty

# #Disallow:/admin/

# #Disallow:/include/

# #Disallow:/html/

# #Disallow:/queue/

# #Disallow:/config/

#开放搜索的目录有 ####################################

# /

#/advice/

#/help/

#/lm/

#/main/

#/myblog/

Can see the new modified robots.txt file in the writing has fully supported all kinds of mainstream search engine crawl blog content, an open Sina blog back!

In the case of previous days, individual speculation may be due to the following reasons:

1, a few days ago most Sina blog by K, may be due to Baidu adjustment algorithm caused.

2, The original robots.txt file is likely to be Sina blog just when it was built, because at that time Google has not officially entered China, Chinese search or Baidu a single large, so the content is set mainly for Baidu, since the document has not been modified, until recently by netizens found, Sina staff only remembered to revise.

Although this "screen door" is just a false alarm, but also to the webmaster sounded the alarm, do not blindly junk spam information, and pollution of the Internet environment and the cost of the network service provider resources, careful lays! It is more effective to cultivate 10 high quality blogs than to raise 100 spam blogs!

The above is my point of view, I hope to have a friend reprint the time to retain one of my links http://www.jfbest.com more than one retention of luck, more than a word of mouth, more than a success, thank you!