Talking about: The big search engine spider's Those things

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

Do SEO daily deal with the most is the search engine put out the small robot, also known as search engine spiders, we have to do is to serve them well, let its intention to cast its good.

First, enumerate the major search engine spider's name

Google Spider: Googlebot, Baidu Spider: Baiduspider, Sogou spider: Sogou spider, search spiders: Sosospider,yahoo spider: Slurp,alexa spider: Ia_archiver, MSN Spider: Msnbot,altavista spider: Scooter,lycos spider: Lycos_spider_ (Rex), AllTheWeb spider: Fast-webcrawler,inktomi spider: Slurp, Youdao Spider: Yodaobot and Outfoxbot, Hot land spider: Adminrtspider. Of course, this is only symbolic, even if we analyze the log is not too strange to see, or you want to forbid them to crawl your site is also possible.

Two or one sentences summarizing the habits of spiders

Google spider: not too fond of crawling, but love included.

Baidu Spider: Crawl cautious, included more cautious.

Search spiders: Love to climb pictures, often around the dynamic address does not come out.

Yahoo Spider: Abide by the rules, each time is first climb robots.txt.

The others don't pay much attention to it, they don't say much.

Iii. level of support for robots.txt

All analysis may not be realistic, take disallow:/*?* analysis (prohibit dynamic page crawl).

Google performance: Write a ban will no longer crawl, will be listed in Google Webmaster Tools it wants to crawl by you blocked, the following figure:

Baidu's performance: write a ban after very little climb, but occasionally will climb, believe it works, because less and fewer, a few times a day now a few days.

Sogou Spider: Can be said to be basically not obedient, also do not know is not to eat this rule, said it completely do not eat it also ate a bit, just take the dynamic address of the question mark off, and then climb, a climb is a large, this does not know what it can climb out of something, the following figure:

Search Spider and Yahoo spider seems to be similar, feeling is quite effective, prohibit after no longer has it crawled traces.

Wen Zhangmingrui (http://www.iyoov.com) original share, said is not very comprehensive, only analyzed the dynamic prohibition, some folders prohibit the overall feeling with the above, and the suffix of the prohibition did not try, hope later to give a supplement.