The nine kinds of spiders that appear on the website log

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

This is the name of the spider that I analyzed in the space IIS log on the major search engines

Probably everyone except the first one is common.

1.gigabot/3.0+ (http://www.gigablast.com/spider.html) This search for a while, as if the Gigabot search engine crawler. Has it been bought by Google?

2. (Compatible;+msie+7.0;+windows+nt+5.1;++embedded+web+browser+from:+http://bsalsa.com/;+mozilla/4.0 ( compatible+mozilla/4.0 (compatible-embeddedwb+14.59+http://bsalsa.com/+embeddedwb-+14.59++from:+http:// bsalsa.com/+;+.net+clr+1.1.4322)

This I first thought is a spider, inquires after, some people said is bsalsa.com development Windows platform Delphi related software, the win host will have the record, does not know that potential energy to explain.

3.mozilla/5.0+ (compatible;+yodaobot/1.0;+http://www.yodao.com/help/webmaster/spider/;+)

This is NetEase Youdao robot, do not see the log, I can not think of a youdao search it.

4.sogou+web+spider/4.0 (+http://www.sogou.com/docs/help/webmasters.htm#07)

This is Sogou spider, every day scan my website (http://www.aistxt.com) Hundreds of times, grabbed 2,438 pages

, the average IP per day is 3. SouGou rank is not very low, just 43.

Scanning volume is very large, the dynamic link to the site is very heavy burden.

5.iaskspider/2.0 (+http://iask.com/help/help_index.html)

mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)

Sina love to ask crawler and Sogou almost, no meaning.

6.mozilla/5.0+ (compatible;+yahoo!+slurp+china;+http://misc.yahoo.com.cn/help.html)

This is the Chinese Yahoo, the following is the U.S. headquarters of the crawler

HTTP://HELP.YAHOO.COM/HELP/US/YSEARCH/SLURP)

7.mediapartners-google This is GG click AD Crawler

The following is the protagonist of the Google crawler

8.mozilla/5.0+ (compatible;+googlebot/2.1;++http://www.google.com/bot.html)

9.baiduspider+ (+http://www.baidu.com/search/spider.htm)

The last one is to let me the most headache, every day the home page is visited dozens of times, but the inner pages are rarely visited.

Some people analysis that Baidu algorithm problem, resulting in Baidu crawler on the same page will be issued multiple requests (especially the homepage).

10. Spiders not found on the log: MSN crawler and Alexa rank crawler

Has Microsoft abandoned its search program?

Search.live.com

This site has not opened, the emergence of China netcom hint does not exist in the Web site, a bit surprising.

As for the Alexa ranking crawler, my website ranking is not enough, people are naturally not back.