Search engine crawlers User-Agent collection

Baidu Crawlers
* Baiduspider + (+ http://www.baidu.com/search/spider.htm ")

Google Crawlers
* Mozilla/5.0 (compatible; googlebot/2.1; + http://www.google.com/bot.html)
* Googlebot/2.1 (+ http://www.googlebot.com/bot.html)
* Googlebot/2.1 (+ http://www.google.com/bot.html)

Yahoo crawlers (crawlers from Yahoo China and the U.S. headquarters)
* Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
* Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Sina love question Crawler
* Iaskspider/2.0 (+ http://iask.com/help/help_index.html ")
* Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)

Sogou Crawler
* Sogou web spider/3.0 (+ http://www.sogou.com/docs/help/webmasters.htm#07)
* Sogou push spider/3.0 (+ http://www.sogou.com/docs/help/webmasters.htm#07)

Netease Crawler
* Mozilla/5.0 (compatible; yodaobot/1.0; http://www.yodao.com/help/webmaster/spider/

MSN Crawler
* Msnbot/1.0 (+ http://search.msn.com/msnbot.htm)