Search engine spider spider related knowledge summary

Intermediary transaction SEO diagnosis Taobao guest Cloud host technology Hall

What is Baiduspider?

Baiduspider is a Baidu search engine of an automatic program. Its role is to access the Internet's HTML Web page, set up an index database, so that users can search the Baidu search engine to your site's Web page.

Baiduspider what is the pressure on a Web server to access?

Baiduspider automatically adjusts the access density based on the server's load capacity. After a continuous visit, Baiduspider pauses for a moment to prevent increased server access pressure. So under normal circumstances, Baiduspider will not cause too much pressure on your site's servers.

Why Baiduspider constantly crawl my site?

For new or continuously updated pages on your site, Baiduspider will continue to crawl. In addition, you can check the Web site access log baiduspider access to the normal, to prevent malicious impersonating baiduspider to frequently crawl your site. If you find Baiduspider is not normal crawl your site, please feedback to webmaster@baidu.com, and please try to give Baiduspider access to your station log, so that we can track processing.

I don't want my site to be visited by Baiduspider.

Baiduspider comply with Internet protocol. You can use the robots.txt file to completely prohibit Baiduspider from accessing your Web site, or to prevent Baiduspider from accessing some of the files on your site. Note: Prohibit Baiduspider to visit your website, will make your website on the web, in Baidu search engine and all Baidu search engine Services search engines can not be searched. For the writing method of robots.txt, please refer to:

Why my website has added robots.txt, still can search out in Baidu?

Because the search engine index database update takes time. Although Baiduspider has stopped visiting the pages on your site, the index information that has been set up in the Baidu search engine database may take 2-4 weeks to clear. Also check to see if your robots are configured correctly.

What is the name of Baidu Spider in robots.txt?

"Baiduspider" is all lowercase letters.

How long will Baiduspider crawl my web page?

Baidu Search engine weekly update, Web page depending on the importance of a different update rate, frequency between a few days to January, Baiduspider will again visit and update a Web page.

Know what is Baidu Spider, then how to know whether the spider has come to your station?

This can be seen from your server or virtual host log, for example, I use the full use of the virtual host log has such a record:

220.181.38.198--[11/nov/2007:04:28:29 +0800] "get/http/1.1" 61083 "" baiduspider+ (+http://www.baidu.com/search /spider.htm) "This means that Baidu spiders have come to my station, if you also want to know that there are no other search engine spider came to your station, you can search in the log file" Spider "The word, or search the spider's IP, I found that Sogou also came to my station, The IIS log is the same as the Apache log and can be traced.

All kinds of spider IP Collection, not necessarily completely accurate.

Ordinal IP Comment

1 202.106.186.* 163 Spiders

2 202.108.36.* 163 Spiders

3 202.108.44.* 163 Spiders

4 202.108.45.* 163 Spiders

5 202.108.5.* 163 Spiders

6 202.108.9.* 163 Spiders

7 220.181.12.* 163 Spiders

8 220.181.13.* 163 Spiders

9 220.181.14.* 163 Spiders

10 220.181.15.* 163 Spiders

11 220.181.28.* 163 Spiders

12 220.181.31.* 163 Spiders

13 222.185.245.* 163 Spiders

14 202.165.100.* 3721 Spiders

15 220.181.19.* Baidu Spider

16 159.226.50.* Baidu Spider

17 202.108.11.* Baidu Spider

18 202.108.22.* Baidu Spider

19 202.108.23.* Baidu Spider

20 202.108.249.* Baidu Spider

21 202.108.250.* Baidu Spider

22 61.135.145.* Baidu Spider

23 61.135.146.* Baidu Spider

64.124.85.* become.com

61.151.243.* the Spider

202.165.96.* gais.cs.ccu.edu.tw

216.239.33.* Google Spider

216.239.35.* Google Spider

216.239.37.* Google Spider

216.239.39.* Google Spider

216.239.51.* Google Spider

216.239.53.* Google Spider

216.239.55.* Google Spider

216.239.57.* Google Spider

216.239.59.* Google Spider

64.233.161.* Google Spider

Panax 64.233.189.* Google spider

66.102.11.* Google Spider

66.102.7.* Google Spider

66.102.9.* Google Spider

66.249.64.* Google Spider

66.249.65.* Google Spider

66.249.66.* Google Spider

66.249.71.* Google Spider

66.249.72.* Google Spider

72.14.207.* Google Spider

61.135.152.* iask Spider

65.54.188.* MSN Spider

65.54.225.* MSN Spider

65.54.226.* MSN Spider

Wuyi 65.54.228.* MSN Spider

65.54.229.* MSN Spider

207.46.98.* MSN Spider

207.68.157.* MSN Spider

194.224.199.* Noxtrumbot

220.181.8.* Outfox

221.239.209.* Outfox

Psbot 217.212.224.*

219.133.40.* QQ Spider

202.96.170.* QQ Spider

202.104.129.* QQ Spider

61.135.157.* QQ Spider

219.142.118.* Sina Spider

219.142.78.* Sina Spider

61.135.132.* Sohu Spider

220.181.26.* Sohu Spider

220.181.19.*

61.135.158.* Tom Spider

66.196.90.* Yahoo Spider

66.196.91.* Yahoo Spider

68.142.249.* Yahoo Spider

68.142.250.* Yahoo Spider

68.142.251.* Yahoo Spider

202.165.102.* Yahoo China spider

202.160.178.* Yahoo China spider

202.160.179.* Yahoo China spider

202.160.180.* Yahoo China spider

202.160.181.* Yahoo China spider

202.160.183.* Yahoo China spider

72.30.101.* Yahoo Spider

72.30.102.* Yahoo Spider

Eight 72.30.103.* yahoo spider

72.30.104.* Yahoo Spider

72.30.107.* Yahoo Spider

72.30.110.* Yahoo Spider

72.30.111.* Yahoo Spider

72.30.128.* Yahoo Spider

72.30.129.* Yahoo Spider

72.30.131.* Yahoo Spider

72.30.133.* Yahoo Spider

72.30.134.* Yahoo Spider

72.30.135.* Yahoo Spider

72.30.216.* Yahoo Spider

72.30.226.* Yahoo Spider

72.30.252.* Yahoo Spider

72.30.97.* Yahoo Spider

72.30.98.* Yahoo Spider

72.30.99.* Yahoo Spider

74.6.74.* Yahoo Spider

Search Spiders in 99 202.108.4.*

Search Spiders in 100 202.108.4.*

Search Spiders in 101 202.108.33.*

Search Spiders in 102 202.96.51.*

Search Spiders in 103 219.142.53.*

Ordinal IP Comment

1 202.106.186 163

2 202.108.36 163

3 202.108.44 163

4 202.108.45 163

5 202.108.5 163

6 202.108.9 163

7 220.181.12 163

8 220.181.13 163

9 220.181.14 163

10 220.181.15 163

11 220.181.28 163

12 220.181.31 163

13 222.185.245 163

14 202.165.100 3721

220.181.19 Baidu

159.226.50 Baidu

202.108.11 Baidu

202.108.22 Baidu

202.108.23 Baidu

202.108.249 Baidu

202.108.250 Baidu

61.135.145 Baidu

61.135.146 Baidu

64.124.85 become.com

61.151.243

202.165.96 gais.cs.ccu.edu.tw

216.239.33 Google

216.239.35 Google

216.239.37 Google

216.239.39 Google

216.239.51 Google

216.239.53 Google

216.239.55 Google

216.239.57 Google

216.239.59 Google

64.233.161 Google

Panax 64.233.189 Google

66.102.11 Google

66.102.7 Google

66.102.9 Google

66.249.64 Google

66.249.65 Google

66.249.66 Google

66.249.71 Google

66.249.72 Google

72.14.207 Google

61.135.152 Iask

65.54.188 MSN

65.54.225 MSN

65.54.226 MSN

65.54.228 MSN

65.54.229 MSN

207.46.98 MSN

207.68.157 MSN

194.224.199 Noxtrumbot

220.181.8 Outfox

221.239.209 Outfox

Psbot 217.212.224

219.133.40 QQ

202.96.170 QQ

202.104.129 QQ

61.135.157 QQ

219.142.118 Sina

219.142.78 Sina

61.135.132 Sohu

220.181.26 Sohu

61.135.158 Tom

66.196.90 Yahoo

66.196.91 Yahoo

68.142.249 Yahoo

68.142.250 Yahoo

68.142.251 Yahoo

202.165.102 Yahoo

202.160.178 Yahoo

202.160.179 Yahoo

202.160.180 Yahoo

202.160.181 Yahoo

202.160.183 Yahoo

72.30.101 Yahoo

72.30.102 Yahoo

Bayi 72.30.103 Yahoo

72.30.104 Yahoo

72.30.107 Yahoo

72.30.110 Yahoo

72.30.111 Yahoo

72.30.128 Yahoo

72.30.129 Yahoo

72.30.131 Yahoo

72.30.133 Yahoo

72.30.134 Yahoo

72.30.135 Yahoo

72.30.216 Yahoo

72.30.226 Yahoo

72.30.252 Yahoo

72.30.97 Yahoo

72.30.98 Yahoo

72.30.99 Yahoo

74.6.74 Yahoo

202.108.4 Zhongsou

202.108.33 Zhongsou

Zhongsou 202.96.51

102 219.142.53 Zhongsou

-------------Baidu-------------

31.135.145.*

61.135.145.*

61.135.146.*

159.226.50.*

202.108.11.*

202.108.22.*

202.108.23.*

202.108.249.*

202.108.250.*

220.181.19.*

-------------Yahoo China-------------

66.196.90.*

66.196.91.*

68.142.249.*

68.142.250.*

68.142.251.*

72.30.101.*

72.30.102.*

72.30.103.*

72.30.104.*

72.30.107.*

72.30.110.*

72.30.111.*

72.30.128.*

72.30.129.*

72.30.131.*

72.30.133.*

72.30.134.*

72.30.135.*

72.30.216.*

72.30.226.*

72.30.252.*

72.30.97.*

72.30.98.*

72.30.99.*

74.6.74.*

202.165.102.*

202.160.178.*

202.160.179.*

202.160.180.*

202.160.181.*

202.160.183.*

-------------Google-------------

64.233.161.*

64.233.189.*

66.102.11.*

66.102.7.*

66.102.9.*

66.249.64.*

66.249.65.*

66.249.66.*

66.249.71.*

66.249.72.*

72.14.207.*

216.239.33.*

216.239.35.*

216.239.37.*

216.239.39.*

216.239.51.*

216.239.53.*

216.239.55.*

216.239.57.*

216.239.59.*

-------------MSN-------------

65.54.188.*

65.54.225.*

65.54.226.*

65.54.228.*

65.54.229.*

207.46.98.*

207.68.157.*

-------------Search-------------

202.108.1.*

202.108.2.*

202.108.3.*

202.108.4.*

202.108.33.*

202.96.51.*

219.142.53.*

-------------QQ-------------

219.133.40.*

202.96.170.*

202.104.129.*

61.135.157.*

-------------163-------------

202.106.186.*

202.108.36.*

202.108.44.*

202.108.45.*

202.108.5.*

202.108.9.*

220.181.12.*

220.181.13.*

220.181.14.*

220.181.15.*

220.181.28.*

220.181.31.*

222.185.245.*

-------------Other-------------

64.124.85.* become.com

61.151.243.*

202.165.96.* gais.cs.ccu.edu.tw

61.135.152.* Iask

194.224.199.* Noxtrumbot

220.181.8.* Outfox

221.239.209.* Outfox

217.212.224.* Psbot

219.142.118.* Sina

219.142.78.* Sina

61.135.132.* Sohu

220.181.26.* Sohu

61.135.158.*, Tom.