Baidu Webmaster Platform: How to identify baiduspider to ensure that the site is properly crawled

A5 Webmaster Network (www.admin5.com) April 24 News, there are many sites have been mistakenly sealed baiduspider behavior, the site is included in the impact. Even have a website to reflect Baidu spider behavior unusual patronage too frequently. Baidu Webmaster Platform recently said Baiduspider to the site's crawl and no exception, and published tutorials to help webmaster identify Baiduspider, and add white list.

Last week, Baidu Webmaster platform received a webmaster for help, said mistakenly banned Baiduspider IP, ask whether there is a way to get Baiduspider all IP, intend to put into the white list to protect, prevent again false seal. Here to tell you webmaster, baiduspider IP pool is constantly changing, we can not provide complete IP.

In addition, before there is a webmaster sent to doubt that Baiduspider patronize too frequently, has exceeded the capacity of the server. and Baidu webmaster platform tracking found, Baiduspider to the site of the crawl and no anomaly, that only spider very likely is a Li ghost.

So, webmaster How to judge by IP to this spider is not from Baidu search engine?

This problem can be solved by the way of DNS counter check. According to the different verification methods of the platform, such as Linux/windows/os three kinds of platform authentication methods are as follows:

1, under the Linux platform, you can use the host IP command to reverse IP to determine whether the baiduspider from the crawl. The hostname of Baiduspider is named in *.baidu.com or *.baidu.jp format, not *.baidu.com or *.baidu.jp as impersonation.

2, in Windows platform or IBM OS/2 platform, you can use nslookup IP command to reverse IP to determine whether from the Baiduspider crawl. Open the command processor input nslookup xxx.xxx.xxx.xxx (IP address) can resolve IP, to determine whether from the Baiduspider crawl, baiduspider hostname to *.baidu.com or *.baidu.jp The format of the name, not *.baidu.com or *.baidu.jp is to impersonate.

3, under the Mac OS platform, you can use the dig command to reverse IP to determine whether to come from the Baiduspider crawl. Open the command processor input dig xxx.xxx.xxx.xxx (IP address) can resolve IP, to determine whether from the Baiduspider crawl, baiduspider hostname in *.baidu.com or *.baidu.jp format named , a *.baidu.com or *.baidu.jp is a pseudo.