Talking about the spider principle of search engine

Web site can be found in search engines, thanks to search engine spiders capture the credit, the weight of high, updated fast Web site, search engine spiders will often crawl, crawl the latest data, after the search engine data collation, search engine on the Web page, in order to better SEO optimization site, To understand the search engine spider crawling rules is also very important, how to know search engine spiders crawling Web site time, crawling Web pages, crawling reflect it, want to view the Web site IIS log files, IIS default log file in c:\windows\system32\loghttp:// Www.aliyun.com/zixun/aggregation/19352.html ">files, from the view log, you can understand the search engine spider crawling through, such as:

2008-08-19 00:09:12 w3svc962713505 203.171.226.111 get/index.html-80-61.135.168.39 baiduspider+

(+http://www.baidu.com/search/spider.htm) 200 0 64

1, 203.171.226.111 is the search engine spider to ask the website IP,

2, 61.135.168.39 Baiduspider representative, Baidu search engine spider IP is 61.135.168.39,

3, the code in the/index.html on behalf of the search engine Spider-proofing page

4, 2008-08-19 00:09:12 on behalf of the search engine spider crawling Date and time

5, http://www.baidu.com/search/spider.htm represents a number of instructions on Baiduspider

6, w3svc962713505 represents the Web site log folder

7, the code of 200 on behalf of the search engine spider crawling back code representatives, the code can understand the spider crawling after the reflection, the code is as follows:

2XX success

200 normal;

201 normal;

202 normal; Accepted for processing, but processing has not yet completed.

203 normal; Partial information-the information returned is only part of the message.

204 normal; No response-received request, but no information to echo.

3xx redirect

301 Moved-The requested data has a new location and the change is permanent.

302 found-The requested data has a different URI temporarily.

303 See other-the response to the request can be found under another URI, and the response should be retrieved using the Get method.

304 unmodified-The document is not modified as expected.

305 Use proxy-the requested resource must be accessed through the agent provided in the Location field.

306 unused-is no longer in use;

Error occurred in 4xx client

400 Error request-There is a syntax problem in the request or cannot satisfy the request.

401 Unauthorized-The client is not authorized to access the data.

402 Payment required-Indicates that the billing system is valid.

403 prohibited-access is not required even if authorized.

404 Not Found-the server could not find the given resource;

407 Proxy authentication Request-The client must first use the proxy authentication itself.

410 The requested Web page does not exist (permanent);

415 Media type not supported-server denial of Service request because the format of the request entity is not supported.

Error occurred in 5xx server

500 INTERNAL Error-The server cannot complete the request because of an unexpected condition.

501 not executed-the server does not support the requested tool.

502 Error Gateway-server received an invalid response from the upstream server.

503 cannot get service-the server cannot process the request due to temporary overload or maintenance.

Author: seoer Xi yuan site: http://www.noseo.org.cn/