How can I view spider? How can I know if spider has been to your site?

Source: Internet
Author: User
This can be seen from the logs of your server or virtual host, for example, the complete Use log of the www.com-edu.cn I use has such a record :( IIS Log File Location: c: windowssystem32LogFilesW3SVCXXXXXXXXexyymmdd. log) 220.181.38.198

This can be seen from the log of your server or virtual host, for example, the complete Use log of the www.com-edu.cn of my site has such a record: (IIS Log File Location: c: /windows/system32/LogFiles/W3SVC XXXXXXXX/ex yymmdd. log) 220.181.38.198--[11/Nov/2007: 04: 28: 29 + 0800] "GE

This can be seen from the log of your server or virtual host, for example, the complete Use log of the www.com-edu.cn of my site has such a record: (IIS Log File Location: c: /windows/system32/LogFiles/W3SVCXXXXXXXX/exyymmdd. log)

220.181.38.198--[11/Nov/2007: 04: 28: 29 + 0800] "GET/HTTP/1.1" 200 61083 "-""Baidusp+

(+ Http://www.baidu.com/search/spider.htm )"This indicates that the spider has come to my site. If you want to know if any other search engine spider has come to your site, you can search for the word "spider" in the log file, or search for the spider's IP address.

+; +. NET + CLR + 1.1.4322; +. NET + CLR + 2.0.50727; + CBA; +. NET + CLR + 3.0.04506.648; +. NET + CLR + 3.5.21022) 200 0 0 01:07:48 running 110.110.110.39 GET/images_index/icon_gaozhong.jpg-8077-110.110.110.36 Mozilla/4.0 + (compatible; + MSIE + 8.0; + Windows + NT + 5.2; + Trident/4.0; + GTB6; + Mozilla/4.0 + (compatible; + MSIE + 6.0; + Windows + NT + 5.1; + SV1)

Other frequently encounteredHTTPThe status code list is as follows:

HTTP status code Summary

The status code of the request is successfully processed 2 ×.
200 the server has successfully processed the request and provided the requested webpage.
204 the server successfully processed the request but did not return any content.
Redirect 3 × each request must contain no more than 5 redirects.
301 the requested webpage has been permanently moved to a new location. Code 301 is used when the URLs changes. Save the new URL in the search engine index.
302 the requested webpage is temporarily moved to a new location. The original URL is saved in the search engine index.
304 if the web page is not updated after the last request, use code 304 to notify the search engine robot, which can save bandwidth and costs.
A client error of 4 × indicates that the request may be incorrect, which hinders server processing.
400 the server does not understand the request syntax.
403 the server rejects the request.
404 The server cannot find the requested webpage. This code is often returned for webpages that do not exist on the server.
410 after the requested resource is permanently deleted, the server returns this response. This code is similar to the 404 (not found) code, but is sometimes used to replace the 404 code when the resource exists before and does not exist. If the resource has been permanently deleted, use 301 to specify a new location for the resource.
A server error 5×× indicates an internal error occurred when processing the request. These errors may be server errors rather than request errors.
500 the server encountered an error and could not complete the request.
503 the server is currently unavailable (due to overload or downtime maintenance ). Generally, this is only a temporary status.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.