Python crawler encounters status code 304,705

304 What is a status code?

If the client sends a conditional GET request and the request is allowed, and the contents of the document (since the last time it was accessed or according to the requested condition) have not changed, the server should return the 304 status code. The simple expression is: The client has performed a get, but the file has not changed.

Under what circumstances will the 304 status code be returned? How does the client know that the content is not updated? In fact, this is not the client thing, but your server's things, we all know that the server can set the caching mechanism, this function is to improve the speed of access to the site, when you send a GET request, the server will call from the cache you want to access the content, This time the server will be able to determine whether this page is updated, if not updated then he will return you a 304 status code.   For example: Some search engines are how to know if our site is updated. The most straightforward way to determine whether a webpage is changing is to set up a section of the page as a monitoring area, crawl the contents of that part of the area each time, and then compare it with the local saved or the most recent crawl, and if there is a discrepancy, the page changes to be resolved. This method is more reliable and can achieve almost foolproof results. However, this way in each scan to download the page content, and to intercept the contents of the monitoring area, and finally a string comparison, the whole process is more time-consuming. In fact, in many Web pages, there are some Web site content is static pages, tablets, Html,js, and so on, these static pages may be the server is already ready, user access is only download. So for this static page, it can be judged only by the 304 status code, whether the content has changed.　　  How to solve? The server should return this status code if the client sends a conditional GET request and the request has been allowed, and the contents of the document (since the last time it was accessed or based on the requested condition) have not changed. 　　　　The 304 response suppresses the inclusion of the message body, so it always ends with the first empty line after the message header. The response must contain the following header information: Date, unless the server does not have a clock. 　　　　If a server without a clock follows these rules, then the proxy server and the client can add the Date field to the received response header (as specified in RFC 2068), and the caching mechanism will work correctly. 　　　　ETag and/or content-location, if the same request should return a 200 response. 　　Expires, Cache-control, and/or vary, if the value may be different than the value of the other response of the same variable as before. 　　　　  If this response request uses strong cache authentication, then this response should not contain other entity headers, otherwise (for example, a conditional GET request uses weak cache validation), this response prohibits the inclusion of other entity headers, which avoids inconsistencies between the cached entity content and the updated entity header information. If a 304 response indicates that the current entity does not have a cache, the cache system must ignore the response andRecurrence sends a request that does not contain a restriction. If a 304 response is received that requires a cache entry to be updated, the cache system must update the entire entry to reflect the value of all fields updated in the response    when a conditional request is made, the client provides the server with a if-modified-since request header, The value is the date value in the Last-modified response header that was last returned by the server, and also provides a If-none-match request header, which is the value of the ETag response header that the server last returned.   When the status code of the site is 304, the crawler or returns 705 of the status information. Indicates that the WAP gateway has failed to establish a connection to the remote server.　　   Reference Status Code information: Http://tool.oschina.net/commons?type=5 https://wenku.baidu.com/view/4e06018483d049649b66581c.html

Python crawler encounters status code 304,705