How to determine whether a url is updated

This survey has found three methods:

1. Use Headers of HttpWebRequest and HttpWebResponse.

The Header contains two Modify-related keys,

One is eTag (http://www.baike.com/wiki/Etag), and this is entity id.

The other is Lasted-Modified. Last modification time.

When using these two headers, it is best to use them at the same time, and Etag takes precedence.

One advantage of doing so is that if the webpage only updates the last-modify but does not change the content, 304 is returned.

// Obtain the two values for the first response: HttpWebRequest req = (HttpWebRequest) WebRequest. create (url); HttpWebResponse res = req. getResonse () as HttpWebResponse; stringeTag = res. headers [HttpWebResponse. ETag]; string lastModified = res. headers [HttpWebResponseHeader. lastModified] // For the second response, write the preceding two values into the Header of WebRequest and pass in. HttpWebRequest req = (HttpWebRequest) WebRequest. create (url); req. methid = "GET"; req. headers [HttpRequestHeader. ifNoneMatch] = eTag; req. ifModifiedSinse = ifModifiedSince; HttpWebResponse res = req. getResonse () as HttpWebResponse; // judge the statusCode if (res. statusCode = HttpStatusCode. notModified ){}

(Note: This method is applicable to static pages, but not dynamic pages. Because the LastModified time of the dynamic page is the time when the server sends the Response, not the last update time of the page. Etag is usually null.

2. Use MD5 (or SHA1) signature for the Stream of Response: the most recommended method.

Perform MD5 signatures on the first and second response streams respectively, and then compare them. If different, the content will change.

The Md5 Algorithm quickly signs text streams, and M-level data can be completed within milliseconds.

Private static string HashData (System. IO. stream stream, string algName) {stream. seek (0, System. IO. seekOrigin. begin); System. security. cryptography. hashAlgorithm algorithm; if (algName = null) {throw new ArgumentNullException ("algName cannot be null");} if (string. compare (algName, "sha1", true) = 0) {algorithm = System. security. cryptography. SHA1.Create ();} else {if (string. compare (algName, "md5", tr Ue )! = 0) {throw new Exception ("algName can only use sha1 or md5");} algorithm = System. security. cryptography. MD5.Create ();} byte [] resultByteAr = algorithm. computeHash (stream); return BitConverter. toString (resultByteAr ). replace ("-","");}

3. There is another method, but the accuracy is not very accurate: Download a length of response byte each time, and then compare it.

Here, we mainly use the Range Header to get a byte [] content for storage, and then get the byte [] at the same position for the second time for comparison.