HttpWebRequest: The resolution problem record of chunked in http1.1, and webrequest asynchronous request

HttpWebRequest: The resolution problem record of chunked in http1.1, and webrequest asynchronous request

Problem: My request cannot get the content corresponding to the URL (another browser can be used ).

 

The first step is to compare the wirshark packet capture to check the HTTP request header. I found that some of the request headers are missing.

Add the following to the official document. Https://msdn.microsoft.com/zh-cn/library/system.net.httpwebrequest (v = vs.110). aspx

Step 2: After the request header is corrected, wireshark returns but cannot be obtained. Debugging finds that the content cannot be parsed.

This place may take a little time and experience to know.

After visual inspection, we found that the website is HTTP1.1 and there is another chunked parameter. In short, it is returned in segments when the content is returned. Do not set the length.

The reading method of the old version cannot be read. Refer to another netizen to solve the problem.

For your reference. The project code will not be pasted, avoiding suspicion.

Static void Main (string [] args) {HttpWebResponse web = MySpider. getResponse ("http: // localhost: 1853/WebForm1.aspx"); DecompressGZip (web); Console. readLine ();} public static MemoryStream DecompressGZip (HttpWebResponse res) {// if the server uses the Transfer-Encoding: chunked buffer output, as long as the server is flushed, this method is triggered instead of waiting until all the content sent by the server is sent,
// It does not have to do with asynchronous HTTP webrequest requests. Conversely, if the server does not use the Transfer-Encoding: chunked buffer output,
// This method is triggered when all the content sent by the server is sent, whether it is an asynchronous HttpWebRequest request or an HTTP webrequest synchronization request. Stream stream = res. getResponseStream (); int length = 0; if (res. contentLength> 0) {length = (int) res. contentLength;} else {length = 3000;} MemoryStream memory = new MemoryStream (length); int count = 0; // read 5000 byte [] buffer = new byte [5000] each time the server returns a stream; while (true) {// if the server uses Transfer-Encoding: chunked buffer output. If the second Flush content has not been received by the server after the first Flush content of the server is read, the current thread will be blocked,
// Wait until the server receives the second Flush content (third, fourth... The same is true for the second Flush operation). Therefore, it is likely that the count returned for one read operation is less than 5000, but the count returned for the next read operation is not 0. count = stream. read (buffer, 0, buffer. length); if (count = 0) {break;} memory. write (buffer, 0, count);} stream. close (); // set the stream's readable position to the starting value of memory. seek (0, SeekOrigin. begin); return memory ;}

 

 

 

 

Appendix

1. About Transfer-Encoding: chunked issues in Http. Routing http://blog.csdn.net/zhangboyj/article/details/62367803. http://www.cnblogs.com/mxw09/archive/2010/12/17/1908753.html