HttpWebRequest catch the data encountered problems

1, some websites slow access, and the number of connections to this site (such as the global underwear, in addition to girls of all kinds of what underwear do not understand can also go up to see HA), because there is no immediate closure, resulting in crawling page data time-out is also serious.

Solution: Put the corresponding httpwebresponse.close (), Httpwebrequest.abort (); and Httpwebrequest.keepalive=false, and the time-out setting is a bit longer, and then the chance of a connection timeout is degraded. The domain name of the host file of the direct C drive directly points to an IP, reducing the time to find the DNS server

2, catch the Chinese suppliers when the multi-threaded run too fast, dozens of appeared to pull the class verification code, this can only reduce the frequency.

The following is the class library that encapsulates the request

public static string getrequest (string url, string charset = "Utf-8")
{
HttpWebRequest myreq = null;
HttpWebResponse myres = null;
StreamReader reader = null;
Stream stream = null;
String result = "";
String code = CharSet;//charset. ToLower ()
//code = "utf-8";
Try
{
Myreq = (HttpWebRequest) webrequest.create (URL);
Myreq. Timeout = 20000;
Myreq. Method = "GET";
Myreq. KeepAlive = false;
Myreq. Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
//myreq. useragent = "mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html) ";
Myreq. Headers.add ("Content", "text/html; charset= "+ code);
//myreq. useragent = "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2;. NET CLR 1.1.4322;. NET CLR 2.0.50727) ";
Myreq. useragent = "mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/51.0.2704.106 safari/537.36 ";

Myreq. KeepAlive = true;
Myres = (HttpWebResponse) myreq. GetResponse ();
stream = Myres. GetResponseStream ();
reader = new StreamReader (stream, System.Text.Encoding.GetEncoding (code));
result = reader. ReadToEnd ();

Reader. Close ();
Reader. Dispose ();

Stream. Close ();
Stream. Dispose ();
}
Catch
{ }
Finally
{
if (myreq! = null)
{
Myres. Close ();
}
if (myreq! = null)
{
Myreq. Abort ();
}

}

return result;
}

HttpWebRequest catch the data encountered problems