How to use HttpWebRequest and HttpWebResponse in C #

keywords: C # HttpWebRequest HttpWebResponse HTTP GET POST Request

This class is written specifically for HTTP GET and post requests, resolves issues such as encoding, certificates, and automatic cookies.
C # Httphelper, Help class, True HttpRequest request ignores encoding, ignores certificates, ignores cookies, web crawls

1. First move, obtain the Web page information according to the URL address
Let's take a look at the code
Get method

public static string geturltohtml (string url,string type) {    try    {        System.Net.WebRequest wReq = System.Net.WebRequest.Create (URL);        Get the response instance.        System.Net.WebResponse Wresp = Wreq.getresponse ();        System.IO.Stream Respstream = Wresp.getresponsestream ();        Dim reader as StreamReader = new StreamReader (respstream)        using (System.IO.StreamReader reader = new System.IO.Stre Amreader (Respstream, encoding.getencoding (type)))        {            return reader. ReadToEnd ();        }    }    catch (System.Exception ex)    {        //errormsg = ex. Message;    }    Return "";}

Post method

<SUMMARY>///uses the HTTPS protocol to access the network///</summary>///<param name= "URL" >url address </param>///<param Name= "Strpostdata" > Sent data </param>///<returns></returns>public string Openreadwithhttps ( String URL, String strpostdata, String strencoding) {    Encoding Encoding = Encoding.default;    HttpWebRequest request = (HttpWebRequest) webrequest.create (URL);    Request. Method = "POST";    Request. Accept = "text/html, Application/xhtml+xml, */*";    Request. ContentType = "application/x-www-form-urlencoded";    byte[] buffer = encoding. GetBytes (strpostdata);    Request. contentlength = buffer. Length;    Request. GetRequestStream (). Write (buffer, 0, buffer. Length);    HttpWebResponse response = (HttpWebResponse) request. GetResponse ();    using (StreamReader reader = new StreamReader (response. GetResponseStream (), System.Text.Encoding.GetEncoding (strencoding))      {           return reader. ReadToEnd ();      }}

This trick is the first style of entry, features:
1. The simplest and most intuitive one, introductory course.
2. Adapt to clear text, no need to log in, no Authentication required to enter the page.
3. The data type obtained is an HTML document.
4. The request method is Get/post

2. Second move, obtain Web page information that requires a certificate to be authenticated according to the URL address
Let's take a look at the code
Get method

 Callback Validation certificate issues public bool CheckValidationResult (object sender, X509Certificate certificate, X509chain chain, Sslpolicyerrors errors) {//Always accept return true;} <summary>///incoming URL returns HTML code for Web page//</summary>///<param name= "Url" >url</param>///<    Returns></returns>public string geturltohtml (String Url) {StringBuilder content = new StringBuilder (); try {//This sentence must be written before creating the connection.        Use the callback method for certificate validation. Servicepointmanager.servercertificatevalidationcallback = new        System.Net.Security.RemoteCertificateValidationCallback (CheckValidationResult);        Creates an HTTP request with the specified URL HttpWebRequest request = (HttpWebRequest) webrequest.create (URL);        Create a certificate file X509Certificate objx509 = new X509Certificate (Application.startuppath + "\\123.cer"); Add to request.        Clientcertificates.add (objx509); Gets the response of the corresponding HTTP request httpwebresponse response = (HttpWebResponse) request.        GetResponse (); Get response stream Stream ResponsestReam = Response.        GetResponseStream ();        Docking response flow (in "GBK" character set) StreamReader Sreader = new StreamReader (Responsestream, encoding.getencoding ("Utf-8"));        Start reading data char[] Sreaderbuffer = new char[256];        int count = Sreader.read (sreaderbuffer, 0, 256);            while (Count > 0) {string tempstr = new String (sreaderbuffer, 0, Count); Content.            Append (TEMPSTR);        Count = Sreader.read (sreaderbuffer, 0, 256);    }//Read end sreader.close ();    } catch (Exception) {content = new StringBuilder ("Runtime Error"); } return content. ToString ();}

Post method

Callback Validation certificate issues public bool CheckValidationResult (object sender, X509Certificate certificate, X509chain chain, Sslpolicyerrors errors) {//Always accept return true;} &LT;SUMMARY&GT;///uses the HTTPS protocol to access the network///</summary>///<param name= "URL" >url address </param>///<param Name= "Strpostdata" > Sent data </param>///<returns></returns>public string Openreadwithhttps ( String URL, String strpostdata, String strencoding) {//This sentence must be written before the connection is created.    Use the callback method for certificate validation. Servicepointmanager.servercertificatevalidationcallback = new    System.Net.Security.RemoteCertificateValidationCallback (CheckValidationResult);    Encoding Encoding = Encoding.default;    HttpWebRequest request = (HttpWebRequest) webrequest.create (URL);    Create a certificate file X509Certificate objx509 = new X509Certificate (Application.startuppath + "\\123.cer"); Load the cookie request.    Cookiecontainer = new Cookiecontainer (); Add to request.    Clientcertificates.add (objx509); Request.    Method = "POST"; Request. Accept = "Text/html, Application/xhtml+xml, */* "; Request.    ContentType = "application/x-www-form-urlencoded"; byte[] buffer = encoding.    GetBytes (Strpostdata); Request. contentlength = buffer.    Length; Request. GetRequestStream (). Write (buffer, 0, buffer.    Length); HttpWebResponse response = (HttpWebResponse) request.    GetResponse (); using (StreamReader reader = new StreamReader (response. GetResponseStream (), System.Text.Encoding.GetEncoding (strencoding)) {return reader.       ReadToEnd (); }}

This trick is learned to be in the door, all need to verify the certificate to enter the page can use this method to enter, I use the certificate callback authentication method, the certificate is verified by the client, so we can use ourselves to define a method to verify, Some people will say that it is not clear how to verify the Ah, the other is very simple, the code is to write their own why should be so difficult for themselves, directly return a true is not finished, always be verified through, so you can disregard the existence of certificates, features:
1. Enter the front of the small problem, the beginner course.
2. Suitable for pages that require no login, clear text but need to verify the certificate to access.
3. The data type obtained is an HTML document.
4. The request method is Get/post

3. Third recruit, according to the URL address to get login to access the Web page information
We first analyze this type of Web page, need to log in to access the Web page, the other is also a kind of authentication, verify what, verify that the client is logged in, whether with the corresponding credentials, need to login to verify SessionID this is every need to log on the page needs to be verified, then we do, Our first step is to have the data contained in the cookie including SessionID, how to get it, this method is many, using ID9 or Firefox browser is easy to get, you can refer to my article
Provide a Web page to crawl hao123 mobile phone number attribution to the example of this in the ID9 for detailed instructions.
If we get the cookie information that is logged in, then it will be very easy to access the corresponding page, and the other is to put the local cookie information on the request to the past.
Look at the code
Get method

<summary>///incoming URL Returns the HTML code of the Web page with a certificate method//</summary>///<param name= "Url" >url</param>///    <returns></returns>public string geturltohtml (String Url) {StringBuilder content = new StringBuilder ();        try {//create HTTP request with specified URL HttpWebRequest request = (HttpWebRequest) webrequest.create (URL); Request. useragent = "mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; trident/5.0; BOIE9;        ZHCN) "; Request.        Method = "GET"; Request.        Accept = "*/*"; If the method validates the source of the page, add this sentence if you do not verify it, you can not write the request.        Referer = "http://txw1958.cnblogs.com";        Cookiecontainer Objcok = new Cookiecontainer (); Objcok.        ADD (New Uri ("http://txw1958.cnblogs.com"), New Cookie ("Key", "value")); Objcok.        ADD (New Uri ("http://txw1958.cnblogs.com"), New Cookie ("Key", "value")); Objcok.        ADD (New Uri ("http://txw1958.cnblogs.com"), New Cookie ("Sidi_sessionid", "360a748941d055bee8c960168c3d4233")); Request.        Cookiecontainer = Objcok;  Do not stay connected      Request.        KeepAlive = true; Gets the response of the corresponding HTTP request httpwebresponse response = (HttpWebResponse) request.        GetResponse (); Gets the response stream stream Responsestream = response.        GetResponseStream ();        Docking response flow (in "GBK" character set) StreamReader Sreader = new StreamReader (Responsestream, encoding.getencoding ("gb2312"));        Start reading data char[] Sreaderbuffer = new char[256];        int count = Sreader.read (sreaderbuffer, 0, 256);            while (Count > 0) {string tempstr = new String (sreaderbuffer, 0, Count); Content.            Append (TEMPSTR);        Count = Sreader.read (sreaderbuffer, 0, 256);    }//Read end sreader.close ();    } catch (Exception) {content = new StringBuilder ("Runtime Error"); } return content. ToString ();}

Post method.

&LT;SUMMARY&GT;///uses the HTTPS protocol to access the network///</summary>///<param name= "URL" >url address </param>///<param Name= "Strpostdata" > Sent data </param>///<returns></returns>public string Openreadwithhttps (    String URL, String strpostdata) {Encoding Encoding = Encoding.default;    HttpWebRequest request = (HttpWebRequest) webrequest.create (URL); Request.    Method = "POST"; Request.    Accept = "text/html, Application/xhtml+xml, */*"; Request.    ContentType = "application/x-www-form-urlencoded";    Cookiecontainer Objcok = new Cookiecontainer (); Objcok.    ADD (New Uri ("http://txw1958.cnblogs.com"), New Cookie ("Key", "value")); Objcok.    ADD (New Uri ("http://txw1958.cnblogs.com"), New Cookie ("Key", "value")); Objcok.    ADD (New Uri ("http://txw1958.cnblogs.com"), New Cookie ("Sidi_sessionid", "360a748941d055bee8c960168c3d4233")); Request.    Cookiecontainer = Objcok; byte[] buffer = encoding.    GetBytes (Strpostdata); Request. contentlength = buffer.    Length; Request. GetRequestStream (). Write(buffer, 0, buffer.)    Length); HttpWebResponse response = (HttpWebResponse) request.    GetResponse (); StreamReader reader = new StreamReader (response.    GetResponseStream (), System.Text.Encoding.GetEncoding ("Utf-8")); Return reader. ReadToEnd ();}

Characteristics:
1. Still a little water type, after the success of the practice can be a calf.
2. Adapt to pages that need to be logged in to access.
3. The data type obtained is an HTML document.
4. The request method is Get/post

To summarize, the other basic skills are these parts, and if you go deeper, that's a combination of basic skills.
Like what
1. Use the Get or post method to log in and then obtain a cookie to access the page to get information, the other is a combination of the above skills,
This is a step that needs to be made after the request.

Response. Cookies

This is the way you can get a cookie when you request it, just get back to the previous method and use it, and we're all built on it, so we can use this cookie directly here.

2. If we come across a webpage that needs to be logged in and verify the certificate, the other one is simple. Just combine the methods above.
Here's the code I use Get for example post example is the same method

<summary>///incoming URL returns HTML code for Web page//</summary>///<param name= "Url" >url</param>///<    Returns></returns>public string geturltohtml (String Url) {StringBuilder content = new StringBuilder (); try {//This sentence must be written before creating the connection.        Use the callback method for certificate validation. Servicepointmanager.servercertificatevalidationcallback = new        System.Net.Security.RemoteCertificateValidationCallback (CheckValidationResult);        Creates an HTTP request with the specified URL HttpWebRequest request = (HttpWebRequest) webrequest.create (URL);        Create a certificate file X509Certificate objx509 = new X509Certificate (Application.startuppath + "\\123.cer"); Add to request.        Clientcertificates.add (objx509);        Cookiecontainer Objcok = new Cookiecontainer (); Objcok.        ADD (New Uri ("http://www.cnblogs.com"), New Cookie ("Key", "value")); Objcok.        ADD (New Uri ("http://www.cnblogs.com"), New Cookie ("Key", "value")); Objcok. ADD (New Uri ("http://www.cnblogs.com"), New Cookie ("Sidi_sessionid", "360a748941d055bee8c960168c3d4233 ")); Request.        Cookiecontainer = Objcok; Gets the response of the corresponding HTTP request httpwebresponse response = (HttpWebResponse) request.        GetResponse (); Gets the response stream stream Responsestream = response.        GetResponseStream ();        Docking response flow (in "GBK" character set) StreamReader Sreader = new StreamReader (Responsestream, encoding.getencoding ("Utf-8"));        Start reading data char[] Sreaderbuffer = new char[256];        int count = Sreader.read (sreaderbuffer, 0, 256);            while (Count > 0) {string tempstr = new String (sreaderbuffer, 0, Count); Content.            Append (TEMPSTR);        Count = Sreader.read (sreaderbuffer, 0, 256);    }//Read end sreader.close ();    } catch (Exception) {content = new StringBuilder ("Runtime Error"); } return content. ToString ();}

3. If we encounter the kind of method that needs to verify the source of the Web page, the other is that some programmers will think you may use the program, automatically to obtain the page information, in order to prevent the use of page source to verify, that is, as long as not from their page or the domain name of the request is not accepted, Some are direct authentication source IP, these can use the following sentence to enter, this is mainly the address can be directly forged

Request. Referer = <a href= "http://txw1958.cnblogs.com" >http://txw1958.cnblogs.com</a>;

Oh other very simple because this address can be directly modified. But if the server is verifying the URL of the source then it is over, we have to modify the packet, this is a bit difficult to discuss for the time being.

4. Provide some methods that are configured with this example
Ways to filter HTML tags

<summary>///filtering HTML tags///</summary>///<param name= "strhtml" >html content </param>///< Returns></returns>public static string striphtml (String stringtostrip) {    //paring using RegEx           //    stringToStrip = Regex.Replace (stringToStrip, "</p" (?: \ \s*) > (?: \ \s*) <p (?: \ \s*) > "," \ n \ nyou ", Regexoptions.ignorecase | regexoptions.compiled);    stringToStrip = Regex.Replace (stringToStrip, "", "\ n", Regexoptions.ignorecase | regexoptions.compiled);    stringToStrip = Regex.Replace (stringToStrip, "\" "," "", Regexoptions.ignorecase | regexoptions.compiled);    stringToStrip = Striphtmlxmltags (stringToStrip);    return stringToStrip;} private static string Striphtmlxmltags (string content) {    return Regex.Replace (content, "<[^>]+>", "", Regexoptions.ignorecase | regexoptions.compiled);}

How to convert URLs

#region convert urlpublic static string UrlDecode (string text) {    return Httputility.urldecode (text, encoding.default);} public static string UrlEncode (string text) {    return Httputility.urlencode (text, encoding.default);} #endregion

To provide a practical example, this is the use of IP138 to query cell phone number attribution to the method, the other in my last article has, here I put up is convenient for everyone to read, this aspect of the technical other research is very interesting, I hope we have more suggestions, I believe there should be more better, more perfect method, Here's a reference for you. Thanks for supporting
Example above

<summary>///Enter the mobile number to get the attribution information///</summary>///<param name= "number" > Mobile phone Numbers </param>///< Returns> Array type 0 for attribution, 1 card type, 2 area code, 3 zip </returns>public static string[] Gettelldate (string number) {try {s Tring strsource = geturltohtml ("http://www.ip138.com:8080/search.asp?action=mobile&mobile=" + number.)        Trim ());        Place of Attribution strsource = Strsource.substring (strsource.indexof (number));        strsource = striphtml (strsource);        strsource = Strsource.replace ("\ R", "");        strsource = Strsource.replace ("\ n", "");        strsource = Strsource.replace ("\ T", "");        strsource = Strsource.replace ("", "");        strsource = Strsource.replace ("--", "" "); string[] Strnumber = Strsource.split (new string[] {"Attribution", "card type", "Zip Code", "Area code", "more detailed", "card Number"}, Stringsplitoptions.removeem        Ptyentries);        string[] Strnumber1 = null; if (Strnumber. Length > 4) {strnumber1 = new string[] {strnumber[1]. Trim (), StrnuMBER[2]. Trim (), strnumber[3]. Trim (), strnumber[4].        Trim ()};    } return strnumber1;    } catch (Exception) {return null; }}

This example writes is not how much, some places can be simplified, this interface and can be directly using XML to get, but my focus here is to let some novice look at methods and ideas cool Ah, hehe

Four strokes, access via socket

<summary>///the public class of the request is used to send a request to the server///</summary>///<param name= "Strsmsrequest" > send the requested string </param >///<returns> returned is the requested information </returns>private static string Smsrequest (String strsmsrequest) {byte[] data = n    EW byte[1024];    string stringdata = null;    Iphostentry gist = Dns.gethostbyname ("www.110.cn"); IPAddress IP = gist.    ADDRESSLIST[0];    Get IP IPEndPoint ipend = new IPEndPoint (IP, 3121);    Default 80 port number Socket SOCKET = new socket (addressfamily.internetwork, SocketType.Stream, protocoltype.tcp); Use the TCP protocol stream type try {socket.    Connect (Ipend); } catch (SocketException ex) {return "Fail to connect server\r\n" + ex.    ToString (); } String path = Strsmsrequest.tostring ().    Trim ();    StringBuilder buf = new StringBuilder (); Buf. Append ("GET"). Append (Path).    Append ("http/1.0\r\n"); Buf.    Append ("content-type:application/x-www-form-urlencoded\r\n"); Buf.    Append ("\ r \ n"); Byte[] ms = System.Text.UTF8EncodiNg. UTF8. GetBytes (BUF.    ToString ()); Submits the requested information to the socket.    Send (MS);    Receive returns String strsms = "";    int recv = 0; do {recv = socket.        Receive (data);        StringData = Encoding.ASCII.GetString (data, 0, recv);        If the page encoding specified in the requested page meta is gb2312, the corresponding encoding is required to convert the bytes () strsms = strsms + stringdata; Strsms + = recv.    ToString ();    } while (recv! = 0); Socket.    Shutdown (Socketshutdown.both); Socket.    Close (); return strsms;}

Original address: http://www.cckan.net/thread-6-1-1.html

How to use HttpWebRequest and HttpWebResponse in C #