C # HttpWebRequest Crawl Web page content application via Proxy server _c# tutorial

Source: Internet
Author: User
Tags readline
Intranet users or proxy users using the Internet
Copy Code code as follows:

Using System.IO;
Using System.Net;
public string get_html ()
{
String urlstr = "http://www.domain.com"; Set the address to be taken
HttpWebRequest HWR = (HttpWebRequest) httpwebrequest.create (URLSTR); Establish HttpWebRequest image
Hwr. Timeout = 60000; Defining a service over time
WebProxy proxy = new WebProxy (); Defining an Internet image
Proxy. Address = new Uri ("http://proxy.domain.com:3128"); Network Service: Ports
Proxy. Credentials = new NetworkCredential ("f3210316", "6978233"); User name, password
Hwr. useDefaultCredentials = true; To activate the Internet to confirm
Hwr. Proxy = proxy; Setup Internet
Try
{
HttpWebResponse Hwrs = (httpwebresponse) hwr. GetResponse (); Get response
}
Catch
{
MessageBox.Show ("Unable to connect agent!") ");
Return
}
To determine the HTTP response status
if (Hwrs. StatusCode!= Httpstatuscode.ok)
{
MessageBox.Show ("Access failed!") ");
Hwrs. Close ();
Return
}
Else
{
Stream s = Hwrs. GetResponseStream (); Get the response to the flow of the image
StreamReader sr = new StreamReader (s, Encoding.UTF8); Read the stream with the UTF-8 encoding
StringBuilder content = new StringBuilder (); //
while (Sr. Peek ()!=-1)//read one row at a time until
{//The next word has no content
Content. Append (Sr. ReadLine () + "" R "N"); Return to stop
} //
return content. ToString ();
}
Outputs all headers (including, of course, cookies from the server output)
for (int ii=0;ii//{
MessageBox.Show (Hwrs. Headers.getkey (ii) + ":" +res. HEADERS[II]);
//}
}

You know, with HttpWebRequest can be used to crawl the Web page through HTTP, but if the intranet, but also through the proxy Internet users, if the direct operation is not feasible.
Is there any way to do that?
Of course there is, hehe, see the following code:
Copy code code as follows:

String urlstr = "http://www.domain.com";//Set the address to be fetched
HttpWebRequest HWR = (HttpWebRequest) Httpwebrequest.create (URLSTR); Establish a HttpWebRequest-like
Hwr. Timeout = 60000; The definition service is over time
WebProxy proxy = new WebProxy ();//define a Web-like
Proxy. Address = new Uri ("http://proxy.domain.com:3128"); Network Service: Port
Proxy. Credentials = new NetworkCredential ("f3210316", "6978233"); User name, password
HWR. useDefaultCredentials = true; To activate the Internet accreditation
HWR. Proxy = proxy; Set the Web
HttpWebResponse Hwrs = (httpwebresponse) hwr. GetResponse (); Gets the response
Stream s = Hwrs. GetResponseStream (); Get the response to the stream like
StreamReader sr = new StreamReader (s, Encoding.UTF8);//Read stream in UTF-8
StringBuilder content = new STR Ingbuilder ();
while (Sr. Peek ()!=-1)//read one row at a time until the
{//Next Word does not have content
content. Append (Sr. ReadLine () + "" R "N"); Returns the
}//
return content. ToString (); Returns the resulting string
Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.