Write a web-grabbing application in C #

This article uses the classes provided by C # and. NET to easily create a program that crawls the content source code of a Web page. HTTP is one of the most basic protocols for data access at WWW, and it provides two object classes in the basic type library class of. NET: HttpWebRequest and HttpWebResponse, which are used to send requests to a resource and get a response, respectively. In order to get the content of a resource, we first specify a URL that we want to crawl, request with the HttpWebRequest object, receive the result of the response with the HttpWebResponse object, and finally use the TextStream object to extract the information we want. and print it out at the console.
Here's how to achieve this:
Step One: Open vs.net, point "file"-"new"-"Project", Project type Select "Visual C # project", template to choose Windows Application, as shown in the following illustration:
Step Two: Add Label1,button1,textbox1,textbox2 four controls to the Form1, TextBox2 multiline property to True, as shown in the following illustration:
Step Three: Right-click on the Form1 form, select "View Code", and then enter at the top:
Using System.IO;
Using System.Net;
Using System.Text;
In
private void Button1_Click (object sender, System.EventArgs e)
{
}
Enter the following code between the parentheses:
byte[] buf = new byte[38192];
HttpWebRequest request = (HttpWebRequest) webrequest.create (TextBox1.Text);
HttpWebResponse response = (HttpWebResponse) request. GetResponse ();
Stream Resstream = Response. GetResponseStream ();
int count = Resstream.read (buf, 0, buf.) Length);
TextBox2.Text = Encoding.Default.GetString (buf, 0, Count);
Resstream.close ();
Step Fourth: Click the "Save All" button, press "F5" to run the application, in the "Please enter the URL address:" After the single-line text box input http://lucky.myrice.com/down.htm, click the "Get HTML Code" button, You can see the code for the address! The results are as follows:
Below, we will do an analysis of the above procedure:
The function of the above program is to crawl the web http://lucky.myrice.com/ The content of down.htm, and the HTML code is displayed in a multiline text box, because the data returned is byte type, so we create an array variable named BUF that stores the result of the request return, where the size of the array is related to the size of the data we are requesting to return. First, we instantiate the HttpWebRequest object, use the static method of the WebRequest class create (), the string parameter of the method is the URL address of the page we want to request, because the Create () method returns the WebRequest type, We have to sculpt it (that is, type conversion) into a HttpWebRequest type, and then assign it to the request variable. Once we have established the HttpWebRequest object, we can use its GetResponse () method to return a WebResponse object and then sculpt it into a HttpWebResponse object to assign to the response variable. Now, you can use the GetResponseStream () method of the response object to get the text stream of the response, and then put the returned response information into the byte array buf we created originally by using the Read () method of the Stream object, and Read () has 3 parameters, is the byte array to put in, the start position of the byte array, and the length of the byte array. Finally, the byte is converted to a string, note: Here we use the default encoding, which uses the defaults encoding, so we don't have to convert between character encodings.
You can also use WebRequest and WebResponse to achieve the above functionality, the code is as follows:
WebRequest request = WebRequest.Create (TextBox1.Text);
WebResponse response = Request. GetResponse ();
Enter the other ur