The spider program that constructs the C # language

C # is especially good for building spider programs because it has built-in HTTP access and multithreading capabilities that are critical to spider programs. The following are the key issues to be addressed in constructing a spider program:

⑴html analysis: Some kind of HTML parser is needed to analyze every page the Spider program encounters.

⑵ page Processing: You need to process each downloaded page. The downloaded content may be saved to disk, or further analysis will be processed.

⑶ Multithreading: Only have multithreading ability, the spider program can really achieve efficiency.

⑷ determine when to finish: Do not underestimate the problem, determine whether the task has been completed is not easy, especially in a multithreaded environment.

First, HTML parsing

The HTML parser provided by this article is implemented by the Parsehtml class, which is easy to use: first create an instance of the class, and then set its Source property to the HTML document to parse:

ParseHTML parse = new ParseHTML();
parse.Source = "
Hello World
";

You can then use loops to examine all the text and markup contained in an HTML document. Typically, the inspection process can begin with a while loop that tests the EOF method:

while(!parse.Eof())
{
char ch = parse.Parse();

The Parse method returns the characters contained in the HTML document-it returns only those characters that are not HTML-tagged, and if an HTML tag is encountered, the Parse method returns a value of 0, indicating that an HTML tag is now encountered. After a tag is encountered, we can use the Gettag () method to handle it.

if(ch==0)
{
HTMLTag tag = parse.GetTag();
}

In general, one of the most important tasks of the Spider program is to find each href attribute, which can be accomplished with the help of C # 's indexing function. For example, the following code extracts the value of the href attribute (if it exists).

Attribute href = tag["href"];

String link = href. Value;

After the attribute object is obtained, the value of the property can be obtained by Attribute.value.

Second, processing HTML page

Let's look at how to work with HTML pages. The first thing to do is to download the HTML page, which can be implemented through the HttpWebRequest class provided by C #:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(m_uri);
response = request.GetResponse();
stream = response.GetResponseStream();

Next, we create a stream stream from the request. Before performing any other processing, we need to determine whether the file is a binary or a text file, and different file types are handled differently. The following code determines whether the file is a binary file.

if( !response.ContentType.ToLower().StartsWith("text/") )
{
SaveBinaryFile(response);
return null;
}
string buffer = "",line;

If the file is not a text file, we will read it as a binary file. If it is a text file, first create a StreamReader from the stream, and then add the contents of the text file to the buffer line by row.

reader = new StreamReader(stream);
while( (line = reader.ReadLine())!=null )
{
buffer+=line+"rn";
}

After you load the entire file, you will save it as a text file.

Savetextfile (buffer);