C # Collect and capture website data

First, you need to be clear: any website page, whether it is php, jsp, aspx dynamic pages or static pages generated by background programs, you can view their HTML source files in a browser.

 

 

Therefore, to develop a data collection program, you must first understand the front-end Page Structure (HTML) of the website you are trying to collect.

 

When you are very familiar with the HTML source file content of the website for data collection, the rest of the procedures will be very easy to do. Because the principle of C # data collection for Web sites is that "download the HTML source file of the page you want to collect, analyze the HTML code, and then capture the data you need, finally, save the data to a local file ".

 

Shows the basic process:

 

 

 

1. Download Page source files

First, reference the System. Net namespace

Using System. Net;
You must also reference

Using System. Text;
Using System. IO;
Instantiate a WebClient object after referencing

Private WebClient wc = new WebClient ();
Call the DownloadData method to download a set of bytes from the source file of the specified webpage, and convert the BYTE array into a string.

// Download the source file on the page and convert it to a UTF-8 encoded STRING
String mainData = Encoding. UTF8.GetString (wc. DownloadData (string. Format ("the URL you want to collect ")));
Alternatively, you can call the DownloadFile method to download the source file to the local device and then read the string.

// Download the webpage source file to the local device
Wc. DownloadFile ("the URL of the webpage you want to collect", "Save the local file path of the source file ");
// Read the HTML string of the downloaded source file
String mainData = File. ReadAllText ("local File path for saving source files", Encoding. UTF8 );
With the HTML string of the webpage, you can analyze and collect the webpage and capture the content you need.

 

2. PAGE analysis collection

PAGE analysis uses a specific or unique character (string) in the webpage source file as the capture point to capture the data on the desired page as the starting point.

For example, if you want to collect the titles and links of the articles listed on the blog homepage, you must use "<a class =" titlelnk "href =" "as the capture point, to capture the title and link of the article.

 

CODE:

// Start collection with "<a class =" titlelnk "href =" "as the capture point
MainData = mainData. Substring (mainData. IndexOf ("<a class =" titlelnk "href =" ") + 26 );

// Obtain the link address of the Article Page
String articleAddr = mainData. Substring (0, mainData. IndexOf ("""));

// Obtain the article title
String articleTitle = mainData. Substring (mainData. IndexOf ("target =" _ blank ">") + 16,
MainData. IndexOf ("</a>")-mainData. IndexOf ("target =" _ blank ">")-16 );
 

Note: When the HTML format of the webpage you want to collect changes in the foreground, the character seek as the capture point also changes accordingly; otherwise, nothing can be collected.

 

3. Data Storage

After you extract the required data from the web page, sort the data in the program and save it to a local file (or insert it into your local database ). In this way, the entire collection work is completed.

// Output data to a local file
File. AppendAllText (CreateFolderIfNot (Settings. Default. OutPath) + articleTitle + ". txt ",
ArticleData,
Encoding. UTF8 );
 

In addition, I attached a small program code that I wrote to collect articles on the homepage of the blog Garden. The function of this program is to collect all the articles published on the homepage of the blog garden.

: CnBlogCollector.rar

Of course, if the format of the front-end page of the blog garden is adjusted, the collection function of the program is definitely invalid. You can only re-adjust the program to continue the collection...

The program effect is as follows: