Use htmlagilitypack to capture web page data in batches

Click to download related software

Login processing. Because some webpage data can be extracted only after login. Here, you need to use iehttpheaders to extract the submission information for logon.

Capture webpages

Htmlagilitypack. htmldocument htmldoc;

If ( ! String . Isnullorempty (logon URL ))
{
Htmldoc=Htmlweb. Load (logon URL, submitted user authentication information, and webpage URL for data retrieval );
}
Else
{
Htmldoc=Htmlweb. Load (URL of the webpage that obtains data );
}

 

Arraylist list =   New Arraylist ();
List. Add ( " // Table/TR [1]/TD " );
List. Add ( " // Table/TR [2]/TD " );
// Obtain the XPath of the cyclic node, for example: // Table/tr
Htmlnodecollection repeatnodes = Htmldoc. documentnode. selectnodes ( " // Table/tr " );

// Cyclic nodes
Foreach (Htmlnode Node In Repeatnodes)
{
// Obtain data cyclically
Foreach ( String Datapath In List)
{

Htmlnode datanode = Node. selectsinglenode (list );
If (Datanode ! =   Null )
{
StringText=Datanode. innertext;
}

}
}

If garbled characters appear, adjust the encoding set to gb2312 or UTF-8.

Htmlweb. defaultencoding = System. Text. encoding. getencoding (strencode );