Examples of HtmlAgility in asp.net for reading webpages, obtaining data, and id

This article learns about its parsing function, and can also simulate user requests, create html, set proxy, and so on.

Entry Code:

The code is as follows:

Copy code

Using System; Using System. Collections. Generic; Using System. Linq; Using System. Text; Using System. Net; Using HtmlAgilityPack; Namespace ConsoleApplication1 { Class Program     { Static void Main (string [] args) { HtmlWeb webClient = new HtmlWeb (); HtmlDocument doc = webClient. Load ("http://www.baidu.com "); Var rootNode = doc. DocumentNode; HtmlNodeCollection categoryNodeList = rootNode. SelectNodes ("// html [1]/body [1]"); Foreach (var item in categoryNodeList)     { Console. WriteLine ("item:" + item. Name );     } Console. Read (); }     } }

2. Read

Then, if it is to load the local Html or directly read the stream, the string. You can do this.

The code is as follows:	Copy code
HtmlDocument doc = new HtmlDocument (); Doc. Load (@ "D: xxx. mht", Encoding. UTF8, false );

 
Public void LoadHtml (string html); // directly read the string-based htmlpublic void Load (Stream stream); // Stream public void Load (string path); // Local path
 

HtmlDocumen also provides the encoding method.

HtmlWeb is mainly used to automatically detect the encoding. If you want to customize the encoding, you can change its attributes. OverrideEncoding, AutoDetectEncoding. HtmlDocument has different encoding operations. In the parameter, it is estimated that the automatic detection encoding is very powerful and seldom needs to be specified by itself ....

3. Select nodes

RootNode. SelectNodes
RootNode. SelectSingleNode
Select a node and a single node.

Taking SelectNodes as an example, let's take a look at the parameters.

RootNode. SelectNodes ("// html [1]/body [1]");
"//" Double slash indicates searching all subnodes from the root node

"/" Single slash indicates that only the first layer of subnodes are searched.

"./" The Dot slash (/) indicates searching from the current node.

 
[] Represents the index of a subnode with the same name in brackets.

The code is as follows:

Copy code

Var resultList = rootNode. selectNodes ("// html [1]/body [1]/div [1]/div [position () <5]"); // obtain the first four elements: resultList = rootNode. selectNodes ("// html [1]/body [1]/div [1]/div [last ()]"); // Obtain the last element resultList = rootNode. selectNodes ("// html [1]/body [1]/div [1]/div [@ id]"); // obtain all the elements with the id attribute resultList = rootNode. selectNodes ("// html [1]/body [1]/div [1]/div [@ id = 'head']"); // Retrieve the element whose property id is head

For more attributes, see W3SCHOOL.
 
Get attributes

The code is as follows:	Copy code
Doc. Attributes ["id"]; Retrieve element Doc. GetElementbyId ("id ");

Supplement

HtmlAgilityPack class library usage

1. First, you need to obtain html page data, which can be obtained through the WebRequest class.

The code is as follows:	Copy code
Public static string GetHtmlStr (string url) {    Try     { WebRequest rGet = WebRequest. Create (url ); WebResponse rSet = rGet. GetResponse (); Stream s = rSet. GetResponseStream (); StreamReader reader = new StreamReader (s, Encoding. UTF8 ); Return reader. ReadToEnd ();     } Catch (WebException)     { // Connection failed Return null;     } }

2. Load html data through the HtmlDocument class

The code is as follows:

Copy code

String htmlstr = GetHtmlStr (http://www.111cn.net ); HtmlAgilityPack. HtmlDocument doc = new HtmlAgilityPack. HtmlDocument (); Doc. LoadHtml (htmlstr ); HtmlNode rootnode = doc. DocumentNode; // XPath path expression, which indicates selecting the last font subnode of all span nodes. The class attribute value of the span node is num. // Set the XPath path expression based on the webpage content String xpathstring = "// span [@ class = 'num']/font [last ()]"; HtmlNodeCollection aa = rootnode. SelectNodes (xpathstring); // all the nodes found are a collection If (aa! = Null) { String innertext = aa [0]. InnerText; String color = aa [0]. GetAttributeValue ("color", ""); // gets the color attribute. The second parameter is the default value. // Try other attributes by yourself }

You can also use the HtmlWeb class to obtain the HtmlDocument

The code is as follows:	Copy code
HtmlWeb web = new HtmlWeb (); HtmlAgilityPack. HtmlDocument doc = web. Load (url ); HtmlNode rootnode = doc. DocumentNode;

 

Supplement:

Query multiple attribute conditions // div [@ align = 'center' and @ height = '24']

The class attribute does not exist. // div [not (@ class)]