Write a message to capture information from the web page.

Write a message to capture information from the web page.

Write a class that captures information (such as the latest headlines, news sources, titles, and content) from a webpage, this article describes how to use this class to capture the information required in a webpage. This article takes the title and link of the blog home page as an example:

The DOM tree of the homepage is displayed. Obviously, you only need to extract the div whose class is post_item, and then extract the sign whose class is titlelnk. This function can be implemented through the following functions:

/// <Summary> /// search for all the tags whose names are tagnames and whose attributes are attrName and whose values are attrValue. // For example: FindTagByAttr (html, "div", "class", "demo") // return all div labels whose classes are demo. // The QQ Group for front-end learning and communication: 461593224
/// </Summary> public static List <HtmlTag> FindTagByAttr (String html, String tagName, String attrName, String attrValue) {String format = String. format (@ "<{0} \ s [^ <>] * {1} \ s * = \ s * (\ x27 | \ x22) {2} (\ x27 | \ x22) [^ <>] *> ", tagName, attrName, attrValue); return FindTag (html, tagName, format );} public static List <HtmlTag> FindTag (String html, String name, String format) {Regex reg = new Regex (format, RegexOptions. IgnoreCase); Regex tagReg = new Regex (String. format (@ "<(\/|) ({0}) (\ s [^ <>] * |)>", name), RegexOptions. ignoreCase); List <HtmlTag> tags = new List <HtmlTag> (); int start = 0; while (true) {Match match = reg. match (html, start); if (match. success) {start = match. index + match. length; Match tagMatch = null; int beginTagCount = 1; while (true) {tagMatch = tagReg. match (html, start); if (! TagMatch. success) {tagMatch = null; break;} start = tagMatch. index + tagMatch. length; if (tagMatch. groups [1]. value = "/") beginTagCount --; else beginTagCount ++; if (beginTagCount = 0) break;} if (tagMatch! = Null) {HtmlTag tag = new HtmlTag (name, match. value, html. substring (match. index + match. length, tagMatch. index-match. index-match. length); tags. add (tag) ;}else {break ;}} else {break ;}} return tags ;}

With the above functions, you can extract the required HTML flag. to capture, you also need a function to download the webpage:

public static String GetHtml(string url){    try    {        HttpWebRequest req = HttpWebRequest.Create(url) as HttpWebRequest;        req.Timeout = 30 * 1000;        HttpWebResponse response = req.GetResponse() as HttpWebResponse;        Stream stream = response.GetResponseStream();        MemoryStream buffer = new MemoryStream();        Byte[] temp = new Byte[4096];        int count = 0;        while ((count = stream.Read(temp, 0, 4096)) > 0)        {            buffer.Write(temp, 0, count);        }        return Encoding.GetEncoding(response.CharacterSet).GetString(buffer.GetBuffer());    }    catch    {        return String.Empty;    }}

/// QQ Group for front-end learning and communication: 461593224

The following describes how to use the HtmlTag class to capture web page information by taking the title and link of the blog homepage as an example:

Class Program {static void Main (string [] args) {String html = HtmlTag. getHtml ("http://www.cnblogs.com"); List <HtmlTag> tags = HtmlTag. findTagByAttr (html, "div", "id", "post_list"); if (tags. count> 0) {List <HtmlTag> item_tags = tags [0]. findTagByAttr ("div", "class", "post_item"); foreach (HtmlTag item_tag in item_tags) {List <HtmlTag> a_tags = item_tag.FindTagByAttr ("a", "class ", "titlelnk"); if (a_tags.Count> 0) {Console. writeLine ("title: {0}", a_tags [0]. innerHTML); Console. writeLine ("link: {0}", a_tags [0]. getAttribute ("href"); Console. writeLine ("");}}}}}

　　

The running result is as follows:

 

You are welcome to study together.

QQ Group for front-end learning and communication: 461593224