C # Crawl data on a Web page

recent job demands regularly crawl the temperature of different cities every day. In fact, through the method of programming to crawl different Web pages for analysis and screening process. NET provides many classes to access and obtain data for remote Web pages, such as the WebClient class and the HttpWebRequest class. These classes are useful for using HTTP to access remote Web pages and downloading them, but it is very weak in terms of the parsing capabilities of the downloaded HTML. Recommend an open Source component HTML Agility Pack (http://htmlagilitypack.codeplex.com/), which is designed to simplify the reading and writing of HTML documents as much as possible. The package itself uses the DOM document Object model to parse the HTML. In this way record a recent collection of crawl history and current weather for the site backup:

• Support Historical Weather query in Chinese website: http://lishi.tianqi.com/
• Similar sites: http://tianqi.2345.com/wea_history/59287.htm (last 2-3 years data)
• Website: http://www.wunderground.com/history/(last 18 years data)
• This site is supported for a longer period of time: http://www.tutiempo.net/en/Climate/

Examples of programming use are: we want to get weather information from the following pages:

Download the HTML Agility Pack component, create a new console program, reference the corresponding framework version of the component in your project, the sample code is as follows:         

stringURL =@"http://lishi.tianqi.com/beijing/201701.html"; varWebGet =NewHtmlweb (); varDocument =webget.load (URL); vardiv = document. Documentnode.selectnodes ("//div[@class = ' tqtongji2 ']/ul"); foreach(Htmlnode nodeinchDiv) {varTmpnode = node. SelectNodes ("Li"); Console.WriteLine (string. Format ("{ 0}-----------{1}---------{2}----------{3}", tmpnode[0]. InnerText, tmpnode[1]. InnerText, tmpnode[2]. InnerText, tmpnode[3].            InnerText)); } console.readkey ();

Program running effect: Chinese is garbled, such as

By parsing the HTML Agility pack source code, in the Htmlweb class get (Uri Uri, string method, String path, HTMLDocument doc) method, the local variable RESP is the response of the HTTP request. Set breakpoints to discover resp. ContentEncoding is empty. So download the data via HttpWebRequest, the sample code is as follows:

stringURL =@"http://lishi.tianqi.com/beijing/201701.html"; HttpWebRequest req= WebRequest.Create (NewUri (URL)) asHttpWebRequest; Req. Method="GET"; WebResponse rs=req.            GetResponse (); Stream RSS=Rs.            GetResponseStream (); HTMLDocument Doc=NewHTMLDocument (); Doc.            Load (RSS); vardiv = doc. Documentnode.selectnodes ("//div[@class = ' tqtongji2 ']/ul"); foreach(Htmlnode nodeinchDiv) {varTmpnode = node. SelectNodes ("Li"); Console.WriteLine (string. Format ("{ 0}-----------{1}---------{2}----------{3}", tmpnode[0]. InnerText, tmpnode[1]. InnerText, tmpnode[2]. InnerText, tmpnode[3].            InnerText)); } console.readkey ();

The code works as follows:

That ' s OK!!!

C # Crawl data on a Web page