C # Crawler Small program

C # Crawler Small program

• Design ideas

  Mainly based on HTTP GET Request Web page data, for analysis. Involves recursive calls, multithreading improves efficiency, daemon threads, and so on.

• Related Technologies

  • Abstract class
  • Multithreading
  • Queue
  • Http GET Request
  • String parsing

• Project structure

  • Abschain responsibility Chain abstract class, responsible for defining HTML processing methods, defining recursive processing methods, etc.
  • Absthreadmanager Thread Management abstract class, responsible for defining the watcher thread, managing multithreading
  • Urlqueue URL queue objects, managing URL queues
  • Crawl Crawler object, responsible for combining URL queue and responsibility chain, run crawl function
  • HttpGet HTTP GET Request class, responsible for getting HTML text
  • Threadentity crawler threads, entity objects

• Simple Crawler Example

  The following example is a simple example of getting HTML page text, can be downloaded text, and analysis, can be said to be the simplest crawler

            WebClient wc = new WebClient();            byte[] response = wc.DownloadData("http://www.weather.com.cn/weather/101120501.shtml");            string ss = Encoding.UTF8.GetString(response);

• Project Code Invocation Example

  • Create inheritance classes, inherit responsibility chains, and be responsible for specific crawler methods

    public class Nodechain:abschain {#region remove head ' with '///<summary>/// </summary>//<param name= "url" ></param>//<returns></returns> PR Ivate string removequotation (string url) {if (URL. IndexOf ("'") = = 0) | | (URL.) IndexOf ("\" ") = = 0)) {url = URL.                Remove (0, 1); if (URL. IndexOf ("'")! =-1) {url = URL. Remove (URL.                IndexOf ("'"), 1); } if (URL. IndexOf ("\" ")! =-1) {url = URL. Remove (URL.                IndexOf ("\" "), 1); }} if (URL. IndexOf ("")! =-1) {url = URL. Remove (URL.            IndexOf (""));        } return URL; } #endregion #region Processing web///<summary>////Processing Web page//</summary>// <param name= "HTML" ></param> protected override void Process (string html) {try {Regex re = New Regex (@ "href= (? <web_url>[\s\s]*?) >|href= "" (? <web_url>[\s\s]*?) "" |                Href= ' (? <web_url>[\s\s]*?) '); MatchCollection mc = Re.                Matches (HTML); foreach (Match m in mc) {string url = m.groups["Web_url"].                    ToString (); URL = this.                    Removequotation (URL); if (URL. IndexOf ("http://")! =-1) {urlqueue.getinstance ().                    Enqueue (URL); }} string title = String.                Empty; Re = new Regex (@ "<title[\s\S]*?> (? <title>[\s\s]*?)                </title> "); Match temp = Re. Match (HTML.                ToLower ()); title = Temp. groups["title"].                ToString (); if (!string. IsNullOrEmpty (title)) {Console.WriteLine (string.Format ("page title: {0}", title)); Console.WriteLine (String. Format ("Page url:{0}", this.                URL)); }} catch {}} #endregion}

- 创建线程管理继承类，负责重写新建职责链对象

    public class ThreadManager:AbsThreadManager    {        protected override AbsChain GetChainHeader()        {            return new NodeChain();        }    }

- 设置URL入口，运行爬虫

            try            {                Console.Title = System.Configuration.ConfigurationManager.AppSettings["Title"].ToString();                Console.WriteLine("Process is running！");                                string url = System.Configuration.ConfigurationManager.AppSettings["URL"].ToString();                UrlQueue.GetInstance().Enqueue(url);                ThreadManager thread = new ThreadManager();                thread.Start();            }            catch (Exception ex)            {            }

• GitHub

  . net-app/netspider/

C # Crawler Small program