Introduction to. Net open-source Web Crawler Abot

. Net also has many open-source crawler tools. abot is one of them. Abot is an open-source. net crawler with high speed and ease of use and expansion. The Project address is https://code.google.com/p/abot/

For the crawled Html, the analysis tool CsQuery is used. CsQuery can be regarded as Jquery implemented in. net, and html pages can be processed using methods similar to Jquery. The CsQuery Project address is https://github.com/afeiship/CsQuery

1. Configure the Abot Crawler

1. Set attributes

First, create a config object, and then set the attributes in config:

CrawlConfiguration crawlConfig = new CrawlConfiguration(); crawlConfig.CrawlTimeoutSeconds = 100; crawlConfig.MaxConcurrentThreads = 10; crawlConfig.MaxPagesToCrawl = 1000; crawlConfig.UserAgentString = "abot v1.0 http://code.google.com/p/abot"; crawlConfig.ConfigurationExtensions.Add("SomeCustomConfigValue1", "1111"); crawlConfig.ConfigurationExtensions.Add("SomeCustomConfigValue2", "2222");

2. Configure through App. config

Read from the configuration file directly, but you can also modify the attributes:

CrawlConfiguration crawlConfig = AbotConfigurationSectionHandler.LoadFromXml().Convert(); crawlConfig.CrawlTimeoutSeconds = 100; crawlConfig.MaxConcurrentThreads = 10;

3. Configure the application to a crawler object

PoliteWebCrawler crawler = new PoliteWebCrawler();PoliteWebCrawler crawler = new PoliteWebCrawler(crawlConfig, null, null, null, null, null, null, null);

2. Use crawlers to register events

The crawler mainly contains four events: Page crawling start, page crawling failure, and page Link crawling failure.

The following is the sample code:

CrawlergeCrawlStartingAsync + = crawler_ProcessPageCrawlStarting; // a single page crawlers. pageCrawlCompletedAsync + = crawler_ProcessPageCrawlCompleted; // The crawler stops crawling a single page. pageCrawlDisallowedAsync + = crawler_PageCrawlDisallowed; // The page cannot be crawled. pageLinksCrawlDisallowedAsync + = crawler_PageLinksCrawlDisallowed; // The Page link does not allow crawling event void crawler_ProcessPageCrawlStarting (object sender, PageCrawlStartingArgs e) {Page ToCrawl pageToCrawl = e. pageToCrawl; Console. writeLine ("About to crawl link {0} which was found on page {1}", pageToCrawl. uri. absoluteUri, pageToCrawl. parentUri. absoluteUri);} void crawler _ processpagecrawlcompleted (object sender, PageCrawlCompletedArgs e) {crawler page crawler ledpage = e. crawledPage; if (crawledPage. webException! = Null | crawledPage. HttpWebResponse. StatusCode! = HttpStatusCode. OK) Console. writeLine ("Crawl of page failed {0}", crawler page. uri. absoluteUri); else Console. writeLine ("Crawl of page succeeded {0}", crawler page. uri. absoluteUri); if (string. isNullOrEmpty (crawledPage. content. text) Console. writeLine ("Page had no content {0}", crawler Page. uri. absoluteUri);} void crawler _ pagelinkscrawldisallowed (object sender, PageLinksCrawlDisallowedArgs e) {crawler page crawler ledpage = e. crawledPage; Console. writeLine ("Did not crawl the links on page {0} due to {1}", crawler page. uri. absoluteUri, e. disallowedReason);} void crawler_PageCrawlDisallowed (object sender, PageCrawlDisallowedArgs e) {PageToCrawl pageToCrawl = e. pageToCrawl; Console. writeLine ("Did not crawl page {0} due to {1}", pageToCrawl. uri. absoluteUri, e. disallowedReason );}

 

3. Add multiple additional objects for the crawler

Abot should refer to ViewBag in Asp.net MVC, and set object-level CrwalBag and Page-level ViewBag for Crawler objects.

PoliteWebCrawler crawler = new PoliteWebCrawler (); crawler. crawlBag. myFoo1 = new Foo (); // CrwalBagcrawler at the object level. crawlBag. myFoo2 = new Foo (); crawler. pageCrawlStartingAsync + = crawler_ProcessPageCrawlStarting ;... void crawler_ProcessPageCrawlStarting (object sender, PageCrawlStartingArgs e) {// obtain the object CrawlContext context = e. crawlContext; context. crawlBag. myFoo1.Bar (); // use CrwalBag context. crawlBag. myFoo2.Bar (); // use PageBag e at the page level. pageToCrawl. pageBag. bar = new Bar ();}

4. Start Crawler

It is very easy to start crawlers. Call the Crawl method and specify the start page.

CrawlResult result = crawler.Crawl(new Uri("http://localhost:1111/"));if (result.ErrorOccurred)        Console.WriteLine("Crawl of {0} completed with error: {1}", result.RootUri.AbsoluteUri, result.ErrorException.Message);else        Console.WriteLine("Crawl of {0} completed without error.", result.RootUri.AbsoluteUri);

V. Introduction to CsQuery

In the PageCrawlCompletedAsync event, e. CrawledPage. CsQueryDocument is a CsQuery object.

Here we will introduce the advantages of CsQuery in analyzing Html:

cqDocument.Select(".bigtitle > h1")

Here, the selector is used exactly the same as Jquery. Here, the h1 tag under the. bittitle class is used. If you are familiar with Jquery, getting started with CsQuery is fast and easy.