(C #) Download the Tianya image in parallel with Scrapysharp

Recently, because of a job need to complete the Cnki crawler, the study of the crawler structure found this is suspected to be transplanted in Python's famous open-source crawler framework Scrapy Scrapysharp, but found on the internet only this F # demo, This C # version of the code is written on a Web site that uses the example in the original text.

PS: The study found that the gap between Scrapysharp and Scrapy is quite large, there is no scrapy as perfect eight components, only to get web content and based on the htmlagilitypack extension of the page parsing function, inexplicable some small disappointment.

Using system;using system.io;using system.linq;using system.threading.tasks;using htmlagilitypack;using Scrapysharp.extensions;using Scrapysharp.network;namespace scrapysharpdemo{class Program {static void Main (            String[] args) {//sample website address var url = "Http://bbs.tianya.cn/post-12-563201-1.shtml";            var web = new Scrapingbrowser (); var html = web.            Downloadstring (new Uri (URL));            var doc = new HTMLDocument (); Doc.            loadhtml (HTML); Get the picture address of the website var urls= doc. Documentnode.cssselect ("Div.bbs-content > img"). Select (node = node). Getattributevalue ("original")).            ToList ();        Parallel download picture Parallel.ForEach (URLs, savepic);            } public static void Savepic (string url) {var web = new Scrapingbrowser (); Because of the restrictions of Tianya website, all external sources can not access the picture, so first set the request header Refer property is the current page address Web.            Headers.add ("Referer", "http://bbs.tianya.cn/post-12-563201-1.shtml"); VAR pic = web. Navigatetopage (new Uri (URL)).            Rawresponse.body; var file = URL. Substring (URL.            LastIndexOf ("/", stringcomparison.ordinal)); if (!            Directory.Exists ("IMGs")) Directory.CreateDirectory ("IMGs");        File.writeallbytes ("IMGs" + file, pic); }    }}

(C #) Download the Tianya image in parallel with Scrapysharp