Autohome Auto Brand logo information grasping dotnetspider actual combat [three]

First, the nagging before the subject

The first actual combat blog, reading volume 1000 +, the second, reading volume 200 +, two articles nearly 5 times times, this difference really makes me very laborious, as of today, I have been thinking why there is such a big gap, is because the dry goods less, or what reason, has not thought clearly, if there are readers found the problem, Can comment write down everyone's point of view, when there is such a gap will be what reason, thank you.

Second, Analysis Autohome brand logo page 2.1 Analysis page structure

First we open the Autohome brand logo Selection page https://car.m.autohome.com.cn/, we take the Hua song as an example, in fact we need to be the class is the inside of the IMG src (image path), And strong inside the text (brand) to get on the line, you can see, this is actually very simple, compared to the last time we get the page, access to interface data is much simpler, why should take an individual as an article, is because this place also involves a file download, this piece has not mentioned before.

2.2 Holes in the page

At the beginning of the crawl, I found that many places src is empty, I am very puzzled why this, and then after the breakpoint debugging only found that Autohome logo image in the page has not been zoned here, IMG is not loaded, just occupy a position in that, until the scroll bar roll to where, which picture will be loaded, So here's the way to crawl the IMG, you need to judge

Third, hands-on development 3.1 preparation processor

Private classGetlogoinfoprocessor:basepageprocessor//Get logo Info        {             Publicgetlogoinfoprocessor () {}protected Override voidHandle (Page page) {List<LogoInfoModel> logoinfolist =NewList<logoinfomodel>(); varlogoinfonodes = page. Selectable.xpath (".//div[@id = ' Div_listbrand ']//div[@class = ' item ']").                Nodes (); foreach(varLogoinfoinchlogoinfonodes) {Logoinfomodel Model=NewLogoinfomodel (); Model. Brandname= Logoinfo.xpath ("./strong").                    GetValue (); Model. Imgpath= Logoinfo.xpath ("./img/@src").                    GetValue (); if(model. Imgpath = =NULL) {model. Imgpath= Logoinfo.xpath ("./img/@data-src").                    GetValue (); }                    if(model. Imgpath.indexof ("HTTPS") == -1) {model. Imgpath="https:"+model.                    Imgpath;                    } logoinfolist.add (model); //page. Addtargetrequest (model. Imgpath); //Site Settings downloadfiles True to automatically download files} page. Addresultitem ("logoinfolist", logoinfolist); }        }

3.2 Preparing pipeline

This place, I didn't use his original download method, wrote a simple download method, because I feel his download way down, not very consistent with my business logic

Private classPrintloginfopipe:basepipeline { Public Override voidProcess (ienumerable<resultitems>resultitems, Ispider spider) {                foreach(varResultiteminchResultitems) {                    varLogoinfolist = Resultitem.getresultitem ("logoinfolist") asList<logoinfomodel>; foreach(varLogoinfoinchlogoinfolist) {Console.WriteLine ($"brand:{logoinfo.brandname} Path:{logoinfo.imgpath}");                    SaveFile (Logoinfo.imgpath, logoinfo.brandname); }                }            }            Private voidSaveFile (stringUrlstringfilename) {Httprequestmessage Httprequestmessage=NewHttprequestmessage (); Httprequestmessage.requesturi=NewUri (URL); Httprequestmessage.method=Httpmethod.get; HttpClient HttpClient=NewHttpClient (); varHttpResponse =Httpclient.sendasync (httprequestmessage); varIntervalpath =NewUri (URL); stringFilePath = Environment.currentdirectory +"/img/"; if(!file.exists (FilePath)) {                    Try                    {                        stringfolder =Path.getdirectoryname (FilePath); if(!string. Isnullorwhitespace (folder)) {if(!directory.exists (folder))                            {directory.createdirectory (folder); }} file.writeallbytes (FilePath+ filename +". jpg", HttpResponse.Result.Content.ReadAsByteArrayAsync ().                    Result); }                    Catch{}} httpclient.dispose (); }        }

Storage entity Classes

Private class logoinfomodel{     publicstringgetset;}       Public string Get Set ; }}

3.3 Construction Crawler

Static voidMain (string[] args) {            varsite =NewSite {cycleretrytimes=1, Sleeptime= $,                //Downloadfiles = True, Dotnetspider set whether to download the fileHeaders =Newdictionary<string,string>()                {                    { "Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" },                    { "Cache-control","No-cache" },                    { "Connection","keep-alive" },                    { "Content-type","application/x-www-form-urlencoded; Charset=utf-8" },                    { "user-agent","mozilla/5.0 (Windows NT 10.0; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/66.0.3359.139 safari/537.36"}                }            }; List<Request> reslist =NewList<request>(); Request Res=NewRequest (); Res. URL="https://car.m.autohome.com.cn/"; Res. Method=System.Net.Http.HttpMethod.Get;            Reslist.add (RES); varSpider = Spider.create (site,NewQueueduplicateremovedscheduler (),NewGetlogoinfoprocessor ())                . Addstartrequests (Reslist.toarray ()). Addpipeline (Newprintloginfopipe ()); Spider. Threadnum=1; Spider.            Run ();        Console.read (); }

Downloadfiles Source Analysis in 3.4 site

Source code in the source code in the Httpclientdownloader will automatically determine whether the Downloadfiles in the site is allowed to download the file, the default is False, if the value of Downloadfiles is not set to true, Then for non-string format interface data, will be ignored directly, if you are interested, can be in my code, two lines of comments canceled, then you can see the dotnetspider in the download mode

Iv. Results of implementation

The results of this implementation, has been uploaded to Bilibili, we are interested to open the onlookers

https://www.bilibili.com/video/av24022630/

V. Summary

This time we will crawl data and file download a small synthesis, also introduced the Dotnetspider original download method, and I wrote a download methods, if you encounter similar requirements can choose their own business logic method, I hope this article can help everyone, If you feel that it is not good to write, welcome to shoot the big bricks

Three times blog Source code I have uploaded GitHub, interested can be downloaded directly down Https://github.com/FunnyBoyDeng/SpiderAutoHome

Six, the next period no notice

As for the next issue, I'm not ready to climb anything, welcome to message that you want to crawl things

2018-05-27

Autohome Auto Brand logo information grasping dotnetspider actual combat [three]