C # write crawler, version V2.0

This version is mainly based on Baidu image for the object, the crawler operation, to achieve the most basic download function, but there are a lot of defects, it will be improved in the future.

Open Baidu Pictures, while opening the developer tools, we will find that the Baidu image is through the following section of Ajax to load the picture.

http://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1& Fm=result&fr=&sf=1&fmq=1466428638972_r&pv=&ic=0&nc=1&z=&se=1&showtab=0 &fb=0&width=&height=&face=0&istype=2&ie=utf-8&word=%E5%94%90%E5%AB%A3&f=3& Oq=tangyan&rsp=0

Here, we just need to understand that word is behind our keywords, then, this is better, combined with a part of the V1.0 code, can be developed quickly, the principle and V1.0 similar.

The background code is as follows:

usingSystem;usingSystem.Collections.Generic;usingSystem.ComponentModel;usingSystem.Data;usingSystem.Drawing;usingSystem.IO;usingSystem.Linq;usingSystem.Net;usingSystem.Text;usingSystem.Threading.Tasks;usingSystem.Windows.Forms;usingNewtonsoft.Json.Linq;usingNewtonsoft.json;usingSystem.Text.RegularExpressions;namespaceDynamic web crawler for Baidu images { Public Partial classForm1:form {Static intCount =0;  PublicForm1 () {InitializeComponent (); }        Private voidBtndo_click (Objectsender, EventArgs e) {            intPagecount=2; stringKeyword = This. Keywords.text;  for(inti =0; I <pageCount; i++) {HttpWebRequest Request= (HttpWebRequest) httpwebrequest.create ("Http://image.baidu.com/search/index?tn=baiduimage&ipn=r&ct=201326592&cl=2&lm=-1&st=-1 &fm=result&fr=&sf=1&fmq=1466307565574_r&pv=&ic=0&nc=1&z=&se=1&showtab= 0&fb=0&width=&height=&face=0&istype=2&ie=utf-8&word="+keyword. ToString ());                using(HttpWebResponse response =(HttpWebResponse) request. GetResponse ()) {if(Response. StatusCode = =Httpstatuscode.ok) {using(Stream stream =Response. GetResponseStream ()) {Try                            {                                //Download all pictures of the specified pageDownloadpage (stream); }                            Catch(Exception ex) {//Txtlogs to access the UI thread across threads                                                           }                        }                    } Else                    {                       //MessageBox.Show ("Get section" + PageCount + "page failed:" + response. StatusCode);}}} MessageBox.Show ("successful execution, a total"+count. ToString () +"Image"); }        Private Static string[] Getlinks (stringhtml) {            Const stringPattern =@"http://([\w-]+\.) +[\w-]+ (/[\w-./?%&=]*)?"; Regex R=NewRegex (pattern, regexoptions.ignorecase);//New Regular ModeMatchCollection m = r.matches (HTML);//Get matching results            string[] links =New string[M.count]; intCount=0;  for(inti =0; i < M.count; i++)            {                if(Isvaliable (m[i). ToString ())) {Links[count]= M[i]. ToString ();//Extract the resultscount++; }                          }            returnlinks; }        Private voidDownloadpage (Stream stream) {using(StreamReader reader=NewStreamReader (Stream)) {                stringR1; StringBuilder SB=NewStringBuilder ();  while(R1=reader. ReadLine ())! =NULL) {sb.                Append (R1); } FileStream afile=NewFileStream (".. /.. /txt.txt", FileMode.OpenOrCreate); StreamWriter SW=NewStreamWriter (Afile);//Store a webpage in a txt text fileSW. WriteLine (sb.)                ToString ()); Sw.                Close (); string[] s; S=getlinks (sb.)                ToString ()); inti =0;  for(i=0; I<s.count (); i++)                {                    if(s[i]!=NULL|| s[i]!="") {Count++;                    SavePicture (S[i]); }                                   }                 This. Label2. Text =count.            ToString (); }        }        Private Static BOOLIsvaliable (stringURL) {            if(URL.) Contains (". jpg") || Url. Contains (". gif") || Url. Contains (". PNG"))            {                return true;//get some resources like pictures            }            return false; }        Private Static voidSavePicture (stringpath) {Dataclasses1datacontext db=NewDataclasses1datacontext (); Uri URL=NewUri (path); HttpWebRequest WebRequest=(HttpWebRequest) httpwebrequest.create (URL); Webrequest.referer="http://image.baidu.com"; HttpWebResponse WebResponse=(HttpWebResponse) webrequest.getresponse (); if(isvaliable (path))//determine if it is a picture, store it in a database. {Bitmap MyImage=NewBitmap (WebResponse.GetResponseStream ()); MemoryStream Ms=NewMemoryStream ();                Myimage.save (MS, System.Drawing.Imaging.ImageFormat.Jpeg); varp =NewPictureurl {pictureUrl1=Ms.                ToArray ()};                Db.pictureUrl.InsertOnSubmit (P); Db.            SubmitChanges (); }        }    }}

Demo Effect:

This program only solves the problem, there are many problems, will continue to solve.

C # write crawler, version V2.0