"Source code" probe into C # crawler, continuous update ...

recently saw someone in the garden using Python to do the crawler software and upload the source code, suffering from the lack of understanding of python, then think of C # also realize a simple crawler software. So last night spent one hours of time to achieve a simple crawler software, the function is very simple, but feel like to share it ... The subsequent landlord will also continue to improve the function.

Nonsense not much to say, directly on the code!

 

In fact, the code is simple:

   Public stringCreateweb (stringURL) {StringBuilder SB=NewStringBuilder (); //Crawling Web pagesWebRequest request =webrequest.create (URL); WebResponse response=request.            GetResponse (); //Read file streamStreamReader reader =NewStreamReader (response. GetResponseStream (), Encoding.GetEncoding ("Utf-8"));//Reader. ReadToEnd () means to obtain the source of the Web page//FileStream fs = new FileStream ("~/baidu.html", FileMode.OpenOrCreate);            stringstrHTML =Reader.            ReadToEnd (); //regular match the picture label of the website            stringRXG =@") *)"; //match a collection of picture labelsMatchCollection MC =regex.matches (strhtml, RXG);  for(inti =0; I < MC. Count; i++) {sb.                        Append (Mc[i]); }            //return picture label HTML output            returnsb.        ToString (); }

Summary: In fact, the crawler is nothing more than crawling the page, and then through some rules to match the elements inside the page.

Source:Supersearch.rar

Liuhuatao( Liuhuatao ' s blog on the blog Park )
Source:http://www.cnblogs.com/Lhuatao/
This work is authored by LIuhuatao and is licensed under the Creative Commons Attribution-NonCommercial use-no derivative of the 2.5 Chinese mainland license Agreement . Welcome reprint, but any reprint must retain the complete article, in the prominent place displays the signature as well as the original text link. If you have any questions or authorization to negotiate, please leave me a message .

"Source code" probe into C # crawler, continuous update ...