Efficient webpage source code Retrieval

Without relying on the web browser interface, you can use the. NET encapsulated WebClient to download the page, or use httprequest to send a request. Of course, you can also return to the COM component XMLHTTP for retrieval. It turns out that the COM component is very fast and easy to use. Because the page encoding is automatically recognized.

If you don't want to talk about the code, you need to add a com reference to the following test code: Microsoft XML can be used in any version. I use 6.0.

The test code is as follows. Very effective

Public static void test () <br/>{< br/> int tick = environment. tickcount; <br/> string html = gethtmlcom ("http://www.csdn.net"); <br/> file. writealltext ("comhtml.txt", HTML); <br/> console. writeline ("com:" + (environment. tickcount-tick ). tostring (); <br/> tick = environment. tickcount; <br/> html = gethtmlwebclient ("http://www.csdn.net"); <br/> file. writealltext ("webclient.txt", HTML); <br/> cons Ole. writeline ("WebClient:" + (environment. tickcount-tick ). tostring (); <br/> tick = environment. tickcount; <br/> html = webfunc. gethtmlex ("http://www.csdn.net", encoding. utf8); <br/> file. writealltext ("webrequest.txt", HTML); <br/> console. writeline ("webrequest:" + (environment. tickcount-tick ). tostring (); <br/>}</P> <p> Public static string gethtmlcom (string URL) <br/>{< br/> XMLHTTP xmlhtt P = new xmlhttpclass (); <br/> XMLHTTP. open ("get", URL, false, null, null); <br/> XMLHTTP. send (""); <br/> while (XMLHTTP. readystate! = 4) thread. sleep (1); <br/> return XMLHTTP. responsetext; <br/>}</P> <p> Public static string gethtmlwebclient (string URL) <br/>{< br/> return encoding. utf8.getstring (New WebClient (). downloaddata (URL); <br/>}</P> <p> static class webfunc <br/>{< br/> Private Static cookiecontainer cookie = new cookiecontainer (); <br/> Private Static string contenttype = "application/X-WWW-form-urlencoded"; <br/> Private Static string accept = "image/GIF, image/X-xbitmap, image/JPEG, image/pjpeg, application/X-Shockwave-flash, application/X-Silverlight, application/vnd. MS-Excel, application/vnd. MS-PowerPoint, application/MSWord, application/X-MS-application, application/X-MS-xbap, application/vnd. MS-xpsdocument, application/XAML + XML, application/x-silverlight-2-b1, */* "; <br/> Private Static string useragent =" Mozilla/4.0 (compatible; MSIE 7.0; windows NT 5.1 ;. net CLR 2.0.50727 ;. net CLR 3.0.04506.648 ;. net CLR 3.5.21022) "; </P> <p> Public static string gethtmlex (string URL, encoding) <br/>{< br/> httpwebrequest request = (httpwebrequest) webrequest. create (URL); <br/> request. useragent = useragent; <br/> request. contenttype = contenttype; <br/> request. cookiecontainer = cookie; <br/> request. accept = accept; <br/> request. method = "get"; </P> <p> webresponse response = request. getresponse (); <br/> stream responsestream = response. getresponsestream (); <br/> streamreader reader = new streamreader (responsestream, encoding); <br/> string html = reader. readtoend (); <br/> response. close (); </P> <p> return HTML; <br/>}< br/>}

 

The output time is as follows:

"Com: 140"
WebClient: 3074
Web Request: 218