) Asp.net C # How to collect the pages to be logged on?

Address: http://www.cnblogs.com/subendong/archive/2013/01/31/2886568.html

Note:CodeFragments are obtained from the network and modified by yourself. I think something should be shared.

First, let's talk about the principle: When we collect pages, if the website to be collected needs to be logged on to collect. Whether based on cookies or sessions, we will first send an HTTP request header that contains the cookie information required by the website. When the website receives the sent HTTP Request Header, it obtains the cookie or session information from the HTTP request header.ProgramTo determine whether you have the permission to access the current page.

Well, the principle is clear, so it is easy to do. All we need to do is to put Cookie Information in the HTTP Request Header during data collection (or when httpwebrequest submits data.

Here I provide two methods.

First, directly store the cookie information in cookiecontainer of httpwebrequest. Check the Code:

 Protected   Void Page_load ( Object Sender, eventargs e ){ //  Set the cookie and store it to hashtable. Hashtable ht =New Hashtable (); Ht. Add ( "  Username  " , "  Youraccount  " ); Ht. Add ( "  ID  " , "  Yourid  " ); This . Collect (HT );} Public   Void Collect (hashtable HT ){ String Content = String . Empty; String Url = " Http://www.ibest100.com /##  " ; String Host = "  Http://www.ibest100.com  " ; Try { //  Get submitted bytes                  Byte [] BS = encoding. utf8.getbytes (content ); //  Set parameters for submission Httpwebrequest Req = (httpwebrequest) httpwebrequest. Create (URL); Req. method = "  Post  " ; Req. contenttype = "  APP/JSON; charset = UTF-8  " ; Req. contentlength = BS. length; //  Put the cookie into cookiecontainer, and then add cookiecontainer to httpwebrequest Cookiecontainer cc = New Cookiecontainer (); CC. Add ( New Uri (host ), New Cookie ( "  Username  " , HT [ "  Username  " ]. Tostring (); CC. Add ( New Uri (host ), New Cookie ( "  ID  " , HT [ "  ID " ]. Tostring (); Req. cookiecontainer = cc; //  Submit request data Stream reqstream = Req. getrequeststream (); reqstream. Write (BS, 0 , BS. Length); reqstream. Close (); //  Receive the returned page, required, cannot be omitted Webresponse wR = Req. getresponse (); system. Io. Stream respstream = Wr. getresponsestream (); system. Io. streamreader reader = New System. Io. streamreader (respstream, system. Text. encoding. getencoding ( "  UTF-8  " )); String T = reader. readtoend (); system. Web. httpcontext. Current. response. Write (t); Wr. Close ();} Catch (Exception ex) {system. Web. httpcontext. Current. response. Write ( "  Exception occurs in getpostrespone:  " + Ex. Source + "  :  " + Ex. Message );}}

Second, each time you open the collection program, You need to log on to the collected website simulated once, get cookiecontainer, and then collect. Check the Code:

 Protected   Void Page_load ( Object Sender, eventargs e ){ Try {Cookiecontainer = New Cookiecontainer (); String Formatstring = "  Username = {0} & Password = {1}  " ; //  ***************                  String Poststring = String . Format (formatstring, "  Youradminaccount  " , "  Yourpassword  " ); //  Converts the submitted string data to a byte array.                  Byte [] Postdata = encoding. utf8.getbytes (poststring );//  Set parameters for submission                  String Uri = "  Http://www.ibest100.com/#page  " ; //  *************** Httpwebrequest request = webrequest. Create (URI) As Httpwebrequest; request. method = "  Post  " ; Request. keepalive = False ; Request. contenttype = "  Application/X-WWW-form-urlencoded  " ; Request. cookiecontainer = cookiecontainer; request. contentlength = postdata. length;//  Submit request data System. Io. Stream outputstream = request. getrequeststream (); outputstream. Write (postdata, 0 , Postdata. Length); outputstream. Close (); //  Receive the returned page, required, cannot be omitted Httpwebresponse response = request. getresponse () As Httpwebresponse; system. Io. Stream responsestream = response. getresponsestream (); system. Io. streamreader reader = New System. Io. streamreader (responsestream, encoding. getencoding ( "  Gb2312  " )); String Srcstring = reader. readtoend ();//  Open the page you want to access Uri = "  Http://www.ibest100.com /##  " ; //  *************** Request = webrequest. Create (URI) As Httpwebrequest; request. method = "  Get  " ; Request. keepalive = False ; Request. cookiecontainer = cookiecontainer; //  Receive returned page Response = request. getresponse () As Httpwebresponse; responsestream = response. getresponsestream (); reader = New System. Io. streamreader (responsestream, encoding. getencoding ( "  Gb2312  " ); Srcstring = reader. readtoend (); //  Output page or process Response. Write (srcstring );} Catch (Webexception we ){ String MSG = We. Message; response. Write (MSG );}}

 

Someone may ask, what if the other party needs a verification code during login? You can use the first method, but you only need to analyze the cookie of the other party.

Application Scope: data collection, forum posting, and blog posting.