Asp.net c # implementation principle and code of Logon pages for collection

Description: The code snippets are obtained from the network and modified by yourself. I think something should be shared.

Implementation Principle: When we collect the page, if the website to be collected needs to log in to collect. Whether based on cookies or sessions, we will first send an Http request header that contains the Cookie information required by the website. When the website receives the sent Http Request Header, it obtains related Cookie or Session information from the Http Request Header, which is then processed by the program to determine whether you have the permission to access the current page.

Well, the principle is clear, so it is easy to do. All we need to do is to put Cookie Information in the Http Request Header during data collection (or when HttpWebRequest submits data.

Here I provide two methods.
FirstDirectly put the Cookie information in CookieContainer of HttpWebRequest. Check the Code:
Copy codeThe Code is as follows:
Protected void Page_Load (object sender, EventArgs e)
{
// Set the Cookie and save it to Hashtable
Hashtable ht = new Hashtable ();
Ht. Add ("username", "youraccount ");
Ht. Add ("id", "yourid ");
This. Collect (ht );
}
Public void Collect (Hashtable ht)
{
String content = string. Empty;
String url = "http://www.ibest100.com/##page that can be collected ";
String host = "http://www.ibest100.com ";
Try
{
// Obtain the submitted bytes
Byte [] bs = Encoding. UTF8.GetBytes (content );
// Set parameters for submission
HttpWebRequest req = (HttpWebRequest) HttpWebRequest. Create (url );
Req. Method = "POST ";
Req. ContentType = "app/json; charset = UTF-8 ";
Req. ContentLength = bs. Length;
// Put the Cookie into CookieContainer, and then add CookieContainer to HttpWebRequest
CookieContainer cc = new CookieContainer ();
Cc. Add (new Uri (host), new Cookie ("username", ht ["username"]. ToString ()));
Cc. Add (new Uri (host), new Cookie ("id", ht ["id"]. ToString ()));
Req. CookieContainer = cc;
// Submit request data
Stream reqStream = req. GetRequestStream ();
ReqStream. Write (bs, 0, bs. Length );
ReqStream. Close ();
// Receive the returned page, which is required and cannot be omitted
WebResponse wr = req. GetResponse ();
System. IO. Stream respStream = wr. GetResponseStream ();
System. IO. StreamReader reader = new System. IO. StreamReader (respStream, System. Text. Encoding. GetEncoding ("UTF-8 "));
String t = reader. ReadToEnd ();
System. Web. HttpContext. Current. Response. Write (t );
Wr. Close ();
}
Catch (Exception ex)
{
System. Web. HttpContext. Current. Response. Write ("exception in getPostRespone:" + ex. Source + ":" + ex. Message );
}
}

SecondEach time you open the collection program, You need to log on to the collected website to obtain CookieContainer and then collect it. Check the Code:
Copy codeThe Code is as follows:
Protected void Page_Load (object sender, EventArgs e)
{
Try
{
CookieContainer cookieContainer = new CookieContainer ();
String formatString = "username = {0} & password = {1 }";//***************
String postString = string. Format (formatString, "youradminaccount", "yourpassword ");
// Convert the submitted string data to a byte array
Byte [] postData = Encoding. UTF8.GetBytes (postString );
// Set parameters for submission
String URI = "http://www.ibest100.com/#page ";//***************
HttpWebRequest request = WebRequest. Create (URI) as HttpWebRequest;
Request. Method = "POST ";
Request. KeepAlive = false;
Request. ContentType = "application/x-www-form-urlencoded ";
Request. CookieContainer = cookieContainer;
Request. ContentLength = postData. Length;
// Submit request data
System. IO. Stream outputStream = request. GetRequestStream ();
OutputStream. Write (postData, 0, postData. Length );
OutputStream. Close ();
// Receive the returned page, which is required and cannot be omitted
HttpWebResponse response = request. GetResponse () as HttpWebResponse;
System. IO. Stream responseStream = response. GetResponseStream ();
System. IO. StreamReader reader = new System. IO. StreamReader (responseStream, Encoding. GetEncoding ("gb2312 "));
String srcString = reader. ReadToEnd ();
// Open the page you want to access
URI = "http://www.ibest100.com/ page ";//***************
Request = WebRequest. Create (URI) as HttpWebRequest;
Request. Method = "GET ";
Request. KeepAlive = false;
Request. CookieContainer = cookieContainer;
// Receive the returned page
Response = request. GetResponse () as HttpWebResponse;
ResponseStream = response. GetResponseStream ();
Reader = new System. IO. StreamReader (responseStream, Encoding. GetEncoding ("gb2312 "));
SrcString = reader. ReadToEnd ();
// Output the obtained page or process
Response. Write (srcString );
}
Catch (WebException we)
{
String msg = we. Message;
Response. Write (msg );
}
}

Someone may ask, what if the other party needs a verification code during login? You can use the first method, but you only need to analyze the Cookie of the other party.

Application Scope: Collect data, post on forums, and post on blogs.