C # How to collect the pages to be logged on,

C # How to collect the pages to be logged on,

First note: the code snippets are obtained from the network and modified by yourself. I think something should be shared.

First, let's talk about the principle: When we collect pages, if the website to be collected needs to be logged on to collect. Whether based on cookies or sessions, we will first send an Http request header that contains the Cookie information required by the website. When the website receives the sent Http Request Header, it obtains related Cookie or Session information from the Http Request Header, which is then processed by the program to determine whether you have the permission to access the current page.

Well, the principle is clear, so it is easy to do. All we need to do is to put Cookie Information in the Http Request Header during data collection (or when HttpWebRequest submits data.

Here I provide two methods.

First, directly store the Cookie information in CookieContainer of HttpWebRequest. Check the Code:

Protected void Page_Load (object sender, EventArgs e)
{
// Set the Cookie and save it to Hashtable
Hashtable ht = new Hashtable ();
Ht. Add ("username", "youraccount ");
Ht. Add ("id", "yourid ");
This. Collect (ht );
}
Public void Collect (Hashtable ht)
{
String content = string. Empty;
String url = "http://www.ibest100.com/##page that can be collected ";
String host = "http://www.ibest100.com ";
Try
{
// Obtain the submitted bytes
Byte [] bs = Encoding. UTF8.GetBytes (content );
// Set parameters for submission
HttpWebRequest req = (HttpWebRequest) HttpWebRequest. Create (url );
Req. Method = "POST ";
Req. ContentType = "app/json; charset = UTF-8 ";
Req. ContentLength = bs. Length;
// Put the Cookie into CookieContainer, and then add CookieContainer to HttpWebRequest
CookieContainer cc = new CookieContainer ();
Cc. Add (new Uri (host), new Cookie ("username", ht ["username"]. ToString ()));
Cc. Add (new Uri (host), new Cookie ("id", ht ["id"]. ToString ()));
Req. CookieContainer = cc;
// Submit request data
Stream reqStream = req. GetRequestStream ();
ReqStream. Write (bs, 0, bs. Length );
ReqStream. Close ();
// Receive the returned page, which is required and cannot be omitted
WebResponse wr = req. GetResponse ();
System. IO. Stream respStream = wr. GetResponseStream ();
System. IO. StreamReader reader = new System. IO. StreamReader (respStream, System. Text. Encoding. GetEncoding ("UTF-8 "));
String t = reader. ReadToEnd ();
System. Web. HttpContext. Current. Response. Write (t );
Wr. Close ();
}
Catch (Exception ex)
{
System. Web. HttpContext. Current. Response. Write ("exception in getPostRespone:" + ex. Source + ":" + ex. Message );
}

}

Second, each time you open the collection program, You need to log on to the collected website simulated once, get CookieContainer, and then collect. Check the Code:

Protected void Page_Load (object sender, EventArgs e)
{
Try
{
CookieContainer cookieContainer = new CookieContainer ();
String formatString = "username = {0} & password = {1 }";//***************
String postString = string. Format (formatString, "youradminaccount", "yourpassword ");
// Convert the submitted string data to a byte array
Byte [] postData = Encoding. UTF8.GetBytes (postString );
// Set parameters for submission
String URI = "http://www.ibest100.com/#page ";//***************
HttpWebRequest request = WebRequest. Create (URI) as HttpWebRequest;
Request. Method = "POST ";
Request. KeepAlive = false;
Request. ContentType = "application/x-www-form-urlencoded ";
Request. CookieContainer = cookieContainer;
Request. ContentLength = postData. Length;
// Submit request data
System. IO. Stream outputStream = request. GetRequestStream ();
OutputStream. Write (postData, 0, postData. Length );
OutputStream. Close ();
// Receive the returned page, which is required and cannot be omitted
HttpWebResponse response = request. GetResponse () as HttpWebResponse;
System. IO. Stream responseStream = response. GetResponseStream ();
System. IO. StreamReader reader = new System. IO. StreamReader (responseStream, Encoding. GetEncoding ("gb2312 "));
String srcString = reader. ReadToEnd ();
// Open the page you want to access
URI = "http://www.ibest100.com/ page ";//***************
Request = WebRequest. Create (URI) as HttpWebRequest;
Request. Method = "GET ";
Request. KeepAlive = false;
Request. CookieContainer = cookieContainer;
// Receive the returned page
Response = request. GetResponse () as HttpWebResponse;
ResponseStream = response. GetResponseStream ();
Reader = new System. IO. StreamReader (responseStream, Encoding. GetEncoding ("gb2312 "));
SrcString = reader. ReadToEnd ();
// Output the obtained page or process
Response. Write (srcString );
}
Catch (WebException we)
{
String msg = we. Message;
Response. Write (msg );
}
}

Someone may ask, what if the other party needs a verification code during login? You can use the first method, but you only need to analyze the Cookie of the other party.

Application Scope: data collection, forum posting, and blog posting.

Thanks for editing the article from the Internet: dezai

Reprinted from: http://www.aspnetjia.com