How C # collects pages that need to be signed in

First of all: The code snippet is obtained from the network and then modified by itself. I think the good things should be shared.

First, the principle: when we collect the page, if the collected sites need to log in to collect. Whether cookie-based or session-based, we will first send an HTTP request header, which contains the cookie information required by the site. When the Web site receives the HTTP request header sent over, it gets the relevant cookie or session information from the HTTP request header, which is then processed by the program to determine whether you have permission to access the current page.

Well, the principle is clear, it's good to run. All we have to do is to put the cookie information in the HTTP request header at the time of collection (or when HttpWebRequest commits the data).

Here I offer 2 ways to do this.

First, the cookie information is placed directly into the HttpWebRequest Cookiecontainer. Look at the code:

protected void Page_Load (object sender, EventArgs e)
{
Set cookies, deposit Hashtable
Hashtable ht = new Hashtable ();
Ht. ADD ("username", "youraccount");
Ht. ADD ("id", "Yourid");
This. Collect (HT);
}
public void Collect (Hashtable ht)
{
String content = String. Empty;
String url = "http://www.ibest100.com/a page that needs to be logged in to capture";
String host = "http://www.ibest100.com";
Try
{
Gets the Committed bytes
byte[] bs = Encoding.UTF8.GetBytes (content);
Set the relevant parameters for the submission
HttpWebRequest req = (HttpWebRequest) httpwebrequest.create (URL);
Req. Method = "POST";
Req. ContentType = "Application/json;charset=utf-8";
Req. ContentLength = BS. Length;
Put the cookie in Cookiecontainer, and then add the Cookiecontainer to the HttpWebRequest
Cookiecontainer cc = new Cookiecontainer ();
Cc. ADD (New Uri (host), New Cookie ("username", ht["username"). ToString ()));
Cc. ADD (New Uri (host), New Cookie ("id", ht["id"). ToString ()));
Req. Cookiecontainer = CC;
Submit Request Data
Stream Reqstream = req. GetRequestStream ();
Reqstream.write (BS, 0, BS. Length);
Reqstream.close ();
Receives the returned page, must, cannot omit
WebResponse WR = req. GetResponse ();
System.IO.Stream Respstream = wr. GetResponseStream ();
System.IO.StreamReader reader = new System.IO.StreamReader (Respstream, System.Text.Encoding.GetEncoding ("Utf-8"));
String t = Reader. ReadToEnd ();
System.Web.HttpContext.Current.Response.Write (t);
wr. Close ();
}
catch (Exception ex)
{
System.Web.HttpContext.Current.Response.Write ("Exception in Getpostrespone:" + ex.) Source + ":" + ex. Message);
}

}

Second, each time you open the acquisition program, you need to go to the site of the acquisition simulation login once, get Cookiecontainer, and then collect. Look at the code:

protected void Page_Load (object sender, EventArgs e)
{
Try
{
Cookiecontainer Cookiecontainer = new Cookiecontainer ();
String formatString = "Username={0}&password={1}";//***************
String poststring = String. Format (formatString, "Youradminaccount", "YourPassword");
Converts a committed string data into a byte array
byte[] PostData = Encoding.UTF8.GetBytes (poststring);
Set the relevant parameters for the submission
String URI = "http://www.ibest100.com/login page";//***************
HttpWebRequest request = WebRequest.Create (URI) as HttpWebRequest;
Request. Method = "POST";
Request. KeepAlive = false;
Request. ContentType = "application/x-www-form-urlencoded";
Request. Cookiecontainer = Cookiecontainer;
Request. ContentLength = Postdata.length;
Submit Request Data
System.IO.Stream outputstream = Request. GetRequestStream ();
Outputstream.write (postdata, 0, postdata.length);
Outputstream.close ();
Receives the returned page, must, cannot omit
HttpWebResponse response = Request. GetResponse () as HttpWebResponse;
System.IO.Stream Responsestream = Response. GetResponseStream ();
System.IO.StreamReader reader = new System.IO.StreamReader (Responsestream, encoding.getencoding ("gb2312"));
String srcstring = reader. ReadToEnd ();
Open the page you want to visit
URI = "Page http://www.ibest100.com/needs to be logged in to capture";//***************
Request = WebRequest.Create (URI) as HttpWebRequest;
Request. Method = "GET";
Request. KeepAlive = false;
Request. Cookiecontainer = Cookiecontainer;
Receive the returned page
Response = Request. GetResponse () as HttpWebResponse;
Responsestream = Response. GetResponseStream ();
reader = new System.IO.StreamReader (Responsestream, encoding.getencoding ("gb2312"));
srcstring = reader. ReadToEnd ();
The output gets the page or processing
Response.Write (srcstring);
}
catch (WebException We)
{
String msg = We. Message;
Response.Write (msg);
}
}

Perhaps some people will ask, if the other side log on when the verification code to do? Then you use the first way, but you need to analyze each other's cookies.

Application scope: Collect data, forum post, blog posts.

Thanks for the article from the Network editor: Dezai

Reprinted from: http://www.aspnetjia.com

How C # collects pages that need to be signed in