ASP. NET captures webpage content and asp.net webpage content

ASP. NET captures webpage content and asp.net webpage content

I. ASP. NET uses HttpWebRequest to capture webpage content

 

This methodCaptureSomePage MEETINGFailed

However, sometimes we will find that this program fails to obtain the required content when crawling some pages, and sometimes even returns the Error 404 page. Why?

In fact, many people ignore a problem, that is, the Default Server Browser. Some servers use the mobile browser by default. When I capture the webpage under this server, it is equivalent to using the mobile browser to open the webpage, if the target webpage to be crawled does not have a corresponding mobile phone webpage, unexpected results will be returned, some will return the 404 error prompt page, and some will return the 403 error prompt page, some even go to other webpages.

 

How can this problem be solved?

To solve this problem, we only need to specify the browser used in the program, that is, to set the parameter value of UserAgent.

 

Complete code:

C # code Replication

/// <Summary> Method 1: It is recommended to use HttpWebRequest to obtain the webpage source code. // It is very effective for webpages with BOM, no matter what the encoding is, you can correctly identify /// </summary> /// <param name = "url"> webpage address "</param> /// <returns> to return the webpage source file </returns> public static string GetHtmlSource2 (string url) {// processing content string html = ""; HttpWebRequest request = (HttpWebRequest) WebRequest. create (url); request. accept = "*/*"; // Accept any file request. userAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2 ;. net clr 1.1.4322) "; // request. allowAutoRedirect = true; // whether 302 is allowed // request. cookieContainer = new CookieContainer (); // cookie container, request. referer = url; // reference of the current page HttpWebResponse response = (HttpWebResponse) request. getResponse (); Stream stream = response. getResponseStream (); StreamReader reader = new StreamReader (stream, Encoding. default); html = reader. readToEnd (); stream. close (); return html ;}

 

Ii. ASP. NET uses WebResponse to capture webpage content

C # code Replication

        public static string GetHttpData2(string Url)        {            string sException = null;            string sRslt = null;            WebResponse oWebRps = null;            WebRequest oWebRqst = WebRequest.Create(Url);            oWebRqst.Timeout = 50000;            try            {                oWebRps = oWebRqst.GetResponse();            }            catch (WebException e)            {                sException = e.Message.ToString();            }            catch (Exception e)            {                sException = e.ToString();            }            finally            {                if (oWebRps != null)                {                    StreamReader oStreamRd = new StreamReader(oWebRps.GetResponseStream(), Encoding.GetEncoding("utf-8"));                    sRslt = oStreamRd.ReadToEnd();                    oStreamRd.Close();                    oWebRps.Close();                }            }            return sRslt;        }

 

3. ASP. NET uses WebClient to capture webpage content

C # code Replication

/// <Param name = "url">/url of the website to be accessed </param> /// <param name = "charSets"> encoding of the target webpage, if the input is null or "", the code of the webpage is automatically analyzed </param> // <returns> </returns> public static string getHtml (string url, params string [] charSets) {try {string charSet = null; if (charSets. length = 1) {charSet = charSets [0];} WebClient myWebClient = new WebClient (); // note the following when creating a WebClient instance myWebClient: // some webpages may not be available, for various reasons such as cookie requirements and encoding problems // This requires specific analysis, such as adding cookies to the header // webclient. headers. add ("Cookie", cookie); // This may require some overload methods. you can write it as needed // obtain or set the network creden. used to authenticate requests to Internet resources. myWebClient. credentials = CredentialCache. defaultCredentials; // if the server wants to verify the user name, password // NetworkCredential mycred = new NetworkCredential (struser, strpassword); // myWebClient. credentials = mycred; // download data from the resource and return a byte array. (add @ because the URL contains the "/" symbol.) byte [] myDataBuffer = myWebClient. download Data (url); string strWebData = Encoding. default. getString (myDataBuffer); // obtain the character encoding description of the webpage. Match charSetMatch = Regex. match (strWebData, "<meta ([^ <] *) charset = ([^ <] *)", RegexOptions. ignoreCase | RegexOptions. multiline); string webCharSet = charSetMatch. groups [2]. value; if (charSet = null | charSet = "") charSet = webCharSet; if (charSet! = Null & charSet! = "" & Encoding. GetEncoding (charSet )! = Encoding. default) {strWebData = Encoding. getEncoding (charSet ). getString (myDataBuffer);} else {strWebData = Encoding. getEncoding ("UTF-8 "). getString (myDataBuffer);} return strWebData;} catch (Exception e) {return "";}}

How does ASPNET capture specified webpage data?

You use httprequest directly in the background ....
To obtain the entire web page.
Then, use a regular expression. Or you can use methods such as string truncation to get what you want.
You can use html for parsing.
Next HTML pack. dll. Available online

How does ASPNET capture data on a specified page?

Private string GetStringByUrl (string strUrl)
{
WebRequest wrt = WebRequest. Create (strUrl );
WebResponse wrse = wrt. GetResponse ();
Stream strM = wrse. GetResponseStream ();
StreamReader SR = new StreamReader (strM, Encoding. GetEncoding ("gb2312 "));
String strallstrm = SR. ReadToEnd ();
Return strallstrm;
}

Obtain the source code of the corresponding page, and then match the TAG content you need through regular expressions.