C # capture webpage data, analyze and remove HTML tags

First, capture the entire webpage content and place the data in byte [] (the Network upload and transmission form is byte) to further convert it to a string to facilitate its operation. The example is as follows:

Private Static string getpagedata (string URL)
{
If (url = NULL | URL. Trim () = "")
Return NULL;
WebClient WC = new WebClient ();
WC. Credentials = credentialcache. defaultcredentials;
Byte [] pagedata = WC. downloaddata (URL );
Return encoding. Default. getstring (pagedata); //. ASCII. getstring
}

Obtain the string form of the data, and then parse the webpage (in fact, it is the application of various string operations and regular expressions ):
// Parse the page and find the link
// Extension is required, and some forms of links are not recognized.
String strref = @ "(href | SRC | action) [] * = [] * ["'] [^" "' #>] + [" '] ";
Matchcollection matches = new RegEx (strref). Matches (strresponse );
Strstatus + = "found:" + matches. Count + "Connections \ r \ n ";

In the above example, links in the web page are parsed. The strref variable represents the regular expression pattern, the variable matches represents the set of items that match the matching, and the subsequent RegEx (strref ). matches (strresponse) is used to create regular rules so that all strings in strresponse that conform to the strref mode are returned. Then, call the matches variable to obtain various information.
Of course, only some basic link forms can be identified here, such as links in scripts and links without "" are not supported. This extension is relatively simple.

Common resolutions include the following:
// Obtain the title
Match titlematch = RegEx. Match (strresponse, "<title> ([^ <] *) </title>", regexoptions. ignorecase | regexoptions. multiline );
Title = titlematch. Groups [1]. value;

// Obtain the description
Match DESC = RegEx. match (strresponse, "<meta name = \" Description \ "content = \" ([^ <] *) \ ">", regexoptions. ignorecase | regexoptions. multiline );
Strdesc = DESC. Groups [1]. value;

// Obtain the webpage size
Size = strresponse. length;

// Remove HTML tags

Private string striphtml (string strhtml)
{
RegEx objregexp = new RegEx ("<(. | \ n) +?> ");
String stroutput = objregexp. Replace (strhtml ,"");
Stroutput = stroutput. Replace ("<", "& lt ;");
Stroutput = stroutput. Replace (">", "& gt ;");
Return stroutput;
}
Some exceptions may make the removal non-clean, so it is recommended that the conversion be performed twice in a row. In this way, HTML tags are converted to spaces. Too many consecutive spaces will affect subsequent string operations. Therefore, add the following statement:
// Convert all spaces into one space
RegEx r = new RegEx (@ "\ s + ");
Wordsonly = R. Replace (strresponse ,"");
Wordsonly. Trim ();