c# Regex對網頁進行有效內容抽取

搜尋引擎中一個比較重要的環節就是從網頁中抽取出有效內容。簡單來說，就是吧HTML文本中的HTML標記去掉,留下我們用IE等瀏覽器開啟HTML文檔看到的部分（我們這裡不考慮圖片）.
將HTML文本中的標記分為:注釋,script ,style，以及其他標記分別去掉：
1.去注釋,正則為:
output = Regex.Replace(input, @"<!--[^-]*-->", string.Empty, RegexOptions.IgnoreCase);
2.去script,正則為:
ouput = Regex.Replace(input, @"<script[^>]*?>.*?</script>", string.Empty, RegexOptions.IgnoreCase | RegexOptions.Singleline);
output2 = Regex.Replace(ouput , @"<noscript[^>]*?>.*?</noscript>", string.Empty, RegexOptions.IgnoreCase | RegexOptions.Singleline);
3.去style,正則為:
output = Regex.Replace(input, @"<style[^>]*?>.*?</style>", string.Empty, RegexOptions.IgnoreCase | RegexOptions.Singleline);
4.去其他HTML標記
result = result.Replace(" ", " ");
result = result.Replace(""", "\"");
result = result.Replace("<", "<");
result = result.Replace(">", ">");
result = result.Replace("&amp", "&");
result = result.Replace("<br>", "\r\n");
result = Regex.Replace(result, @"<[\s\S]*?>", string.Empty, RegexOptions.IgnoreCase);
以上的代碼中大家可以看到,我使用了RegexOptions.Singleline參數，這個參數很重要，他主要是為了讓"."(小圓點)可以匹配分行符號.如果沒有這個參數，大多數情況下，用上面列Regex來消除網頁HTML標記是無效的.
HTML發展至今，文法已經相當複雜,上面只列出了幾種最主要的標記,更多的去HTML標記的正則我將在
Rost WebSpider 的開發過程中補充進來。
下面用c#實現了一個從HTML字串中提取有效內容的類:
using System;
using System.Collections.Generic;
using System.Text;
using System.Text.RegularExpressions;
class HtmlExtract
{
#region private attributes
private string _strHtml;
#endregion
#region public mehtods
public HtmlExtract(string inStrHtml)
{
_strHtml = inStrHtml
}
public override string ExtractText()
{
string result = _strHtml;
result = RemoveComment(result);
result = RemoveScript(result);
result = RemoveStyle(result);
result = RemoveTags(result);
return result.Trim();
}
#endregion
#region private methods
private string RemoveComment(string input)
{
string result = input;
//remove comment
result = Regex.Replace(result, @"<!--[^-]*-->", string.Empty, RegexOptions.IgnoreCase);
return result;
}
private string RemoveStyle(string input)
{
string result = input;
//remove all styles
result = Regex.Replace(result, @"<style[^>]*?>.*?</style>", string.Empty, RegexOptions.IgnoreCase | RegexOptions.Singleline);
return result;
}
private string RemoveScript(string input)
{
string result = input;
result = Regex.Replace(result, @"<script[^>]*?>.*?</script>", string.Empty, RegexOptions.IgnoreCase | RegexOptions.Singleline);
result = Regex.Replace(result, @"<noscript[^>]*?>.*?</noscript>", string.Empty, RegexOptions.IgnoreCase | RegexOptions.Singleline);
return result;
}
private string RemoveTags(string input)
{
string result = input;
result = result.Replace(" ", " ");
result = result.Replace(""", "\"");
result = result.Replace("<", "<");
result = result.Replace(">", ">");
result = result.Replace("&amp", "&");
result = result.Replace("<br>", "\r\n");
result = Regex.Replace(result, @"<[\s\S]*?>", string.Empty, RegexOptions.IgnoreCase);
return result;
}
#endregion