Data acquisition Program (web Thief) bit of experience

Collection | program | data | web | experience

The so-called data acquisition program is the Web page thief program (everyone don't scold me oh), finished writing here to send something, I hope we have any ideas to study together.

1. At the beginning of the download data, some sites are to log in to see the corresponding data, this will require us to send the login username and password, but I was logged in, but his server is not garbage, where he redirected, a total of 2 sessions, This 2nd session I don't know how to catch. So I was speculating, using software to capture the session 1 software called Ethereal, with the following code added to the head of the HTTP request
WebClient mywebclient = new WebClient ();
String Sessionkey=textbox78.text;
String Refererurl=textbox77.text;
MyWebClient.Headers.Clear ();
MYWEBCLIENT.HEADERS.ADD ("Cookie", SessionKey);
MYWEBCLIENT.HEADERS.ADD ("Referer", Refererurl);
MYWEBCLIENT.HEADERS.ADD ("User-agent", "mozilla/5.0" (X11; U Linux i686; En-us; rv:1.5) gecko/20031107 debian/1.5-3 ");
This will deceive the server, haha

2. The second part is the code download
byte[] Mydatabuffer = Mywebclient.downloaddata (Remoteuri);
Download = Encoding.Default.GetString (Mydatabuffer);

3. The 3rd is the matching of data, I read the stream to the data, and then use IndexOf to get 2 key fields of the location, and then take out with substring, I know this is stupid, but with regular expression difficult ah (who will guide me), Match the resulting string I removed the HTML code with the following function:
private string striphtml (string strhtml)
{
string [] Aryreg ={
@ "<script[^>]*?>.*?</script>",
@ "< (\/\s*)?!? ((\w+:) \w+) (\w+ (\s*=?\s* ([[""]) (\\[""] tbnr]|[ ^\7] *?\7|\w+) |. {0}) | \s) *? (\/\s*)?> ",
@ "([\ r \ n]) [\s]+",
@ "& (quot| #34);",
@ "& (amp| #38);",
@ "& (lt| #60);",
@ "& (gt| #62);",
@ "& (nbsp| #160);",
@ "& (iexcl| #161);",
@ "& (cent| #162);",
@ "& (pound| #163);",
@ "& (copy| #169);",
@ "&# (\d+);",
@ "-->",
@ "<!--. *\n"
};

string [] Aryrep = {
"",
"",
"",
"\"",
"&",
"<",
">",
" ",
"\xa1",//CHR (161),
"\xa2",//CHR (162),
"\xa3",//CHR (163),
"\xa9",//CHR (169),
"",
"\ r \ n",
""
};

String Newreg =aryreg[0];
String stroutput=strhtml;
for (int i = 0;i<aryreg.length;i++)
{
Regex regex = new Regex (aryreg[i],regexoptions.ignorecase);
Stroutput = Regex. Replace (Stroutput,aryrep[i]);

}

Stroutput.replace ("<", "");
Stroutput.replace (">", "");
Stroutput.replace ("\ r \ n", "");

return stroutput;
}

4. In the back is the warehousing, this everyone understand it. But I still have a problem is that when I write the data, out of the exception, said that my field is too long, can not be written into the database, I use Access, I experimented with SQL bar.

5. Do you have any good suggestions for me to leave a word for the game. Make progress together.