A program has been programmed to be used in the data extraction phase of mathematical modeling. is summarized as follows, the thought writes more detailed, looks like the tutorial-_-.
The purpose is to do a BBS traffic statistics, the data to be analyzed is every minute, what ID is online, the IP of these IDs is how many, and require ID and IP one by one correspond, cannot have duplicate ID and duplicate IP, and require every minute to generate a matrix, is ID IPA segment IPB segment IPC section IPD paragraph, coexist into a text file, file name is hh:mm format. Then the text file into the MATLAB for analysis and processing, the previous data acquisition phase of the implementation of the method is discussed.
The data source obtains from the http://bbs.pku.edu.cn/cgi-bin/bbsusr?to=* (Beida's server is rather rotten, does not go up today)
1, the first data need to download, here using the Msxml2.serverxmlhttp control to asynchronous download to obtain the data source, the reason is not microsoft.xmlhttp because msxml2.serverxmlhttp this control can set the timeout time. This is completely inspired by the process of developing sxna. Because the data source is characterized by changes in the amount of data at any time, not fixed capacity, it is necessary to set the timeout. Otherwise, the program will not execute correctly and easily cause the panic.
2, the second need to automatically download every minute of the data source, and require accurate storage time. That is, you must automatically drive the download program every minute, which requires the program to realize the automatic refresh function. There are several ways to implement the automatic refresh feature, most based on the use of META tags to implement the refresh, but this program requires precise control time download, must be guaranteed in every minute of the download, so the META tag does not apply to this program. Other methods, such as JavaScript inside the reload method, Window.location.href method, etc., take into account the compatibility of the problem to choose Window.location.href to achieve automatic driver. Data updates in Sxna are also used in almost this way.
3, in order to achieve automatic download every minute need to record the next download time, there is application ("MyTime") inside, and then use JavaScript settimeout to determine whether the time to the next download time, and the current remaining time to display on the screen, If the download time is reached, the driver is automatically refreshed. There is a problem in that JavaScript only recognizes the RFC time, so we have to use a ISOTORFC time conversion subroutine. The application ("MyTime") is automatically added for 60 seconds before each drive is downloaded, thus relying on application ("MyTime") to accurately control the download time.
4, after the data comes back, use HTTP. ResponseText to extract the data source information, because the resulting code is HTML code, so here the regular expression for effective data extraction. First analyze the distribution of IDs, notice that each ID is followed by "Bbsqry?name=", so the search code uses "bbsqry\?name= (\w*)" "", because each ID to repeat two times, So the following is the value traversal extraction time Matches.count to mark Step 2. The same method is also used to extract IP, search code using "> (\d+) \. (\d+) \. (\d+) \. (\d+) ", it is worth mentioning that because of the need to extract each IP segment, so to use the submatches, to get each of the child matching values.
5, the extraction of the data is not finished, because the requirements do not appear duplicate ID and IP so I have to find ways to remove duplication, also to ensure that one by one corresponding, how to remove I think of several ways, start to use regular expression, but in view of such a cycle of too many times, the efficiency of the basic row can not, do not have a crash. When I think of the sxna, I put link as the primary key index of the database to avoid duplicate link, so this time also intend to divert. First open access to create a data table, the establishment of an ID IP IPA segment IPB segment IPC segment IPD section 6 columns, Id,ip indexed, no duplicate columns. Automatic number columns are not established because each drive deletes the data and does not compress the database after the data is deleted. This number will grow indefinitely, and the numbering is useless.
6, the data has been removed after the completion of all storage, now you have to make them into TXT file, and at the current time as the file name of the text file. Taking into account the problem of a single digital time, the front should be 0 (suddenly thought of the digital signal processing inside the sequence of the supplement 0 problem-_-), Specifically write right ("0" &minute (Application ("Mytime2")), 2, the contents of the file is very simple to use the database inside the ADODB.stream to write a text file on it, The rest is the basic operation of the database.
In fact, the idea is quite simple, but did a 3-hour-_-, which mainly spent time on the issue of time, for each minute to accurately store a file of the problem I made several implementation plans, and eventually I screened out, leaving behind a kind of most reliable.
(reprint please indicate the origin: http://www.dc9.cn/post/ASPMathematicalModeling.asp)
The above is just the most basic idea, it is written here. Here's the whole code.
<title> automatically save/remove duplicate IDs and ip/accurate on-time storage/in IP ascending order (Sipo made for Xia) v1</title>
How much longer <input type= "text" name= "MyTime" id= "MyTime" "size=" "value=" ">
<br>
<%
' Www.dc9.cn Sipo QQ17862153
' This is to remove the duplicate Id,ip version 13:15
' If you want to sort by ID, change the to name
On Error Resume Next
Dim nowstr
Const TIMEINTERVAL=60
' Set time interval
' If the download time is slow, write 120 seconds
response.lcid=2052
Const LRESOLVE=6
' Resolving domain Name Timeout time, sec
Const LCONNECT=6
' Connect site timeout, seconds
Const LSEND=6
' Send data request timeout, seconds
Const LRECEIVE=40
' Download Data Timeout time, sec
Const Myurl= "http://bbs.pku.edu.cn/cgi-bin/bbsusr?to=*"
' Const myurl= ' http://localhost/test.htm '
If IsEmpty (Application ("Mytime2")) Then
Nowstr=now ()
Application. Lock
Application ("MyTime") =datetostr (Nowstr, "w,d m y h:i:s")
Application ("Mytime2") =nowstr
Application.UnLock
ElseIf DateDiff ("s", Application ("Mytime2"), now >timeinterval Then
Response.Write "Time Reset"
Nowstr=now ()
Application. Lock
Application ("MyTime") =datetostr (Nowstr, "w,d m y h:i:s")
Application ("Mytime2") =nowstr
Application.UnLock
Else
If DateDiff ("s", Application ("Mytime2"), now >=timeinterval Then
Application. Lock
Addstr=dateadd ("S", Timeinterval,application ("mytime2"))
Application ("MyTime") =datetostr (Addstr, "w,d m y h:i:s")
Application ("Mytime2") =addstr
Application.UnLock
End If
End If
Response.Write "Last Save Time" &application ("Mytime2")
Response.Write "<br>"
Response.Write "Next Save Time" &dateadd ("s", Timeinterval,application ("mytime2"))
Response.Write "<br>"
Function Datetostr (Datetime,showtype)
Dim Datemonth,dateday,datehour,dateminute,dateweek,datesecond
Dim Fullweekday,shortweekday,fullmonth,shortmonth,timezone1,timezone2
timezone1= "+0800"
Timezone2= "+08:00"
Fullweekday=array ("Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday")
Shortweekday=array ("Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat")
Fullmonth=array ("January", "February", "March", "April", "may", "June", "July", "August", "September", "October", " November "," December ")
Shortmonth=array ("The", "Feb", "Mar", "APR", "may", "June", "may", "Aug", "Sep", "Oct", "Nov", "Dec")
Datemonth=month (DateTime)
Dateday=day (Date
