Does it often have to analyze user behavior? Do you often encounter multiple server uploads with the logs analyzed together? Is the time interval for data statistics very short? There is also wood because the log file is too large, and need to block processing?
1, note that in the log writing must be in accordance with a strict format, so that in the time of parsing, only good segmentation. For example gameid:123 gameid:2333. Split the unified standard can be.
2, in the generation of the file name of the log should also be in accordance with certain rules, in the analysis of the time, is a good match, such as Serverhostname_date.log in the match when the only need to Glob (*-date.log); Glob See the PHP function manual to find the file path that matches the pattern.
3. Why use NoSQL? In fact, the project teacher is not only limited to know how to achieve, but to think more about what kind of business to use what kind of tools to solve. Non-relational data is very suitable for this, the log often add new behavior, you use Key-value way, do not need the log new to analyze the behavior, you have to manually change the configuration of your program, so I personally think not too good. ~ If you use MySQL, you design the database vertically,
Structure: ID GameID count Createtime
1 1001 3000 2013-03-23 12:22:21
2 1002 2222 2013-03-23 12:22:21
..................
If this is the case, then it will not change the data table because of the new GameID, so what is the disadvantage? That is, every time the data is inserted very much, if 30 seconds to insert, one time to insert 30 game statistics, then the day increment 2*30*60*24 = 86,400 data, this is obviously unreasonable.
So horizontal design, insert one piece of data at a time.
ID gameid_1001 gameid_1002 gameid_1003 ... createtime
1 3000 2222 40000 2013-03-23 09:08:56
2 4000 1800 4000 2013-03-23 09:09:20
......
The disadvantage is that every time you add a game ID, you have to change the data table structure, plus the field, of course, you can do a good job of the utility program to achieve, but I think it is not well.
This embedded document is very cool in MONGO. Recommended use of Hadoop
Storage structures such as the following
+{
"_id": 3E3ESS3SAZXCDSDSFDF,
"Createtime": "2013-03-23 09:13:02",
"Data": {
"gameid_1001": 2000,
"gameid_1002": 3000,
......
}
}
Insert only one piece of data at a time, new game types do not need to do whatever changes, perfect~
4. Why use rsync? It is easier to synchronize logs from multiple servers to a single folder for processing.
5, need to use several functions, Glob, Fopen,fget,isset,explode
The program is best not to write very rigid,
Bulk read-in log files
$sLogfileName = '/path/. /*_date.log ';
$aLogfileName = Glop ($sLogfileName); Match the log file to be processed and read into the array.
......
fopen ();
while ()//with a while loop, processing a row of data in the file and then going to the file, assuming that the array is read in a single foreach, the Memory overflows.
{
......
}
......
$aCountResult = Array ();
$iNum = 100;
if (Isset ($aCountResult [$iGameId]))
$aCountResult [$iGameId] = (int) $aCountResult [$iGameId] + $iNum;
Else
$aCountResult [$iGameId] = $iNum;
......
End of statistics Insert:
Then add the scheduler, OK.
The main or different business is solved in different ways.
@update 2013-3-25 21:31:45
In log analysis \ n is a very important delimiter, avoid memory overflow, do not
EOF as a separator, colleagues to strictly in accordance with the standard format of the log written, so that in the resolution is better than parsing. It is obtained in fgets mode and cannot be read into memory at one time.