Log analysis (php+nosql+rsync+crontable)

Source: Internet
Author: User

Does it often have to analyze user behavior? Do you often encounter multiple server uploads with the logs analyzed together? Is the time interval for data statistics very short? There is also wood because the log file is too large, and need to block processing?

1, note that in the log writing must be in accordance with a strict format, so that in the time of parsing, only good segmentation. For example gameid:123 gameid:2333. Split the unified standard can be.

2, in the generation of the file name of the log should also be in accordance with certain rules, in the analysis of the time, is a good match, such as Serverhostname_date.log in the match when the only need to Glob (*-date.log); Glob See the PHP function manual to find the file path that matches the pattern.

3. Why use NoSQL? In fact, the project teacher is not only limited to know how to achieve, but to think more about what kind of business to use what kind of tools to solve. Non-relational data is very suitable for this, the log often add new behavior, you use Key-value way, do not need the log new to analyze the behavior, you have to manually change the configuration of your program, so I personally think not too good. ~ If you use MySQL, you design the database vertically,

Structure: ID GameID count Createtime

1 1001 3000 2013-03-23 12:22:21

2 1002 2222 2013-03-23 12:22:21

..................

If this is the case, then it will not change the data table because of the new GameID, so what is the disadvantage? That is, every time the data is inserted very much, if 30 seconds to insert, one time to insert 30 game statistics, then the day increment 2*30*60*24 = 86,400 data, this is obviously unreasonable.

So horizontal design, insert one piece of data at a time.

ID gameid_1001 gameid_1002 gameid_1003 ... createtime

1 3000 2222 40000 2013-03-23 09:08:56

2 4000 1800 4000 2013-03-23 09:09:20

......

The disadvantage is that every time you add a game ID, you have to change the data table structure, plus the field, of course, you can do a good job of the utility program to achieve, but I think it is not well.

This embedded document is very cool in MONGO. Recommended use of Hadoop

Storage structures such as the following

+{

"_id": 3E3ESS3SAZXCDSDSFDF,

"Createtime": "2013-03-23 09:13:02",

"Data": {

"gameid_1001": 2000,

"gameid_1002": 3000,

......

}


}

Insert only one piece of data at a time, new game types do not need to do whatever changes, perfect~

4. Why use rsync? It is easier to synchronize logs from multiple servers to a single folder for processing.

5, need to use several functions, Glob, Fopen,fget,isset,explode

The program is best not to write very rigid,

Bulk read-in log files

$sLogfileName = '/path/. /*_date.log ';

$aLogfileName = Glop ($sLogfileName); Match the log file to be processed and read into the array.

......

fopen ();

while ()//with a while loop, processing a row of data in the file and then going to the file, assuming that the array is read in a single foreach, the Memory overflows.

{

......

}

......

$aCountResult = Array ();

$iNum = 100;

if (Isset ($aCountResult [$iGameId]))

$aCountResult [$iGameId] = (int) $aCountResult [$iGameId] + $iNum;

Else

$aCountResult [$iGameId] = $iNum;

......

End of statistics Insert:

Then add the scheduler, OK.

The main or different business is solved in different ways.

@update 2013-3-25 21:31:45

In log analysis \ n is a very important delimiter, avoid memory overflow, do not

EOF as a separator, colleagues to strictly in accordance with the standard format of the log written, so that in the resolution is better than parsing. It is obtained in fgets mode and cannot be read into memory at one time.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.