In the most recent project, the log of the project was saved in JSON format for easy analysis. Before the log directly exists in the file, and MongoDB timely broke into my sight, so the log into the MongoDB. Log only save is meaningless, the most important thing is to find out from the log business trends, system performance vulnerabilities. There is a Java-written analysis module that runs under Tomcat. Achieving considerable weight, the process of adding a new metric is cumbersome, and causes analysis to fail due to NFS. Always wanted to rewrite, initially wanted to use Ruby on Rails, but there was no time to learn and develop (excuse me!). ）。 In Hangzhou Qcon 2011 and met Node.js, although also heard before, but did not in-depth research, after listening to Taobao Suchi speech, then have to use Node.js to achieve this log analysis system ideas. The front-end with JS, the server with JS, even the database shell is JS, think is cool--of course, the key is the small amount of code.
First, implement server-side code with Node.js
In order to have a good style and fast code writing, it is inevitable that a simple framework should be adopted. Express has achieved most of the functionality, but it takes a while to be familiar with it and seems to be a bit heavyweight for the project. There is a chat demo on Node.js's official website, which simply moves to encapsulate the processing of URLs and return JSON. So I used the fu.js directly, rewriting the Server.js:
var fu = require ("./fu"),
SYS = require ("Util"),
url = require ("url"),
MONGO = require ("./request_handler");
Fu.listen (Process.env.PORT | | PORT), HOST);
Fu.get ("/", Fu.statichandler ("index.html"));
Is it too easy?! But it is true that a server has been set up.
The Request_handler.js code that handles the request is shown below:
Top User Action
Fu.get ("/useractiontop10", Function (req, res) {
Mongodb.connect (' Mongodb://localhost:27017/log ', function (err, conn) {
Conn.collection (' Action_count ', function (err, coll) {
Coll.find ({"value.action": {$in: User_action}}). Sort ({"Value.count": -1}). Limit. ToArray (function (err, docs) {
if (!err) {
var action = [];
var count = [];
for (var i = 0; i < docs.length i + +) {
Console.log (Docs[i]);
Action.push (docs[i].value.action);
Count.push (Docs[i].value.count);
}
Res.simplejson ({action:action, count:count});
Be sure to close the database connection
Conn.close ();
}
});
});
});
});
Second, the client
The most important of the journaling system is the visualization display, which uses a jquery plugin Jqplot Chart. Start with a static HTML page that is used as a graphic display container:
It's almost a full copy of the Jqplot example, OK, I admit I'm too lazy.
Here is a look at the chart.js used to display the generated graphics:
/****************************** top User Action Start *********************************/
document.write (' <div id= ' Useractiontop10chart "></div>");
var drawuseractiontop10chart = function () {
if (!$ ("#userActionTop10Chart"). attr (' class ') {
$ ("#userActionTop10Chart"). attr (' class ', ' Small_chart ');
}
$.ajax ({
Async:false,
URL: '/useractiontop10 ',
DataType: ' JSON ',
Cache:false,
Success:function (data) {
try{
$ (' #userActionTop10Chart '). html (');
$.jqplot (' Useractiontop10chart ', [Data.count], {
Title: "Top User Action",
seriesdefaults:{
Renderer:$.jqplot. Barrenderer,
Rendereroptions: {filltozero:true},
Pointlabels: {
Show:true,
Ypadding:1
}
},
axesdefaults:{
Tickrenderer:$.jqplot. Canvasaxistickrenderer,
Tickoptions: {
Angle:-30,
FontSize: ' 12px '
}
},
Axes: {
Xaxis: {
Renderer: $.jqplot. Categoryaxisrenderer,
Ticks:data.action
},
YAxis: {
pad:1.05
}
}
});
}catch (e) {
alert (e.message);
}
}
});
}
Draws.push (' Drawuseractiontop10chart ');
/******************************* top User Action End ************************************/
/*********** Chart Start *****************/
Put your chart drawing function
1. Insert a div for the chart
2. Implement the function drawing chart
3. Push the function name into the array draws
/*********** Chart End *******************/
Draw All charts
var drawallcharts = function () {
for (var i = 0; i < draws.length i + +) {
Eval (Draws[i] + "()");
}
Recall itself in 5 minute.
Window.settimeout (Drawallcharts, 5 * 60 * 1000);
}
//
$ (function () {
Drawallcharts ();
});
Server-side and client code are available, then run to see the effect of it:
Seems to have forgotten something? Analysis code for the log.
Third, the use of MongoDB incremental MapReduce implementation of log analysis
An introduction to incremental MapReduce is available in the MongoDB documentation. Just beginning always thought MongoDB implement streaming processing, can automatically perform incremental mapreduce. Finally, I found out that I was wrong, the document does not write this, but it shows how to set up to incrementally execute mapreduce.
For convenience, I wrote the MapReduce using MongoDB JavaScript in a separate JS file, and then executed it by crontab timing. Code for Stats.js:
var action_count_reduce = function (key, values) {
var count = 0;
Values.foreach (function (value) {
Count + = Value.count;
});
return {action:key, count:count};
}
Db.log.mapReduce (Action_count_map, action_count_reduce, {query: {' Action_count ': {$ne: 1}},out: {reduce: ' Action_ Count '}});
Db.log.update ({' Action_count ': {$ne: 1}}, {$set: {' Action_count ': 1}}, False, True);
The idea is simple:
1. Set each action access number to 1 in the map
2. Reduce, count the number of visits to the same action
3. Implement MapReduce. Specifies that the query is ' action_count ' not equal to 1, that is, the statistic is not executed, the result is stored in the ' Action_count ' collection, and the reduce option is used to represent the result set as input for the next reduce.
4. The value of ' action_count ' for all current logging settings is 1, indicating that the statistic has been executed. Do not know if this will cause no statistics have been recorded also updated?? Hope to have experience of the warrior enlighten!
Timed execution of Stats.js shell:
Well, that's the whole code, there's nothing particularly mysterious about it, but Node.js really is a good thing.
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
