Play to MongoDB (ix): Unified log Management for distributed systems via Log4jmongo

Background

In distributed systems, we have multiple web apps that may be deployed on separate physical servers and have their own log output. When production problems come, it is often necessary to go to each log file to look for possible exceptions, which is very labor-intensive. Log storage in the form of text files, when there is a need for log analysis and mining, this processing is also a lot of inconvenience, and inefficient.

In order to facilitate the unified management and analysis of these logs, we can output the log uniformly to the specified database system, and then by the log analysis system to manage. Because this is the chapter of MongoDB, so it is subjective to do log data storage with MongoDB, objectively, because it is light and simple, easy to integrate with log4j, and low in intrusion of system . Second, because it has many advantages compared with the large-scale relational database, such as fast query ,Bson storage structure is advantageous to extend , free and so on.

Solution Solutions

Integrate MongoDB and log4j

1, install the MongoDB database, and start locally, the default port is 27017, for details please refer to: play to MongoDB (a): The first knowledge of MongoDB

2, a new Maven (Maven version requires more than 3.0) project, select Maven-archetype-quickstart, project name: Log4j2mongo

3, in the Pom.xml file, add log4j, Log4mongo-java, mongo-java-driver three dependencies. The specific code is as follows:

1 <Projectxmlns= "http://maven.apache.org/POM/4.0.0"Xmlns:xsi= "Http://www.w3.org/2001/XMLSchema-instance"2 xsi:schemalocation= "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">3   <modelversion>4.0.0</modelversion>4 5   <groupId>Com.manyjar</groupId>6   <Artifactid>Log4j2mongo</Artifactid>7   <version>0.0.1-snapshot</version>8   <Packaging>Jar</Packaging>9 Ten   <name>Log4j2mongo</name> One   <URL>http://maven.apache.org</URL> A    -   <Properties>   -     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>   the     <junit.version>3.8.1</junit.version>   -     <log4j.version>1.2.17</log4j.version>   -     <log4mongo.version>0.7.4</log4mongo.version>   -     <mongo-java-driver.version>2.8.0</mongo-java-driver.version>   +   </Properties>  -  +   <Dependencies> A     <Dependency>   at       <groupId>Junit</groupId>   -       <Artifactid>Junit</Artifactid>   -       <version>${junit.version}</version>   -       <Scope>Test</Scope>   -     </Dependency>   -    in       <Dependency>   -           <groupId>Log4j</groupId>   to           <Artifactid>Log4j</Artifactid>   +           <version>${log4j.version}</version>   -       </Dependency>   the    *       <Dependency>   $           <groupId>Org.log4mongo</groupId>  Panax Notoginseng           <Artifactid>Log4mongo-java</Artifactid>   -           <version>${log4mongo.version}</version>   the       </Dependency>   +    A       <Dependency>   the           <groupId>Org.mongodb</groupId>   +           <Artifactid>Mongo-java-driver</Artifactid>   -           <version>${mongo-java-driver.version}</version>   $       </Dependency> $   </Dependencies> - </Project>

Pom.xml

4. In the Resources folder, add the Log4j.properties file. The file primarily adds log4j to the MongoDB adapter Org.log4mongo.MongoDbAppender. The adapter here is Log4mongo-java this jar package is provided. ip:127.0.0.1,port:27017 of the MongoDB database, library name: logs, set name: Log. The specific configuration is as follows:

1Log4j.rootlogger=DEBUG,MongoDB2Log4j.appender.mongodb=org.log4mongo.Mongodbappender3Log4j.appender.mongodb.databasename=logs4Log4j.appender.mongodb.collectionname=Log5log4j.appender.mongodb.hostname=127.0.0.16log4j.appender.mongodb.port=270177    8Log4j.appender.stdout=org.apache.log4j.Consoleappender9Log4j.appender.stdout.layout=org.apache.log4j.PatternlayoutTenLOG4J.APPENDER.STDOUT.LAYOUT.CONVERSIONPATTERN=%D{YYYY-MM-DD HH: Mm:ss, SSS}%5P%c{1}:%l-%m%n

log4j.properties

5. In the Java folder, Com.manyjar.log4j2mongo This package, add Main.java file, observe the code in the file can be found, and we usually use log4j to write the log exactly the same, the log stream to MongoDB this thing, the business development programmer completely transparent. The specific code is as follows:

1  PackageCom.manyjar.log4j2mongo;2 3 ImportOrg.apache.log4j.Logger;4 5 ImportCom.mongodb.BasicDBObject;6 ImportCom.mongodb.DBObject;7 8  Public classMain {9     Ten      Public Static voidMain (string[] args) { OneLogger Logger = Logger.getlogger (Main.class); A              -              for(inti = 0; i < 10000000; i++) { -DBObject Bson =NewBasicdbobject (); theBson.put ("name", "Ryan" +i); - Logger.debug (Bson); -             } -     } +}

Main.java

6, in step 5, we performed 10 million log insertions, the data structure is as follows:

The default data volume size is 10G:

Here, we can see that the amount of data stored in the log is relatively small. If you need to modify the storage structure of the log data, you can use the source code of Log4mongo to develop two times.

If the amount of data is too large, we can use the TTL index (auto-deletion expired) or fixed collection size two ways to resolve:

TTL索引:db.log_events.createIndex({"timestamp": 1},{expireAfterSeconds: 60*60*24*30}) #1个月后过期后删除

Modify the log collection to a fixed size collection:DB. runCommand({"converttocapped":"Log" , size:10000}).

Play to MongoDB (ix): Unified log Management for distributed systems via Log4jmongo