MongoDB Log Server Scenario

Describe

Currently to do is a number of servers on the program log (such as the order journal, the transaction log, the interface is successful, etc.) summarized to 1 MongoDB server, daily about 100 million of the volume, and then a real-time chart display, and report display log information

Attention:

Did not put all the logs into 1 tables, because log insertion is often caused by the lock table, and the second is a large amount of impact query speed. No consideration is given to sharding, because the Shard does not have a high efficiency for our business queries. The use of MongoDB replica set to ensure that the server stable operation and master-slave library read and write separation to improve efficiency.

Scheme

1: Simplify warehousing, use log4j configuration to save the log4j log in a unified JSON format directly to Mogodb

2: Build 4 tables

A: Total table (all logs) "To be saved by a separate library to prevent write global lock impact query"

B: Hour temp table:

Format: 2013122106 Note: December 21, 2013 6 o'clock log summary written by log4j

Hour table: (crontab or quartz timing statistics of the first hour before the temporary table business data required, such as the total number of hours of trading, success rate, various aggregation values, etc.)

C: Set up a fixed set to hold the data for the last 2 hours. Fixed size, high efficiency, automatically replace the old data "ready to insert data, so will affect the query, it is recommended to put into the MongoDB memory database and fixed set, improve efficiency"

D: Day table (Summary of the hour table)

Business Processing

Real-time display of icons within 1 hours (fixed set to meet)

Real-time Chart display (hour table + fixed set to meet)

Report Statistics (hour table or day table, need + month table or chronology)

Encounter problems

MongoDB Imports data using import speed is not stable, so if the import data have time requirements, it is recommended to use the program batch import,

MongoDB to insert will appear lock table, locked collection, affect the collection query, do not put log4j log directly to the total table, in the statistics hours business aggregation information, because the total number of tables, efficiency is slow. It is recommended that you log4j a temporary collection that is written by the hour, and the hourly business data to the hour table in the staging collection

MongoDB Log Server Scenario