NET Distributed System Seven: Log Capture System (1)

Logs are especially important for large application systems or platforms, and system log collection and analysis is the basis of system operation, maintenance and user analysis.

First, the System log classification

The general system log can be divided into three main categories:

1, user behavior log: Through the collection system users use the system process, a series of operational logs.

2, Business change log: A specific business scenario needs to collect a user at a certain time using a function, a business (object, data) to do something, from A to B.

3, the system operation log: The system runs the server resources, the network and the basic middleware situation carries on the timing collection log analysis.

Second, the common log analysis application scenario

Log acquisition analysis is driven by demand, according to a certain scenario needs log collection, collected logs for targeted analysis. Common log analysis results scenarios include:

1, analysis system or platform which features are most popular: when to use the most users, a region, a category of users to use the most. facilitates the promotion of functions and improves the user experience by boosting server resources.

2, Content recommendation: According to the user's usual reading content, collect related logs, and through analysis, the user is interested in the content system automatically recommended to users, thereby enhancing the user station stickiness.

3, System Audit: For the application system, the acquisition of operational logs, business change log, for reference and provide relevant security audit functions.

4, automated operation: the system or platform of the scene micro-service architecture, the requirements of operation and maintenance investment, automatic deployment and operation, can reduce the workload and pressure of operation and maintenance. The system Operation Environment Log collects, analyzes, can realize the early warning, the server resources dynamic deployment, facilitates the quick localization troubleshooting.

The requirements for operating environments, functional scenarios, and the need to collect and analyze logs for different systems vary. The log content, the collection method exists the multiplicity, the log data quantity is large, therefore needs to design a set of log collection system, satisfies the log collection request, supports the convenient analysis result to catch in the application function.

Third, the overall design of the log collection system

Based on the preliminary analysis of the above log collection, especially in the micro-service architecture, the micro-service governance capability and the service load need to rely on the collection and analysis of service logs, so the log collection subsystem is the basic support function of the microservices architecture. The overall design of the log collection system is as follows:

Figure 1-The overall framework of the log capture system

As shown, according to the entire process of the log, the system consists of three parts: Log capture, log storage & Analysis and Results application. Log collection is mainly responsible for providing a variety of ways to collect logs; log storage & Analysis is the main implementation of the analysis of unified storage and custom scene analysis log; the results of application implementation of the log analysis results provided by the service interface or the default management functions, the provision of functional use.

The whole process of log acquisition: After the log acquisition, through the online or offline transmission to the message queue, by the log consumption application pull after storage. According to the analysis needs to customize the automatic job task to implement log data extraction, conversion, statistics, the results of data written to the relational database. Provides a service interface or log query default management function for use.

1. Log Collection

Log capture achieves three ways of collecting logs:

(1) Webapi method: Realize the restful way of collecting log data based on HTTP protocol, and send to message queue. Mainly used to provide mobile, public and small amount of log capture use, in the net distributed system can be combined with "API Gateway".

(2) Service proxy mode: Based on log4.net excellent log components and Message Queuing client driver, encapsulation as a logging service agent, providing convenient, unified interface supply for use. Support for logging to the application local and online real-time send to the message queue, which is recorded to the application local, can be combined with the third way to complete the function of log collection.

(3) Lcclient method: Realize the client batch crawl log data, send to Lcserver. The Lcclient client communicates with the Lcserver server based on the TCP protocol and is built on the NIO framework to support high concurrency processing capabilities. The Lcserver then writes the log data to the message queue.

Log capture uses capture logs by providing three ways to meet different business scenarios.

2. Log Storage & Analysis

Log storage and analysis consists of five parts: Message Queuing (MQ), log receive server (Lcserver), log Store (Elasticsearch), Task Job (quartz.net), and relational database (MySQL).

(1) Message Queuing (MQ): The log through the message queue to achieve unified message reception, to do a buffer to meet the log data concurrent reception capacity.

(2) Log receive server (Lcserver): Receive Lcclient client sends log data in bulk and sends it to message queue.

(3) Log storage (Elasticsearch): Choose ES to do log storage, mainly consider ES is a document distributed search service, support petabytes of data storage, support hundreds of nodes of distributed cluster capabilities, and provide a wealth of API use, data query has superior performance.

(4) Task Assignment (quartz.net): After extracting, transforming and analyzing ES log data through custom development job task, the result is stored in relational database. Quartz.net Middleware is an industry-recognized job task component.

(5) relational database (MySQL): the implementation of log results data storage for log application query use. MySQL has a superior master-slave replication mechanism to solve single-node query performance bottlenecks, while providing traditional SQL script operational data capabilities, to provide development efficiency.

After the log data is stored and analyzed, the ultimate goal is to provide and use the application functionality.

3. Application of Analysis results

The analysis result application mainly provides the service interface and the management function two kinds of ways.

(1) Service interface mode: Provide the interface in restful way based on HTTP protocol, and support the application function to make the call easily.

(2) Management module mode: The default provides a set of log query management functions, easy to use directly integrated application.

Iv. Summary

Log capture itself is also a distributed system, each node of the server supports distributed cluster deployment, which can achieve high concurrency and high availability. This article is just a kind of introduction, each component of the system needs to be deepened in detail, also includes implementation deployment.

Author: Liu Zaitao
Source: Http://www.cnblogs.com/Andon_liu
About the focus on Microsoft Platform Project architecture, management. Familiar with design patterns, domain drivers, architecture design, agile development and project management. is mainly engaged in ASP, Wcf/web API, SOA, MSSQL, Redis aspects of project development, architecture, management work. If you have questions or suggestions, please learn to discuss together!
This article is copyrighted by the author and the blog Park, Welcome to reprint, but without the consent of the author must retain this paragraph, and in the article page obvious location to the original link.
If you have questions, you can email: [Email protected] contact me, thank you.

No.

NET Distributed System Seven: Log Capture System (1)