. The Operation log Analysis system of NET micro-service architecture

First, Introduction

. NET technology stack does not yet have a complete micro-service architecture stack like Spring Cloud, which is built on its own as the evolution of the business development system architecture. NET technology system of micro-service architecture, supporting the relevant core components. Because the platform is based on the micro-service architecture approach, each domain service adheres to the unified Platform standard, the respective development, the independent deployment operation, the service running logs are recorded by recording the file. Program log can not be consulted in time, need to log on to the server to view, at the same time, not conducive to unified log management, due to the development and operation of log analysis system, log unified analysis management, easy to quickly locate program operation problems timely processing, to ensure the stability of the platform operation. Although there are also some log architectures in the industry, such as the more famous Lek (Logstash, Elasticsearch, Kibana), due to some of the following personalized requirements, or self-developed running log analysis system.

Ii. Principles of Design

System in the overall micro-service architecture design, adhere to the platform standards, and the actual problem-solving needs, design and research and development, the system in strict accordance with the following design principles:

1, Modular: According to the responsibility and attribution of clear, Split module, module function of high cohesion.

2, Service: Each module communication, through the service mode of invocation, communication between services, compliance with platform standards.

3, Loose coupling: Communication between the system modules, based on the industry common standards, according to the "agreement better than the configuration" thought, fully reflects the system module loosely coupled relationship.

4, High performance: The asynchronous way to log and analysis, does not affect the overall performance of the platform.

5, easy to expand: Log using a unified access mode, support flexible expansion.

6, cross-platform: support different technical language, platform for collecting logs.

Iii. Overall architecture

Figure-Run log analysis system overall architecture

, the system based on the above design principles, mainly consists of five parts: logging, log capture, log access, log analysis and storage, log management services. Each part of the responsibility of the clear, mutual support, mutual cooperation, together constitute a running log analysis system. Logging to a local file, through the log collector to extract the local log files, sent to Kafka, while the log Analysis service for consumption, analysis and storage in Elasticsearch, log management services provided by the user access & analysis log information. Under the Platform Unified Communication protocol, the contents of the running log are extended, and the contents of the data include: IP, port, service ID, recording time and log content.

1. Log Messages

2. Log records

Day record, support a variety of logging components: Log4.net (this system selection),. NET core comes with Nlog, Microsoft Enterprise Library, etc., for the Java Technology language development services, the use of log4j components, as long as the contract specification to print the log. Log information is logged to a local file.

3. Log Collection

Log capture, select the Filebeat component, Filebeat is a lightweight log transport component that provides a lightweight way to forward and centralize logs and files to help you simplify the simple collection and delivery of things. Essentially filebeat is a log information porter that does not perform log filtering and analysis work. Support cross-platform at the same time.

4, log access

Log access, considering the amount of log in a fault condition, there may be large concurrent access, so the use of Kafka components as a unified portal for log information access analysis, the Kafka component is a high-performance MQ component, with high concurrency, high availability features. Unified Access Scheme provides a more flexible collection log for the system, not limited by a platform, technology or log collection components, can be run log access analysis and storage.

5. Log analysis and Storage

Log analysis, the use of. Net core development, in the form of back-office services to run, subscribe to Kafka log messages for bulk consumption, the analysis program supports scale-out deployment in multiple programs, improve the consumption capacity of the log, to ensure the ability to receive the log and analysis capacity is comparable. For log analysis, stored in Elasticsearch (hereinafter referred to as: es), ES is a highly extensible, open-source full-text search and analysis engine that allows the storage, search, and analysis of large amounts of data. Ideal for storing semi-structured log data to facilitate full-text search of log information.

6. Log Management Service

Log Management Service, implement query es in the running log data, through the WEBAPI way to provide interface function calls. This system uses "before and after separation" application development technology, the front-end interface function uses H5+JS, the backend service provides the API interface in the restful way.

Iv. Summary

According to different application scenarios and requirements, the use of different log analysis architecture or self-development, the ultimate goal is to quickly and easily capture the system log, help to analyze the system operation situation. Especially for microservices architectures with large platforms, running logs is more important. This article mainly is the system design principle, the thought carries on the record, at the same time provides the reference for the schoolmate who needs, may again practical application, still has many places to be perfect or the optimization.

Public Number: