How to solve log management?

A slightly larger system usually needs to be embedded with a log system. no matter how simple the system is, at least the problem can be located for developers. The common method is not to use the console's printf and file record fwrite functions. In a large system, how to manage logs...

A slightly larger system usually needs to be embedded with a log system. no matter how simple the system is, at least the problem can be located for developers. The common method is not to use the console's printf and file record fwrite functions.
 
 
In a large system, how should I manage logs? when the system runs for n days and finds a BUG, do I have to intercept printf printing on the console? Generate a huge log file? Is that true? Is it feasible? The answer to this question is very obvious. we cannot use printf for log management of large systems. it can only be used for simple problem locating in the development stage.
 
Log management needs to be addressed from the system architecture, and this architecture ultimately depends on the purpose and purpose of the log. Let's take a look at what log management actually does?
 
Log Management mainly aims to provide a method to manage a large amount of log information generated by computers. The main process involves information collection, storage and analysis. The main purpose is security check, audit check, and problem locating.
 
 
 
Developers usually build their own log systems and analysis tools to support module development. However, there are often no fixed formats and specifications to follow. Therefore, system-level log management is a complex process. if proper control is not performed during development, obfuscation is often triggered, reducing the efficiency of system-level development and positioning. Therefore, effective system-level log analysis involves many challenges: a large number of logs, log types, and log formats.
 
To solve system-level log analysis [1], there are many solutions:
 
 
 
Pattern recognition)
Normalization)
Classification and tagging)
Correlation analysis)
Artificial Ignorance)
 
 
However, let's look back at the purpose and process of log management: logs from applications, operating systems, and hardware devices are organized into logs by means of message streams, the log management system collects and stores the logs to enter the log analysis stage. It can be said that the main purpose of log collection and storage is to analyze, and analysis is to solve the purpose of log management: security check, audit check, and problem locating.
 
 
 
Log Management is a system architecture problem in large systems. In this case, log management should be based on the system architecture. The following describes the main steps and steps involved in log management architecture development.
 
Main steps and steps for log management architecture development:
 
 
 
Define requirements and objectives: such as security log analysis, application problem analysis, and control management reports;
Define the log framework, type, format, and specifications;
Are you sure you want to perform log management: collect, analyze, report, and remote monitoring? How long does it take to archive logs? Need compression?
What information does the log contain: Module type? Log level? Error message?
Evaluate the current technology and select or build the most appropriate Log Management Solution based on open-source projects.
 
 
The log management is standardized from the source, and the log management architecture is well designed. the system development and positioning of subsequent modules will have a profound impact.
 
 
 
If you have any questions, you can discuss a system-level Log Management Solution.
 
 
 


Reference:

[1] Log Analysis

From lida2003's column