Exploring new ideas for IT system maintenance-centralized processing of IT system logs

With the increasingly fierce competition in the communication market, operators constantly strengthen their IT support system construction to support the needs of rapid business development. The number of IT support systems that operators rely on to carry out their businesses is increasing and the scale is growing. It system users have higher requirements for the background IT system maintenance department: on the one hand, the background maintenance department must ensure the uninterrupted operation of various IT systems and support the development requirements of business departments. Early detection and early handling of system problems must be implemented, minimize the impact of system faults on business development. On the other hand, with the increasingly fierce competition of telecom operators, the human and financial resources are constantly tilted towards the market, and the number of staff in the background maintenance department is decreasing, the number of IT systems to be maintained increases, and the workload increases. The maintenance department urgently needs to use technical innovation and maintenance mode innovation to reduce manual operation workload and improve maintenance efficiency.

In the daily work of the maintenance department, periodic viewing of server operating system logs, application systems and database operation logs of each IT system is the key to ensuring early detection of problems and timely exception handling. In the past, IT support systems were often built independently, and the networks between systems were different. Even if the same IT system was mostly used on different hosts and servers, to regularly check the operating status of the IT system, maintenance personnel need to log on to each important host of each system, analyze the main log information, and identify the hidden fault. This can be done manually when the number of IT systems is small and the scale is small. However, the number of IT systems is constantly expanding, such as the carrier's signaling monitoring system and large-scale Centralized Billing System, A system requires dozens or even hundreds of host servers. The original log processing method cannot meet the new needs of business development, so it is imperative to improve maintenance efficiency and log processing.

In the daily check of the above three categories of logs, the main factors restricting the efficiency improvement include the following:

1. The log information of each IT system is not collected and displayed in a centralized manner. You need to manually log on to the key hosts of each system. After system authentication, view the application system logs, host logs, and database operation logs one by one. A large amount of time is consumed during Remote logon;

2. There are a large number of General prompts or general alarms that can be ignored in the log information regularly checked. Many of these alarms are not filtered, occupying a large amount of space in the alarm log file, A large amount of repetitive work is added to routine checks;

3. Keyword segments of log messages of various IT systems are not displayed in segments. For example, a log message generally contains time, log source, and log description. This information is mixed in a message, and the fields to be followed cannot be found at a glance, causing unnecessary trouble for routine checks.

To address these problems, combined with daily IT system maintenance experience, we consider using a centralized log Message Processing Method to Improve the disadvantages of the original work. The specific ideas are as follows:

1. In consideration of system security, the isolation status between the original IT systems is broken to achieve centralized collection of log messages.

2. Based on daily maintenance experience, simple processing and filtering of unnecessary parts of log messages can reduce unnecessary workload.

3. Based on maintenance experience, different fields of log messages are segmented and saved to the database for analysis and processing.

4. The log information is displayed on the web page to facilitate subsequent troubleshooting.

Based on this idea, we implement centralized log processing step by step:

Step 1. Collect logs from the main IT systems through the network:

A pc server is installed separately for centralized collection of log messages. Because the main IT systems are connected to the DCN network, the log server uses an intranet IP address to access the DCN network, which is physically isolated from the interconnected network to ensure its security. Through MPLS/VPN technology, the log server is connected with the main application servers and database servers of various IT systems, providing a channel for log message transmission.

Step 2: implement centralized collection of log messages

Considering that all existing IT system servers use Unix platforms, log messages must be centrally collected based on UNIX platforms, you must transmit all logs of the UNIX operating system, application system running logs, and database logs to the log server. The specific method is as follows:

The operating system logs are saved locally by modifying the syslog service configuration file of the UNIX operating system, and a copy is also sent to the log server.

For application system and database logs, these log files are mostly text files, write scripts, regularly collect incremental parts of these text files, and store them into temporary files. Call the logger process of the UNIX system to automatically read the log messages in the temporary files. After reading the logs, delete the original temporary files and send the log messages to the log server.

Step 3. Implement multipart storage of log messages to facilitate subsequent analysis and processing

The log server uses the Linux operating system, installs free rsyslog tool software, and stores operating system log messages into the database based on the message time, origin, message importance, and other fields.

For the segmentation of application system logs and database log messages, you need to write a script program to extract log time, log content, and other fields from the log messages, and import them to the database for storage using the rsyslog tool.

Step 4. Create a page to extract log messages from the database for unified log display

Use tools such as PHP and Apache to extract collected log information from the database. Based on the importance of log messages, you can make a special display effect or issue a sound alarm to display the log messages that need to be focused, reminding maintenance personnel to process them in a timely manner.

After log messages are collected and processed in a centralized manner, maintenance personnel do not need to log on to each IT system to view log messages separately. You can use the unified log display portal to regularly view log alarm information on web pages to handle exceptions in a timely manner, greatly improving maintenance efficiency.