Logminner: A log analysis tool based on EMF and STAF

Introduction: Large software in the running process will produce a large number of various log information, testers often have to analyze a large number of lengthy logs, and these logs are often distributed in different locations of different platforms in different directories, so that the test log analysis is cumbersome and tedious, in order to facilitate the tester to quickly analyze the log, The author of this paper has developed a log analysis tool ――logminner based on EMF and STAF technology.

1. Overview

Distributed heterogeneous software system testers often have to analyze a large number of logs, such as installation logs, database logs and so on, these logs are distributed in different locations of different operating system hosts, testers in order to analyze these logs need to travel between the various machines in the laboratory, even if there is remote operation, Testers also have to master a variety of operating systems, and log on to multiple machines for analysis, and many log files, lengthy content, making the test work is cumbersome and tedious. For testers, viewing these log files and analyzing them will be a huge challenge.

In order to realize the fast and efficient and simple log analysis of the testers, we have developed a small and easy-to-use log analysis tool--logminner.

With Logminner, you simply edit the XML-formatted configuration file, specifying in the configuration file the machine name, path, file name pattern, and keyword pattern and attributes of the log that you want to analyze, Logminner to the specified machine to search the specified directory and its subdirectories. Files that specify the file name pattern, perform a log scan, search for keywords that match the keyword pattern, and export the profiling results information to the specified or default HTML file.

The Logminner tool includes an EMF-based Profile editor (Config file Editor), a java-based and Staf Logminner Console (Logminner console), and STAF based Logminner Service three components.

This paper introduces the architecture, design implementation and configuration use of Logminner, and puts forward further development prospects.

2. Logminner

The architecture of 2.1LogMinner

Logminner is a distributed Peer-to-peer system based on the STAF (Software testing Automation Framework). All hosts in the system are STAF Client. Where Logminner Servvice is a custom Java STAF Service. While the Logminner console is a Java application running in a STAF environment, communication between the Logminner console and the Logminner Servvice is a STAF-mode synchronous communication implemented by Request/response. The Logminner console presents a log analysis request to a specified set of hosts that deploy the Logminner service based on the XML format configuration file, Logminner the service response request and returns the analysis results to Logminner Console. The deployment of the system is shown in the following illustration.

Figure 2.1 Logminner Deployment diagram

Staf is an open source, multi-platform, multi-language software testing framework designed for reusable components based on service. Based on the STAF implementation of the Logminner system dependencies and the edge of the active collaboration of equipment, rather than the central server, direct communication between members, breaking the traditional C/s mode, members have the dual characteristics of client and server. This decentralized architecture realizes the distributed computation of log analysis, and effectively utilizes the computational resources in the network environment. At the same time, no central server, not only save the hardware resources, and no need for transit service time resource consumption makes the system has good communication timeliness, high data transmission rate advantages. Logminner based on Staf, in addition to End-to-end communication mode, the reusable components developed for service concept make this software have good reusability and scalability.

2.2LogMinner function Description of each component

1 as the server role of the Logminner service to achieve the log analysis work. The Logminner service receives services requests from the Logminner console, searches the specified directory and its subdirectories for legitimate requests, and matches the specified regular expression pattern for the file name. A search that matches a keyword that conforms to a specified regular expression. Generates a structured result message collection that returns the Logminner Console.

2 The Logminner console, which serves as the client role, simply parses the XML-formatted configuration file, submits the request to each server according to the profile generation standard Logminner service request, and reads the response data resolution to write to the specified or default result file.

3 the config file Editor as a Visual Profile Editor leverages the EMF (Eclipse Model Framework) technology to generate XML that conforms to the machineset.xsd defined profile through visualization XML document for the schema. is an Eclipse plug-in tool that generates Logminner configuration files, which can also be generated and modified by any other XML document editor, and Config file editor is not a logminner system.