Log Collection System Overview

A log analysis system is built based on hadoop, Zookeeper, hbase, and flume frameworks.
Use HDFS in hadoop as the storage of raw data, and flume as the real-time data collection and conversion functions.
Hbase is a data storage database. The data on the query page is obtained from a table in the distributed database.
Query different hbase tables based on different applications. The following functions are implemented:
1. Data collection, conversion, and storage. As described above, flume, HDFS, and hbase are used.
Build and implement this function.
2. query the log information of different systems. Provides a Web interface to query the corresponding log information.
3. Periodically collect error log information and send it by email.

Log Collection System Overview