Facebook scribe log Aggregation

With the arrival of the cloud computing era, our internet system servers are getting bigger and bigger. A large system is usually composed of thousands of machine clusters, and our system will be deployed on these thousands of machines, in this case, you need to monitor the running data of each system at all times. We may need to know how many transactions are made by each system every day, and make summary, analysis, statistics, and reports. A typical application is to collect logs generated by thousands of systems, analyze the data in statistics, and mine and forecast data.

A simple method is to synchronize logs on each machine to an offline server, periodically analyze offline logs, store offline analysis results reports, and back up logs. In this way, real-time analysis and monitoring cannot be achieved. In addition, we need to focus on all systems.

However, there is also a simple way: Report the logs of each system to the central server in real time, and use large disk storage or distributed storage on the central server to manage, analyze, mine, and report, you can do whatever you want. All these requirements have been discovered by Facebook (Facebook logs tens of billions of pieces of information every day, millions of logs per second), and the implementation result is the scribe system.

In a distributed computing environment, each node must be monitored at the system level and service level,System-level monitoringEnsures system server stability, such as CPU, memory, network, and database;Service-level monitoringIs a relatively fine-grained monitoring, such as how many service errors, warnings, alarms, and so on within a period of time. Service-level monitoring of the system in the cluster is mainly intended forLog AggregationTo monitor logs.

 

Below are some introductions of Facebook Scribe:

Scribe is the server used to collect logs. It has strong scalability, and network faults and server node faults do not affect log collection. A scribe server is running on each node in a large-scale cluster system. This scribe server can collect information and send the information to a central scribe server (or multiple central scribe servers ), if a fault occurs on the central scribe server (or central server group) and becomes unavailable, the SCIBE server of each node will write the log information to the local disk and send it again when the central scribe server returns to normal. The central scribe server stores the information written to the final disk address, which is generally an NFS file system or a distributed file system, these log files are sometimes transmitted to the scribe server group on other layers. the unique feature of scribe is that the client log instance contains two strings: A category and a message ). Category is a high-level description of the expected target information. You can configure it on the scribe server. This allows you to transfer data without changing the code by changing the configuration file. The scribe server can also be configured based on the category prefix. By default, the category name can be inserted in the file path. Flexibility and scalability can be abstracted through storage. Stores can be statically configured through a configuration file, or you do not need to stop the server during running. Scribe is the implementation of a thrift service that uses a non-blocking C ++ server. Facebook runs the scribe service on thousands of servers, collecting and transmitting billions of information every day. Scribe itself is implemented by C ++, And the RPC communication between the client and the server is implemented by python by default. To allow scribe to be used in multiple languages, it uses the thrift framework for communication. The server has already implemented it, and the client only implements the Python version, we can easily export the Java implementation version based on the thrift framework and apply it to systems in various languages.

 

Next, let's take a look at scribe. There are too few materials in China. Let's find a way to go over the wall!