Distributed log Collection System: log collection solution for Facebook scribe

My independent blog website is:Http://wuyouqiang.sinaapp.com/.

My Sina Weibo:Http://weibo.com/freshairbrucewoo.

You are welcome to exchange ideas and improve your technology together.

Solution for writing logs to scribe

1. Overview

The scribe log collection server is only responsible for collecting the logs that are actively written to it. It does not actively capture a log, so to write the log to the scribe server, we must actively send log information to the scribe server. Because the scribe server is implemented based on the thrift framework, and thrift supports multipleProgramming LanguageTherefore, you can use multiple languages for the client implementation that writes logs to the scribe server, which provides good support for integrating the client that writes logs into various application systems. Integrating the functions of writing logs to the scribe server into the application system is a feasible solution, but it is not the only solution. We can also implement a separate client, it is used to capture the log files generated by the application system and then write them to the scribe server. The following describes the two solutions.

 

2. Scribe log writing and Application System Integration

(1) integration with Java application systems

There are two solutions for integration with the java system: one is self-developed through the Development API provided by scribe, and the other is integrated with log4j (open-source implementation is available on the Internet ).

Impact on the application system: Because this method is added as a function module of the application system, additional jar files need to be added and overhead of the application system is required, in addition, you also need to consider the exception handling when the scribe server cannot connect properly.

 

(2) integration with C # Application System

Integration with C # system encapsulates the development API provided by scribe into a DLL file, and then C # Application System imports the DLL file, use the provided API to develop the function module for writing scribe logs.

Impact on the application system: DLL files need to be imported, additional execution overhead of the application system is increased, and Exception Handling linked to the scribe server exists.

 

(3) integration with other application systems

Because the thrift framework supports multiple languages and the scribe is implemented based on thrift, the development languages supported by Thrift can be integrated with the corresponding application system for development.

 

3. separate client for log file capturing

Writing a separate client is a solution suitable for any application system, provided that the application system needs to generate corresponding log files. This separate client can be implemented in any language supported by Thrift, but it is usually implemented in Python to facilitate modification, expansion, and deployment.

There are two ways to implement this solution: one is to cyclically detect log files or folders. If a new log is generated, the log file is read and uploaded to the scribe server; second, the event response mechanism is used to monitor files or folders.

 

4. Comparison of the two solutions

First, we can be certain that both solutions can write logs to the scribe.

The difference between the two is:

(1) it is more efficient to write logs in a solution integrated with the application system, because you do not need to write logs directly through the application system through a forwarding link, and this solution reduces the possibility of errors, you do not need to maintain oneProgram. However, this solution will have a certain impact on the application system. The specific impact described earlier is that it is not universal, each application system needs to develop such a function module separately (the development idea and method are basically the same ).

(2) Independent Log File capturing client: it has good versatility. It does not require every application system to develop a log Writing module separately, but only requires the application system to generate a log file. This solution has no impact on the application system. Of course, this solution is less efficient, because you need to use a file to transfer the log information, and because a separate program may also be suspended.

 

5. Problems to be Solved in both solutions

(1) single point of failure

(2) log loss