Distributed Log Collection system: Facebook scribe Log collection program

My independent blog URL is: http://wuyouqiang.sinaapp.com/.

My Sina microblog: Http://weibo.com/freshairbrucewoo.

Welcome everyone to communicate with each other and improve technology together.

Write log to scribe solution

1. Overview

The Scribe Log collection server is only responsible for collecting the log that actively writes to it, it will not actively crawl a log, so in order to write the log to the scribe server, we must actively send the log information to the scribe server. Because the scribe server is implemented based on the thrift Framework, and Thrift supports communication in a variety of programming languages, the client implementation of the write scribe server can also use multiple languages, which provides good support for integrating log-writing clients into a variety of application systems. Integrating the functionality of writing logs into the scribe server into the application system is a viable solution, but not the only solution, and we can also realistically have a single client dedicated to capturing the application-generated log files and then writing to the scribe server. The following is a specific description of these two scenarios.

2.scribe log write and application system integration

(1) Integration with Java application system

There are two solutions to integration with the Java system, one through the development API provided by Scribe, and the other through integration with log4j (open source implementation on the Web).

Impact on Application Systems: Because this approach is a functional module of the application system, it requires additional jars and additional usage overhead, in addition to taking into account exception handling when the scribe server is not properly linked.

(2) integration with C # application system

Integration with C # Systems is done by encapsulating the development API provided by scribe into a DLL file, and then the C # application system imports DLL files and uses the provided APIs to develop functional modules written to the scribe log.

Impact on Application Systems: You need to import DLL files, add additional overhead to application systems, and have exception handling for linked scribe servers.

(3) Integration with other application systems

Because the thrift framework supports more languages and scribe is based on thrift implementations, as long as the thrift supported development languages can be integrated with the corresponding application systems.

3. The client of a separate crawl log file

Writing a single client is a solution for any application system, provided that the application system needs to generate a corresponding log file. This individual client can be implemented in any language supported by thrift, but is typically implemented in Python to facilitate modification, scaling, and deployment.

This solution can be implemented in two ways: one is to detect log files or folders in a circular way, and if there is a new log generation, read the log files and upload them to the scribe server; the second is to monitor files or folders through the mechanism of event response.

4. Comparison of two kinds of solutions

The first thing to be sure is that both solutions can achieve the goal of writing logs to scribe.

The difference between the two is:

(1) It is more efficient to write to a program that integrates with an application system because it does not need to be written directly through the application system through a forwarding link, and this scheme reduces the likelihood of error and does not require a separate program to be maintained. However, this solution will have a certain impact on the application system, the previous description of the specific impact, there is not universal, each application system needs to develop a separate functional modules (the development of ideas and methods are basically the same).

(2) Separate capture log file client: With good versatility, do not need each application system to develop a separate log write module, only the application system to generate log files. And this scheme has no effect on the application system. Of course, this solution is less efficient because it requires a file to relay log information, and also because a separate program may be hung up.

5. Issues to be addressed in all two programmes

(1) Single point of failure

(2) Log loss problem