Facebook scribe Introduction

 

Facebook scribe Introduction

-------------------

1. Introduction

Scribe is an open-source Real-Time Distributed log collection system on Facebook. It improves the reliability and scalability of large-scale log collection. You can install the scribe service on different nodes, and then these services will publish the collected information to the central service cluster. When the central service is unavailable, the local scribe service temporarily stores the collected information locally, and then uploads the information after the Central Service recovers. The central service cluster can write the collected information to a local disk or distributed file system, such as HDFS, or distribute it to another layer of the scribe service cluster.

 

Scribe provides a non-blocking thrift service, which relies on the libevent library and thrift Library at the underlying layer.

 

2. Installation

The installation method is. You can find it in the contrib of thrfit source code. The installation method is the same as thrift installation. The second is that if you have installed scribe, the python module cannot be found when running the example. Be sure to configure the environment variable pythonpath.

3. Example

There is the examples directory in the scribe source code package. There are three examples.

The first is how to start the scribe service and send messages to the scribe service.

# Create a directory to log messages: <br/> mkdir/tmp/scribetest <br/> # Start scribe using the configuration in example1.conf: <br/> src/scribed examples/example1.conf <br/> # from another terminal, use scribe_cat to send a message to Scribe: <br/> echo "Hello World" |. /scribe_cat test <br/> # If the previous command failed, make sure you did a 'Make install' from the <br/> # Root scribe directory and that $ pythonpath is set correctly (see readme. build) <br/> # verify that the message got logged: <br/> CAT/tmp/scribetest/test/test_current <br/> # check the status of scribe (requires root): <br/>. /scribe_ctrl status <br/> # Check scribe's counters (you shocould see 1 message 'ordered ed good'): <br/>. /scribe_ctrl counters <br/> # shutdown Scribe: <br/>. /scribe_ctrl stop 

The second example is how to transmit messages on multiple scribe instances.

# This example shows you how to log messages between multiple scribe instances. <br/> # in this example, we will run each scribe server on a different port to simulate <br/> # Running scribe on multiple machines. <br/> 'client' 'center' <br/> ------------------------------------------ <br/> | Port 1464 | Port 1463 | <br/> | ---------------- | <br /> |-> | scribe server | -- | ---> | scribe server | <br/> | ---------------- | <br/> | <br/> | Temp File | <br/> | ----------------------------- | ------------------- <br/> | <br/> ------------------- <br/> |/tmp/scribetest /| <br/> ----------------- </P> <p> # create a directory for the second scribe instance: <br/> mkdir/tmp/scribetest2 <br/> # Start up the 'Central' instance of scribe on port 1463 to write messages to disk <br/> # (See example2central. conf): <br/> src/scribed examples/example2central. conf <br/> # Start up the 'client' instance of scribe on port 1464 to forward messages to <br/> # the 'Central' scribe server (see example2client. conf): <br/> src/scribed examples/example2client. conf <br/> # Use scribe_cat to send some messages to the 'client' scribe instance: <br/> echo "Test message" |. /scribe_cat-H localhost: 1464 Test2 <br/> echo "this message will be ignored" |. /scribe_cat-H localhost: 1464 ignore_me <br/> echo "123: This message will be bucketed" |. /scribe_cat-H localhost: 1464 bucket_me <br/> # The first message will be logged similar to Example 1. <br/> # the second message will not get logged. <br/> # The third message will be bucketized into 1 of 5 buckets <br/> # (See example2central. conf) <br/> # verify that the first message got logged: <br/> CAT/tmp/scribetest/Test2/test2_current <br/> # verify that the third message got logged into a subdirectory: <br/> CAT/tmp/scribetest/bucket */bucket_me_current <br/> # Check the status and counters of both instances: <br/>. /scribe_ctrl status 1463 <br/>. /scribe_ctrl status 1464 <br/>. /scribe_ctrl counters 1463 <br/>. /scribe_ctrl counters 1464 <br/> # Shutdown both servers: <br/>. /scribe_ctrl stop 1463 <br/>. /scribe_ctrl stop 1464 <br/>. /scribe_ctrl stop <br/> 

The third example is to check a Buffer Response of the local scribe when the central service is closed.

# Test scribe buffering <br/> # startup the two scribe instances used in example 2. <br/> # Start the 'Central' server first: <br/> src/scribed examples/example2central. conf <br/> # then start the 'client': <br/> src/scribed examples/example2client. conf <br/> # log a message to the 'client' scribe instance: <br/> echo "Test message 1" |. /scribe_cat-H localhost: 1464 test3 <br/> # verify that the message got logged: <br/> CAT/tmp/scribetest/test3/test3_current <br/> # Stop the 'Central' scribe instance: <br/>. /scribe_ctrl stop 1463 <br/> # attempting to check the status of this server will return failure since it not running: <br/>. /scribe_ctrl status 1463 <br/> # Try to log another message: <br/> echo "Test message 2" |. /scribe_cat-H localhost: 1464 test3 <br/> # This message will be buffered by the 'client' since it cannot be forwarded to <br/> # the 'Central' server. scribe will keep retrying until it is able to send. <br/> # after a couple seconds, the status of the 'client' will be set to a warning message: <br/>. /scribe_ctrl status 1464 <br/> # Try to log yet another message (which will also get buffered): <br/> echo "Test message 3" |. /scribe_cat-H localhost: 1464 test3 <br/> # restart the 'Central' instance: <br/> src/scribed examples/example2central. conf <br/> # Wait for both scribe instance's statuses to change to alive: <br/>. /scribe_ctrl status 1463 <br/>. /scribe_ctrl status 1464 <br/> # verify that all 3 messages have now been already ed by the 'Central' Server: <br/> CAT/tmp/scribetest/test3/test3_current <br/> # shutdown: <br/>. /scribe_ctrl stop 1463 <br/>. /scribe_ctrl stop 1464 

 

Since the C ++ example is not found, I wrote a simple example here for your reference only.

# Include <iostream> <br/> // scribe headers <br/> # include "scribe. H "// This header is in the Gen-CPP directory <br/> // thrift headers <br/> # include <protocol/tbinaryprotocol. h> <br/> # include <transport/tsocket. h> <br/> # include <transport/ttransportutils. h> <br/> using namespace STD; <br/> using namespace boost; <br/> using namespace Apache: thrift:: Protocol; <br/> using namespace Apache: thrift: Transport; <br/> using namespace Scribe: thrift; <br/> int main (INT argc, char ** argv) <br/>{< br/> STD: String category; // The log Category <br/> STD: String host; // The IP host <br/> int port; // The port </P> <p> If (argc = 2) <br/> {<br/> category. assign (argv [1]); <br/> host = "localhost"; <br/> Port = 1463; <br/>} else if (argc = 4 & strcmp (argv [1], "-h") = 0) <br/> {<br/> category. assign (argv [3]); <br/> char * Sep = strstr (argv [2], ":"); <br/> If (SEP = NULL) <br/> {<br/> host. assign (argv [2]); <br/> Port = 1463; <br/>}< br/> else <br/> {<br/> char TMP [20]; <br/> strncpy (TMP, argv [2], sep-argv [2]); <br/> Port = atoi (SEP + 1); <br/> host. assign (TMP); <br/>}< br/>}else <br/>{< br/> printf ("usage (message is stdin ): scribe_cat [-H host [: Port] category "); <br/> exit (1 ); <br/>}</P> <p> shared_ptr <tTransport> socket (New tsocket (host, Port )); <br/> shared_ptr <tTransport> transport (New tframedtransport (socket); <br/> shared_ptr <tprotocol> protocol (New tbinaryprotocol (Transport )); <br/> // use boost shared pointer <br/> shared_ptr <scribeclient> sbclient (New scribeclient (Protocol )); <br/> try <br/> {<br/> transport-> open (); <br/> // create a log entry and push it to vector <br/> logentry le; <br/> le. message = "This Is A Test message"; <br/> le. category = category; </P> <p> STD: vector <logentry> messages; <br/> messages. push_back (LE); <br/> // write to scribe server <br/> sbclient-> log (messages ); <br/> // close the transport <br/> transport-> close (); <br/>} catch (texception & Tx) <br/>{< br/> printf ("error: % s/n", TX. what (); <br/>}< br/>} 

 

Related makefile content:

 Cc = g ++ <br/> libs =-lthrift-lfb303 <br/> include =-I/usr/local/include/thrift/fb303-I/usr/local/ include/thrift <br/> source = scribe_cat.cpp scribe. CPP scribe_types.cpp <br/> scribe_cat: $ (source) <br/> $ (CC) $ (source)-o $ @ $ (include) $ (libs) <br/> clean: <br/> RM scribe_cat 

 

4. Reference

 

Https://github.com/facebook/scribe

Http://blog.nosqlfan.com/html/1287.html

Http://incubator.apache.org/thrift/