Scribe core analysis

The Facebook scribe log aggregation system has three core contents: configuration, running scribe server, and logging messages)

Logging messages (logging)

Scribe executes the following thrift interface:

enum ResultCode
{
  OK,
  TRY_LATER
}

struct LogEntry
{
1:  string category,
2:  string message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list messages);
}

To send a message to the scribe server with the specified host and port, we need to create a scribe client and call the log () method. The following is an example written in Python:

from scribe import scribe
from thrift.transport import TTransport, TSocket
from thrift.protocol import TBinaryProtocol

category='test'
message='hello world'
log_entry = scribe.LogEntry(category, message)
# depending on thrift version
#   log_entry = scribe.LogEntry(dict(category=category, message=message))

socket = TSocket.TSocket(host='localhost', port=1463)
transport = TTransport.TFramedTransport(socket)
protocol = TBinaryProtocol.TBinaryProtocol(trans=transport, strictRead=False, strictWrite=False)
client = scribe.Client(iprot=protocol, oprot=protocol)

transport.open()
result = client.Log(messages=[log_entry])
transport.close()

Running scribe server (running server)

$ Scribed [-p] [-C] # Start the scribe Server
$ Scribe_ctrl stop [] # Stop the scribe Server
$ Scribe_ctrl {command} [] # monitor and manage the scribe Server

Subcommand of scribe_ctrl:

Status-if the server runs normally, 'alive' is returned'

Version-returns the version of the current scribe server.

Alive-return the server running time

Stop-stop the scribe Server

Reload-Reload the scribe configuration file

Counters-returns the following statistics (if not zero ):

• Received good: returns the number of messages received after the scribe server is started.
• Received ed bad: number of illegal messages received
• Sent: number of messages sent to another scribe Server
• Denied for queue size: Number of forbidden requests due to full information queue
• Denied for rate: Number of requests prohibited due to speed restrictions
• Retries
• Requeue: the number of times a message is sent to a store by scribe (if must_succeed is enabled ).
• Lost: Number of unrecorded messages. (Recommended Configuration: Use buffer stores to avoid information loss)
• Received blank category: the number of messages that have not been received.

 

Configurarion (configuration)

The scribe server is started through the configuration file.

• Null: discards all messages.
• Multi: send messages to all multi-type stores.
• File: write messages to files.
• Thriftfile: similar to file, but the written file type is thrift tfiletransport.
• Network: forward messages to another scribe server.
• Buffer: contains two sub-stores (primary and secondary). When the primary store is working, it directly uses the primary store to record logs. The secondary store record is used only when the primary store is unavailable. In addition, once the primary store recovers, the logs recorded by the secondary stroe will be transferred to the primary store.
• Bucket: contains a series of other types of stroe, which is determined by the defined Hash Function

For more detailed configuration information, see: https://github.com/facebook/scribe/wiki/Scribe-Configuration

3. server startup process flowchart:

Log process flowchart:

The client calls the log (messages) function to send messages. The remote method scribehandler: log (const vector & messages) is called through the implementation of the thrift framework. The process is as follows:

 

 

Reference: http://blog.csdn.net/kohaku/article/details/6049183

Https://github.com/facebook/scribe/wiki/