C + + Thrift Client and Flume Thrift Source docking

The project requires C + + code to interface with the Flume, which in turn writes the log to HDFs.
Flume native to Java code, the original solution was to invoke the Flume Java method via JNI.
But because of the concern about the efficiency of JNI calls, and the fact that the C + + call JNI needs to take care of the local reference and GC issues, the headache has been caused.
Rage, rewrite the code, use C + + with the remote Java flume docking.

In the choice of protocol, AVRO C + +, although there are Apache open source projects, but currently only support read and write files, but not the use of RPC.
The thrift is used to communicate with the remote flume thrift source.
Here are some implementation-specific methods:

Flume Thrift Protocol Preparation:

1. Install the thrift and install the flume.
2. Download the flume source package and unzip it.
3. Under Extract directory Flume-ng-sdk/src/main/thrift, there is a file flume.thrift.
This is the Thrift Protocol rule file for Flume, which executes the command: Thrift-r--gen cpp./flume.thrift
In the./gen-cpp/directory, some. h. cpp files are generated, after which the methods in the C + + code need to be called for RPC communication.

Note : If you are developing under Windows/visual Studio, you need to modify the Flume.thrift because of the error in conflict with the keyword.

Enum Status {
Ok
FAILED,
Ferror, //original error, modified to ferror.
UNKNOWN
}

If you're working under Linux, you don't need to change any code.

C + + code implementation:

1. Include some header files, and namespaces using thrift

1#include <thrift/protocol/TBinaryProtocol.h>2#include <thrift/protocol/TCompactProtocol.h>3#include <thrift/transport/TSocket.h>4#include <thrift/transport/TTransportUtils.h>5 6 using namespacestd;7 using namespaceApache::thrift;8 using namespaceapache::thrift::p rotocol;9 using namespaceApache::thrift::transport;

2. Create thrift socket, transport, protocol, client.

 1  class   thriftclient{ private  :  3  /*   Thrift protocol needings ...  */ 4  boost::shared_ptr<ttransport> socket;  5  boost::shared_ptr<ttransport> Transport;  6  boost::shared_ptr<tprotocol> protocol ; 7  thriftsourceprotocolclient* pclient;  8 } 

1Thriftclient::thriftclient (std::stringInipaddress, std::stringinPort):2SocketNewTsocket (Inipaddress.c_str (), Atoi (Inport.c_str () )),3TransportNewTframedtransport (socket)),4ProtocolNewTcompactprotocol (transport))5     {6Pclient =Newthriftsourceprotocolclient (protocol);7}

3. Communicate with the remote flume thrift Source:

1       BOOLThriftclient::sendevent (Constevent*Event)2       {3           //Build the head4STD::MAP&LT;STD::string, std::string>headers;5 Std::ostringstream Timebuffer;6Timebuffer <<Event->timestamp <<" the";7Headers.insert (Std::make_pair ("timestamp", Timebuffer.str ()));8Headers.insert (Std::make_pair ("appId", AppId));9           //Build the BodyTen Std::ostringstream osbody; OneOsbody << *Event; ASTD::stringSbody =osbody.str (); -   - thriftflumeevent tfevent; the tfevent.__set_body (sbody); - tfevent.__set_headers (headers); -           if(!transport->IsOpen ()) -           { +Transport->open (); -           } +Status::type res=pclient->append (tfevent); A           if(res = =Status::ok) at           { -               return true; -           } -           Else -           { -printf"Warning:send Event via thrift failed, return code:%d\n", res); in               return false; -           } to}

Other points of note:

1. The Append method is used to send an event:

Status::type thriftsourceprotocolclient::append (constevent)

There is also a method appendbatch used to send more than one event at a time:

Status::type Thriftsourceprotocolclient::appendbatch (const std::vector<thriftflumeevent> & Events

2. Both the Append and the Appendbatch methods are blocking methods.

3. You may be able to send an event without confirmation of success (not tested) via Send_append,send_appendbatch.

void Thriftsourceprotocolclient::send_append (constevent)

void Thriftsourceprotocolclient::send_appendbatch (const std::vector<thriftflumeevent> & Events)

4. Thrift only provides protocol RPC functionality and does not provide flume channel functionality, as well as multiple source load balance functions. These all need to be achieved by themselves.

About Performance:

Test environment: VMWARE+UBUNTU,I3-4150 CPU, configure 1G memory, dual core CPU.

In the native configuration two flume thrift source (load balance), network loopback.

In the case of approximately 50 characters per event, 16,000 throughput per second can be reached, at which time the CPU is exhausted (two flume thrift source consumes approximately 30%cpu).

C + + Thrift Client and Flume Thrift Source docking