Custom Flume Sink:elasticsearch Sink

The purpose of the Flume sink is to obtain data from the Flume channel and then output it to storage or other Flume source. When the Flume agent starts, it launches a Sinkrunner object for each sink, and the Sinkrunner.start () method initiates a new thread to manage each sink life cycle. Each sink needs to implement the start (), Stop (), and process () methods. You can initialize the parameters and states of the sink in the start method and clean the sink resources in the Stop method. The most critical is the process approach, which will handle the data taken out of the channel. Also, if Sink has some configuration, it needs to implement the configurable interface.

Since Flume's official sink often fail to meet the requirements, we customize the sink to achieve the needs of customization, Elasticsearch for example. The simple Insert function of the document is implemented in sink. The example uses Flume 1.7.

1. Writing code

The new class Elasticsearchsink class inherits the Abstractsink class, and the configurable interface is implemented because you also want the configuration of the custom sink .

 Public class extends Implements Configurable

elasticsearch IP and the name of the index can be configured in the configuration file, the configuration file is to use the Flume conf file. You can rewrite the configurable configure method to get the configuration, the code is as follows:

@Override      Public void Configure (context context)    {        = context.getstring ("Es_host");         = Context.getstring ("Es_index");    }

Note The syntax inside the configuration entry "Es_host" and "Es_index" in the Conf configuration file:

Agent.sinks = Sink1agent.sinks.sink1.type = Nick.test.flume.ElasticSearchSinkagent.sinks.sink1.es_host = 192.168.50.213agent.sinks.sink1.es_index = Vehicle_event_test

The next step is to implement the Process method, in which the channel is required because the data is obtained from the channel. Before you get a message, you need to get a channel that is a transaction, and then commit and close the transaction after processing is complete. This allows the channel to know that the message has been consumed and that it can delete the message from its internal queue. If consumption fails and needs to be re-consumed, the transaction can be rollback. The introduction of transactions is the key to the reliability assurance of Flume.

The process method needs to return an enumeration of status types, ready and backoff. If you get to a message and handle it properly, you need to use ready. If the received message is NULL, you can return Backoff. The so-called Backoff (failure compensation) is that when sink gets no message, Sink's Pollingrunner thread waits for a backoff time, and the data in the channel is compensated for pollling operation.

The complete code is as follows:

 Public classElasticsearchsinkextendsAbstractsinkImplementsconfigurable{PrivateString Eshost; PrivateString Esindex; Privatetransportclient Client; @Override PublicStatus process ()throwseventdeliveryexception {Status Status=NULL; //Start TransactionChannel ch =Getchannel (); Transaction Txn=ch.gettransaction ();        Txn.begin (); Try{Event Event=Ch.take (); if(Event! =NULL) {String body=NewString (Event.getbody (), "UTF-8"); Bulkrequestbuilder bulkrequest=Client.preparebulk (); List<JSONObject> jsons =NewArraylist<jsonobject>(); Jsonobject obj=Jsonobject.parseobject (body); String Vehicleid= Obj.getstring ("vehicle_id"); String Eventbegincode= Obj.getstring ("Event_begin_code"); String Eventbegintime= Obj.getstring ("Event_begin_time"); //Doc ID in indexString id = (Vehicleid + "_" + Eventbegintime + "_" +eventbegincode). Trim (); Jsonobject JSON=NewJsonobject (); Json.put ("VEHICLE_ID", Vehicleid);                Bulkrequest.add (Client.prepareindex (Esindex, Esindex). SetSource (JSON)); Bulkresponse Bulkresponse=Bulkrequest.get (); Status=Status.ready; }            Else{Status=Status.backoff;        } txn.commit (); }        Catch(Throwable t) {txn.rollback ();            T.getcause (). Printstacktrace (); Status=Status.backoff; }        finally{txn.close (); }        returnstatus; } @Override Public voidConfigure (Context context) {Eshost= Context.getstring ("Es_host"); Esindex= Context.getstring ("Es_index"); } @Override Public synchronized voidStop () {Super. Stop (); } @Override Public synchronized voidstart () {Try{Settings Settings= Settings.builder (). Put ("Cluster.name", "Elasticsearch"). build (); Client=NewPrebuilttransportclient (Settings). addtransportaddress (NewInetsockettransportaddress (Inetaddress.getbyname (eshost), 9300)); Super. Start (); System.out.println ("Finish Start"); }        Catch(Exception ex) {ex.printstacktrace (); }    }}

2. Package, configure and run

Because it is a custom sink, you need to make a jar package and copy it to the Flume Lib folder. Then configure the agent configuration file, and finally start the flume. In this example, I used the Kafkasource, Memorychannel, and custom sink, and the complete configuration file is as follows:

agent.sources = Source1agent.channels = Channel1agent.sinks = Sink1agent.sources.source1.type = Org.apache.flume.source.kafka.KafkaSourceagent.sources.source1.channels = Channel1agent.sources.source1.batchSize = 1agent.sources.source1.batchdurationmillis = 2000agent.sources.source1.kafka.bootstrap.servers =  192.168.50.116:9092,192.168.50.117:9092,192.168.50.118:9092,192.168.50.226:9092agent.sources.source1.kafka.topics = Iov-vehicle-eventagent.sources.source1.kafka.consumer.group.id = Flume-vehicle-event-nickagent.sinks.sink1.type = Nick.test.flume.ElasticSearchSinkagent.sinks.sink1.es_host = 192.168.50.213agent.sinks.sink1.es_index = Vehicle_ Event_testagent.sinks.sink1.channel = Channel1agent.channels.channel1.type = memoryagent.channels.channel1.capacity = 1000

Custom Flume Sink:elasticsearch Sink