Flume Transaction Implementation

Flume as the log collection end, which is essentially a producer consumer structure, source as the message of the Producer,sink as the message of the consumer, the middle channel for the storage of messages
In order to ensure the correctness of the message consumption, flume used the mechanism of the transaction, mainly related to the class:
1) Org.apache.flume.Transaction interface class that provides transactional functionality for accessing the channel (either put or take)
First, an enum class transactionstate is defined, defining several states of time {Started, Committed, Rolledback, Closed}
The abstract method defined:

Begin//Start transaction commit//Transaction complete rollback//rollback close//close transaction

Methods of use, such as:

Channel ch = ...//construct a channel object transaction tx = Ch.gettransaction ();  Gets the transaction object of this channel object try {tx.begin ();//Transaction start ... ch.put (event) or ch.take ()//Data Manipulation ... tx.commit ();//COMMIT TRANSACTION} catch (Channelexception ex) {tx.rollback ();//exception ROLLBACK TRANSACTION ...} finally {tx.close ();}

2) Org.apache.flume.channel.BasicTransactionSemantics implements the abstract class of the transaction interface (and Basicchannelsemantics common use)
Defines the class state of an enum, identifies the status of transaction new (new), OPEN (can perform commit or rollback), completed (the state after commit or rollback), CLOSED
Several abstract method dobegin/doput/dotake/docommit/dorollback/doclose are defined, and the Put/take/begin/commit/rollback/close method is defined.

Each method uses Preconditions.checkstate to detect if it is the same thread operation and whether the state is a required state, such as

  protected basictransactionsemantics ()  {     state =  state.new; //first set State to New    initialthreadid = thread.currentthread () in the constructor method . GetId ();   }   public void begin ()  { //begin methods      preconditions.checkstate (Thread.CurrentThread (). GetId ()  == initialThreadId,          "Begin ()  called from different thread than  Gettransaction ()! ");  //detects whether the same thread is operating     preconditions.checkstate (State.equals (state.new),          "Begin ()  called when transaction is "  + state  +  "!");  //detects if the current state is New    try {      dobegin ();  // Call Dobegin method     } catch  (interruptedexception e)  {       thread.currentthread (). interrupt ();       throw  new channelexception (E.tostring (),  e);     }    state  = state.open; //set the status to OPEN  }

Order of transactions:
Begin-->put/take-->commit/rollback-->close

3) The subclass of Basictransactionsemantics includes the following, which correspond to specific channel

Filechannel-->filebackedtranscationmemorychannel-->memorytranscationspillablememorychannel--> Spillablememorytransaction

Transaction class:
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/5A/17/wKiom1T1wiqTAyovAAEruW-ZXYA796.jpg "title=" 1.png " alt= "Wkiom1t1wiqtayovaaeruw-zxya796.jpg"/>

Channel class:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/5A/17/wKiom1T1wkey5obqAAErrxff4IM198.jpg "title=" 2.png " alt= "Wkiom1t1wkey5obqaaerrxff4im198.jpg"/>

Each specific basictransactionsemantics subclass implements the specific Doput/dotake/docommit/dorollback method
to see how it is called:
Invoke Channel instance put/ Take/commit/rollback method, use put as an example
1) in Org.apache.flume.channel.ChannelProcessor call Processevent (Operation an event) or Processeventbatch (insert a batch of event) method
such as processevent:

    // Process required channels    List<Channel>  requiredchannels = selector.getrequiredchannels (event);    for  ( Channel reqchannel : requiredchannels)  {      transaction  tx = reqchannel.gettransaction ();       Preconditions.checknotnull (tx,  "Transaction object must not be null");  // The transaction object cannot be null      try {         tx.begin ();  //transaction begins         reqchannel.put (event);  // Call the channel's Put method, here is org.apache.flume.channel.basicchannelsemantics         tx.commit ();  //transaction commit, put has commit operation, take also has commit operation       } catch   (throwable t)  {        tx.rollback ();        if  (t  Instanceof error)  {          log.error ("Error  while writing to required channel:  " +       &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;REQCHANNEL,&NBSP;T);           throw  (Error)  t;        } else {           throw new channelexception ("Unable  to put event on required  " +                "channel: "  + reqchannel, t);         }      } finally {         if (tx != null )  {          tx.close ();         }      }     }

2) Call the parent class Org.apache.flume.channel.BasicChannelSemantics's Put method:

public void put (event event) throws Channelexception {basictransactionsemantics transaction = Currenttransaction.get ()    ;    Preconditions.checkstate (transaction! = NULL, "No transaction exists for this thread"); Transaction.put (event); Call the Put method of the Basictransactionsemantics implementation class}

3) Org.apache.flume.channel.BasicTransactionSemantics put method (subclass does not have a specific put implementation)

  protected void put (event event)  {     Preconditions.checkstate (Thread.CurrentThread (). GetId ()  == initialThreadId,          put ()  called from different thread than gettransaction ( )!"  )     preconditions.checkstate (State .equals (State.open),          "put ()  called when transaction is %s!"  , state );     preconditions.checkargument (event != null,          "put ()  called with null event!"  )  //will begin to make some state judgments, such as whether the state of transcation is correct, etc.     try {       doput (event);  //call DoPut method     } catch  (interruptedexception  e)  {      thread.currentthread (). inteRrupt ();       throw new channelexception (E.toString (),  e);     }  }

4) Then call the Doput method of the Memorytransaction specific implementation class in the channel
For example, Org.apache.flume.channel.MemoryChannel's inner class memorytransaction Doput:

    protected void doput (event event)  throws interruptedexception  {      channelcounter.incrementeventputattemptcount ();       int eventByteSize =  (int) Math.ceil (estimateeventsize (event)/  bytecapacityslotsize);      if  (! putlist.offer (event))  {         throw new channelexception (            "put queue for memorytransaction of capacity "  +            putlist.size ()  +  "  full, consider committing more frequently,  " +              "Increasing capacity or increasing thread  count " );      }      putByteCounter += eventByteSize;     }

This article is from the "Food and Light Blog" blog, please make sure to keep this source http://caiguangguang.blog.51cto.com/1652935/1617017

Flume Transaction Implementation