Flume-ng source reading: Sinkgroups and Sinkrunner

In the Abstractconfigurationprovider class, the Loadsinks method invokes the Loadsinkgroups method to put all sink and sinkgroup into map<string, Sinkrunner > Sinkrunnermap.

Sinkrunner can correspond to one sink can also correspond to one sinkgroup. Because if there is a sinkgroup in the configuration file, the sink of this sinkgroup will form a group and then be encapsulated as a sinkrunner, then sinkgroup in Sink will become a sinkrunner themselves. The argument for each Sinkrunner's construction method is a sinkprocessor that is used to handle multiple sink.

One, if a sinkrunner corresponds to a sink. Sinkprocessor PR = new Defaultsinkprocessor () is the default sinkprocessor. The relevant code in the Loadsinkgroups method is as follows:

1 sinkprocessor PR = new defaultsinkprocessor ();
2 list<sink> sinkmap = new arraylist<sink> ();
3 Sinkmap.add (Entry.getvalue ());
4 Pr.setsinks (SINKMAP);
5 configurables.configure (PR, new context ());
6 Sinkrunnermap.put (Entry.getkey (), New Sinkrunner (PR));

Defaultsinkprocessor.configure (context context) is an empty method, and all this configurables.configure (PR, new context ()) has no effect, The important thing is the defaultsinkprocessor.process () method, which means that the return sink.process () directly invokes the sink process. The Setsinks method will only set a sink. The Defaultsinkprocessor Start () method also invokes Sink.start () directly to start the sink.

Second, if a sinkrunner corresponds to multiple sink. A sinkgroup group = new Sinkgroup (groupsinks) is constructed and then SinkProcessor:group.getProcessor () is acquired. The relevant code in the Loadsinkgroups method is as follows:

1 Sinkgroup group = new Sinkgroup (groupsinks);
2 Configurables.configure (group, groupconf);
3 Sinkrunnermap.put (Comp.getcomponentname (), New Sinkrunner (Group.getprocessor ()));

Configurables.configure (Group, groupconf) invokes Sinkgroupconfiguration.configure, which gets the configuration file about " Processor. " The properties of the Sinkprocessortype (either Failoversinkprocessor or loadbalancingsinkprocessor) are obtained by the Getknownsinkprocessor method. and executes the sinkprocessor.configure (Processorcontext) for instantiation and configuration.

1, if Sinkprocessor is Loadbalancingsinkprocessor, this is a load-balanced processor that will send channel to all specified sink. By configuring selector selector to choose which way to load balance, 1.3 has two kinds: round_robin, polling, is to send the channel data in turn, RANDOM, randomly select channel send data. Only this sinkprocessor has a selector.

(1) Configure (context-context) method. First get the type of selector selector, the default is Round_robin, poll, get Backoff (whether you use a deferred algorithm, or a Boolean value that sets the penalty time for the sink after the problem is no longer considered to be active) ( The default false is not enabled), the corresponding Selector object roundrobinsinkselector based on the type (actually constructs a roundrobinorderselector) or Randomordersinkselector ( Actually constructs a randomorderselector), then instantiates and sets the sinks, and finally initializes the selector execution Configure (context) method.

A, the actual operator of Roundrobinsinkselector is Roundrobinorderselector extends Orderselector, it implements the Createiterator () method, This method is used to elect all sink and its active Sinkl index encapsulated into a specificorderiterator<t> (Indexorder, GetObjects ()) and return, You can use the Specificorderiterator.hasnext () method to determine if there is a sink, and to get the next sink with the next () method. This allows sink to be fetched sequentially, sequentially, in the order in which the index is incremented. The main specificorderiterator are two: one is an indexed array, the other is a sink list. The Createiterator () method code is as follows:

@Override public
  iterator<t> createiterator () {
    list<integer> activeindices = Getindexlist (); Gets the latest Active sink index list
    int size = Activeindices.size ();
    Possible that's size has shrunk so gotta adjust nexthead for this
    if (nexthead >= size) {    //There may be a total number of sink adjustments , so gotta getindexlist () and adjust nexthead
      nexthead = 0;
    }
    int begin = nexthead++;    Note + + in the following note is the first assignment, in the Self
    -add if (Nexthead = = Activeindices.size ()) {
      nexthead = 0;
    }
    
    int[] Indexorder = new Int[size];
    
    for (int i = 0; i < size; i++) {
      Indexorder[i] = Activeindices.get ((begin + i)% size);
    
    return new Specificorderiterator<t> (Indexorder, getobjects ());    Make up two arrays, all the same size
  }

The Getindexlist () method is always invoked in the Createiterator () method because of the possibility of sink interruption, or sinkgroup readjustment, which causes the Sinkgroup number of actual activities in the sink to change. Nexthead always points to the next active sink index. The Indexorder is an indexed array of new active sink, and getobjects () returns all sink lists, which can be indexed to get the corresponding sink in this list.

B, the actual operator of Randomordersinkselector is Randomorderselector extends Orderselector, which implements the Createiterator () method:

Public synchronized iterator<t> Createiterator () {
    list<integer> indexlist = Getindexlist ();
    
    int size = Indexlist.size ();
    int[] Indexorder = new Int[size];
    Indexlist because the remove operation changes dynamically, using indexlist.size () always obtains the actual size while
    (Indexlist.size ()!= 1) {
      int pick = Random.nextint (Indexlist.size ());
      Indexorder[indexlist.size ()-1] = Indexlist.remove (pick);    Take out the index of the pick position, which always causes the Indexorder to insert data from the back forward
    }
    
    indexorder[0] = indexlist.get (0);    Place the last index into the Indexorder return
    
    new Specificorderiterator<t> (Indexorder, getobjects ());