Flume-ng source reading: Sourcerunner and selector selector and interceptor Interceptor execution

In the Abstractconfigurationprovider class The Loadsources method encapsulates all of the source into Sourcerunner map<string, sourcerunner> In the midst of Sourcerunnermap. The relevant code is as follows:

map<string, string> selectorconfig = context.getsubproperties (
              basicconfigurationconstants.config_source _channelselector_prefix);
    
          Channelselector selector = channelselectorfactory.create (
              sourcechannels, selectorconfig);
    
          Channelprocessor channelprocessor = new Channelprocessor (selector);
          Configurables.configure (channelprocessor, context);
          Source.setchannelprocessor (channelprocessor);
          Sourcerunnermap.put (SourceName,
              sourcerunner.forsource (source));

Each source has a selector. The code above obtains the selector configuration information about source in the configuration file, and then constructs the Channelselector object selector , and encapsulates selector objects into Channelprocessor objects Channelprocessor To perform the configuration of the Channelprocessor.configure method, set the channelprocessor of the soure, and finally encapsulate the sourcerunner and source names together into the Sourcerunnermap.

One, Channelselector selector = channelselectorfactory.create (sourcechannels, Selectorconfig) Instantiates a channelselector based on the type specified in the configuration file (two kinds of replicatingchannelselector replication and Multiplexingchannelselector multiplexing) If you do not specify a type default is Replicatingchannelselector, that is, the configuration file does not have to be configured selector sends each event replication to multiple channel;selector.setchannels ( Channels), configure the Configure (context) for this slector. Both of these selector implement three methods Getrequiredchannels (event event), Getoptionalchannels (event event), and configure. In fact, the event to send to the channel there are two components: Requiredchannels and optionalchannels, corresponding to two methods.

The Configure (context) method of the

(1) Replicatingchannelselector is given an optional channels (can be separated by a space) specified in the configuration file by "optional". ; Gets requiredchannels is the list of channel that can be active in the channel corresponding to this source, and then gets the names of all channel and their mappings with channel Channelnamemap And then add the optional channel to the Optionalchannels and remove the corresponding channel from the Requiredchannels, There is no check for the legality of the optional channel and the Channel,requiredchannels and optionalchannels that can be configured with this source channel cannot have intersections. The Getoptionalchannels method is to return directly to the Optionalchannels list. The Getrequiredchannels method returns the list of Requiredchannels, and if requiredchannels is null, returns all the channel lists that can be active.

(2) Multiplexingchannelselector's Configure (context) obtains the header,headername of the event to match, and gets the channel list sent by default Defaultchannels To obtain mapping values and corresponding channel name Mapconfig, which are used to store the different header values and their corresponding channel lists to be sent to multiple channel Optionalchannels is an optional sent channel,channelmapping that has already appeared in the configuration channel not allowed to appear again in Optionalchannels, Optionalchannels stores the list of selectable channel to be sent to each value corresponding to the header and the event that is equal to the value. The Getoptionalchannels (Event event) method returns a list of selectable channel corresponding to the specified header for the event in Optionalchannels. The Getrequiredchannels (Event event) method returns the list of channel that corresponds to the specified header for the event in channelmapping, if NULL ( Indicates that the default Send list Defaultchannels is returned because the headers of the event does not have a matching channel sent to the default channel.

Second, channelprocessor channelprocessor = new Channelprocessor (selector) This is the encapsulation selector construct channelprocessor. Its construction method assigns a value selector and constructs a Interceptorchain object interceptorchain. The Channelprocessor class is responsible for managing selector selector and Interceptor interceptor.

Third, the implementation channelprocessor.configure (context) for the necessary configuration, the method will call Channelprocessor.configureinterceptors (context) To get and configure interceptors, the Configureinterceptors method first obtains the Interceptor component name interceptornames[] (multiple) from the configuration file, and then gets all the interceptors. Configuration information interceptorcontexts, and then iterate through all interceptornames to get the configuration information and type (type) from the configuration file that belong to this interceptor. Build the appropriate interceptor based on the type and configure configure, add the Interceptors list (to hold the instantiated interceptor), and finally pass the list to Interceptorchain. More interceptor information can be seen in this article Flume-ng source reading Interceptor (original).

Four, Source.setchannelprocessor (channelprocessor) Assign value. Each source uses the Getchannelprocessor () method to get processor to invoke its Processeventbatch (events) or processevent (event) to send the event to channel.

Five, Sourcerunnermap.put (source) Sourcename,sourcerunner.forsource the source into Sourcerunner into the sourcerunnermap. Sourcerunner.forsource will be encapsulated into different runner based on the interface implemented by this source, and there are two interfaces Pollablesource and Eventdrivensource, the former having their own threads to drive to implement the process method, which is not No process method is driven by a separate thread.

public static Sourcerunner Forsource (source source) {
    Sourcerunner runner = null;
    
    if (source instanceof Pollablesource) {
      runner = new Pollablesourcerunner ();
      ((Pollablesourcerunner) runner). SetSource ((pollablesource) source);
    else if (source instanceof Eventdrivensource) {
      runner = new Eventdrivensourcerunner ();
      ((Eventdrivensourcerunner) runner). SetSource ((eventdrivensource) source);
    else {
      throw new IllegalArgumentException ("No known runner type for source"
          + source);
    
    return runner;
  }

(1) The Pollablesourcerunner start () method takes the channelprocessor of source and then executes its initialize () method, which invokes the Interceptorchain.initialize () method initializes the interceptor (traverses all interceptors and then executes the Initialize () method of the Interceptor), then executes Source.start () to start source, and then starts a thread pollingrunner. Its run method always executes source.process () and does some statistical work based on the returned state value.