"Java" "Flume" flume-ng Start process Source Analysis (i) __java zone

From Bin/flume this shell script can see Flume starting from the Org.apache.flume.node.Application class, which is where the main function of Flume is.

The main method first resolves the shell command, throwing an exception if the specified configuration file does not exist.

According to the command contains "no-reload-conf" parameters, decide which way to load the configuration file: First, without this parameter, the configuration file is dynamically loaded, the configuration file is loaded every 30 seconds by default, so the configuration file can be modified dynamically, and with this parameter, the configuration file is loaded only once at startup. The implementation of dynamic loading features a publish subscription model, which is implemented using Eventbus in guava.

Eventbus Eventbus = new Eventbus (agentname + "-event-bus");
        Pollingpropertiesfileconfigurationprovider Configurationprovider =
            New Pollingpropertiesfileconfigurationprovider (AgentName,
                configurationfile, Eventbus, 30);//This is the class that publishes the event, Here 30 is the dynamic load configuration file time interval, the unit is S
        components.add (configurationprovider);
        application = new application (components);
        Eventbus.register (application);    Register the subscription class with the bus

Subscription class is application = new application (components) ; Publish the code in the Filewatcherrunnable.run method in Pollingpropertiesfileconfigurationprovider.  In this just first build a Pollingpropertiesfileconfigurationprovider object, Pollingpropertiesfileconfigurationprovider extends Propertiesfileconfigurationprovider implements Lifecycleaware, continue tracking Propertiesfileconfigurationprovider extends Abstractconfigurationprovider, and then trace Abstractconfigurationprovider implements Configurationprovider can see that the constructors for these classes are all initialized , the Abstractconfigurationprovid construction method initializes the factory class for Sink, channel, and source.

Application.handleconfigurationevent (materializedconfiguration conf) has @subscribe annotations, is the subscription method, when Eventbus.post ( Materializedconfiguration conf) executes, the execution Handleconfigurationevent method is triggered.

When new application (components), an object is built supervisor = new Lifecyclesupervisor () starts 10 threads to execute the components in the configuration file and monitors the entire operation of the component.

The Application.start () method initiates the loading process of the configuration file Supervisor.supervise (component, New Supervisorpolicy.alwaysrestartpolicy (), Lifecyclestate.start); Lifecyclestate.start began to run, in which the component was the Pollingpropertiesfileconfigurationprovider object. The supervise method creates a monitorrunnable process for component and puts it into the default 10-thread Monitorservice to execute

    Supervisoree process = new Supervisoree ();
    Process.status = new status ();

    Process.policy = policy;
    Process.status.desiredState = desiredstate;
    Process.status.error = false;

    Monitorrunnable monitorrunnable = new monitorrunnable ();
    Monitorrunnable.lifecycleaware = lifecycleaware;//component
    monitorrunnable.supervisoree = process;
    Monitorrunnable.monitorservice = Monitorservice;

    Supervisedprocesses.put (lifecycleaware, process);
    Creates and executes a periodic operation that was first enabled after a given initial delay, followed by a given delay between execution termination and the start of the next execution. If any of the execution of a task encounters an exception, subsequent execution is canceled.
    scheduledfuture<?> future = Monitorservice.schedulewithfixeddelay (
        monitorrunnable, 0, 3, Timeunit.seconds);  Start monitorrunnable, 3 seconds after the end of reboot, can be used to retry
    monitorfutures.put (Lifecycleaware, future);

Look at the Monitorrunnable class, its run method is mainly based on the value of Supervisoree.status.desiredState to perform the corresponding operation. The lifecycleaware here is the component,lifecycleaware of the above supervise method will be lifecyclestate=idle,application.start at the beginning of the construction () Method Supervisoree.status.desiredstate=start by Supervisor.supervise method. So the Lifecycleaware.start (), the Pollingpropertiesfileconfigurationprovider.start () method, is executed in the Run method.

The Pollingpropertiesfileconfigurationprovider.start () method starts a single-threaded filewatcherrunnable every 30s to load a configuration file (if there are modifications to the configuration file) : Eventbus.post (GetConfiguration ()). GetConfiguration () is the Abstractconfigurationprovider.getconfiguration () method that resolves the configuration file to obtain all the components and their configuration properties. This method is more complicated and is explained later.

The Application.handleconfigurationevent method is triggered after Eventbus.post (GetConfiguration ()):

  @Subscribe public
  synchronized void Handleconfigurationevent (Materializedconfiguration conf) {
    Stopallcomponents ();
    Startallcomponents (conf);
  

The Stopallcomponents () method stops the operation of each component sequentially: source, sink, channel. The reason why there is Order is: one, source is constantly reading data into the channel, sink is constantly from channel to take data, channel both ends in use should be the last stop, stop to channel send data after the sink stop will not lose data. The stop is done through the Supervisor.unsupervise method.

Startallcomponents (conf) is the start of each component, the order is exactly the same as the stopallcomponents () stop order, I believe it is easy for everyone to understand. is to start the component through Supervisor.supervise. Also note that you need to wait a certain amount of time to start the channel component in order for all channel to start.

In addition, why stop and start first. Because you want to load the configuration file dynamically, you need to reboot all the components after loading the configuration file, so stop all and restart all.

The end of the main method also has a hook function runtime.getruntime (). Addshutdownhook, mainly used for memory cleaning, object destruction and other operations.