Flume Service Management Implementation analysis

The flume can monitor and manage the running state of the component, which can be pulled automatically when the component is closed, by starting a scheduled task thread pool (Monitorservice, the maximum number of threads is 30), running the monitoring thread (monitorrunnable thread), Each 3s determines whether the state of the component (including Channel,sinkrunner) meets the requirements (the available state consists of two start and stop), calls the corresponding component different methods according to different requirements, start invokes the Start method, and stop calls the Stop method, If you want to monitor the state of a component, simply call the supervise method on the component, and if you want to stop monitoring a component, simply call the Unsupervise method on the component, At the same time, there is a thread that removes inspection tasks for components that are no longer monitoring (called the Unsupervise method) every two hours.

This function is mainly implemented by Org.apache.flume.lifecycle.LifecycleSupervisor
The object of the Lifecyclesupervisor class is initialized in the constructor of the Org.apache.flume.node.Application class:

Public application (list<lifecycleaware>) {the. components = components;  Supervisor = new Lifecyclesupervisor (); }

Called when the flume process starts

Org.apache.flume.node.Application.main--->org.apache.flume.node.application.start public synchronized void The start () {//start method calls the Lifecyclesupervisor.supervise method on each component, the parameter is the component, the Alwaysrestartpolicy and the start state (that is, the expected state is start) for ( Lifecycleaware component:components) {supervisor.supervise (component, New SUPERVISORPOLICY.ALWAYSRESTARTP     Olicy (), Lifecyclestate.start); }  }

Analysis Lifecyclesupervisor class:
Org.apache.flume.lifecycle.LifecycleSupervisor implements the Lifecycleaware interface, which itself also has a concept of life cycle (providing methods such as Start/stop)

It defines several important inner classes:
1.MonitorRunnable, implements the thread class of the Runnable interface
1) 3 Properties Scheduledexecutorservice monitorservice,lifecycleaware lifecycleaware,supervisoree Supervisoree;
2) Analysis of the main run method

    public void run ()  {      long now  = system.currenttimemillis ();      try {         if  (supervisoree.status.firstseen == null)  {           logger.debug ("first time seeing {}",  Lifecycleaware);           supervisoree.status.firstseen  = now; //first Run, set Firstseen to System.currenttimemillis ()          }        supervisoree.status.lastSeen = now;  //set Lastseen to now        synchronized  (lifecycleAware)   {          if  (Supervisoree.status.discard)  {  //If the status's discard or error value is true, it exits directly...            return;           } else if  (Supervisoree.status.error)  {...             return;           }           Supervisoree.status.lastseenstate = lifecycleaware.getlifecyclestate ();  //setting the value of LastSeenState           if  (!lifecycleaware.getlifecyclestate (). Equals (               supervisoree.status.desiredState))  { //If the Lifecycleaware object state obtained is not the desiredstate state you want to set ...             switch  (Supervisoree.status.desiredState)  { //the different methods of calling Lifecycleaware according to the Desiredstate state of the set, there are only two values for desiredstate start and Stop              case start:                 try {                    Lifecycleaware.start (); Set run Start method when  //status is start                  } catch  (throwable e)  {...                    Supervisoree.status.failures++; //start method Exception when failures value plus 1                 }                 break;               case stop:                try {                   lifecycleaware.stop ();  //the Stop method is set when the status is Stop                  } catch  (throwable e)  {...                   supervisoree.status.failures++; // Stop method exception when failures value plus 1                 }                 break;              default:...             }             if  (!SUPervisoree.policy.isValid (lifecycleaware, supervisoree.status))  {                //calls Supervisorpolicy's IsValid method, such as Onceonlypolicy The IsValid method of   will determine the value of status.failures , if 0 returns true, otherwise false               logger.error (                    "policy {} of {} has been  violated - supervisor should exit! ",                   supervisoree.policy, lifecycleaware);             }           }        }      } catch (throwable t)  {...             }...    } 

2.Purger, implements the thread class of the Runnable interface
Run Method:

public void Run () {if (Needtopurge) {//If Needtopurge is set to true Monitorservice.purge ();//scheduledthreadpoolexe The Cutor.purge method is used to remove the Java.util.concurrent.Future object (free queue space) that has been cancel from the work queue Needtopurge = false; and set Needtopurge to False}}

The 3.Status inner class defines several state attributes that represent the state of the Supervisoree

Public Long Firstseen;    Public Long Lastseen;    Public Lifecyclestate lastseenstate;    Public Lifecyclestate desiredstate;    public int failures;    public Boolean discard; public volatile Boolean error;

4. Supervisorpolicy is an abstract class that defines the abstract method IsValid (Lifecycleaware object, status status), which contains two extension classes Alwaysrestartpolicy and Onceonlypolicy
Alwaysrestartpolicy's IsValid will always return True,onceonlypolicy IsValid method will determine the value of Status.failures, if 0 returns true, otherwise false

5.Supervisoree contains Supervisorpolicy and status attributes

The Main method analysis:
Several important properties are initialized in the constructor method:

  public lifecyclesupervisor ()  {    lifecycleState =  Lifecyclestate.idle;    supervisedprocesses = new hashmap<lifecycleaware The, supervisoree> (); // supervisedprocesses  is used to hold key-value pairs for lifecycleaware and Supervisoree objects, Represents an already managed component     monitorFutures = new HashMap<LifecycleAware,  Scheduledfuture<?>> (); //monitorfutures  for storing key-value pairs of Lifecycleaware objects and Scheduledfuture objects      monitorservice = new scheduledthreadpoolexecutor (10,         new threadfactorybuilder (). Setnameformat (              "lifecyclesupervisor-"  + thread.currentthread (). GetId ()  +   "-%d")             .build ()); //  monitorservice  is used to call the Purger thread and periodically removes the CA in the thread poolNcel's Task    monitorservice.setmaximumpoolsize (;    ) Monitorservice.setkeepalivetime (30, timeunit.seconds);     purger = new  purger ();    needtopurge = false; //  initially false, in task  When Cancel is set to true  }

The Start method is used to start the detection thread pool:

Public synchronized void Start () {.... monitorservice.schedulewithfixeddelay (Purger, 2, 2, Timeunit. HOURS); Run Purger every two hours after two hours, releasing the worker queue Lifecyclestate = Lifecyclestate of the thread pool. START; Set status to start ...}

The Stop method first closes the thread pool and then closes the individual components
1) thread pool shutdown

Monitorservice.shutdown ();

2) each component is closed

For (final Entry<lifecycleaware, supervisoree> entry:supervisedprocesses. EntrySet ()) {//Traversal Supervisedproc The individual components in the Esses if (Entry.getkey (). Getlifecyclestate (). Equals (Lifecyclestate.start)) {//If the current state of the component is START, You first set the Stop method that it needs to change to a stop and call the component Entry.getvalue (). Status.         Desiredstate = Lifecyclestate.stop;      Entry.getkey (). Stop (); }    }

The supervise method is used to monitor the corresponding component, with 3 parameters Lifecycleaware Lifecycleaware, supervisorpolicy policy, Lifecyclestate desiredstate

 public synchronized void supervise (lifecycleaware lifecycleaware,       supervisorpolicy policy, lifecyclestate desiredstate)  {     if (This. monitorservice.isshutdown ()         | |  this.monitorservice .isterminated ()         | |  this.monitorservice .isterminating ()) { //detects if the monitoring thread pool is healthy       throw  new flumeexception ("supervise called on "  + lifecycleAware +  "   " +          " after shutdown has  been initiated.  " + lifecycleAware + "  will not " +            " be started");    }     preconditions.checkstate (!supervIsedprocesses .containskey (Lifecycleaware),         "Refusing  to supervise  " + lifecycleAware + "  more than once " );  //detection has been managed .....     supervisoree process = new supervisoree ();  //Initialize Supervisoree object     process.status = new status ();  // and instantiate the Supervisoree object's Status property     process.policy = policy; //set the properties of the Supervisoree     process.status.desiredState = desiredState;     process.status.error = false;    monitorrunnable monitorrunnable =  new monitorrunnable ();  //Initializes a monitorrunnable  object (thread) and sets the object's properties      Monitorrunnable.lifecycleaware = lifecycleaware;    monitorrunnable.supervisoree  = process;   &nbsP;monitorrunnable.monitorservice = monitorservice;    supervisedprocesses.put ( lifecycleaware, process);  //inserts a key-value pair into the supervisedprocesses, representing the components that have already been managed      Scheduledfuture<?> future = monitorservice.schedulewithfixeddelay (         monitorrunnable, 0, 3, timeunit. seconds); //  Set up a scheduled task thread pool, running Monitorrunnable    monitorfutures.put (lifecycleaware, future) after every 3s;  //  inserting key values into Monitorfutures   }

The

Unsupervise method is used to stop the component and remove it from the monitoring container:

    synchronized  (Lifecycleaware)  {    Supervisoree  Supervisoree = supervisedprocesses.get (Lifecycleaware);  //from the already managed supervisoree   Gets the Supervisoree object in HashMap     supervisoree.status.discard = true;  // The discard  value for the status property of the Supervisoree object is set to Discard      this.setdesiredstate ( Lifecycleaware, lifecyclestate.stop);        //calls the Setdesiredstate method, Set the desiredstate  value of the Supervisoree object's Status property to stop (Supervisoree.status.desiredstate = desiredstate)       logger.info ("stopping component: {}",  lifecycleAware);       lifecycleaware.stop ();  //call component's Stop method     }     supervisedprocesses.remove (lifecycleaware);  //remove this component from the Supervisedprocesses hashmap     monitorfutures.geT (Lifecycleaware). Cancel (false);      //calls the component corresponding to the Scheduledfuture cancellation method of the task (a  Future represents the result of an asynchronous computation. ,cancel  :attempts to cancel execution of this task.)      needtopurge = true; //sets the property of needtopurge  to True so that the Scheduledfuture object that has been cancel can be deleted in purge     monitorfutures.remove (Lifecycleaware);

This article is from the "Food and Light Blog" blog, please make sure to keep this source http://caiguangguang.blog.51cto.com/1652935/1619527

Flume Service Management Implementation analysis