Detailed analysis of Execsource source code in "Flume" Flume--Execute terminal command to get data

Let's look directly at the source's Start method.

public void Start () {    logger.info ("Exec source starting with command:{}", command);    Executor = Executors.newsinglethreadexecutor ();    Runner = new Execrunnable (shell, Command, Getchannelprocessor (), Sourcecounter,        restart, Restartthrottle, Logstderr, BufferCount, BatchTimeout, CharSet);    Fixme:use a callback-like executor/future to signal us upon failure.    Runnerfuture = Executor.submit (runner);    /     * * Nb:this comes at the end rather than the beginning of the method because     * It sets we state to running. We want to make sure the executor are alive * and well first     .     *    /Sourcecounter.start ();    Super.start ();    Logger.debug ("Exec source Started");  }

Start a thread run, run the detailed process look runner

It is a thread that implements the Runnable interface, so look directly at the logic of the Run method it overrides, and we look at it one piece at a glance:

if (shell! = null) {            string[] Commandargs = formulateshellcommand (shell, command);            Process = Runtime.getruntime (). exec (Commandargs);          }  else {            string[] Commandargs = Command.split ("\\s+");            Process = new Processbuilder (Commandargs). Start ();          }          reader = new BufferedReader (              new InputStreamReader (Process.getinputstream (), CharSet));

This is the execution of the shell command, and the output of the shell command is read into reader as an input stream, and InputStreamReader is a bridge of bytes flowing to the character stream, which reads the bytes and decodes them into characters using the specified charset. Each call to the Read method reads one or more bytes from the underlying input stream.

while (line = Reader.readline ()) = null) {            synchronized (eventlist) {              Sourcecounter.incrementeventreceivedcount ();              Eventlist.add (Eventbuilder.withbody (Line.getbytes (CharSet)));              if (Eventlist.size () >= BufferCount | | timeout ()) {                flusheventbatch (eventlist);}}          }

If the read-in content is not empty, synchronize the EventList first, if the eventlist exceeds a certain range, it will flush

private void Flusheventbatch (list<event> eventlist) {      channelprocessor.processeventbatch (eventList);      Sourcecounter.addtoeventacceptedcount (Eventlist.size ());      Eventlist.clear ();      Lastpushtochannel = Systemclock.currenttimemillis ();    }

Flush is the event in the EventList that is now being saved and emptied

1. Put the event into the configured channel

For (event event:events) {      list<channel> reqchannels = Selector.getrequiredchannels (event);      for (Channel ch:reqchannels) {        list<event> eventqueue = reqchannelqueue.get (CH);        if (EventQueue = = null) {          eventqueue = new arraylist<event> ();          Reqchannelqueue.put (CH, eventqueue);        }        Eventqueue.add (event);      }      list<channel> optchannels = Selector.getoptionalchannels (event);      for (Channel ch:optchannels) {        list<event> eventqueue = optchannelqueue.get (CH);        if (EventQueue = = null) {          eventqueue = new arraylist<event> ();          Optchannelqueue.put (CH, eventqueue);        }        Eventqueue.add (event);      }    }

Here is the detailed procedure for putting the event into the channel, but here you notice that there are two selector getchannel methods, because there are two types of channel selector modes: Multiplexing and Replication

  if (restart) {          logger.info ("Restarting in {}ms, exit code {}", Restartthrottle,              exitCode);          try {            thread.sleep (restartthrottle);          } catch (Interruptedexception e) {            thread.currentthread (). Interrupt ();          }        } else {          logger.info ("command [" + COMMAND + "] exited with" + ExitCode);        }      } while (restart);

The meaning of the restart parameter is that when the shell command executes, the process is dead, if the process of restarting the command is restarted, the default is False

Configured to True, all the code just now loops over

Summarize:

1. How is the event produced?

Eventlist.add (Eventbuilder.withbody (Line.getbytes (CharSet)));

public static event Withbody (byte[] body, map<string, string> headers) {    Event event = new SimpleEvent ();    if (BODY = = null) {      BODY = new byte[0];    }    Event.setbody (body);    if (headers! = null) {      event.setheaders (new hashmap<string, string> (headers));    }    return event;  }

2. How does the event fit into the channel?

private void Flusheventbatch (list<event> eventlist) {      channelprocessor.processeventbatch (eventList);      Sourcecounter.addtoeventacceptedcount (Eventlist.size ());      Eventlist.clear ();      Lastpushtochannel = Systemclock.currenttimemillis ();    }

Detailed analysis of Execsource source code in "Flume" Flume--Execute terminal command to get data