Source code Analysis of HTTP monitoring types in "Flume" Flume, metric information analysis, and Flume event bus

In flume1.5.2, if you want to get flume related metrics through HTTP monitoring, add the following after the startup script:

-dflume.monitoring.type=http-dflume.monitoring.port=34545

Monitoring

The-D attribute can be obtained directly through system.getproerties (), so the above two properties are read by Method Loadmonitoring (), and the method is flume in the portal application

 private void Loadmonitoring () {Properties systemprops = system.getproperties ();    set<string> keys = Systemprops.stringpropertynames (); try {if (Keys.contains (Conf_monitor_class)) {String MonitorType = Systemprops.getproperty (conf_monitor_class        ); class<?        Extends Monitorservice> Klass;          try {//is it a known type?        Klass = Monitoringtype.valueof (Monitortype.touppercase ()). Getmonitorclass (); } catch (Exception e) {//not a known type, use FQCN klass = (class<? extends monitorservice>) Cla        Ss.forname (MonitorType);        } This.monitorserver = Klass.newinstance ();        Context context = new context (); for (String Key:keys) {if (Key.startswith (Conf_monitor_prefix)) {Context.put (key.substring (conf_mo          Nitor_prefix.length ()), Systemprops.getproperty (key));        }} monitorserver.configure (context); MOnitorserver.start ();              }} catch (Exception e) {Logger.warn ("Error starting monitoring.")    + "Monitoring might not being available.", e); }  }

The monitorserver here is a httpmetricsserver, in which a jettyserver instance is started to listen, see its start () method

public void Start () {    jettyserver = new Server ();    We can use contexts etc if we had many URLs to handle. For one URL,    //specifying a handler directly are the most efficient.    Selectchannelconnector connector = new Selectchannelconnector ();    Connector.setreuseaddress (true);    Connector.setport (port);    Jettyserver.setconnectors (new connector[] {Connector});    Jettyserver.sethandler (New Httpmetricshandler ());    try {      jettyserver.start ();      while (!jettyserver.isstarted ()) {        thread.sleep (+)}    } catch (Exception ex) {      Log.error (" Error starting Jetty. JSON Metrics May is available. ", ex);    }

In fact, this is the same as our socket in Java, where we start the server, listening to the configured port number
Watch out for a processor Httpmetricshandler

Look at its handle () method

public void handle (String target, httpservletrequest request, httpservletresponse response,      int dispatch) throws IOException, servletexception {///metrics is the-the-place-to-pull metrics.  If we want to use any other URLs for something else, we should make sure//that for metrics only/metrics are used to      Prevent backward//compatibility issues. if (Request.getmethod (). Equalsignorecase ("TRACE") | | request.getmethod (). Equalsignorecase ("Options") {RESP        Onse.senderror (Httpservletresponse.sc_forbidden);        Response.flushbuffer ();        Request). SetHandled (True);      Return        } if (Target.equals ("/")) {Response.setcontenttype ("text/html;charset=utf-8");        Response.setstatus (HTTPSERVLETRESPONSE.SC_OK); Response.getwriter (). Write ("For Flume metrics click" + "<a href = \"./metrics\ "> HERE&LT;/A&G        t;. ");        Response.flushbuffer (); ((Request) request). SetHandled (True);      Return        } else if (Target.equalsignorecase ("/metrics")) {Response.setcontenttype ("application/json;charset=utf-8");        Response.setstatus (HTTPSERVLETRESPONSE.SC_OK);        Map<string, map<string, string>> metricsmap = Jmxpollutil.getallmbeans ();        String json = Gson.tojson (Metricsmap, Maptype);        Response.getwriter (). write (JSON);        Response.flushbuffer ();        Request). SetHandled (True);      Return      } response.senderror (Httpservletresponse.sc_not_found);      Response.flushbuffer ();    Not handling the request returns a does not found error page. }

This approach is the key to getting those metrics

1, the first is to prohibit the trace and options two kinds of requests

trace- Echo the requests received by the server, primarily for testing or diagnostics.
OPTIONS- returns the HTTP request methods that the server supports for a specific resource. You can also test the functionality of your server with a request to send a ' * ' to the Web server.

2, if you enter the address is Ip:port, did not enter the following metrics, the effect is as follows:

Click here to see the measurement information

3, if you enter the metrics request, the following is the specific processing logic

Map<string, map<string, string>> metricsmap = Jmxpollutil.getallmbeans ();

That's the core of getting JSON information.
The content behind this is JMX related, and this article does not have much to do with the details, see: Http://baike.baidu.com/link?url= K1fgrfj4e720xbcekizt1ev1zfct5lckufbwkyj1s5mfvak7ks0vlb-lfdmkxowfjjb9vya1pwgypyupkirzj_

Measure

Now let's take a look at how a certain metric information is acquired in a step-by-step way to

Openconnectioncount

As an example

In the class Sourcecounter

Construction method

Public Sourcecounter (String name) {    super (MonitoredCounterGroup.Type.SOURCE, name, ATTRIBUTES);  }

Called by the Doconfigure (context context) of the class Jmssource

And this method is called by the Configure of the Basicsourcesemantics class.
At the meeting back to the entrance application Loadmonitoring method, this chain will pass

The above constructor method calls the parent class construction method

Protected Monitoredcountergroup (type type, string name, string ... attrs) {    this.type = Type;    this.name = name;    map<string, atomiclong> counterinitmap = new hashmap<string, atomiclong> ();    Initialize the Counters    for (String attribute:attrs) {      counterinitmap.put (attribute, new Atomiclong (0L)); c6/>}    countermap = Collections.unmodifiablemap (counterinitmap);    StartTime = new Atomiclong (0L);    StopTime = new Atomiclong (0L);  }

The default value for all properties here is 0, declared with Atomiclong.

In the whole Sourcecounter class, we see a lot of increment,add,get methods, so when are these methods called?

Let's look at a specific Avrosource class

Public Status Append (avroflumeevent avroevent) {    logger.debug ("Avro Source {}: Received Avro event: {}", GetName (), 
   avroevent);    Sourcecounter.incrementappendreceivedcount ();    Sourcecounter.incrementeventreceivedcount ();    Event event = Eventbuilder.withbody (Avroevent.getbody (). Array (),        Tostringmap (Avroevent.getheaders ()));    try {      getchannelprocessor (). processevent (Event),    } catch (Channelexception ex) {      Logger.warn ("Avro SOURCE "+ getName () +": Unable to process event. "+          " Exception follows. ", ex);      return status.failed;    }    Sourcecounter.incrementappendacceptedcount ();    Sourcecounter.incrementeventacceptedcount ();    return status.ok;  }

Here you can see the increment method called Sourcecounter, the source code is easy to see the meaning of the various properties

Each property is assigned in real time in the process of component operation.

{
"Source.r1": {
"Openconnectioncount": "0", the number of currently active connections
' type ': ' SOURCE ', component type
"Appendbatchacceptedcount": "0", the source side just append the number of batches into the channel
"Appendbatchreceivedcount": "0", the source side just append the number of batches, such as a batch of 100, the measure is 2, is the source side received 200 events
"Eventacceptedcount": "Ten", the source side is currently successfully placed in the channel number of event
"StopTime": "0",
"Appendreceivedcount": "0", the source side just append the current number of received event
"StartTime": "1422502242340", the time when the source component started
"Eventreceivedcount": "Ten", the source side has received the number of event
"Appendacceptedcount": "0" source side just append the number of event in channel
},
"Source.r2": {
"Openconnectioncount": "0",
"Type": "SOURCE",
"Appendbatchacceptedcount": "0",
"Appendbatchreceivedcount": "0",
"Eventacceptedcount": "10",
"StopTime": "0",
"Appendreceivedcount": "0",
"StartTime": "1422502242341",
"Eventreceivedcount": "10",
"Appendacceptedcount": "0"
},
"Channel.c1": {
"Eventputsuccesscount": "20", number of event successfully placed in channel
"Channelfillpercentage": "0.0", Channel usage ratio
"Type": "CHANNEL",
"StopTime": "0",
"Eventputattemptcount": "20", number of event being put into channel
"Channelsize": "0",
"StartTime": "1422502242328",
"Eventtakesuccesscount": "20", the number of event successfully removed from the channel
"Channelcapacity": "10000000",
"Eventtakeattemptcount": "2105" is fetching the number of event from the channel
},
"Sink.k1": {
"Batchcompletecount": "0", the number of batch events to complete the output successfully
"Connectionfailedcount": "0", number of sink end connection failures
"Eventdrainattemptcount": "20", the number of events attempted to consume, taken from the channel to consume
"Connectioncreatedcount": "1", number of sink end connections
"Type": "SINK",
"Batchemptycount": "1042", Number of empty batches
"Connectionclosedcount": "0", number of connections closed
"Eventdrainsuccesscount": "20", Number of successfully processed event
"StopTime": "0",
"StartTime": "1422502242340",
"Batchunderflowcount": "1" internal event is in the batch process of batch quantity, such as two batch, the internal event is processing, is equal to 2
},
"Sink.k2": {
"Batchcompletecount": "0",
"Connectionfailedcount": "0",
"Eventdrainattemptcount": "0",
"Connectioncreatedcount": "1",
"Type": "SINK",
"Batchemptycount": "1042",
"Connectionclosedcount": "0",
"Eventdrainsuccesscount": "0",
"StopTime": "0",
"StartTime": "1422502243048",
"Batchunderflowcount": "0"
}
}

Event Bus

The above is the monitoring related metric information, then back to the beginning of this article, loadmonitoring method, we look at its invocation process, analysis of eventbus related content

if (reload) {        Eventbus Eventbus = new Eventbus (agentname + "-event-bus");        Pollingpropertiesfileconfigurationprovider Configurationprovider =            New Pollingpropertiesfileconfigurationprovider (AgentName,                configurationfile, Eventbus,);        Components.add (Configurationprovider);        application = new application (components);        Eventbus.register (application);      } else {        Propertiesfileconfigurationprovider configurationprovider =            new Propertiesfileconfigurationprovider (AgentName,                configurationfile);        application = new application ();        Application.handleconfigurationevent (Configurationprovider.getconfiguration ());      }

or the upper entry, Else branch we can easily find the entire call tree structure of the loadmonitoring method, but in the if branch it is difficult to catch the tree structure of the call, let's take a closer look:

Eventbus is an event bus

Eventbus's Code:

public void Register (object object) {
Multimap<class<?>, eventhandler> Methodsinlistener =
Finder. findallhandlers (object);
Handlersbytype.putall (Methodsinlistener);
}

When the register is Eventbus, a finder is used to find the @subscribe method in the object of the register. and classify according to EventType, put in Handlersbytype. This way, when Eventbus post a new event, the corresponding EventHandler can be called according to the EventType.

The event bus registers the application and, as explained above, looks for a way to annotate @subscribe in the application class.

@Subscribe public  synchronized void Handleconfigurationevent (Materializedconfiguration conf) {    Stopallcomponents ();    Startallcomponents (conf);  }

There is a startallcomponents method in this method, the last line of the method calls the This.loadmonitoring () method, so the tree structure is out.

For more information about event bus, please refer to: http://blog.csdn.net/kiwi_coder/article/details/9338721

Source code Analysis of HTTP monitoring types in "Flume" Flume, metric information analysis, and Flume event bus