Monitoring of Flume

Flume, as a Log collection tool, exhibits a very powerful capability in data collection. Its source, SINK, channel three components of this mode, to complete the data reception, caching, sending this process, has a very perfect fit. But here, we want to say is not flume how good or flume have what merit, we want to talk about is flume monitoring. 1, why need flume monitoring.

Flume as a powerful data collection tool, although it is very powerful and practical, but can not see the details of flume collection of data, so we need a can show flume real-time data collection dynamic information interface, including Flume successfully collected the number of logs, the number of successfully sent logs, Flume start time, stop time, and flume some specific configuration information, such as channel capacity, so smooth Cheng Zhang monitoring can help us do these, with these data, in the face of data collection bottlenecks or data loss, through the analysis of monitoring data to analyze and solve problems. 2. What are the monitoring methods of flume? (1), HTTP monitoring

With this monitoring method, you only need to add a monitoring configuration to the boot parameters when starting flume, for example:

[Java] view plain copy bin/flume-ng agent--conf conf--conf-file conf/flume_conf.properties--name COLLECT-DF lume.monitoring.type=http-dflume.monitoring.port=1234 where-dflume.monitoring.type=http means to monitor using HTTP, The following-dflume.monitoring.port=1234 indicates that we need to start the Monitoring service port number is 1234, this port number can be configured on its own. After starting Flume, a JSON-formatted monitoring data of Flume can be obtained through http://ip:1234/metrics.

(2), ganglia monitoring

This monitoring method requires installing ganglia and then starting ganglia, and then starting flume with the monitoring configuration, for example:

[Java] view plain copy bin/flume-ng agent--conf conf--conf-file conf/producer.properties--name Collect-dflu Me.monitoring.type=ganglia-dflume.monitoring.hosts=ip:port where-dflume.monitoring.type=ganglia means to monitor using ganglia, Instead,-dflume.monitoring.hosts=ip:port represents the ganglia installed IP and the port number that was started.

Flume monitoring can also use the Zabbix, but this way needs to add the monitoring module in Flume source code, relatively troublesome, because not flume comes with the monitoring method, here does not discuss this way.

Therefore, the flume comes from the monitoring method is actually HTTP, ganglia two, HTTP monitoring can only be accessed through an HTTP address to get a JSON-formatted monitoring data, and ganglia monitoring is to get this data in the form of interface display, relatively intuitive. 3, Flume monitor which components, can get the information of the component. (1), SOURCE

Source as the data source component of the Flume, where all the first of the collected logs arrives, its monitoring information is very important. By monitoring the data we can get, we have these:

Openconnectioncount (number of open connections), type (component type), Appendbatchacceptedcount (number of batches appended to the channel), Appendbatchreceivedcount (Number of batches just appended to source end), Eventacceptedcount (number of event successfully placed in channel), Appendreceivedcount (source append current quantity received), StartTime (component start time), StopTime (component stop time), Eventreceivedcount (number of event received by source side successfully), Appendacceptedcount (number of event placed in channel), etc. Of course these are only flume monitoring source has been brought in the monitoring elements, if you need other monitoring information, such as IP, port number, there are two methods, the first, modify the monitoring source, add the monitoring elements you need, this method is based on the original code, add some of the monitoring elements to meet their needs, simpler, but less flexible; the second is the custom monitoring component, which implements its own monitoring component in the original monitoring framework, so that it can fully meet its own needs and is highly flexible. As to how these two methods operate, the following flume monitoring how implementation has been discussed.

The monitoring of the two components of the same channel and sink can also be used to add the monitoring elements that you want. (2), CHANNEL

Channel is a flume component that has a caching effect on the data. The data that can be obtained:

Eventputsuccesscount (number of event successfully placed in the channel), Channelfillpercentage (channel usage scale), type (component type), Eventputattemptcount (attempts to put the event into the channel), Channelsize (number of event currently in channel), StartTime (component start time), StopTime (Component Stop Time), Eventtakesuccesscount (number of event successfully removed from channel), channelcapacity (channel capacity), Eventtakeattemptcount (attempts to take the event from the channel) and so on.

(3), SINK Sink is the last component of the data that is about to leave Flume, which takes data from the channel and then sends it to the cache system or persistent database. Can get the data:

Batchcompletecount (number of batches completed), Connectionfailedcount (number of connection failures), Eventdrainattemptcount (number of event attempts submitted), Connectioncreatedcount (number of connections created), type (component type), Batchemptycount (number of bulk empties), Connectionclosedcount (number of connections closed), Eventdrainsuccesscount (number of successfully sent event), StartTime (component start time), StopTime (component Stop Time), Batchunderflowcount (Number of batches in batch processing), etc.
4, flume monitoring is how to achieve.

First, in the main method of org.apache.flume.node.Application in Flume-ng-node, there is a startallcomponents () method:

[Java]  View Plain  copy private void startallcomponents (                materializedconfiguration materializedconfiguration)  {            logger.info ("Starting new configuration: {} ",  materializedconfiguration);               this.materializedconfiguration = materializedconfiguration;               for  (entry<string, channel> entry :  materializedconfiguration                    .getchannels (). EntrySet ())  {                try {                &Nbsp;   logger.info ("starting channel "  + entry.getkey ());                    Supervisor.supervise (Entry.getvalue (),                            new  Supervisorpolicy.alwaysrestartpolicy (),                            lifecyclestate.start );               } catch  ( Exception e)  {                    logger.error ("error while starting {}",  entry.getvalue (),  e);                }           }              /*            * Wait for all channels to start.            */           for  (Channel  ch : materializedconfiguration.getchannels (). VALUES ())  {                while  (Ch.getlifecyclestate ()  !=  lifecyclestate.start                        && !supervisor.iscomponentinerrorstate (CH))  {                     try {                        logger.info ("waiting for channel: "  + ch.getname ()                                  +  " to start. sleeping for 500  ms "