Several monitoring methods of flume

Flume mainly by the following types of monitoring methods:
JMX Monitoring

JMX High detonation can modify the JAVA_OPTS environment variables in the flume-env.sh file as follows:

Export java_opts= "-dcom.sun.management.jmxremote-dcom.sun.management.jmxremote.port=5445- Dcom.sun.management.jmxremote.authenticate=false-dcom.sun.management.jmxremote.ssl=false "

Ganglia monitoring

Flume can also report metrics to ganglia 3 or ganglia 3.1 metanodes. To report metrics to ganglia, the Flume Agent must be supported at boot time. The flume agent uses flume.monitoring as the prefix and starts with the following parameters. Of course, you can also set the following in flume-env.sh:

Property	default	Description
type	–	Component Name: Ganglia
hosts	–	A comma-delimited list of hostname:port ganglia servers.
Pollfrequency	60	How many seconds to send data to ganglia.
IsGanglia3	False	If the server is 3, the default is the format of ganglia 3.1.

If you want to support ganglia, you can start by using the following command.

$ bin/flume-ng Agent--conf-file example.conf--name a1-dflume.monitoring.type=ganglia-dflume.monitoring.hosts= com.example:1234,com.example2:5455

JSON monitoring

Flume can report metrics in JSON form, enable JSON, Flume need to configure a port. The following is a format for reporting metrics in JSON format:

{"
typename1.componentname1": {"Metric1": "MetricValue1", "Metric2": "MetricValue2"},
" Typename2.componentname2 ": {" metric3 ":" MetricValue3 "," Metric4 ":" MetricValue4 "}
}

For example:

{"
Channel.filechannel": {"Eventputsuccesscount": "468085",
                      "Type": "CHANNEL",
                      "stoptime": "0",
                      " Eventputattemptcount ":" 468086 ","
                      channelsize ":" 233428 ",
                      " StartTime ":" 1344882233070 ",
                      " Eventtakesuccesscount ":" 458200 ","
                      channelcapacity ":" 600000 ",
                      " Eventtakeattemptcount ":" 458288 "},
" Channel.memchannel ": {" Eventputsuccesscount ":" 22948908 ",
                   " Type ":" CHANNEL ",
                   " stoptime ":" 0 ",
                   " Eventputattemptcount ":" 22948908 ","
                   channelsize ":" 5 ",
                   " StartTime ":" 1344882209413 ",
                   " Eventtakesuccesscount ":" 22948900 ","
                   channelcapacity ":" The ","
                   eventtakeattemptcount ":" 22948908 "}
}

Property name	default	Description
Type	–	Name of component: HTTP
Port	41414	Port on which to start the service

You can start flume with the following command:

$ bin/flume-ng Agent--conf-file example.conf--name a1-dflume.monitoring.type=http-dflume.monitoring.port=34545

Custom Monitoring

Custom monitoring requires implementation of the Org.apache.flume.instrumentation.MonitorService interface. For example, there is an HTTP monitor class called httpreporting, and I can start this monitoring in the following way.

$ bin/flume-ng Agent--conf-file example.conf--name a1-dflume.monitoring.type=com.example.reporting.httpreporting- dflume.monitoring.node=com.example:332

Report metrics We can also customize components, but be sure to inherit org.apache.flume.instrumentation.MonitoredCounterGroup virtual classes. Flume already implemented classes, as shown in the following figure:

Based on the above specification, I can develop custom monitoring components.