Flume Reference Document (1)

FlumeConfiguration FileFlume-site.conf)

1. Watchdog

Watchdog. restarts. Max

What is the maximum number of watchdog restarts per minute ???

2. Common Node

Flume. config. Heartbeat. Period	Node sends a heartbeat cycle. The default value is 5000 (MS)
Flume. node. Status. Port	Node Web port
Flume. node. Heartbeat. backoff. Ceiling	The maximum duration of waiting for feedback after the node sends a heartbeat request to the master node. The default value is 60000 (milliseconds)
Flume. node. http. autofindport	If a node already starts, the second node is allowed to automatically select an unused port for Web Services. The interface ports of multiple nodes are extended from 35862 to 35863.

3. Agent

Flume. Agent. logdir	Agent Log Path
Flume. Agent. logdir. maxage	The duration of Agent log file collection in the enabled status. After this time, the log file will be closed and data will be sent to the network. The default value is 10000 (milliseconds)
Flume. Agent. logdir. retransmit	In end-to-end mode, the interval for sending data again after the agent fails to send data to collector. The default value is 60000 (MS). We recommend that you set the interval to at least flume. collector. roll. twice of millis
Flume. Agent. failover. backoff. Initial	When the primary sink (which can be considered as the first collector) fails, restart a delay time of the primary sink. During this period, the agent will send data to the secondary sink (possibly the second collector)
Flume. Agent. failover. backoff. Max	After a failed attempt to connect to a faulty node within a certain period of time, the agent forwards the data to the standby node.

4. Collector

Flume. Collector. event. Host	Default collector address
Flume. Collector. Port	Default collector Port
Flume. Collector. dfs. dir	The final data sending directory (default) can be local, HDFS, And/tmp by default
Flume. Collector. dfs. Compress. Codec	The compression format is gzipcodec, defaultcodec (deflate), and bzip2codec. The default value is none.
Flume. Collector. Roll. millis	Duration of HDFS file switching (New after Shutdown)
Flume. Collector. Output. Format	Collector sends data in Avro, avrojson (default), avrodata...

5. Master

Flume. master. Servers	Separate multiple master address lists with commas
Flume. master. Store	Master configuration storage mode (zookeeper/memory) zookeeper ensures that the master configuration is synchronized between multiple master nodes, and memory is saved in the memory, and its configuration is lost as the master goes down
Flume. master. serverid	Unique Master ID
Flume. master. http. Port	Http port
Flume. master. Heartbeat. Missed. Max	Maximum heartbeat count of failed nodes
Flume. master. SaveFile	Path of the current flume configuration file. The default value is CONF/current. flume.
Flume. master. SaveFile. autoload	Whether to load current. flume at startup. The default value is false.
Flume. master. Gossip. Period	Master communication cycle (MS)
Flume. master. Heartbeat. RPC	Thrift/Avro
Flume. event. RPC	Thrift/Avro
Flume. Report. server. rpc. Type	Thrift/Avro

6. zookeeper

Flume. master. zk. logdir

Zookeeper Log Path

7. Thrift

Flume. Thrift. Socket. Timeout. Ms

Thrift network connection timeout (MS)