Flume Reference Document (3)

Flume Source

1. Flume's tiered event sources

Collectorsource [(Port)]	Collector Source: collects data on listening ports.
Autocollectorsource	Automatically aggregates data by coordinating physical nodes through the master
Logicalsource	Logical source: the master allocates ports and listens to rpcsink.

 

2. Flume's basic sources

Null
Console	Listen to user editing history and shortcut key input, only available in node_nowatch Mode
Stdin	Standard input for listening, only available in node_nowatch mode, each row serves as an event Source
Rpcsource (Port)	Listen to TCP ports by the RPC framework (thrift/Avro)
Text ("Filename")	Read one text at a time, and each action has an event
Tail ("Filename"[,Startfromend= False])	Each action has an event. Append row at the end of the listening file. If startfromend is true, tail reads data from the end of the file. If it is false, tail reads all data from the file.
Multitail ("Filename"[,File2[,File3... ])	Same as above, listening to the end of multiple files at the same time
Taildir ("Dirname"[, Fileregex =". * "[, startfromend = false [, recursedepth = 0])	At the end of the file in the listening directory, use regular expressions to select the file to be listened to (excluding the directory). recursedepth indicates the depth of the subdirectory under the recursive listening directory.
Seqfile ("Filename")	Listen to the sequencefile of HDFS, full path
Syslogudp (Port)	Listen to UDP port
Syslogtcp (Port)	Listen to TCP port
Syslogtcp1 (Port)	Only listen to one link of the TCP port
Execperiodic ("Cmdline",MS)	Periodical invocation of commands, listening for the output of commands, the entire output is used as an event
Execstream ("Cmdline")	Execute the command, listen to the output of the command, and each line of the output is used as an event
Exec ("Cmdline"[,Aggregate= False [,Restart= False [,Period= 0])	Execute the command and listen on the output of the command. If aggregate is true and the output is a single event, if it is false, each line serves as an event. If restart is true, it is re-run according to the period
Synth (Msgcount,Msgsize)	The string event is generated. msgcount indicates the number of logs generated, and msgsize indicates the string length.
Synthrndsize (Msgcount,Minsize,Maxsize)	Same as above, minsize-maxsize
Nonlsynth (Msgcount,Msgsize)
Asciisynth (Msgcount,Msgsize)	ASCII characters
Twitter ("Username","PW"[,"URL"])	Nima Twitter plug-in
IRC ("Server",Port,"Nick","Chan")
Scribe [(+Port)]	Scribe plug-in
Report [(periodmillis)]	Generate all physical node reports as event sources