Hello Flume (Ubuntu under flume1.5 Standalone installation and simple Getting Started example)

1. Download the latest flume installation package:

wget http://www.apache.org/dist/flume/stable/apache-flume-1.5.2-bin.tar.gz

2, unzip in the installation directory:

TAR-ZXVF apache-flume-1.5.2-bin.tar.gz

3, setting environment variables

Export JAVA_HOME=/USR;

Export Flume_home=/home/joeyon/apache-flume-1.5.2-bin;

Export path= $PATH: Flume_home/bin

4, configure a simple single-machine test case

</pre>4.1 Create a new file in the Conf directory, example-conf.properties, with the following contents: <pre name= "code" class= "HTML" ># Describe the SOURCE a1.sources = R1a1.sinks = K1a1.channels = c1# describe/configure the Sourcea1.sources.r1.type = avroa1.sources.r1.b IND = Localhosta1.sources.r1.port = 44444# Describe The sink# outputs the data to the log A1.sinks.k1.type = Loggera1.channels.c1.type = Memo rya1.channels.c1.capacity = 1000a1.channels.c1.transactioncapacity = 100# Bind The source and sink to the Channela1.source S.r1.channels = C1a1.sinks.k1.channel = C1

4.2 Create a new source file file01 and write data to it in the Conf directory

# Use a channel which buffers events in memory   touch file01;   

4.3 Starting agent Agent

Flume-ng agent-c. -f/home/joeyon/apache-flume-1.5.2-bin/conf/example-conf.properties-  N a1-dflume.root.logger=info,console

The startup success log is as follows:

+ EXEC/USR/BIN/JAVA-XMX20M-DFLUME.ROOT.LOGGER=INFO,CONSOLE-CP '/data/projects/apache-flume-1.5.2-bin/conf:/data/ projects/apache-flume-1.5.2-bin/lib/* '-djava.library.path= org.apache.flume.node.application-f/data/projects/ Apache-flume-1.5.2-bin/conf/example-conf.properties-n a12015-01-21 13:44:43,231 (lifecycleSupervisor-1-0) [INFO- Org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start ( POLLINGPROPERTIESFILECONFIGURATIONPROVIDER.JAVA:61)] Configuration provider starting2015-01-21 13:44:43,239 ( conf-file-poller-0) [info-org.apache.flume.node.pollingpropertiesfileconfigurationprovider$ Filewatcherrunnable.run (pollingpropertiesfileconfigurationprovider.java:133)] reloading configuration file:/data/ Projects/apache-flume-1.5.2-bin/conf/example-conf.properties2015-01-21 13:44:43,248 (conf-file-poller-0) [INFO- Org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty (flumeconfiguration.java:930)] Added Sinks:k1 agent:a12015-01-21 13:44:43,249 (conf-file-poller-0) [Info-org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty ( flumeconfiguration.java:1016)] processing:k12015-01-21 13:44:43,249 (conf-file-poller-0) [INFO- Org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty (flumeconfiguration.java:1016)] Processing:k12015-01-21 13:44:43,261 (conf-file-poller-0) [INFO- Org.apache.flume.conf.FlumeConfiguration.validateConfiguration (flumeconfiguration.java:140)] Post-validation Flume configuration contains configuration for agents: [a1]2015-01-21 13:44:43,262 (conf-file-poller-0) [Info-org.apach E.flume.node.abstractconfigurationprovider.loadchannels (abstractconfigurationprovider.java:150)] Creating Channels2015-01-21 13:44:43,271 (conf-file-poller-0) [Info-org.apache.flume.channel.defaultchannelfactory.create ( DEFAULTCHANNELFACTORY.JAVA:40)] Creating instance of channel C1 type MEMORY2015-01-21 13:44:43,276 (conf-file-poller-0) [Info-org.apache.flume.node.abstractconfigurationprovider.loadchannEls (abstractconfigurationprovider.java:205)] Created channel c12015-01-21 13:44:43,277 (conf-file-poller-0) [INFO- Org.apache.flume.source.DefaultSourceFactory.create (defaultsourcefactory.java:39)] Creating instance of source R1, Type avro2015-01-21 13:44:43,297 (conf-file-poller-0) [Info-org.apache.flume.sink.defaultsinkfactory.create ( DEFAULTSINKFACTORY.JAVA:40)] Creating instance of SINK:K1, type:logger2015-01-21 13:44:43,301 (conf-file-poller-0) [ Info-org.apache.flume.node.abstractconfigurationprovider.getconfiguration (Abstractconfigurationprovider.java : 119)] Channel C1 connected to [R1, k1]2015-01-21 13:44:43,309 (conf-file-poller-0) [Info-org.apache.flume.node.applica Tion.startallcomponents (application.java:138)] starting new configuration:{sourcerunners:{r1=  Eventdrivensourcerunner: {Source:avro source r1: {bindaddress:localhost, port:44444}}} Sinkrunners:{k1=sinkrunner: { Policy:[email protected] countergroup:{name:null counters:{}}} Channels:{c1=org.apacHE.FLUME.CHANNEL.MEMORYCHANNEL{NAME:C1}}}2015-01-21 13:44:43,320 (conf-file-poller-0) [INFO- Org.apache.flume.node.Application.startAllComponents (application.java:145)] starting Channel c12015-01-21 13:44:43,383 (lifecyclesupervisor-1-0) [Info-org.apache.flume.instrumentation.monitoredcountergroup.register ( monitoredcountergroup.java:119)] Monitored counter Group for Type:channel, name:c1:Successfully registered new mbean.20 15-01-21 13:44:43,383 (lifecyclesupervisor-1-0) [INFO- Org.apache.flume.instrumentation.MonitoredCounterGroup.start (monitoredcountergroup.java:95)] Component type: CHANNEL, Name:c1 started2015-01-21 13:44:43,387 (conf-file-poller-0) [INFO- Org.apache.flume.node.Application.startAllComponents (application.java:173)] starting Sink k12015-01-21 13:44:43,388 (conf-file-poller-0) [Info-org.apache.flume.node.application.startallcomponents (Application.java : 184)] Starting Source r12015-01-21 13:44:43,388 (lifecyclesupervisor-1-0) [Info-org.apache.flume.source.aVrosource.start (avrosource.java:228)] starting Avro source R1: {bindaddress:localhost, port:44444} ... 2015-01-21 13:44:43,771 (lifecyclesupervisor-1-0) [INFO- Org.apache.flume.instrumentation.MonitoredCounterGroup.register (monitoredcountergroup.java:119)] monitored Counter group for Type:source, name:r1:Successfully registered new mbean.2015-01-21 13:44:43,771 (lifecycleSupervisor-1 -0) [Info-org.apache.flume.instrumentation.monitoredcountergroup.start (monitoredcountergroup.java:95)] Component Type:source, Name:r1 started2015-01-21 13:44:43,772 (lifecyclesupervisor-1-0) [INFO- Org.apache.flume.source.AvroSource.start (avrosource.java:253)] Avro source R1 started.

4.4. Start the Avro-client client to send data to the agent proxy:

Flume-ng avro-client-c. -H localhost-p 44444-f/home/joeyon/apache-flume-1.5.2-bin/conf/file01

The logs are as follows:

+ EXEC/USR/BIN/JAVA-XMX20M-CP '/data/projects/apache-flume-1.5.2-bin/conf:/data/projects/apache-flume-1.5.2-bin/ lib/* '-djava.library.path= org.apache.flume.client.avro.avrocliclient-h localhost-p 44444-f/home/joeyon/ Apache-flume-1.5.2-bin/conf/file01

4.5 View Agen Console output The agent-side output is collected successfully as follows:

2015-01-21 13:50:13,276 (New I/O server boss #1 ([ID:0XC0B7FFFB,/127.0.0.1:44444]) [Info-org.apache.avro.ipc.nettyser Ver$nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809 =/ 127.0.0.1:44444] open2015-01-21 13:50:13,277 (New I/O worker #1) [info-org.apache.avro.ipc.nettyserver$ Nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809 =/ 127.0.0.1:44444] BOUND:/127.0.0.1:444442015-01-21 13:50:13,278 (New I/O worker #1) [Info-org.apache.avro.ipc.nettyser Ver$nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809 =/ 127.0.0.1:44444] CONNECTED:/127.0.0.1:38092015-01-21 13:50:13,703 (New I/O worker #1) [Info-org.apache.avro.ipc.netty Server$nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809:>/ 127.0.0.1:44444] disconnected2015-01-21 13:50:13,703 (New I/O worker #1) [Info-org.apache.avro.ipc.nettyservEr$nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809:>/ 127.0.0.1:44444] unbound2015-01-21 13:50:13,703 (New I/O worker #1) [info-org.apache.avro.ipc.nettyserver$ Nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809:>/ 127.0.0.1:44444] closed2015-01-21 13:50:13,704 (New I/O worker #1) [info-org.apache.avro.ipc.nettyserver$ Nettyserveravrohandler.channelclosed (nettyserver.java:209)] Connection to/127.0.0.1:3809 disconnected.2015-01-21 13:50:17,411 (Sinkrunner-pollingrunner-defaultsinkprocessor) [Info-org.apache.flume.sink.loggersink.process ( LOGGERSINK.JAVA:70)] Event: {headers:{} body:68 6C 6C 6F, 6F, 6C, 3E 3E 3E 3E 3E Hello world>>>> >}2015-01-21 13:50:17,411 (sinkrunner-pollingrunner-defaultsinkprocessor) [INFO- Org.apache.flume.sink.LoggerSink.process (loggersink.java:70)] Event: {headers:{} body:6d 6E 6D 65 20 69 73 20 6 A 6F 6F My name is Joeyo}2015-01-21 13:50:17,412 (sinkrunner-pollingrunner-defaultsinkprocessor) [INFO- Org.apache.flume.sink.LoggerSink.process (loggersink.java:70)] Event: {headers:{} body:77 6C 6F 6D, 6F 20 63 6 8 6E 3E welcom to China>}

Hello Flume (Ubuntu under flume1.5 Standalone installation and simple Getting Started example)