Hello Flume (Ubuntu under flume1.5 Standalone installation and simple Getting Started example)

Source: Internet
Author: User

1. Download the latest flume installation package:

wget http://www.apache.org/dist/flume/stable/apache-flume-1.5.2-bin.tar.gz



2, unzip in the installation directory:

TAR-ZXVF apache-flume-1.5.2-bin.tar.gz



3, setting environment variables

Export JAVA_HOME=/USR;
Export Flume_home=/home/joeyon/apache-flume-1.5.2-bin;
Export path= $PATH: Flume_home/bin

4, configure a simple single-machine test case

</pre>4.1 Create a new file in the Conf directory, example-conf.properties, with the following contents: <pre name= "code" class= "HTML" ># Describe the SOURCE a1.sources = R1a1.sinks = K1a1.channels = c1# describe/configure the Sourcea1.sources.r1.type = avroa1.sources.r1.b IND = Localhosta1.sources.r1.port = 44444# Describe The sink# outputs the data to the log A1.sinks.k1.type = Loggera1.channels.c1.type = Memo rya1.channels.c1.capacity = 1000a1.channels.c1.transactioncapacity = 100# Bind The source and sink to the Channela1.source S.r1.channels = C1a1.sinks.k1.channel = C1


4.2 Create a new source file file01 and write data to it in the Conf directory

# Use a channel which buffers events in memory   touch file01;   

4.3 Starting agent Agent

Flume-ng agent-c. -f/home/joeyon/apache-flume-1.5.2-bin/conf/example-conf.properties-  N a1-dflume.root.logger=info,console
The startup success log is as follows:

+ EXEC/USR/BIN/JAVA-XMX20M-DFLUME.ROOT.LOGGER=INFO,CONSOLE-CP '/data/projects/apache-flume-1.5.2-bin/conf:/data/ projects/apache-flume-1.5.2-bin/lib/* '-djava.library.path= org.apache.flume.node.application-f/data/projects/ Apache-flume-1.5.2-bin/conf/example-conf.properties-n a12015-01-21 13:44:43,231 (lifecycleSupervisor-1-0) [INFO- Org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start ( POLLINGPROPERTIESFILECONFIGURATIONPROVIDER.JAVA:61)] Configuration provider starting2015-01-21 13:44:43,239 ( conf-file-poller-0) [info-org.apache.flume.node.pollingpropertiesfileconfigurationprovider$ Filewatcherrunnable.run (pollingpropertiesfileconfigurationprovider.java:133)] reloading configuration file:/data/ Projects/apache-flume-1.5.2-bin/conf/example-conf.properties2015-01-21 13:44:43,248 (conf-file-poller-0) [INFO- Org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty (flumeconfiguration.java:930)] Added Sinks:k1 agent:a12015-01-21 13:44:43,249 (conf-file-poller-0) [Info-org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty ( flumeconfiguration.java:1016)] processing:k12015-01-21 13:44:43,249 (conf-file-poller-0) [INFO- Org.apache.flume.conf.flumeconfiguration$agentconfiguration.addproperty (flumeconfiguration.java:1016)] Processing:k12015-01-21 13:44:43,261 (conf-file-poller-0) [INFO- Org.apache.flume.conf.FlumeConfiguration.validateConfiguration (flumeconfiguration.java:140)] Post-validation Flume configuration contains configuration for agents: [a1]2015-01-21 13:44:43,262 (conf-file-poller-0) [Info-org.apach E.flume.node.abstractconfigurationprovider.loadchannels (abstractconfigurationprovider.java:150)] Creating Channels2015-01-21 13:44:43,271 (conf-file-poller-0) [Info-org.apache.flume.channel.defaultchannelfactory.create ( DEFAULTCHANNELFACTORY.JAVA:40)] Creating instance of channel C1 type MEMORY2015-01-21 13:44:43,276 (conf-file-poller-0) [Info-org.apache.flume.node.abstractconfigurationprovider.loadchannEls (abstractconfigurationprovider.java:205)] Created channel c12015-01-21 13:44:43,277 (conf-file-poller-0) [INFO- Org.apache.flume.source.DefaultSourceFactory.create (defaultsourcefactory.java:39)] Creating instance of source R1, Type avro2015-01-21 13:44:43,297 (conf-file-poller-0) [Info-org.apache.flume.sink.defaultsinkfactory.create ( DEFAULTSINKFACTORY.JAVA:40)] Creating instance of SINK:K1, type:logger2015-01-21 13:44:43,301 (conf-file-poller-0) [ Info-org.apache.flume.node.abstractconfigurationprovider.getconfiguration (Abstractconfigurationprovider.java : 119)] Channel C1 connected to [R1, k1]2015-01-21 13:44:43,309 (conf-file-poller-0) [Info-org.apache.flume.node.applica Tion.startallcomponents (application.java:138)] starting new configuration:{sourcerunners:{r1=  Eventdrivensourcerunner: {Source:avro source r1: {bindaddress:localhost, port:44444}}} Sinkrunners:{k1=sinkrunner: { Policy:[email protected] countergroup:{name:null counters:{}}} Channels:{c1=org.apacHE.FLUME.CHANNEL.MEMORYCHANNEL{NAME:C1}}}2015-01-21 13:44:43,320 (conf-file-poller-0) [INFO- Org.apache.flume.node.Application.startAllComponents (application.java:145)] starting Channel c12015-01-21 13:44:43,383 (lifecyclesupervisor-1-0) [Info-org.apache.flume.instrumentation.monitoredcountergroup.register ( monitoredcountergroup.java:119)] Monitored counter Group for Type:channel, name:c1:Successfully registered new mbean.20 15-01-21 13:44:43,383 (lifecyclesupervisor-1-0) [INFO- Org.apache.flume.instrumentation.MonitoredCounterGroup.start (monitoredcountergroup.java:95)] Component type: CHANNEL, Name:c1 started2015-01-21 13:44:43,387 (conf-file-poller-0) [INFO- Org.apache.flume.node.Application.startAllComponents (application.java:173)] starting Sink k12015-01-21 13:44:43,388 (conf-file-poller-0) [Info-org.apache.flume.node.application.startallcomponents (Application.java : 184)] Starting Source r12015-01-21 13:44:43,388 (lifecyclesupervisor-1-0) [Info-org.apache.flume.source.aVrosource.start (avrosource.java:228)] starting Avro source R1: {bindaddress:localhost, port:44444} ... 2015-01-21 13:44:43,771 (lifecyclesupervisor-1-0) [INFO- Org.apache.flume.instrumentation.MonitoredCounterGroup.register (monitoredcountergroup.java:119)] monitored Counter group for Type:source, name:r1:Successfully registered new mbean.2015-01-21 13:44:43,771 (lifecycleSupervisor-1 -0) [Info-org.apache.flume.instrumentation.monitoredcountergroup.start (monitoredcountergroup.java:95)] Component Type:source, Name:r1 started2015-01-21 13:44:43,772 (lifecyclesupervisor-1-0) [INFO- Org.apache.flume.source.AvroSource.start (avrosource.java:253)] Avro source R1 started.

4.4. Start the Avro-client client to send data to the agent proxy:

Flume-ng avro-client-c. -H localhost-p 44444-f/home/joeyon/apache-flume-1.5.2-bin/conf/file01


The logs are as follows:

+ EXEC/USR/BIN/JAVA-XMX20M-CP '/data/projects/apache-flume-1.5.2-bin/conf:/data/projects/apache-flume-1.5.2-bin/ lib/* '-djava.library.path= org.apache.flume.client.avro.avrocliclient-h localhost-p 44444-f/home/joeyon/ Apache-flume-1.5.2-bin/conf/file01


4.5 View Agen Console output The agent-side output is collected successfully as follows:

2015-01-21 13:50:13,276 (New I/O server boss #1 ([ID:0XC0B7FFFB,/127.0.0.1:44444]) [Info-org.apache.avro.ipc.nettyser Ver$nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809 =/ 127.0.0.1:44444] open2015-01-21 13:50:13,277 (New I/O worker #1) [info-org.apache.avro.ipc.nettyserver$ Nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809 =/ 127.0.0.1:44444] BOUND:/127.0.0.1:444442015-01-21 13:50:13,278 (New I/O worker #1) [Info-org.apache.avro.ipc.nettyser Ver$nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809 =/ 127.0.0.1:44444] CONNECTED:/127.0.0.1:38092015-01-21 13:50:13,703 (New I/O worker #1) [Info-org.apache.avro.ipc.netty Server$nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809:>/ 127.0.0.1:44444] disconnected2015-01-21 13:50:13,703 (New I/O worker #1) [Info-org.apache.avro.ipc.nettyservEr$nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809:>/ 127.0.0.1:44444] unbound2015-01-21 13:50:13,703 (New I/O worker #1) [info-org.apache.avro.ipc.nettyserver$ Nettyserveravrohandler.handleupstream (nettyserver.java:171)] [id:0x9d115219,/127.0.0.1:3809:>/ 127.0.0.1:44444] closed2015-01-21 13:50:13,704 (New I/O worker #1) [info-org.apache.avro.ipc.nettyserver$ Nettyserveravrohandler.channelclosed (nettyserver.java:209)] Connection to/127.0.0.1:3809 disconnected.2015-01-21 13:50:17,411 (Sinkrunner-pollingrunner-defaultsinkprocessor) [Info-org.apache.flume.sink.loggersink.process ( LOGGERSINK.JAVA:70)] Event: {headers:{} body:68 6C 6C 6F, 6F, 6C, 3E 3E 3E 3E 3E Hello world>>>> >}2015-01-21 13:50:17,411 (sinkrunner-pollingrunner-defaultsinkprocessor) [INFO- Org.apache.flume.sink.LoggerSink.process (loggersink.java:70)] Event: {headers:{} body:6d 6E 6D 65 20 69 73 20 6 A 6F 6F My name is Joeyo}2015-01-21 13:50:17,412 (sinkrunner-pollingrunner-defaultsinkprocessor) [INFO- Org.apache.flume.sink.LoggerSink.process (loggersink.java:70)] Event: {headers:{} body:77 6C 6F 6D, 6F 20 63 6 8 6E 3E welcom to China>}


Hello Flume (Ubuntu under flume1.5 Standalone installation and simple Getting Started example)

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.