Flume ng 1.3 mounting (RPM)

Source: Internet
Author: User

http://blog.csdn.net/hijk139/article/details/8308224

Business systems need to collect monitoring system logs and think of the flume of Hadoop. After testing, although the function is not strong enough, but basically can meet the functional requirements. Flume is a distributed, reliable and highly available service Log Collection tool, capable of completing log collection, storage, analysis, and other tasks such as hadoop,hive, and more detailed descriptions can be found on the Apache website. The following is an introduction to the simple installation configuration method

1, download the Flume-ng installation package on the Internet, deploy on the server that collects and receive log files separately, the server needs to install JDK 1.6 above,

Http://flume.apache.org/download.html
TAR-ZXVF apache-flume-1.3.0-bin.tar.gz
2, the specific configuration of the new Conf/flume-conf.properties server end of the log file receive end is as follows

Receives data from the Avro source side and writes to the HDFs file system

[HTML]View Plaincopy
  1. [email protected] conf]$ cat flume-conf.properties
  2. agent.sources = avrosrc
  3. Agent.channels = MemoryChanne3
  4. Agent.sinks = Hdfssink
  5. # for each one of the sources, the type is defined
  6. Agent.sources.avrosrc.type = Avro
  7. Agent.sources.avrosrc.bind = 172.16.251.1
  8. Agent.sources.avrosrc.port = 44444
  9. # The channel can be defined as follows.
  10. Agent.sources.avrosrc.channels = MemoryChanne3
  11. # each channel ' s type is defined.
  12. Agent.channels.memoryChanne3.type = Memory
  13. Agent.channels.memorychanne3.keep-alive = Ten
  14. agent.channels.memoryChanne3.capacity = 100000
  15. agent.channels.memoryChanne3.transactionCapacity =100000
  16. # each sink ' s type must be defined
  17. Agent.sinks.hdfsSink.type = HDFs
  18. Agent.sinks.hdfsSink.channel = MemoryChanne3
  19. Agent.sinks.hdfsSink.hdfs.path =/logdata/%{hostname}_linux/%y%m%d_date
  20. Agent.sinks.hdfsSink.hdfs.filePrefix =%{datacenter}_
  21. Agent.sinks.hdfsSink.hdfs.rollInterval = 0
  22. Agent.sinks.hdfsSink.hdfs.rollSize = 4000000
  23. Agent.sinks.hdfsSink.hdfs.rollCount = 0
  24. Agent.sinks.hdfsSink.hdfs.writeFormat = Text
  25. Agent.sinks.hdfsSink.hdfs.fileType = DataStream
  26. Agent.sinks.hdfsSink.hdfs.batchSize = Ten


If Flume and Hadoop are not the same users, you need to be aware of related permissions issues

3, log collection end of the Conf/flume-conf.properties server file configuration, here collect two log files to the collection end

[HTML]View Plaincopy
  1. agent.sources = tailsource-1 tailsource-2
  2. Agent.channels = memoryChannel-1 memoryChannel-2
  3. Agent.sinks = Remotesink remotesink-2
  4. Agent.sources.tailsource-1.type = exec
  5. Agent.sources.tailsource-1.command = tail-f/tmp/linux2.log
  6. Agent.sources.tailsource-1.channels = MemoryChannel-1
  7. Agent.sources.tailsource-2.type = exec
  8. Agent.sources.tailsource-2.command = tail-f/tmp/linux2_2.log
  9. Agent.sources.tailsource-2.channels = MemoryChannel-2
  10. Agent.sources.tailsource-1.interceptors = host_int timestamp_int inter1
  11. Agent.sources.tailsource-1.interceptors.host_int.type = Host
  12. Agent.sources.tailsource-1.interceptors.host_int.hostheader = hostname
  13. Agent.sources.tailsource-1.interceptors.timestamp_int.type = Org.apache.flume.interceptor.timestampinterceptor$builder
  14. #agent.sources.tailsource-1.interceptors = inter1
  15. Agent.sources.tailsource-1.interceptors.inter1.type = static
  16. Agent.sources.tailsource-1.interceptors.inter1.key = Datacenter
  17. Agent.sources.tailsource-1.interceptors.inter1.value = Beijing
  18. Agent.sources.tailsource-2.interceptors = host_int timestamp_int inter1
  19. Agent.sources.tailsource-2.interceptors.host_int.type = Host
  20. Agent.sources.tailsource-2.interceptors.host_int.hostheader = hostname
  21. Agent.sources.tailsource-2.interceptors.timestamp_int.type = Org.apache.flume.interceptor.timestampinterceptor$builder
  22. #agent.sources.tailsource-1.interceptors = inter1
  23. Agent.sources.tailsource-2.interceptors.inter1.type = static
  24. Agent.sources.tailsource-2.interceptors.inter1.key = Datacenter
  25. Agent.sources.tailsource-2.interceptors.inter1.value = linux2_2
  26. Agent.channels.memorychannel-1.type = Memory
  27. Agent.channels.memorychannel-1.keep-alive = Ten
  28. agent.channels.memorychannel-1.capacity = 100000
  29. agent.channels.memorychannel-1.transactioncapacity =100000
  30. Agent.channels.memorychannel-2.type = Memory
  31. Agent.channels.memorychannel-2.keep-alive = Ten
  32. agent.channels.memorychannel-2.capacity = 100000
  33. agent.channels.memorychannel-2.transactioncapacity =100000
  34. Agent.sinks.remotesink.type = Avro
  35. Agent.sinks.remotesink.hostname = 172.16.251.1
  36. Agent.sinks.remotesink.port = 44444
  37. Agent.sinks.remotesink.channel = MemoryChannel-1
  38. Agent.sinks.remotesink-2.type = Avro
  39. Agent.sinks.remotesink-2.hostname = 172.16.251.1
  40. Agent.sinks.remotesink-2.port = 44444
  41. Agent.sinks.remotesink-2.channel = MemoryChannel-2


4, running in the background
Nohup bin/flume-ng agent-n agent-c conf-f conf/flume-conf.properties >1.log &

View Log VI Flume.log
Port connection Condition Netstat-an|grep 44444
[Email protected] flume-1.4]$ Netstat-an|grep 44444
TCP 0 0:: ffff:172.16.251.1:44444:::* LISTEN

5, test method

You can use a script like the following to periodically write to a log file for testing

For i in {1..1000000}; Do echo "LINUX2 press ************* Flume log rotation $i" >>/tmp/linux3.log; Sleep 0.0001; Done

Resources:
Http://flume.apache.org/FlumeUserGuide.html

Flume ng 1.3 mounting (RPM)

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.