Flume Learning application: Write log data to MongoDB and flumemongodb in Java

Flume Learning application: Write log data to MongoDB and flumemongodb in Java
Overview

Windows: Java writes logs to Flume, and Flume writes the logs to MongoDB. System Environment

• Operating System: win7 64
• JDK: 1.6.0 _ 43

Download Resources

• Maven: 3.3.3
  Download, install, and get started: 1. Maven-start and 2. Create a simple Maven Project
• Flume: 1.6.0
  Download apache-flume-1.6.0-bin.tar.gz and decompress it to the appropriate directory (e.g. F: \ temp \ apache-flume-1.6.0-bin)
• MongoDB: 3.2.3
  Download the mongodb-win32-x86_64-3.2.3-signed.msi and install it to the appropriate directory (e.g. D: \ develop \ MongoDB)

Auxiliary resource download

• MongoDB Java Driver: 2.8.0
  : Mongo-java-driver-2.13.0.jar
  Download the package and place it in the lib of the Flume installation directory.
• Flume-ng-mongodb-sink
  This is required for Flume to write data to MongoDB.
  : Flume-ng-mongodb-sink
  This is a Maven-based source code. After package, put the jar package under the lib of the Flume installation directory.

Initialization Specify dbpath for data initialization to MongoDB. Configure Flume implementation Flume to MongoDB under the conf of the Flume installation directory, add mongo-agent.properties:

agent.sources = so1agent.channels = c1agent.sinks = s1agent.sources.so1.type = avroagent.sources.so1.bind = 0.0.0.0agent.sources.so1.port = 44444agent.sources.so1.channels = c1agent.channels.c1.type = memoryagent.channels.c1.capacity = 1000agent.channels.c1.transactionCapacity = 100agent.sinks.s1.type = org.riderzen.flume.sink.MongoSinkagent.sinks.s1.host = localhostagent.sinks.s1.port = 27017agent.sinks.s1.model = singleagent.sinks.s1.db = testagent.sinks.s1.collection = logagent.sinks.s1.batch = 100agent.sinks.s1.channel = c1

Simple Description:

Java to Flume this is a Maven-based simple project, you can get the source code from the flume-ng-java2mongodb. File structure

src/main/java    |---- cn.sinobest.flume.client.demo              |---- LogDemo.javasrc/main/resources    |---- log4j.propertiespom.xml

Document Content test Appendix References