flume dc

1. Development environment 1.1. Package Download 1.1.1. JDKHttp://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.htmlInstall to the D:\GreenSoftware\Java\Java8X64\jdk1.8.0_91 directory 1.1.2. Mavenhttps://maven.apache.org/download.cgiUnzip to the D:\GreenSoftware\apache-maven-3.3.9 directory 1.1.3. Scalahttps://www.scala-lang.org/download/Unzip to the D:\GreenSoftware\Java\scala-2.12.6 directory 1.1.4. ThriftHttp://thrift.apache.org/downloadPlace the downloaded Thrift-0.

scribe, Chukwa, Kafka, flume log System comparison1. Background informationMany of the company's platforms generate a large number of logs per day (typically streaming data, such as search engine PV, queries, etc.), processing these logs requires a specific logging system, in general, these systems need to have the following characteristics: (1) Build the bridge of application system and analysis system, and decouple the association between them; (2)

./pssh-h./host/all.txt-p Mkdir/usr/local/app./pssh-h./host/all.txt-p Tar zxf/usr/local/software/apache-flume-1.6.0-bin.tar.gz-c/usr/local/app./pssh-h./host/all.txt-p mv/usr/local/app/apache-flume-1.6.0-bin/usr/local/app/apache-flume-1.6.0Vi/etc/profileAdd Flume environment variable configuration#set

Flume compared with Logstash, the personal experience is as follows: Logstash more emphasis on the preprocessing of the field, while flume emphasis on data transmission; Logstash has dozens of plug-ins, flexible configuration, Flume is to emphasize the user's custom development (source and sink kind also has ten or twenty, the channel is relatively s

the high-level interface, which hides the details of the broker, allowing consumer to push data from the broker without having to care about the network topology. More importantly, for most log systems, the data information that consumer has acquired is saved by the broker, while in Kafka, the data information is maintained by consumer itself. Cloudera's Flume Flume is Cloudera's Open source log

1, download the latest flume on the official website of Flumewget http://124.205.69.169/files/A1540000011ED5DB/mirror.bit.edu.cn/apache/flume/1.6.0/ apache-flume-1.6.0-bin.tar.gz 2. Solve Flume installation packagecd/export/software/TAR-ZXVF apache-flume-1.6.0-bin.tar.gz-c/e

The project requires C + + code to interface with the Flume, which in turn writes the log to HDFs.Flume native to Java code, the original solution was to invoke the Flume Java method via JNI.But because of the concern about the efficiency of JNI calls, and the fact that the C + + call JNI needs to take care of the local reference and GC issues, the headache has been caused.Rage, rewrite the code, use C + +

The recent project team has the need to tap the stream log to collect, learn a bit flume and install successfully. The relevant information to record a bit.1) Download flume1.5 versionwget http://www.apache.org/dyn/closer.cgi/flume/1.5.0.1/apache-flume-1.5.0.1-bin.tar.gz2) Unzip the flume1.5TAR-ZXVF apache-flume-1.5.0.

IP implementation.Paste the configuration of the testThe configuration is the same, use the time to open or close sinkgroup comments.This is the configuration of the collection node.#flume配置文件Agent1.sources=execsourceagent1.sinks= Avrosink1 Avrosink2Agent1.channels=filechannel#sink groups affect performance very much#agent1. Sinkgroups=avrogroup#agent1. sinkgroups.avroGroup.sinks = Avrosink1 Avrosink2#sink调度模式 load_balance Failover#agent1. sinkgroups

The previous introduction of how to use thrift source production data, today describes how to use Kafka sink consumption data.In fact, in the Flume configuration file has been set up with Kafka sink consumption dataAgent1.sinks.kafkaSink.type =Org.apache.flume.sink.kafka.KafkaSinkagent1.sinks.kafkaSink.topic=TRAFFIC_LOGagent1.sinks.kafkaSink.brokerList=10.208.129.3:9092,10.208.129.4:9092,10.208.129.5:9092agent1.sinks.kafkaSink.metadata.broker.list=10.

First, the architecture scheme such as:Second, the installation of the various components of the program are as follows:1), Zookeeper+kafkaHttp://www.cnblogs.com/super-d2/p/4534323.html2) HBaseHttp://www.cnblogs.com/super-d2/p/4755932.html3) Flume Installation:Installing and installing the JDKFlume operating system requires more than 1.6 of the Java operating environment, download the JDK installation package from the Oracle Web site, unzip the instal

Copyright notice: This article by Wang Liang original article, reprint please indicate source:Article original link: https://www.qcloud.com/community/article/214Source: Tengyun https://www.qcloud.com/communityPhenomenonThe long-running operation found that the disk full of the flume cluster was deployed and was found to be caused by the Flume log directory.Specific questionsSpecifically, Flume's large file