flume cd

First, the architecture scheme such as:Second, the installation of the various components of the program are as follows:1), Zookeeper+kafkaHttp://www.cnblogs.com/super-d2/p/4534323.html2) HBaseHttp://www.cnblogs.com/super-d2/p/4755932.html3) Flume Installation:Installing and installing the JDKFlume operating system requires more than 1.6 of the Java operating environment, download the JDK installation package from the Oracle Web site, unzip the instal

Copyright notice: This article by Wang Liang original article, reprint please indicate source:Article original link: https://www.qcloud.com/community/article/214Source: Tengyun https://www.qcloud.com/communityPhenomenonThe long-running operation found that the disk full of the flume cluster was deployed and was found to be caused by the Flume log directory.Specific questionsSpecifically, Flume's large file

apache Flume is a distributed, reliable, and efficient log data collection component ; we typically use flume to distribute log files scattered across multiple servers in the cluster into a central data platform to address the problem of" Viewing from discrete log files, statistical data. " Of course, flume not only collects log files, it also supports the colle

http://blog.csdn.net/hijk139/article/details/8308224Business systems need to collect monitoring system logs and think of the flume of Hadoop. After testing, although the function is not strong enough, but basically can meet the functional requirements. Flume is a distributed, reliable and highly available service Log Collection tool, capable of completing log collection, storage, analysis, and other tasks s

1 ... Cache file backlog occurs in the/flume/fchannel/spool/data/directoryPossible causes: same time the same client under the two monitoring directory MV file, or at the same time multiple clients to the server to upload files2. Clear: /flume/fchannel/spool/data/directory After the file restart, the monitoring directory file backlog, no uploadRepeat an exception inside the Flume.log:Java.lang.IllegalStateE

data loss. Try to use tail-F. Note that it is in uppercase; 2. About channel: 1. We recommend that you use the new composite spillablememorychannel for the collection node. We recommend that you use memory channel for the summary node, depending on the actual data volume, generally, memory channel is recommended for Flume agents whose data volume exceeds MB per minute (the file channel processing speed is about 2 m/s, which may vary with machin

How is the built-in monitoring of flume integrated? Many people have asked this question. Currently, you can use the cloudera manager and ganglia graphical monitoring tools to obtain JSON strings from the browser or customize the reports to other monitoring systems. What is the monitoring information? Is the statistical information of each component, such as the number of successfully received events, the number of successfully sent events, and the nu

Flume is a highly available, highly reliable, distributed mass log capture, aggregation, and transmission system provided by Cloudera, Flume supports the customization of various data senders in the log system for data collection, while Flume provides simple processing of data The ability to write to various data-receiving parties (customizable). Currently belong

Apache-flume Restart the script, Apache-flume Restart the regular start of multiple processes, kill not clean, write a restart script.#echo-e parameter output is red, online can search the shell output with color font encoding a lot.Catobi-track_restart.sh#!/bin/bashpid= ' lsof-i:8787|grepjava| awk ' {print$2} ' if[-n ' ${pid} ' ];thenecho-e ' ############\033[31m kill${pid}\033[0m############# "foriin" ${p

Log4j Direct output log to FlumeThis jar is a tool class provided by the CDH release of Cloudera, which can be configured to output log4j logs directly to the flume for easy log acquisition.In the CDH5.3.0 version, it is: Flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jarDirectory is:/opt/cloudera/parcels/cdh/lib/flume-ng/tools/Specific Use examplesl

1. Overview Flume is a high-performance, highly possible distributed log collection system for Cloudera company. The core of Flume is to collect data from the data source and send it to the destination. In order to ensure that the transmission must be successful, before sending to the destination, will first cache the data, waiting for the data to really arrive at the destination, delete their own cached da

avro_event or the FQCN of a implementation of EVENTSERIALIZER.B Uilder interface.BatchSize 100 Instance: A1.sources = R1 a1.sinks = K1 a1.channels = C1 A1.sources.r1.type = HTTP a1.sources.r1.port = 6666 a1.sources.r1.channels = C1 a1.channels.c1.type = Memory a1.channels.c1.capacity = a1.channels.c1.transactionCapacity = a1.sinks.k1.type = file_roll a1.sinks.k1.sink.directory =/home/park/work/ Apache-flume-1.6.0-bin/mysink A1.sink

!! "); url = arg0.getstring ("url"); Preconditions.checknotnull (URL, "URL must be set!!"); user = arg0.getstring ("user"); Preconditions.checknotnull (user, "User must be set!!"); Password = arg0.getstring ("password"); Preconditions.checknotnull (password, "Password must be set!!"); TableName = arg0.getstring ("TableName"); Preconditions.checknotnull (TableName, "TableName must be set!!");}}agent.sources = S1 agent.channels = C1 agent.sinks

Todo:The sink of Flume is reconstructed, and the consumer producer (producer) of Kafka is called to send the message;Inherit the Irichspout interface in SOTRM's spout, call Kafka's message consumer (Consumer) to receive the message, and then go through several custom bolts to output the custom contentWriting KafkasinkCopy from $kafka_home/libKafka_2.10-0.8.2.1.jarKafka-clients-0.8.2.1.jarScala-library-2.10.4.jarTo $flume_home/libNew project in Eclipse

Structure:Nginx-flume->kafka->flume->kafka (because involved in the cross-room problem, between the two Kafka added a flume, egg pain. )Phenomenon:In the second layer, write Kafka topic and read Kafka topic same, manually set sink topic does not take effectTo open the debug log:SOURCE instantiation:APR 19:24:03,146 INFO [conf-file-poller-0] (org.apache.flume.sour

Flume write HDFs operation in the Hdfseventsink.process method, the path creation is done by BucketpathAnalyze its source code (ref.: http://caiguangguang.blog.51cto.com/1652935/1619539)Can be implemented using%{} variable substitution, only need to get the time field in the event (the Nginx log of the local times) incoming Hdfs.path can beThe specific implementation is as follows:1. In the Kafkasource process method, add:DT = Kafkasourceutil.getdatem

Transferred from: http://www.cnblogs.com/lxf20061900/p/4014281.htmlThe pathname of the HDFs sink in Flume-ng (the corresponding parameter "Hdfs.path", which is not allowed to be empty) and the file prefix (corresponding to the parameter "Hdfs.fileprefix") support the regular parsing timestamp to automatically create the directory and file prefix by time.In practice, it is found that the flume built-in parsi

Flume Architecture and Core components（1）Source 收集 负责从什么地方采集数据（2）Channel 记录 （3）Sink 输出Official documentsHttp://flume.apache.org/FlumeUserGuide.htmlHttp://flume.apache.org/FlumeUserGuide.html#starting-an-agentFlume Use IdeasThe key to using Flume is to write the configuration file (1) Configuring the source (2) Configuration Channerl (3) configuration sink (4) string The above three comp

Capture Directory to HDFsUsing flume to capture a directory requires an HDFS cluster to be startedVI spool-hdfs.conf# Name the components on Thisagenta1.sources=r1a1.sinks=K1a1.channels=c1# Describe/Configure the source# #注意: You can not repeat the same name in the monitoring target file A1.sources.r1.type=Spooldira1.sources.r1.spoolDir=/root/Logs2a1.sources.r1.fileHeader=true# Describe The Sinka1.sinks.k1.type=Hdfsa1.sinks.k1.channel=C1a1.sinks.k1.hd

Label:Flume is a highly available, highly reliable, distributed mass log collection, aggregation and transmission system. You can look at the model: Each flume agent can provide a flume service. Each agent has three members: source, channel, sink As shown, fetching data from source and sending it to Channel,channel is like a buffer, from which sink reads data from the channel. --------------------------