flume hadoop

Copyright notice: This article by Wang Liang original article, reprint please indicate source:Article original link: https://www.qcloud.com/community/article/214Source: Tengyun https://www.qcloud.com/communityPhenomenonThe long-running operation found that the disk full of the flume cluster was deployed and was found to be caused by the Flume log directory.Specific questionsSpecifically, Flume's large file

apache Flume is a distributed, reliable, and efficient log data collection component ; we typically use flume to distribute log files scattered across multiple servers in the cluster into a central data platform to address the problem of" Viewing from discrete log files, statistical data. " Of course, flume not only collects log files, it also supports the colle

I. Overview1. By building a highly available flume for data collection and storage on HDFs, the frame is composed as follows:650) this.width=650; "src=" Https://s5.51cto.com/wyfs02/M01/05/CC/wKiom1msukvhD4OfAACMzR0FBDM139.png "title=" 301254248495863 (1). png "alt=" Wkiom1msukvhd4ofaacmzr0fbdm139.png "/>Second, the configuration agent1.cat flume-client.properties#name thecomponentsonthisagent Declare the na

Wang Jialin's in-depth case-driven practice of cloud computing distributed Big Data hadoop in July 6-7 in Shanghai Wang Jialin Lecture 4HadoopGraphic and text training course: Build a true practiceHadoopDistributed Cluster EnvironmentHadoopThe specific solution steps are as follows: Step 1: QueryHadoopTo see the cause of the error; Step 2: Stop the cluster; Step 3: Solve the Problem Based on the reasons indicated in the log. We need to clear th

1 ... Cache file backlog occurs in the/flume/fchannel/spool/data/directoryPossible causes: same time the same client under the two monitoring directory MV file, or at the same time multiple clients to the server to upload files2. Clear: /flume/fchannel/spool/data/directory After the file restart, the monitoring directory file backlog, no uploadRepeat an exception inside the Flume.log:Java.lang.IllegalStateE

data loss. Try to use tail-F. Note that it is in uppercase; 2. About channel: 1. We recommend that you use the new composite spillablememorychannel for the collection node. We recommend that you use memory channel for the summary node, depending on the actual data volume, generally, memory channel is recommended for Flume agents whose data volume exceeds MB per minute (the file channel processing speed is about 2 m/s, which may vary with machin

How is the built-in monitoring of flume integrated? Many people have asked this question. Currently, you can use the cloudera manager and ganglia graphical monitoring tools to obtain JSON strings from the browser or customize the reports to other monitoring systems. What is the monitoring information? Is the statistical information of each component, such as the number of successfully received events, the number of successfully sent events, and the nu

Flume is a highly available, highly reliable, distributed mass log capture, aggregation, and transmission system provided by Cloudera, Flume supports the customization of various data senders in the log system for data collection, while Flume provides simple processing of data The ability to write to various data-receiving parties (customizable). Currently belong

Apache-flume Restart the script, Apache-flume Restart the regular start of multiple processes, kill not clean, write a restart script.#echo-e parameter output is red, online can search the shell output with color font encoding a lot.Catobi-track_restart.sh#!/bin/bashpid= ' lsof-i:8787|grepjava| awk ' {print$2} ' if[-n ' ${pid} ' ];thenecho-e ' ############\033[31m kill${pid}\033[0m############# "foriin" ${p

Log4j Direct output log to FlumeThis jar is a tool class provided by the CDH release of Cloudera, which can be configured to output log4j logs directly to the flume for easy log acquisition.In the CDH5.3.0 version, it is: Flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jarDirectory is:/opt/cloudera/parcels/cdh/lib/flume-ng/tools/Specific Use examplesl

First, the demand Use flume to capture the file information under Linux and pass it into the Kafka cluster. Environment ready Zookeeper cluster and Kafka cluster are installed well. Second, the configuration flume Download Flume website. The blogger himself is using flume1.6.0. Official Address http://flume.apache.org/download.html

win7+ubuntu16.04+flume1.8.01. Download apache-flume-1.8.0-bin.tar.gzHttp://flume.apache.org/download.html2. Unzip into the/usr/local/flume3. Set the configuration file/etc/profile file to increase the path of flume①vi/etc/profileExport flume_home=/usr/local/flumeexport PATH= $PATH: $FLUME _home/bin② make the configuration file effective immediatelySource/etc/prof

Original link: Http://www.tuicool.com/articles/Z73UZf6 The data collected on the HADOOP2 and HADOOP3 are sent to the HADOOP1 cluster and HADOOP1 to a number of different purposes. I. Overview 1, now there are three machines, respectively: HADOOP1,HADOOP2,HADOOP3, HADOOP1 for the log summary 2, HADOOP1 Summary of the simultaneous output to multiple targets 3, flume a data source corresponding to multiple channel, multiple sink, is configured in th

Todo:The sink of Flume is reconstructed, and the consumer producer (producer) of Kafka is called to send the message;Inherit the Irichspout interface in SOTRM's spout, call Kafka's message consumer (Consumer) to receive the message, and then go through several custom bolts to output the custom contentWriting KafkasinkCopy from $kafka_home/libKafka_2.10-0.8.2.1.jarKafka-clients-0.8.2.1.jarScala-library-2.10.4.jarTo $flume_home/libNew project in Eclipse

Structure:Nginx-flume->kafka->flume->kafka (because involved in the cross-room problem, between the two Kafka added a flume, egg pain. )Phenomenon:In the second layer, write Kafka topic and read Kafka topic same, manually set sink topic does not take effectTo open the debug log:SOURCE instantiation:APR 19:24:03,146 INFO [conf-file-poller-0] (org.apache.flume.sour

Flume write HDFs operation in the Hdfseventsink.process method, the path creation is done by BucketpathAnalyze its source code (ref.: http://caiguangguang.blog.51cto.com/1652935/1619539)Can be implemented using%{} variable substitution, only need to get the time field in the event (the Nginx log of the local times) incoming Hdfs.path can beThe specific implementation is as follows:1. In the Kafkasource process method, add:DT = Kafkasourceutil.getdatem

Transferred from: http://www.cnblogs.com/lxf20061900/p/4014281.htmlThe pathname of the HDFs sink in Flume-ng (the corresponding parameter "Hdfs.path", which is not allowed to be empty) and the file prefix (corresponding to the parameter "Hdfs.fileprefix") support the regular parsing timestamp to automatically create the directory and file prefix by time.In practice, it is found that the flume built-in parsi

Capture Directory to HDFsUsing flume to capture a directory requires an HDFS cluster to be startedVI spool-hdfs.conf# Name the components on Thisagenta1.sources=r1a1.sinks=K1a1.channels=c1# Describe/Configure the source# #注意: You can not repeat the same name in the monitoring target file A1.sources.r1.type=Spooldira1.sources.r1.spoolDir=/root/Logs2a1.sources.r1.fileHeader=true# Describe The Sinka1.sinks.k1.type=Hdfsa1.sinks.k1.channel=C1a1.sinks.k1.hd

[Hadoop] how to install Hadoop and install hadoop Hadoop is a distributed system infrastructure that allows users to develop distributed programs without understanding the details of the distributed underlying layer. Important core of Hadoop: HDFS and MapReduce. HDFS is res

Recently, flume is used for data collection. The spooldir source has the following problems: If a line of the file contains garbled characters and does not comply with the specified encoding specification, flume throws an exception and stops there. Once the files in the folder specified by spooldir are modified, flume throws an exception and stops there. In f