how to upload on flume

Flume is a highly available, highly reliable, distributed mass log capture, aggregation, and transmission system provided by Cloudera, Flume supports the customization of various data senders in the log system for data collection, while Flume provides simple processing of data The ability to write to various data-receiving parties (customizable). The current

1) hostname error: 2011-11-14 11:44:55,497 ERROR com.cloudera.util.NetUtils: Unable to get canonical host name! test: test java.net.UnknownHostException: test: test at java.net.InetAddress.getLocalHost(InetAddress.java:1354) at com.cloudera.util.NetUtils. Error cause: IP address cannot be obtained from hostname Solution: add the host name to IP address ing in the/etc/hosts file. 2) Java does not follow line 234: exec: java: not found Error cause: the Java command does not exist.

In the distributed system, each machine has the local log that the program runs, sometimes in order to analyze the demand, have to these scattered log summary requirements, I believe many people will choose RSYNC,SCP, but they are not strong in real-time, but also bring the problem of name conflict. The scalability is not satisfactory, not elegant at all.In reality, we are confronted with the need to summarize the Nginx logs of multiple servers on the line in real time.

In a complete large data processing system, in addition to the core of the Hdfs+mapreduce+hive composition Analysis system, data acquisition, result data export, task scheduling and other indispensable auxiliary systems are needed, and these auxiliary tools are There is a convenient open source framework in the Hadoop ecosystem. Log capture framework FlumeFlume is a distributed, reliable, and highly available system for collecting, aggregating, and transmitting large volumes of logs.

Download apache-flume-1.7.0-bin.tar.gz, withTar -zxvfUnzip, add the settings in the/etc/profile file:Export Flume_home=/opt/apache-flume-1.7.0-binexport path= $PATH: $FLUME _home/binModify the two files under $flume_home/conf/and increase the java_home in flume-env.sh:java_home=/opt/jdk1.8.0_121Most importantly, modify

{ //no event, that is Backoffresult =Status.backoff; } //Commit a transactionTransaction.commit (); } Catch(Exception ex) {//rolling back a transactionTransaction.rollback (); Throw NewEventdeliveryexception ("Failed to log event:" +event, ex); } finally { //Close TransactionTransaction.close (); } returnresult; } } 3. Pack and place in/soft/flume/Lib under4, using the custom s

1. Background introduction Many of the company's platforms generate a large number of logs per day (typically streaming data, for example, the search engine PV, query, etc.), the processing of these logs requires a specific log system, in general, these systems need to have the following characteristics: (1) The construction of application systems and analysis systems of the bridge, and the correlation between them decoupling (2) support for near real-time online analysis system and off-line ana

1. Background information Many of the company's platforms generate a large number of logs (typically streaming data, such as the PV of search engines, queries, etc.), which require a specific log system, which in general requires the following characteristics: (1) Construct the bridge of application system and analysis system, and decouple the correlation between them; (2) support the near real-time on-line analysis system and the off-line analysis system similar to Hadoop; (3) with high scalabi

This article introduces flume data insert hdfs and common directory (), this article continues to introduce flume-ng to insert data into the hbase-0.96.0. First, modify the flume-node.conf file in the conf directory under the flume folder in node (for the original configuration, refer to the above) and make the followi

#!/bin/bash# Author:xirong # Date: -- Geneva- .##### Build a flume cluster script # Note: #1JDK7 Environment is required, if there is no Java environment, please configure #2. Have/home/Work directory, otherwise unable to install ###### compressed file unzip tar-ZXF apache-flume-1.5.2-bin.tar.gz-c/home/work/flume_cluster/# Configure Flume environment echo'# #

Unify the time before building, turn off the firewall, use the jar package version is 1.6.0There are two ways to configure a serviceThe first type: The following steps:1. Pass the jar package to the Node1 and extract it to the root directory2. Change the directory name by using the following command: MV apache-flume-1.6.0-bin/home/install/flume-1.63. After entering the