flume samples

Flume is a highly available, highly reliable, distributed mass log capture, aggregation, and transmission system provided by Cloudera, Flume supports the customization of various data senders in the log system for data collection, while Flume provides simple processing of data The ability to write to various data-receiving parties (customizable). The current

1) hostname error: 2011-11-14 11:44:55,497 ERROR com.cloudera.util.NetUtils: Unable to get canonical host name! test: test java.net.UnknownHostException: test: test at java.net.InetAddress.getLocalHost(InetAddress.java:1354) at com.cloudera.util.NetUtils. Error cause: IP address cannot be obtained from hostname Solution: add the host name to IP address ing in the/etc/hosts file. 2) Java does not follow line 234: exec: java: not found Error cause: the Java command does not exist.

In the distributed system, each machine has the local log that the program runs, sometimes in order to analyze the demand, have to these scattered log summary requirements, I believe many people will choose RSYNC,SCP, but they are not strong in real-time, but also bring the problem of name conflict. The scalability is not satisfactory, not elegant at all.In reality, we are confronted with the need to summarize the Nginx logs of multiple servers on the line in real time.

In a complete large data processing system, in addition to the core of the Hdfs+mapreduce+hive composition Analysis system, data acquisition, result data export, task scheduling and other indispensable auxiliary systems are needed, and these auxiliary tools are There is a convenient open source framework in the Hadoop ecosystem. Log capture framework FlumeFlume is a distributed, reliable, and highly available system for collecting, aggregating, and transmitting large volumes of logs.

Download apache-flume-1.7.0-bin.tar.gz, withTar -zxvfUnzip, add the settings in the/etc/profile file:Export Flume_home=/opt/apache-flume-1.7.0-binexport path= $PATH: $FLUME _home/binModify the two files under $flume_home/conf/and increase the java_home in flume-env.sh:java_home=/opt/jdk1.8.0_121Most importantly, modify

{ //no event, that is Backoffresult =Status.backoff; } //Commit a transactionTransaction.commit (); } Catch(Exception ex) {//rolling back a transactionTransaction.rollback (); Throw NewEventdeliveryexception ("Failed to log event:" +event, ex); } finally { //Close TransactionTransaction.close (); } returnresult; } } 3. Pack and place in/soft/flume/Lib under4, using the custom s

1. Background introduction Many of the company's platforms generate a large number of logs per day (typically streaming data, for example, the search engine PV, query, etc.), the processing of these logs requires a specific log system, in general, these systems need to have the following characteristics: (1) The construction of application systems and analysis systems of the bridge, and the correlation between them decoupling (2) support for near real-time online analysis system and off-line ana

1. Background information Many of the company's platforms generate a large number of logs (typically streaming data, such as the PV of search engines, queries, etc.), which require a specific log system, which in general requires the following characteristics: (1) Construct the bridge of application system and analysis system, and decouple the correlation between them; (2) support the near real-time on-line analysis system and the off-line analysis system similar to Hadoop; (3) with high scalabi

1. overview-"three Functions of flume"collecting, aggregating, and movingCollect aggregation Moves2. Block diagram　　3. Architectural Features-"on Streaming Data flowsstreaming-based dataData flow: job-"get Data continuously"Task Flow: JOB1->JOB2->JOB3JOB4-"for Online analytic application.-"flume is only running in the Linux environmentWhat if my log server is windows?-"very SimpleWrite a configuration file,

This sections contain samples for the. NET Compact framework using Microsoft Visual Studio 2005. You can deploy the samples either to a Pocket PC or to a Pocket PC emulator.In this section Bubble technology sample Provides an application that demonstrates the drawing capability of the. NET Compact framework. Com callback sample Demonstrates how to call a managed object from

1. Haier TrainingNow, we use haartraining.exe to train our classifier. The training statement is as follows: Usage:./haartraining-Data -VEC -BG [-NPOs [-Nneg [-Nstages [-Nsplits [-MEM [-Sym (default)] [-nonsym][-Minhitrate [-Maxfalsealarm [-Weighttrimming [-Eqw][-Mode [-W [-H [-BT [-Err [-Maxtreesplits [-Minpos Kuranov et. al. pointed out that 20*20 samples have the highest accuracy rate. In addition, the four split nodes have the

Flume Knowledge Points:Event is a row of data1.flume is a distributed log collection system that transmits collected data to its destination.2.flume has a core concept, called an agent. The agent is a Java process that runs on the Log collection node.The 3.agent consists of 3 core components: source, channel, sink.The 3.1 source component is dedicated to collecti

Flume Introduction and use (i)Flume IntroductionFlume is a distributed, reliable, and practical service that efficiently collects, integrates, and moves massive amounts of data from different data sources.　　Distributed: Multiple machines can simultaneously run the acquisition data, different agents before the transmission of data over the networkReliable: Flume w