flume app

Reprint: http://blog.csdn.net/liuxiao723846/article/details/78133375First, the scene of a description:The Online API interface service prints logs on the local disk via log4j, installs Flume on the interface server, collects logs through the exec source, and then sends the flume to the rollup server via Avro Sink, Flume through Avro on the rollup server Source re

Scenario 1. What is Flume 1.1 backgroundFlume, as a real-time log collection system developed by Cloudera, has been recognized and widely used by the industry. The initial release version of Flume is now collectively known as Flume OG (original Generation), which belongs to Cloudera. But with the expansion of the FLume

Questions Guide:1.flume-ng and Scribe, where is the advantage of Flume-ng?2. What issues should be considered in architecture design considerations?3.Agent How can I fix it?Does 4.Collector panic have an impact?What are the measures for 5.flume-ng reliability (reliability)?The U.S. mission's log collection system is responsible for the collection of all business

Copyright NOTICE: This article is Yunshuxueyuan original article.If you want to reprint please indicate the source: http://www.cnblogs.com/sxt-zkys/QQ Technology Group: 299142667 the concept of flume1. As a real-time log collection system developed by Flume, Cloudera has been recognized and widely used by the industry. The initial release version of Flume is now collectively known as

* Flume Framework FoundationIntroduction to the framework:* * Flume provides a distributed, reliable, and efficient collection, aggregation, and mobile service for large data volumes, flume can only be run in a UNIX environment.* * Flume is based on streaming architecture, fault-tolerant, and flexible and simple, mainl

START: Flume is a high-availability, highly reliable, open-source, distributed, high-volume log collection system provided by Cloudera, where log data can flow through flume to storage terminal destinations. The log here is a general term, refers to the file, Operation Records and many other data.First, flume basic Theory 1.1 Common distributed log Collection sys

1.flume conceptFlume is a distributed, reliable, highly available system for efficient collection, aggregation, and movement of large amounts of log data from different sources, and centralized data storage.Flume is currently a top-level project for Apache.Flume need Java running environment, require java1.6 above, recommended java1.7.Unzip the downloaded Flume installation package to the specified director

START: Flume is a high-availability, highly reliable, open-source, distributed, high-volume log collection system provided by Cloudera, where log data can flow through flume to storage terminal destinations. The log here is a general term, refers to the file, Operation Records and many other data.First, flume basic Theory 1.1 Common distributed log Collection sys

Installation and configuration of flumeFirst, Resources DownloadResource Address: http://flume.apache.org/download.htmlProgram Address: http://apache.fayea.com/flume/1.6.0/apache-flume-1.6.0-bin.tar.gzSource Address: http://mirrors.hust.edu.cn/apache/flume/1.6.0/apache-flume-1.6.0-src.tar.gzSecond, Installation and Con

Original link: Kafka combat-flume to KAFKA1. OverviewIn front of you to introduce the entire Kafka project development process, today to share Kafka how to get the data source, that is, Kafka production data. Here are the directories to share today: Data sources Flume to Kafka Data source Loading Preview Let's start today's shared content.2. Data sourcesThe data produced by Kafka i

Flume Installation ConfigurationOne: Download: http://www.apache.org/dyn/closer.lua/flume/1.8.0/apache-flume-1.8.0-bin.tar.gzTwo: Unzip[[Email protected] ~] # TAR-ZXVF apache-flume-1.8.0-bin.tar.gz-c/usr/local/Renamed Flume convenient for later operation[[email protected] lo

Example 1: Type Avro, create a avro.conf for testing in the Conf of Flume, as follows:A1.sources = R1A1.sinks = K1A1.channels = C1 # Describe/configure The sourceA1.sources.r1.type = AvroA1.sources.r1.channels = C1A1.sources.r1.bind = 0.0.0.0A1.sources.r1.port = 44444 # Describe The sinkA1.sinks.k1.type = Logger # Use a channel which buffers events in memoryA1.channels.c1.type = Memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity =

This article describes the initial process of using flume to transfer data to MongoDB, covering environment deployment and considerations.1 Environment Constructionrequires JDK, flume-ng, MongoDB java driver, Flume-ng-mongodb-sink(1) jdk:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html(2) flune-ng:http://www.apache.org/dyn/close

Flume custom hbasesink class Reference (to the original author) http://ydt619.blog.51cto.com/316163/1230586Https://blogs.apache.org/flume/entry/streaming_data_into_apache_hbaseSample configuration file of flume 1.5 # Name the components on this agenta1.sources = r1a1. sinks = k1a1. channels = c1 # Describe/configure the sourcea1.sources. r1.type = spooldira1.sour

Flume integrated Kafka:flume capture business log, sent to Kafka installation deployment KafkaDownload1.0.0 is the latest release. The current stable version was 1.0.0.You can verify your download by following these procedures and using these keys.1.0.0 Released November 1, 2017 Source download:kafka-1.0.0-src.tgz (ASC, SHA512) Binary Downloads: Scala 2.11-kafka_2.11-1.0.0.tgz (ASC, SHA512) Scala 2.12-kafka_2

Machine list:192.168.137.115 slave0 (Agent) 192.168.137.116 slave1 (agent) 192.168.137.117 slave2 (agent) 192.168.137.11 8 Slave3 (collector) 192.168.137.119 Slave4 (collector)Create a directory on each machineMkdir-p/home/qun/data/flume/logsMkdir-p/home/qun/data/flume/dataMkdir-p/home/qun/data/flume/checkpointDownload the latest

Flume IntroductionFlume installation 1. Unzip the flume installation package into the/itcast/directoryTAR-ZXVF/*flume Installation package *//itcast/2. Modify the Flume configuration file: 2.1 flume-env.shModify file Name:MV Flume

Flume, as a Log collection tool, exhibits a very powerful capability in data collection. Its source, SINK, channel three components of this mode, to complete the data reception, caching, sending this process, has a very perfect fit. But here, we want to say is not flume how good or flume have what merit, we want to talk about is