Use flume + kafka + storm to build a real-time log analysis system. Using flume + kafka + storm to build a real-time log analysis system this article only involves the combination of flume and kafka. for the combination of kafka and storm, refer to other blogs 1. install and download flume install and use
Flume is a distributed, reliable, and highly available system for collecting, aggregating, and transmitting large volumes of logs. Support for customizing various data senders in the log system for data collection, while Flume provides the ability to simply process the data and write to various data recipients (such as text, HDFS, hbase, etc.).First, what is Flume
Reprint: http://blog.csdn.net/liuxiao723846/article/details/78133375First, the scene of a description:The Online API interface service prints logs on the local disk via log4j, installs Flume on the interface server, collects logs through the exec source, and then sends the flume to the rollup server via Avro Sink, Flume through Avro on the rollup server Source re
Scenario 1. What is Flume 1.1 backgroundFlume, as a real-time log collection system developed by Cloudera, has been recognized and widely used by the industry. The initial release version of Flume is now collectively known as Flume OG (original Generation), which belongs to Cloudera. But with the expansion of the FLume
Questions Guide:1.flume-ng and Scribe, where is the advantage of Flume-ng?2. What issues should be considered in architecture design considerations?3.Agent How can I fix it?Does 4.Collector panic have an impact?What are the measures for 5.flume-ng reliability (reliability)?The U.S. mission's log collection system is responsible for the collection of all business
Copyright NOTICE: This article is Yunshuxueyuan original article.If you want to reprint please indicate the source: http://www.cnblogs.com/sxt-zkys/QQ Technology Group: 299142667
the concept of flume1. As a real-time log collection system developed by Flume, Cloudera has been recognized and widely used by the industry. The initial release version of Flume is now collectively known as
* Flume Framework FoundationIntroduction to the framework:* * Flume provides a distributed, reliable, and efficient collection, aggregation, and mobile service for large data volumes, flume can only be run in a UNIX environment.* * Flume is based on streaming architecture, fault-tolerant, and flexible and simple, mainl
START: Flume is a high-availability, highly reliable, open-source, distributed, high-volume log collection system provided by Cloudera, where log data can flow through flume to storage terminal destinations. The log here is a general term, refers to the file, Operation Records and many other data.First, flume basic Theory 1.1 Common distributed log Collection sys
1.flume conceptFlume is a distributed, reliable, highly available system for efficient collection, aggregation, and movement of large amounts of log data from different sources, and centralized data storage.Flume is currently a top-level project for Apache.Flume need Java running environment, require java1.6 above, recommended java1.7.Unzip the downloaded Flume installation package to the specified director
START: Flume is a high-availability, highly reliable, open-source, distributed, high-volume log collection system provided by Cloudera, where log data can flow through flume to storage terminal destinations. The log here is a general term, refers to the file, Operation Records and many other data.First, flume basic Theory 1.1 Common distributed log Collection sys
Installation and configuration of flumeFirst, Resources DownloadResource Address: http://flume.apache.org/download.htmlProgram Address: http://apache.fayea.com/flume/1.6.0/apache-flume-1.6.0-bin.tar.gzSource Address: http://mirrors.hust.edu.cn/apache/flume/1.6.0/apache-flume-1.6.0-src.tar.gzSecond, Installation and Con
Original link: Kafka combat-flume to KAFKA1. OverviewIn front of you to introduce the entire Kafka project development process, today to share Kafka how to get the data source, that is, Kafka production data. Here are the directories to share today:
Data sources
Flume to Kafka
Data source Loading
Preview
Let's start today's shared content.2. Data sourcesThe data produced by Kafka i
Example 1: Type Avro, create a avro.conf for testing in the Conf of Flume, as follows:A1.sources = R1A1.sinks = K1A1.channels = C1
# Describe/configure The sourceA1.sources.r1.type = AvroA1.sources.r1.channels = C1A1.sources.r1.bind = 0.0.0.0A1.sources.r1.port = 44444
# Describe The sinkA1.sinks.k1.type = Logger
# Use a channel which buffers events in memoryA1.channels.c1.type = Memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity =
This article describes the initial process of using flume to transfer data to MongoDB, covering environment deployment and considerations.1 Environment Constructionrequires JDK, flume-ng, MongoDB java driver, Flume-ng-mongodb-sink(1) jdk:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html(2) flune-ng:http://www.apache.org/dyn/close
Flume custom hbasesink class
Reference (to the original author) http://ydt619.blog.51cto.com/316163/1230586Https://blogs.apache.org/flume/entry/streaming_data_into_apache_hbaseSample configuration file of flume 1.5
# Name the components on this agenta1.sources = r1a1. sinks = k1a1. channels = c1 # Describe/configure the sourcea1.sources. r1.type = spooldira1.sour
Flume integrated Kafka:flume capture business log, sent to Kafka installation deployment KafkaDownload1.0.0 is the latest release. The current stable version was 1.0.0.You can verify your download by following these procedures and using these keys.1.0.0
Released November 1, 2017
Source download:kafka-1.0.0-src.tgz (ASC, SHA512)
Binary Downloads:
Scala 2.11-kafka_2.11-1.0.0.tgz (ASC, SHA512)
Scala 2.12-kafka_2
Flume, as a Log collection tool, exhibits a very powerful capability in data collection. Its source, SINK, channel three components of this mode, to complete the data reception, caching, sending this process, has a very perfect fit. But here, we want to say is not flume how good or flume have what merit, we want to talk about is
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.