Scala + thrift+ Zookeeper+flume+kafka Configuration notes

1. Development environment 1.1. Package Download 1.1.1. JDK

Http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Install to the D:\GreenSoftware\Java\Java8X64\jdk1.8.0_91 directory 1.1.2. Maven

https://maven.apache.org/download.cgi

Unzip to the D:\GreenSoftware\apache-maven-3.3.9 directory 1.1.3. Scala

https://www.scala-lang.org/download/

Unzip to the D:\GreenSoftware\Java\scala-2.12.6 directory 1.1.4. Thrift

Http://thrift.apache.org/download

Place the downloaded Thrift-0.11.0.exe file in the D:\Project\ServiceMiddleWare\thrift directory and rename it to thrift.exe1.1.5. Zookeeper

https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/current/

Unzip to the D:\Project\ServiceMiddleWare\zookeeper-3.4.10 directory 1.1.6. Flume

Http://flume.apache.org/download.html

Unzip to the D:\Project\ServiceMiddleWare\flume-1.8.0 directory 1.1.7. Kafka

Http://kafka.apache.org/downloads

Unzip to D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0 directory 1.2. jdk+maven+scala+thrift1.2.1. Environment variables

Java_home D:\GreenSoftware\Java\Java8X64\jdk1.8.0_91

CLASSPATH.; %java_home%\lib\dt.jar;%java_home%\lib\tools.jar

Maven_home D:\GreenSoftware\apache-maven-3.3.9

Scala_home D:\GreenSoftware\Java\scala-2.12.6

Add in PATH

%java_home%\bin;

%maven_home%\bin;

%scala_home%\bin;

d:\project\servicemiddleware\thrift;1.3. Install, configure, and verify the 1.3.1. Jdk

c:\users\zyx>java-version

Java Version "1.8.0_91"

Java (TM) SE Runtime Environment (build 1.8.0_91-b14)

Java HotSpot (TM) 64-bit Server VM (build 25.91-b14, Mixed mode)

c:\users\zyx>1.3.2. Maven

c:\users\zyx>mvn-version

Apache Maven 3.3.9 (BB52D8502B132EC0A5A3F4C09453C07478323DC5; 2015-11-11t00:41:47+08:00)

Maven home:d:\greensoftware\apache-maven-3.3.9\bin\.

Java version:1.8.0_91, Vendor:oracle Corporation

Java Home:d:\greensoftware\java\java8x64\jdk1.8.0_91\jre

Default LOCALE:ZH_CN, Platform ENCODING:GBK

OS name: "Windows Ten", Version: "10.0", Arch: "AMD64", Family: "DOS" 1.3.3. Scala

c:\users\zyx>scala-version

Scala Code Runner version 2.12.6-Copyright 2002-2018, LAMP/EPFL and Lightbend, Inc.

C:\users\zyx>Scala

Welcome to Scala 2.12.6 (Java HotSpot (TM) 64-bit Server VM, Java 1.8.0_91).

Type in expressions for evaluation. Or Try:help.

Scala> : Quit

c:\users\zyx>1.3.4. Thrift

c:\users\zyx>thrift-version

Thrift version 0.11.01.3.5. Zookeeper1.3.5.1. Configuration

In the D:\Project\ServiceMiddleWare\zookeeper-3.4.10\conf directory, create a zoo.cfg file that reads as follows

ticktime=2000

datadir=d:/project/servicemiddleware/zookeeper-3.4.10/data/db

Datalogdir=d:/project/servicemiddleware/zookeeper-3.4.10/data/log

clientport=2181

# Zookeeper Cluster

# server.1=127.0.0.1:12888:1388

# server.2=127.0.0.1:12889:1389

# server.3=127.0.0.1:12887:13871.3.5.2. Start Zookeeper

D:\Project\ServiceMiddleWare\zookeeper-3.4.10\bin\zkServer.cmd1.3.5.3. Start the client

D:\Project\ServiceMiddleWare\zookeeper-3.4.10\bin\zkCli.cmd-server 127.0.0.1:21811.3.5.4. Basic commands

View Catalog

ls/1.3.5.5. Create a Node

Create/config 01.3.5.6. Delete a node

delete/config1.3.5.7. Exit Client

quit1.3.6. Kafka 1.3.6.1. Configuring the configuration file for 1.3.6.1.1. Kafka

Modify the Server.properties file in the D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\config directory as follows:

Broker.id=0

port=9092

host.name=127.0.0.1

# listeners=plaintext://127.0.0.1:9092

# Register Zookeeper ' s node data

# advertised.listeners=plaintext://127.0.0.1:9092

# Log.dirs=/tmp/kafka-logs

Log.dirs=d:/project/servicemiddleware/kafka_2.12-1.1.0/data/log

Log.dir = D:/project/servicemiddleware/kafka_2.12-1.1.0/data/log

# zookeeper.connect=localhost:2181

zookeeper.connect=127.0.0.1:2181

# Zookeeper Cluster

# zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:21831.3.6.1.2. Configuration of the Log

Modify the Log4j.properties file in the D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\config directory as follows:

Kafka.logs.dir=d:/project/servicemiddleware/kafka_2.12-1.1.0/data/log

Log.dir =/tmp/kafka-logs1.3.6.1.3. Port configuration (required if multiple Kafka are configured on the same server)

To modify the Kafka-run-class.bat file in the D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\bin\windows directory, add the following at the beginning of the file

Set jmx_port=19093

Set Java_debug_port =50051.3.6.2. Start Kafka

D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\bin\windows\kafka-server-start.bat D:\Project\ServiceMiddleWare\ kafka_2.12-1.1.0\config\server.properties1.3.6.3. Create a Theme (Topic)

D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\bin\windows\kafka-topics.bat--create--zookeeper 127.0.0.1:2181-- Replication-factor 1--partitions 1--topic testtopic

1.3.6.4. Creating a producer (Producer)

D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\bin\windows\kafka-console-producer.bat--broker-list 127.0.0.1:9092--topic testTopic1.3.6.5. Create consumer (Consumer)

D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\bin\windows\kafka-console-consumer.bat--zookeeper 127.0.0.1:2181- -topic testTopic1.3.6.6. The generator sends a message

Hello World

How are you doing

1.3.6.7. The consumer receives the message

1.3.6.8. Viewing the topic (TOPIC) status

D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\bin\windows\kafka-topics.bat--describe--zookeeper 127.0.0.1:2181- -topic Testtopic

1.3.6.9. Completely Remove topic

616155541.3.7. Flume1.3.7.1. Configuring the configuration file for 1.3.7.1.1. Flume

Under the D:\Project\ServiceMiddleWare\flume-1.8.0\conf directory, create a flume.agent1.conf file that reads as follows:

# flume.agent1.conf:a Single-node Flume Listen netcat configuration

# Name The components in this agent

Agent1.sources = Sources1

Agent1.sinks = Sinks1

Agent1.channels = CHANNELS1

# Describe/configure The source

# Agent1.sources.sources1.type = Avro, Exec, HTTP, JMS, Netcat, Sequence generator, spooling directory, Syslog, Thrift, Tw Itter

Agent1.sources.sources1.type = Netcat

Agent1.sources.sources1.bind = 127.0.0.1

Agent1.sources.sources1.port = 44444

Agent1.sources.source1.interceptors = Interceptor1

# Agent1.sources.source1.interceptors.interceptor1.type = Host, morphline, regex extractor, regex filtering, Static, Timestamp, UUID

Agent1.sources.source1.interceptors.interceptor1.type = Timestamp

# Describe The sink

# Agent1.sinks.sinks1.type = Avro, Elasticsearch, File roll, HBase, HDFS, IRC, Logger, Morphline (Solor), Null, Thrift

Agent1.sinks.sinks1.type = Logger

# Use a channel which buffers events in memory

# Agent1.channels.channels1.type = File, JDBC, Memory

Agent1.channels.channels1.type = Memory

agent1.channels.channels1.capacity = 1000

agent1.channels.channels1.transactionCapacity = 100

# Bind the source and sink to the channel

Agent1.sources.sources1.channels = CHANNELS1

Agent1.sinks.sinks1.channel = channels11.3.7.1.2. Configuration of the Log

Modify the Log4j.properties file in the D:\Project\ServiceMiddleWare\flume-1.8.0\conf directory as follows:

# Flume.log.dir=./logs

flume.log.dir=d:/project/servicemiddleware/flume-1.8.0/data/log1.3.7.2. Start flume

D:\Project\ServiceMiddleWare\flume-1.8.0\bin\flume-ng.cmd agent-conf d:/project/servicemiddleware/flume-1.8.0/ Conf-conf-file D:/project/servicemiddleware/flume-1.8.0/conf/flume.agent1.conf-name Agent1-property Flume.root.logger=info,console

Or

CD D:\Project\ServiceMiddleWare\flume-1.8.0\bin

Flume-ng.cmd agent-conf. /conf-conf-file. /conf/flume.agent1.conf-name Agent1-property flume.root.logger=info,console1.3.7.3. Access to 44444 ports, analog data input

Telnet 127.0.0.1 44444

then enter any Content

View Flume Receive results:

1.3.8. Flume and Kafka connect 1.3.8.1. Configuration of the Flume

In the D:\Project\ServiceMiddleWare\flume-1.8.0\conf directory, modify the flume.agent1.conf file as follows:

# flume.agent1.conf:a Single-node Flume Listen netcat configuration

# Name The components in this agent

Agent1.sources = Sources1

Agent1.sinks = Sinks1

Agent1.channels = CHANNELS1

# Describe/configure The source

# Agent1.sources.sources1.type = Avro, Exec, HTTP, JMS, Netcat, Sequence generator, spooling directory, Syslog, Thrift, Tw Itter

Agent1.sources.sources1.type = Netcat

Agent1.sources.sources1.bind = 127.0.0.1

Agent1.sources.sources1.port = 44444

Agent1.sources.source1.interceptors = Interceptor1

# Agent1.sources.source1.interceptors.interceptor1.type = Host, morphline, regex extractor, regex filtering, Static, Timestamp, UUID

Agent1.sources.source1.interceptors.interceptor1.type = Timestamp

# Describe The sink

# Agent1.sinks.sinks1.type = Avro, Elasticsearch, File roll, HBase, HDFS, IRC, Logger, Morphline (Solor), Null, Thrift

# Agent1.sinks.sinks1.type = Logger

Agent1.sinks.sinks1.type = Org.apache.flume.sink.kafka.KafkaSink

Agent1.sinks.sinks1.kafka.topic = Testtopic

Agent1.sinks.sinks1.kafka.bootstrap.servers = 127.0.0.1:9092

Agent1.sinks.sinks1.kafka.flumeBatchSize = 20

Agent1.sinks.sinks1.kafka.producer.acks = 1

agent1.sinks.sinks1.kafka.producer.linger.ms = 1

Agent1.sinks.sinks1.kafka.producer.compression.type = Snappy

# Use a channel which buffers events in memory

# Agent1.channels.channels1.type = File, JDBC, Memory

Agent1.channels.channels1.type = Memory

agent1.channels.channels1.capacity = 1000

agent1.channels.channels1.transactionCapacity = 100

# Bind the source and sink to the channel

Agent1.sources.sources1.channels = CHANNELS1

Agent1.sinks.sinks1.channel = channels11.3.8.2. Start Zookeeper

D:\Project\ServiceMiddleWare\zookeeper-3.4.10\bin\zkServer.cmd

1.3.8.3. Starting Kafka

D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\bin\windows\kafka-server-start.bat D:\Project\ServiceMiddleWare\ Kafka_2.12-1.1.0\config\server.properties

1.3.8.4. Creating a consumer (Consumer)

D:\Project\ServiceMiddleWare\kafka_2.12-1.1.0\bin\windows\kafka-console-consumer.bat--zookeeper 127.0.0.1:2181- -topic Testtopic

1.3.8.5. Starting Flume

D:\Project\ServiceMiddleWare\flume-1.8.0\bin\flume-ng.cmd agent-conf d:/project/servicemiddleware/flume-1.8.0/ Conf-conf-file D:/project/servicemiddleware/flume-1.8.0/conf/flume.agent1.conf-name Agent1-property Flume.root.logger=info,console

1.3.8.6. Start Flume data producer (access 44444 port, analog data input)

Telnet 127.0.0.1 44444

then enter any Content

1.3.8.7. Viewing the results of messages received by consumers in Kafka

Scala + thrift+ Zookeeper+flume+kafka Configuration notes