Architecture introduction and installation of Kafka Series 1

Introduction and installation of Kafka Architecture

Preface
Or, before you learn a new thing, you must know what it is? What can this thing be used? Then you will learn and use it. To put it simply, Kafka is a message queue and now it has evolved into a distributed stream processing platform, which is amazing. Therefore, learning Kafka is very beneficial for Big Data friends.
(1) Overview of Kafka
1. kafka is a distributed stream processing platform with a message subscription system (after subscription, once a new message is generated, you will receive the message ), data can be stored in multiple copies, real-time data pipelines can be built, and real-time stream processing can be performed, featuring horizontal scaling and fault tolerance. Kafka is often used for real-time stream processing of big data.

The above are images on the official website.
Kafka is similar to the message system. message-oriented middleware has the concept of producer and consumer.
(2) Kafka Architecture

From the pictures on the official website above, we know that Kafka has the following main
1) Producer: producer,

2) Consumer: consumer,

3) Broker: a broker is a Kafka,

4) topic: a topic that adds a tag to the message to be produced, representing who produced the message.

First a few concepts: [several concepts]
Kafka is run as a cluster on one or more servers.

The Kafka cluster stores streams of records in categories called topics.

Each record consists of a key, a value, and a timestamp.

(3) installation and deployment of Kafka
We first go to the official website, http://kafka.apache.org/quickstart
From this quick start, we know that our Kafka depends on zookeeper, so we will first install zookpeer and then install Kafka

Zookeeper Installation

1. Download zookeeper at the http://archive.cloudera.com/cdh5/cdh/5/ on the cdh5 page
2. Decompress the package to a specified directory ~ /APP. This is a good habit. Here I change the name of the decompressed file to ZK,
3. Export the decompressed directory,

[[Email protected] ZK] $ Vim ~ /. Bash_profile #. bash_profile # Get the aliases and functionsif [-f ~ /. Bashrc]; then .~ /. Bashrcfi # user specific environment and startup programsexport java_home =/home/hadoop/APP/jdk1.8.0 _ 144 export Path = $ java_home/bin: $ pathexport flume_home =/home/hadoop/APP/flumeexport Path = $ flume_home/bin: $ pathexport zk_home =/home/hadoop/APP/zkexport Path = $ zk_home/bin: $ path #: WQ save and exit

4. Another source system environment

[[email protected] zk]$ source ~/.bash_profile [[email protected] zk]$ 

5. Configure zookeeper to decompress the conf file in the directory.
Copy one copy

[[Email protected] conf] $ CP zoo_sample.cfg Zoo. CFG [[email protected] conf] $ Vim Zoo. CFG # modify the temporary data storage directory. By default, datadir =/home/hadoop/APP/tmp/ZK under the tmp directory #: WQ save and exit

6. Start zookeeper

[[email protected] conf]$ cd ../bin/[[email protected] bin]$ zkServer.sh startJMX enabled by defaultUsing config: /home/hadoop/app/zk/bin/../conf/zoo.cfgStarting zookeeper ... already running as process 10943.[[email protected] bin]$ 

7. View Processes

[[Email protected] bin] $ jps27179 jps1_43 quorumpeermain [[email protected] bin] $ # When quorumpeermain is displayed, it indicates that the startup is successful.

Kafka installation and deployment

According to the Guide on the official website, we first download Kafka and click downloadhttp: // kafka.apache.org/downloads on the official website. When selecting a version, we recommend that you use version 0.9.0.0, currently, this version is commonly used in production,

1. Download https://archive.apache.org/dist/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz because Kafka is written in Scala language, we may also use Scala later, because Scala I installed is version 2.11. This version is mainly based on your own situation. You can select a suitable version on the official website.
2. Decompress the package to the specified directory and export it to the system environment''

[[Email protected] ~] $ Vim ~ /. Bash_profile #. bash_profile # Get the aliases and functionsif [-f ~ /. Bashrc]; then .~ /. Bashrcfi # user specific environment and startup programsexport java_home =/home/hadoop/APP/jdk1.8.0 _ 144 export Path = $ java_home/bin: $ pathexport flume_home =/home/hadoop/APP/flumeexport Path = $ flume_home/bin: $ pathexport zk_home =/home/hadoop/APP/zkexport Path = $ zk_home/bin: $ pathexport kafka_home =/home/hadoop/APP/kafkaexport Path = $ kafka_home/bin: $ path #: WQ save and exit

3. Click "Source ".

4. Configure and modify the config configuration file in the decompressed directory.

Configure server. properties [Notes] broker. id = 0 Description: Kafka, A brokerlisteners explanation: the listening port host. name Description: current machine log. dirs explanation: the log storage directory zookeeper of Kafka. after the connect ZK address is modified, save and exit.

5. Now, you can start it.
How to start it ?? See how the official website said, http://kafka.apache.org/quickstart

Bin/zookeeper-server-start.sh config/zookeeper. Properties
2). Start Kafka again.
Bin/kafka-server-start.sh config/server. Properties

# Start Kafka command kafka-server-start.sh $ kafka_home/config/server. Properties

6. After startup

[[Email protected] ~] $ Jps271_kafka27334 jps000043 quorumpeermain [[email protected] ~] $ # Check the Kafka process.

Okay. Now we have installed Kafka. Next, we will start broker deployment...
Recommended: deployment and use of Kafka Series 2
Http://blog.csdn.net/liuge36/article/details/78593404

Architecture introduction and installation of Kafka Series 1