Kafka ~ Deployment in Linux, kafkalinux

Kafka ~ Deployment in Linux, kafkalinux
Concept

Kafka is a high-throughput distributed publish/subscribe message system that can process all the action flow data of a website with a consumer scale. Such actions (Web browsing, search, and other user actions) are a key factor in many social functions on modern networks. This data is usually solved by processing logs and log aggregation due to throughput requirements. This is a feasible solution for log data and offline analysis systems like Hadoop that require real-time processing. Kafka aims to unify online and offline message processing through the parallel loading mechanism of Hadoop, and also to provide real-time consumption through clusters.

 
 

  
  

    Software name 
  
  
  

    Apache Kafka 
  
  
  

    Developer 
  
  
  

    Apache Software Foundation 
  
  
  

    Software Platform 
  
  
  

    Cross-platform 
  
  
  

    Software Version 
  
  
  

    0.8.2.2 
  
 
 
 
 

  
  

    Update Time 
  
  
  

    2015-10-0 
  
  
  

    Software Language 
  
  
  

    Scala 
  
  
  

    Software size 
  
  
  

    15 M 
  
  
  

    Software authorization 
  
  
  

    Apache License 2.0 
  
 
 

Features

Kafka is a high-throughput distributed message publishing and subscription system with the following features:

Related Knowledge points

Broker
A Kafka cluster contains one or more servers, which are called broker [5].
Topic
Each message published to the Kafka cluster has a category called Topic. (Messages of different topics are stored separately physically, messages of a Topic in logic are stored on one or more brokers, but you only need to specify the Topic of the message to produce or consume data without worrying about where the data is stored)
Partition
Partition is a physical concept. Each Topic contains one or more partitions.
Producer
Publishes messages to Kafka broker.
Consumer
The consumer that reads messages from the Kafka broker.
Consumer Group
Each Consumer belongs to a specific Consumer Group (you can specify a group name for each Consumer. If no group name is specified, it belongs to the default group ).

Installation and deployment

Prerequisites

Kafka requires support for the zookeeper Distributed Scheduling component, so we need to install it first. For details, please refer to uncle's article Zookeeper ~ Deployment in Linux

Download
Curl-L-O http://mirrors.cnnic.cn/apache/kafka/0.9.0.0/kafka_2.10-0.9.0.0.tgz
Extract
Tar zxvf kafka_2.10-0.9.0.0.tgz

View directory structure

Start Kafka

Bin/kafka-server-start.sh config/server. properties & // & indicates that the process will be executed at the backend

Check whether the service is successfully started.

Netstat-tunlp | egrep "(2181 | 9092 )"

This indicates that our 9092 is that the data communication port of kafka has been started, and 2181 is our zookeeper communication port, indicating that they are in normal listening status!

I hope you will learn and understand kafka, zookeepr, and centos through this article!

Thank you for reading this article!