Installation and use of Kafka

Source: Internet
Author: User

Simply put, Kafka is a high-throughput, partial-messaging system that provides persistence.

Architecture of the Kafka

Producer: Message survivor
Consumer: Message Consumers
Broker:kafka Cluster Server, responsible for processing message read, write requests, store messages
topic: Message Queuing/classification
Queue has a producer consumer model inside.
Broker is the agent, in the Kafka cluster this layer here, in fact, there are a lot of broker
topic is the equivalent of a queue
• There is no picture in fact there are zookeeper, the structure of some meta-information is present zookeeper above, the whole cluster management and zookeeper have a great relationship

Kafka's message storage and production consumption model

• A topic is divided into multiple partition
• Each partition internal message is strong and orderly, with each message having a sequence number called offset
• One partition only corresponds to one broker, and one broker can control multiple partition
• Messages are written directly to the file without memory buffering
• Deleted based on time policy, not after consumption
producer himself decides which partition to write the message, either polling load balancing or hash-based partition strategy

There are two important features: (1) The performance of disk continuous reading and writing is much higher than that of random read and write, (2) concurrency, splitting one topic into multiple partition (the Kafka read-write unit is partition).

1. After entering the kafka2.10 directory, view the boot directory cat startkafka.sh

2. Change the configuration file conf/server.properties, modify the Broker.id (each unique) and change the host name of the zookeeper, the port number does not change, modify the execution permissions: chmod +x./bin/*

3. Start Kafda:bash startkafka.sh

4. Use of the process

Create Topic 20160118
./bin/kafka-topics.sh--create--zookeeper node1:2181,node2:2181,node3:2181--replication-factor 2--partitions 2-- Topic 20160118
./bin/kafka-topics.sh--zookeeper node1:2181,node2:2181,node3:2181--describe--topic 20160118
View Tocpic's Directory
./bin/kafka-topics.sh--list--zookeeper node1:2181,node2:2181,node3:2181 (Result 20160118)
See Tocpic's Specific information
./bin/kafka-topics.sh--describe--zookeeper node1:2181,node2:2181,node4:2181
The result is:
topic:20160118 partitioncount:2 replicationfactor:2 configs:
topic:20160118 partition:0 leader:2 replicas:2,0 isr:2,0
topic:20160118 partition:1 leader:0 replicas:0,1 isr:0,1
View Log directory: ll/kafka-logs/
Send message: bin/kafka-console-producer.sh--broker-list node1:9092,node2:9092,node3:9092--topic 20160118
Receive information: bin/kafka-console-consumer.sh--zookeeper node1:2181,node2:2181,node3:2181--topic 20160118
Receive all information (including messages sent before and when this command is not opened):
bin/kafka-console-consumer.sh--zookeeper node1:2181,node2:2181,node3:2181--topic 20160118--from-beginning

Installation and use of Kafka

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.