Installation and use of Kafka

Simply put, Kafka is a high-throughput, partial-messaging system that provides persistence.

Architecture of the Kafka

Producer: Message survivor
Consumer: Message Consumers
Broker:kafka Cluster Server, responsible for processing message read, write requests, store messages
topic: Message Queuing/classification
Queue has a producer consumer model inside.
Broker is the agent, in the Kafka cluster this layer here, in fact, there are a lot of broker
topic is the equivalent of a queue
• There is no picture in fact there are zookeeper, the structure of some meta-information is present zookeeper above, the whole cluster management and zookeeper have a great relationship

Kafka's message storage and production consumption model

• A topic is divided into multiple partition
• Each partition internal message is strong and orderly, with each message having a sequence number called offset
• One partition only corresponds to one broker, and one broker can control multiple partition
• Messages are written directly to the file without memory buffering
• Deleted based on time policy, not after consumption
producer himself decides which partition to write the message, either polling load balancing or hash-based partition strategy

There are two important features: (1) The performance of disk continuous reading and writing is much higher than that of random read and write, (2) concurrency, splitting one topic into multiple partition (the Kafka read-write unit is partition).

1. After entering the kafka2.10 directory, view the boot directory cat startkafka.sh

2. Change the configuration file conf/server.properties, modify the Broker.id (each unique) and change the host name of the zookeeper, the port number does not change, modify the execution permissions: chmod +x./bin/*

3. Start Kafda:bash startkafka.sh

4. Use of the process

Create Topic 20160118
./bin/kafka-topics.sh--create--zookeeper node1:2181,node2:2181,node3:2181--replication-factor 2--partitions 2-- Topic 20160118
./bin/kafka-topics.sh--zookeeper node1:2181,node2:2181,node3:2181--describe--topic 20160118
View Tocpic's Directory
./bin/kafka-topics.sh--list--zookeeper node1:2181,node2:2181,node3:2181 (Result 20160118)
See Tocpic's Specific information
./bin/kafka-topics.sh--describe--zookeeper node1:2181,node2:2181,node4:2181
The result is:
topic:20160118 partitioncount:2 replicationfactor:2 configs:
topic:20160118 partition:0 leader:2 replicas:2,0 isr:2,0
topic:20160118 partition:1 leader:0 replicas:0,1 isr:0,1
View Log directory: ll/kafka-logs/
Send message: bin/kafka-console-producer.sh--broker-list node1:9092,node2:9092,node3:9092--topic 20160118
Receive information: bin/kafka-console-consumer.sh--zookeeper node1:2181,node2:2181,node3:2181--topic 20160118
Receive all information (including messages sent before and when this command is not opened):
bin/kafka-console-consumer.sh--zookeeper node1:2181,node2:2181,node3:2181--topic 20160118--from-beginning

Installation and use of Kafka