Kafka Basic Introduction

1About Kafka Message Queue1.1Message Queuing1.1.1Basic Features

1. Can be extended
   
   • Capacity expansion without the need for downline
   • Data flow Partitioning (partition) is stored on multiple machines
2. Performance
   
   • A single broker can serve thousands of clients
   • Single broker Reads/writes per second up to hundreds of megabytes per second
   • A cluster of multiple brokers will achieve very high throughput capacity
   • Stable performance, no matter how big the data
3. Persistent storage
   
   • Stored on disk
   • Redundant backups to other servers to prevent loss

1.1.2Message format

1. A topic corresponds to a message format, so messages are classified by topic
   
2. A topic represents a message that consists of 1 or more patition (s)
   
3. a partition.
   
   • One partition should be stored on one or more servers
     • If there is only one server, there is no redundant backup, it is a single machine instead of a cluster
     • If you have more than one server
       • One server for leader
       • Other servers for followers
       • Leader need to accept read and write requests
       • Followers for redundant backups only
       • Leader failure, will automatically elect a follower as leader to ensure uninterrupted service
       • Each server may play some partitions leader and other partitions follower roles, so that the entire cluster will achieve load balancing effect
   • Messages are stored sequentially
   • Message order is not variable
   • Only append messages, cannot insert
   • Each message has an offset, which is used as the message ID, and is unique in a partition
   • Offset has consumer preservation and management, so the reading order is actually completely consumer determined, not necessarily linear
   • Message has a time-out date and is deleted after expiration

1.1.3Producer Producer

• Producer writing messages to Kafka
• Write to specify topic and partition
• How the message is divided into different partition, the algorithm is specified by producer

1.1.4Consumer Consumer

• Consumer reading messages and processing them
• Consumer group
  • This concept was introduced to support two scenarios: Each message was distributed to a consumer, and each message was broadcast to all consumers
  • Multiple consumer group subscribes to a topic, the TOPCI message is broadcast to all consumer group
  • Once a message is sent to a consumer group, it can only be received and used by one consumer of the group
  • Each consumer in a group corresponds to a partition to provide the following benefits
    • can be processed concurrently according to the number of partition
    • Each partition has only one consumer read, thus guaranteeing that the order in which the messages are processed is carried out in the order of partition, noting that the order is affected by the algorithm of the producer holding the message.

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Kafka Basic Introduction