Kafka Design Analysis (i) Kafka Background and architecture introduction

Source: Internet
Author: User
Tags rabbitmq

This article has authorized INFOQ exclusive release, if you need to reprint please specify the source. For more information, please refer to the author's personal site http://www.jasongj.com, or subscribe to the public number "Big Data Architecture"

Summary

Kafka, a distributed messaging system developed and open-source by LinkedIn, is widely used for its distributed and high throughput rates and is now integrated with Cloudera Hadoop,apache Storm,apache Spark. This paper introduces the background of Kafka, the design goal, the advantage of using the message system and the current popular message system contrast. It also introduces the architecture of Kafka, producer message routing, Consumer group and different message distribution methods implemented by it, Topic & Partition, and finally introduces Kafka Why consumer uses pull mode and the three delivery guarantee provided by Kafka.

Background introduction Kafka Creating a background

Kafka is a messaging system that was originally developed from LinkedIn as the basis for the activity stream of LinkedIn and the Operational Data Processing pipeline (Pipeline). It has now been used by several different types of companies as multiple types of data pipelines and messaging systems.
Activity flow data is the most common part of data that almost all sites use to make reports about their site usage. Activity data includes content such as page views, information about the content being viewed, and search conditions. This data is typically handled by writing various activities to a file in the form of a log, and then periodically analyzing the files in a statistical manner. Operational data refers to the performance data of the server (CPU, IO Utilization, request time, service log, and so on). There are a wide variety of statistical methods for operating data.
In recent years, activity and operational data processing has become a critical component of the website's software product features, which requires a slightly more complex infrastructure to support it.

Kafka Introduction

Kafka is a distributed, publish/subscribe-based messaging system. The main design objectives are as follows:

    • Provides message persistence in a time-complexity O (1) manner, guaranteeing constant-time complexity of access performance even for terabytes or more data
    • High throughput rates. Capable of single-machine support for transmission of messages up to 100K per second, even on very inexpensive commercial machines
    • Supports message partitioning between Kafka servers, and distributed consumption, while guaranteeing the sequential transmission of messages within each partition
    • Support offline data processing and real-time data processing
    • Scale out: Supports online horizontal scaling
Why use a messaging system
  • Decoupling
    It is extremely difficult to anticipate what needs to be encountered in future projects at the start of the project. The message system inserts an implicit, data-based interface layer in the middle of the processing process, which is implemented on both sides of the processing. This allows you to independently extend or modify the processing on both sides, as long as you ensure that they adhere to the same interface constraints.

  • Redundancy
    In some cases, the process of processing data will fail. Unless the data is persisted, it is lost. Message Queuing persists the data until it has been fully processed, bypassing the risk of data loss in this way. In the insert-get-delete paradigm used by many message queues, it is necessary for your processing system to explicitly indicate that the message has been processed before it is removed from the queue, ensuring that your data is safely saved until you are finished using it.

  • Scalability
    Because Message Queuing decouples your processing, it is easy to increase the number of messages queued and processed, as long as additional processing is required. No need to change the code, do not need to adjust parameters. Expansion is as simple as adjusting the power button.

  • Flexibility & Peak Handling capability
    Applications still need to continue to function in the event of a surge in traffic, but such bursts are not common, and it is a huge waste to be ready to invest in resources that can handle such peak access. Using Message Queuing enables critical components to withstand burst access pressure without crashing completely due to sudden and overloaded requests.

  • Recoverability
    When a part of the system fails, it does not affect the entire system. Message Queuing reduces the degree of coupling between processes, so even if a process that processes messages is hung up, messages queued to the queue can still be processed after the system resumes.

  • Order Guarantee
    In most usage scenarios, the order of data processing is important. Most message queues are inherently sorted and ensure that the data is handled in a specific order. Kafka guarantees the ordering of messages within a partition.

  • Buffer
    In any important system, there will be elements that require different processing times. For example, loading a picture takes less time than applying a filter. Message Queuing uses a buffer layer to help the task perform the most efficient execution ——— the processing of the write queue is as fast as possible. This buffering helps to control and optimize the speed of the data flow through the system.

  • Asynchronous communication
    Many times, users do not want or need to process messages immediately. Message Queuing provides an asynchronous processing mechanism that allows a user to put a message into a queue, but does not immediately process it. How many messages you want to put into the queue, and then deal with them when you need them.

Common Message Queue Comparison
  • RabbitMQ
    RABBITMQ is an open source message queue written using Erlang, which itself supports a lot of protocols: Amqp,xmpp, SMTP, STOMP, and that's why it's very heavyweight and more suitable for enterprise-class development. The broker framework is also implemented, which means that messages are queued in the central queue before being sent to the client. Good support for routing, load balancing, or data persistence.

  • Redis
    Redis is a NoSQL database based on Key-value, which is very active in development and maintenance. Although it is a Key-value database storage system, it natively supports MQ functionality, so it can be used as a lightweight queue service. For RABBITMQ and Redis on-board and out-of-team operations, each execution 1 million times, every 100,000 times the execution time is recorded. The test data is divided into 128Bytes, 512Bytes, 1K and 10K four different sizes of data. The experiment shows that when the data is compared with the RABBITMQ, the performance of the Redis is higher than that of the 10k,redis when the data is larger than the size of the queue, and the Redis shows very good performance regardless of the data size. And RABBITMQ's performance is much lower than Redis.

  • ZeroMQ
    ZEROMQ claims to be the fastest Message Queuing system, especially for large throughput scenarios. ZMQ can implement advanced/complex queues that rabbitmq are not good at, but developers need to assemble multiple technical frameworks themselves, and technical complexity is a challenge to the success of this MQ application. ZEROMQ has a unique non-middleware model that you do not need to install and run a messaging server or middleware because your application will play this server role. You just need to simply reference the ZEROMQ library, you can use NuGet to install it, and then you can happily send messages between applications. However, ZEROMQ only provides a non-persistent queue, meaning that if the outage occurs, the data will be lost. Of these, Twitter's storm 0.9.0 used Zeromq as a transport for data streams by default in previous versions (Storm has supported both ZEROMQ and Netty as a transport module since version 0.9).

  • ActiveMQ
    ACTIVEMQ is a sub-project under Apache. Similar to ZEROMQ, it is able to implement queues with agent and peer-to-peer technology. It is also similar to RABBITMQ, where a small amount of code makes it possible to implement high-level scenarios efficiently.

  • Kafka/jafka
    Kafka is a sub-project under Apache and is a high-performance cross-language distributed publish/Subscribe Message Queuing system, and Jafka is hatched on Kafka, an upgraded version of Kafka. Has the following characteristics: Fast persistence, can be in the O (1) of the system overhead for message persistence, high throughput, on a common server can achieve 10W/S throughput rate, complete distributed system, Broker, Producer, consumer are native automatically support distributed, Automatic load balancing, support for parallel loading of Hadoop data, the same log data as Hadoop and offline analysis system, but also the limitations of real-time processing, this is a feasible solution. Kafka unifies online and offline message processing through Hadoop's parallel loading mechanism. Apache Kafka is a very lightweight messaging system in contrast to ACTIVEMQ, and is a well-functioning distributed system in addition to its very good performance.

Kafka Architecture Terminology
    • Broker
      The Kafka cluster contains one or more servers, which are called broker
    • Topic
      Each message published to the Kafka Cluster has a category, which is called topic. (Physically different topic messages are stored separately, logically a topic message is saved on one or more brokers but the user only needs to specify the topic of the message to produce or consume data without worrying about where the data is stored)
    • Partition
      Parition is a physical concept, and each topic contains one or more partition.
    • Producer
      Responsible for publishing messages to Kafka broker
    • Consumer
      The message consumer, the client that reads the message to Kafka broker.
    • Consumer Group
      Each consumer belongs to a specific consumer group (the group name can be specified for each consumer, and the default group if the group name is not specified).
Kafka topological structure


As shown, a typical Kafka cluster contains several producer (which can be page View generated by the Web front end, or server logs, System CPUs, memory, etc.), and several brokers (Kafka support horizontal expansion, the more general broker number, The higher the cluster throughput, several consumer Group, and one zookeeper cluster. Kafka manages the cluster configuration through zookeeper, elects leader, and rebalance when the consumer group is changed. Producer uses push mode to publish messages to Broker,consumer to subscribe to and consume messages from broker using pull mode.

Topic & Partition

Topic can logically be thought of as a queue, and each consumption must specify its topic, which can be simply understood to indicate which queue to put the message in. In order to make the Kafka throughput can be linearly improved, the topic is physically divided into one or more partition, each partition in the physical corresponding to a folder, the folder stores all messages and index files of this partition. If you create Topic1 and Topic2 two topic, with 13 and 19 partitions respectively, a total of 32 folders will be generated on the entire cluster (a total of 8 nodes are used in this article, where Topic1 and Topic2 replication-factor are 1). As shown in.
  
  
Each log file is alog entrieSequence, eachlog entrieContains a 4-byte integer (value n+5), 1-byte "Magic Value", a 4-byte CRC checksum followed by a n-byte message body. Each message has a unique 64-byte offset under the current partition, which indicates where the message starts. The message format stored on the disk is as follows:
Message length:4 bytes (value:1+4+n)
"Magic" value:1 byte
Crc:4 bytes
Payload:n bytes
This onelog entriesInstead of a file, it is divided into multiple segment, each segment named with the offset of the segment first message and suffixed with ". Kafka". There will also be an index file that identifies the segment contained under eachlog entryThe offset range, as shown in.
  
  
Because each message is append to the partition, it is a sequential write disk, so the efficiency is very high (proven, sequential write disk efficiency is higher than the random write memory, which is a very important guarantee of Kafka high throughput rate).
  
  
For a traditional message queue, messages that have already been consumed are generally deleted, and the Kafka cluster retains all messages, regardless of whether they are consumed or not. Of course, because of disk limitations, it is not possible to keep all of the data permanently (not really necessary), so Kafka provides two strategies to delete old data. One is based on the time and the second is based on the partition file size. For example, you can configure$KAFKA_HOME/config/server.properties, let Kafka delete the data from a week ago, or delete the old data when the partition file exceeds 1GB, as shown in the following configuration.

# The minimum age ofALogFileTobe eligibleFor deletionlog.retention.hours=168# the maximum size ofALog segmentFile. When this sizeis reachedAnew log segment would be created.log.segment.bytes=1073741824# the interval at which log Segments is checked to see if they can be deleted According to the retention policieslog.retention.check.interval.ms=< Span class= "number" >300000# If log.cleaner.enable=true is set The Cleaner would be enabled and individual logs can then be marked for log compaction. Log.cleaner.enable=false             

  
It is important to note that because the time complexity of Kafka reading a particular message is O (1), which is irrelevant to the size of the file, deleting the expired file here is irrelevant to improving Kafka performance. The choice of how to delete the policy is only related to the disk and the specific requirements. In addition, Kafka will retain some metadata information for each consumer group-the position of messages currently consumed, or offset. This offset is controlled by consumer. Under normal circumstances, consumer will increment the offset after consuming a message. Of course, consumer can also set offset to a smaller value and re-consume some messages. Because Offet is controlled by consumer, Kafka broker is stateless, it does not need to flag which messages are consumed by which, and does not require the broker to ensure that only one consumer of the same consumer group can consume a message. Therefore, there is no need to lock mechanism, which also provides a strong guarantee for the high throughput rate of Kafka.

Producer Message Routing

When producer sends a message to the broker, it chooses which partition to store to it according to the paritition mechanism. If the partition mechanism is set up properly, all messages can be distributed evenly across different partition, thus achieving load balancing. If a topic corresponds to a file, the machine I/O to which this file resides will become a performance bottleneck for this topic, and with partition, different messages can be written in parallel to different partition of different brokers, greatly improving throughput rates. You can $KAFKA_HOME/config/server.properties num.partitions specify the default number of partition for a new topic by configuring the item in, or you can specify it by parameter when you create the topic, or you can modify it with the tools provided by Kafka after the topic is created.
  
When sending a message, you can specify the key,producer of the message based on the key and partition mechanism to determine which parition the message should be sent to. The paritition mechanism can be specified by specifying this parameter of the producer paritition. class , which must implement the kafka.producer.Partitioner interface. In this example, if key can be resolved to an integer, the corresponding integer is partition to the total number of the sum, and the message is sent to the corresponding partition of that number. (Each parition will have an ordinal number starting from 0)

Import Kafka.producer.Partitioner;Import kafka.utils.VerifiableProperties;public class  Jasonpartitioner<t> implements partitioner { span class= "keyword" >public jasonpartitioner (Verifiableproperties verifiableproperties) {} span class= "annotation" > @Override public int  Partition (Object key, int numpartitions) {try { int partitionnum = Integer.parseint (String) key; return math.abs (Integer.parseint (String) key)% numpartitions);} catch (Exception e) {return math.abs (Key.hashcode ()% numpartitions);}}  


If the class in the previous example is Partition.class , and you send 20 messages (key 0,1,2,3) to TOPIC3 (with 4 partition) through the following code.

  public void sendmessage () throws interruptedexception{for (int i =  1; I <= 5; i++) {List messagelist = new arraylist<keyedmessage<string   , string>> ();   for (int j = 0; j < 4; j + +) { Messagelist.add (new keyedmessage<string, string> ( "Topic2", j+<   Span class= "string" "",  "the" + i +  "message for key" + j); } producer.send (Messagelist); } producer.close ();}               

The same message is sent and stored in the same partition, and the key is the same as the partition sequence number. (The partition sequence number starts at 0, and the key in this example starts from 0). Shown is a list of messages printed after calling consumer through a Java program.
    

Consumer Group

(All the descriptions in this section are based on the consumer Hight level API, not the low-level API.)
When using the consumer High level API, a message from the same topic can only be consumed by one consumer within the same consumer group, but multiple consumer group can consume the message at the same time.
  
This is the means by which Kafka is used to implement a topic message broadcast (to all consumer) and unicast (sent to a consumer). A topic can correspond to multiple consumer Group. If you need to implement a broadcast, as long as each consumer has a separate group. To implement unicast as long as all the consumer are in the same group. The consumer group also allows consumer to be freely grouped without having to send messages to different topic multiple times.
In fact, one of the design concepts of Kafka is to provide offline processing and real-time processing. Based on this feature, the real-time streaming system of storm can be used to process messages on-line, use the batch system of Hadoop for offline processing, and simultaneously back up data to another data center in real time. Just make sure that the consumer used by these three operations belong to a different consumer group. is a simplified deployment of Kafka in LinkedIn.
  
  
The following example shows the features of the Kafka Consumer group more clearly. First create a topic (named Topic1, containing 3 partition), then create a consumer instance that belongs to Group1, and create three group2 instances that belong to consumer. Finally, the key is a three-in-one message sent to Topic1 via producer. It was found that consumer, who belonged to Group1, received all of the three messages, while the 3 consumer in Group2 received a key of three-way messages respectively. As shown in.
    

Push vs. pull

As a messaging system, Kafka follows the traditional way of choosing to push messages from producer to broker and pull messages from broker by consumer. Some logging-centric system, such as Facebook's scribe and Cloudera's flume, use push mode. In fact, the push mode and pull mode each have advantages and disadvantages.
Push mode is difficult to adapt to consumers with different consumption rates because the message sending rate is determined by the broker. The goal of Push mode is to deliver messages as fast as possible, but this can easily cause consumer to process messages, typically denial of service and network congestion. The pull mode can consume messages at the right rate based on consumer's spending power.
For Kafka, the pull mode is more appropriate. Pull mode simplifies the design of the broker, consumer can control the rate of consumer messages autonomously, while consumer can control the consumption mode by itself--can be consumed in batches, and can choose different submission methods to realize different transmission semantics.

Kafka Delivery Guarantee

There are several possible delivery guarantee:

  • At most onceMessages may be lost, but they will never be transferred again
  • At least oneMessages will never be lost, but may be transmitted repeatedly
  • exactly once   each message is bound to be transmitted once and only once, and many times this is what the user wants.

    When producer sends a message to the broker, it will not be lost once the message is commit and the factor replication exists. However, if producer sends the data to the broker and the communication is interrupted by a network problem, the producer cannot tell if the message has been commit. Although Kafka cannot determine what happened during a network failure, producer can generate something like a primary key, and retry multiple times with idempotent when it fails, so that exactly once is done. As of now (Kafka 0.8.2, 2015-03-04), this feature has not been implemented and hopefully will be implemented in a future version of Kafka. (So, by default, a message from producer to broker is guaranteed to be at least once , which can be implemented by setting the producer asynchronous send to at most once ). The
    next discussion is delivery guarantee semantics of messages from broker to consumer. (For Kafka consumer-level API only). Consumer after reading a message from broker, you can select Commit, which saves the offset of the message that consumer reads in the partition in zookeeper. The consumer will read the next time the partition is read from the next one. If not committed, the starting position of the next read will be the same as the beginning of the last commit. Of course, the consumer can be set to autocommit, that is, consumer once read the data automatically commit. If you are only discussing the process of reading the message, then Kafka is ensuring exactly once . In practice, however, the application does not end up reading the data in consumer, but it is intended to be processed further, and the order of data processing and commit largely determines the delivery guarantee semantic of the message from broker and consumer.
  • After reading the message, commit and process the message first. In this mode, if consumer has not been able to process the message after the commit, the next time you start working again, you cannot read the message that has just been submitted and not processed, which corresponds to the crash.At most once
  • After you read the message, process the commit again. In this mode, if the commit is consumer crash before the message is processed, the next time you start working again, the message that you just did not commit is processed. This corresponds to the At least once . In many usage scenarios, the message has a primary key, so the processing of the message tends to be idempotent, that is, multiple processing of this message is equivalent to processing only once, which can be considered Exactly once . (I think this argument is far-fetched, after all, it is not the mechanism provided by Kafka itself, the primary key itself can not fully guarantee the power of operation. And in fact we say that delivery guarantee semantics is how many times the discussion is handled, not how the results are handled, because there are a variety of processing methods, we should not treat the characteristics of the process-such as whether it is idempotent, as the feature of Kafka itself)
  • If you do Exactly once , you need to reconcile the output of offset and the actual operation. The classic approach is to introduce a two-phase commit. If you can have offset and operation input in the same place, it will be more concise and universal. This approach may be better because many output systems may not support two-phase commits. For example, consumer can put data into HDFs when it gets the data, if the latest offset and the data itself are written to HDFs, it can ensure that the output of the data and offset updates are either completed, or not completed, indirect implementation Exactly once . (Currently, for the high-level API, offset is stored in zookeeper and cannot be stored in HDFs, and the low-level API's offset is maintained by itself and can be stored in HDFs)
    In summary, Kafka is guaranteed by default At least once and is enabled by setting up producer asynchronous commits At most once . While Exactly once requiring collaboration with external storage systems, fortunately Kafka provides offset that can be very straightforward and very easy to use this way.
Next trailer

The next article will explain in depth how Kafka is doing replication and leader election. In previous versions of Kafka0.8, if a broker was down or there was a problem with the disk, all the partition data on that broker would be lost. Kafka0.8 later joined the replication mechanism, which can backup multiple copies of each partition, even if some broker outages guarantee the availability of the system and the integrity of the data.

Kafka Design Analysis (i) Kafka Background and architecture introduction

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.