Kafka Getting Started

Article source

Kafka Getting Started classic tutorial http://www.aboutyun.com/thread-12882-1-1.html

Kafka Official Website Introduction http://kafka.apache.org/documentation.html#introduction

Kafka Anatomy (i): Kafka Background and architecture Introduction http://www.infoq.com/cn/articles/kafka-analysis-part-1/, this introduction is very comprehensive, focus on it

1. Partitioning

Each partition has replicas in several services in the Kafka cluster, so that the services that hold the replicas can work together to process data and requests, and the number of replicas is configurable. Replicas make Kafka a fault-tolerant capability.

Each partition is made up of one server as "leader", 0 or several servers as "followers", leader is responsible for processing the read and write of messages, and followers to replicate leader. If leader down, One of the followers will automatically become leader.

Each service in a cluster plays two roles at the same time: as the leader of a portion of the partition it holds, as well as the followers of other partitions, the cluster will have good load balancing.

2. Kafka can be a good guarantee of order compared to the traditional message system.

Kafka can only guarantee the ordering of messages within a partition, which is not possible between different partitions, which can meet the needs of most applications. If the order of all messages in the topic is required, then only one partition is allowed for this topic, and of course only one consumer group consumes it.

3. Persistence of data

Don't be afraid of the file system

To improve performance, modern operating systems tend to use memory as a disk cache, and modern operating systems are happy to make all free memory available as disk caches, although this may sacrifice some performance in cache recycling and redistribution.
All disk read and write operations pass through this cache, which is unlikely to be bypassed unless I/O is used directly. So while each program caches only one copy of the data in its own thread, there's one in the OS cache,
This is equivalent to saving two copies of the data.
In addition to discussing the JVM, the following two facts are well known:
The A.java object occupies a very large space, which is almost twice times more or higher than the data to be stored.
B. As the amount of data in the heap increases, it becomes increasingly difficult for garbage collection to change back.
Based on the above analysis, if the data is cached in memory, because the need to store two copies, have to use twice times the memory space, Kafka based on the JVM, but also have to double the space again, and to avoid the performance impact of GC, in a 32G memory of the machine, to use the 28-30g memory space. And when the system restarts, you have to brush the data into memory (10GB memory for almost 10 minutes), even if the use of cold refresh (not a one-time brush into the memory, but in the use of the data without a brush to memory) will also cause the initial time of the new can be very slow. However, with the file system, you do not need to refresh the data even if the system restarts. The use of file systems also simplifies the logic of maintaining data consistency. So unlike the traditional design that caches the data in memory and then brushes it to the hard disk,

Kafka directly writes the data to the file system's log.

Operation efficiency of constant time

In most messaging systems, data persistence is often a mechanism for each cosumer to provide a B-tree or other random read-write data structure. B-Tree is great, of course, but it also comes with some price: for example, the complexity of B-Tree is O (log n), O (log n) is often considered to be a constant complexity, but not for hard disk operations. A search on a disk takes 10ms, and each hard drive can search only once at a time, so concurrent processing becomes a problem. Although the storage system uses caching for a lot of optimizations, the observation of the performance of the tree structure shows that its performance tends to decrease linearly as the data grows, and the data grows one time, and the speed is reduced by one-fold.

Intuitively speaking, for a message system that is primarily used for log processing, data persistence can be done simply by appending data to a file and reading it from a file. The benefit of this is that both the read and write are O (1), and the read operation does not block writes and other operations. The performance benefits are obvious because the performance and size of the data are not related.

Since it is possible to build a message system with a hard disk space that has little capacity limitations (relative to memory), you can provide features that are not available in the general messaging system without a performance penalty. For example, the general message system is deleted immediately after the message is consumed, but Kafka can save the message for a period of time (for example, a week), which gives consumer good maneuverability and flexibility, as detailed in future articles.

4. Transaction definition for message transport

Before discussing how consumer and producer work, now let's discuss the data transfer aspect. Transaction definitions for data transfer typically have the following three levels:

• At most one time: messages are not sent repeatedly and are transmitted at most once, but they may not be transmitted at one time.
• At least once: messages are not sent out, they are transmitted at least once, but they can also be transmitted repeatedly.
• Accurate once (exactly once): does not leak the transmission also does not repeat the transmission, each message transmits once and only then transmits once, this is everybody hoped.

Most messaging systems claim to be "accurate once", but reading their documents carefully can be misleading, such as not explaining what happens when consumer or producer fail, or when multiple consumer are parallel. Or when writing to the hard disk data is lost. Kafka's approach should be more advanced. When publishing a message, Kafka has a concept of "committed", and once the message is committed, the data is not lost as long as the copy of the partition where the message is written is active. The concept of the activity of a replica is discussed in the next section of the document. Now assume that the broker is not down.

If a network error occurs when producer publishes a message, but it is not certain that the actual commit occurred before or after the commit, although this is not common, it must be considered, and now the Kafka version has not resolved the issue, and future versions are trying to resolve it.

Not all situations require a high level of "exact once", Kafka allows producer to specify a flexible level. For example, producer can specify that a notification must wait for a message to be committed, or to send the message completely asynchronously without waiting for any notification, or just wait for leader to declare that it has received the message (followers is not necessary).

Now consider this issue from the consumer aspect, all replicas have the same log file and the same offset,consumer maintain their own consumption of the message offset, if the consumer will not crash of course, can save this value in memory, of course, there is no guarantee of this. If consumer crashes, there will be another consumer. Then consume the message, and it needs to continue processing from a suitable offset. In this case, you have the following options:
1. Consumer can read the message first, then write offset to the log file, and then process the message. There is a possibility that after the storage of offset has not processed the message is crash, the new consumer continue to handle from this offset, then some messages will never be processed, this is said "at most once."

2. Consumer can read the message first, process the message, and finally record the offset, of course, if the crash before the record offset, the new consumer will repeat the consumption of some messages, which is said above "at least once."

3. "Exact once" can be resolved by dividing the submission into two phases: once the offset is saved, the message is processed successfully and then submitted again. But there's a simpler way to do this: Save the offset of the message and the result of the message being processed. For example, when processing a message with Hadoop ETL, the processed result and offset are stored in HDFS simultaneously, so that both the message and the Offser are processed at the same time.

5. Consumption status tracking

Most messaging systems maintain messages consumed on broker side records: A message is distributed to consumer after the broker is immediately tagged or waits for customer notification to be flagged. This can also be deleted immediately after the message is consumed to reduce space consumption.

If a message is sent out and immediately marked as consumed, the message is lost once consumer processes the message (such as a program crash). To solve this problem, many messaging systems provide another feature: When a message is sent out, it is only marked as sent, and is marked as being consumed when the consumer has been notified of the successful consumption. This solves the problem of message loss, but creates a new problem, first of all, if consumer handles the message successfully but fails to send a response to the broker, the message will be consumed two times. On the second issue, the broker must maintain the state of each message and lock the message and then change the state and then release the lock each time. This trouble again, and do not say to maintain a large number of state data, such as if the message sent out but did not receive a notification of the success of the consumer, this message will remain locked in the state,
Kafka has adopted different strategies. Topic are divided into partitions, each of which is consumed by only one consumer at a time. This means that each partition is consumed by the location of the message in the log is simply a simple integer: Offset. This makes it easy to mark the consumption status of each partition, which requires just an integer. So the tracking of consumption status is very simple.

This brings another benefit: consumer can turn offset into an older value to re-consume old messages. This may seem inconceivable to the traditional messaging system, but it is really useful, who has set a message that can only be consumed once? Consumer found that parsing the data of the program has a bug, after modifying the bug and then to parse a message, seems to be a reasonable amount!

6. Master-Slave synchronization

Kafka allows the topic partition to have several replicas, which is configurable, and you can configure the number of replicas for each topci. Kafka automatically backs up data on each copy, so the data remains available when a node is down.
The copy function of Kafka is not required, you can configure only one copy, so it is actually equivalent to only one piece of data.
The units that create the replicas are topic partitions, each with one leader and 0 or more followers. All read and write operations are handled by leader, the number of general partitions is much more than the number of brokers, the leader of each partition is evenly distributed in brokers

All followers copy the leader log, and the message and order in the log are consistent with the leader. Flowers from leader to the normal consumer and saves it in his own log file. The cloth-type message system automatically handles the failed request, and it points to whether the

• Nodes must be able to maintain and zookeeper connections, and zookeeper check the connection of each node through the heartbeat mechanism.
• If the node is a follower, he must be able to synchronize leader writes in a timely manner, the delay can not be too long.

Nodes that meet the above criteria should be accurately said to be "in sync" instead of "alive" or "failed". Leader will track all "in Sync" nodes, and once one is down, or stuck, or delayed too long, leader will remove it. As for how long the delay is "too long", is determined by the parameter replica.lag.max.messages, how to be stuck, how is determined by the parameter replica.lag.time.max.ms.

Only when the message is added to the log by all copies is "committed", only the committed message is sent to consumer, so there is no need to worry that the message will be lost if leader down. Producer can also choose whether to wait for a message to be submitted for notification, which is determined by the parameter request.required.acks.
Kafka guarantees that as long as there is a "in sync" node, the "committed" message is not lost.

The choice of leader
The core of Kafka is log files, and the synchronization of log files in a cluster is the most basic element of distributed Data System.

If leaders never down, we don't need followers! Once the leader down, You need to select a new leader in the followers. But followers itself is likely to delay too long or crash, so must choose the high quality follower as leader. It must be ensured that once a message has been submitted, but leader Down, the newly elected leader must be able to provide the message. Most of the distributed systems use the majority voting law to choose the new leader, and for most voting rules, it is the choice of the most suitable as leader according to the condition of all replica nodes. Kafka does not use this method.

Kafaka dynamically maintains a set of copies of a synchronous state (a set of In-sync replicas), called an ISR, in which the nodes in this set are highly consistent with the leader, and any message must be read and appended to the log by each node in the collection. Only to notify the outside this message has been submitted. Therefore, any node in this set can be selected as leader at any time. The ISR is maintained in the zookeeper. With F+1 nodes in the ISR, it is possible to allow the F node to drop without losing the message and serving properly. The members of the ISR are dynamic, and if a node is eliminated, he can rejoin the ISR when it re-reaches the "in sync" state. This leader selection method is very fast and suitable for Kafka application scenarios.

An evil idea: what if all the nodes are down? Kafka the assurance that data will not be lost is based on the fact that at least one node is alive, and once all nodes are down, this is not guaranteed.
In practice, when all replicas are down, a response must be made in a timely manner. The following two options are available:

• Wait for any node in the ISR to recover and act as leader.
• Select all nodes (not just the ISR) of the first recovered node as leader.

This is a trade-off between usability and continuity. If you wait for the node in the ISR to recover, the cluster will never recover once the nodes in the ISR are not up or the data is. If you wait for an unexpected ISR node to recover, this node's data will be used as the data on the line, possibly in connection with the actual data, because some of the data may not be synchronized. Kafka currently chooses the second strategy, which will make the choice of this strategy configurable in future releases, and can be flexibly selected based on the scenario.
This dilemma is not only Kafka, but almost all distributed data systems will encounter.

Replica Management
The above is discussed simply as an example of a topic partition, But in fact a Kafka will manage thousands of topic partitions. Kafka try to make all the partitions evenly distributed to all nodes of the cluster instead of concentrating on some nodes, as well as the master-slave relationship as much as possible so that each of the points will serve a certain proportion of the leader of the partition.
It is also important to optimize the leader selection process, which determines how long the empty window period is when the system fails. Kafka Select a node as a "controller", when it is found that the node is down when it is responsible for all nodes in the swimming partition to select the new leader, which allows Kafka to efficiently manage the master-slave relationship of all the partition nodes in batch. If the controller is down, one of the surviving nodes will switch to the new controller.

7. Client API

A.kafka producer API

There are two types of Procuder APIs: Kafka.producer.SyncProducer and Kafka.producer.async.AsyncProducer. They all implement the same interface:

Class Producer {/* sends the message to the specified partition */publicvoid send (kafka.javaapi.producer.producerdata<k,v> producerdata);/* Bulk send a batch of messages */publicvoid send (java.util.list<kafka.javaapi.producer.producerdata<k,v>> ProducerData);/* Close producer */publicvoid close ();}

The Producer API provides the following features:

• Multiple messages can be cached to the local queue and then sent asynchronously to the broker, which can be done by parameter producer.type=async. The size of the cache can be specified by some parameters: Queue.time and Batch.size. A Background thread ( Kafka.producer.async.ProducerSendThread) Take the data out of the queue and have Kafka.producer.EventHandler send the message to the broker, or you can customize the handler via the Parameters Event.handler, The producer side handles different stages of data processing to register the processor, such as log tracking for this process, or some monitoring. Simply implement the Kafka.producer.async.CallbackHandler interface and configure it in the Callback.handler.
• Write your own encoder to serialize the message, just implement the following interface. The default encoder is Kafka.serializer.DefaultEncoder.
  
  • Interface Encoder<t> {
  • Public Message tomessage (T data);
  • }
• Provides zookeeper-based broker auto-sensing capability, which can be implemented by parameter zk.connect. If you do not use zookeeper, you can also use the broker.list parameter to specify a static brokers list so that the message is sent randomly to a broker, and once the selected broker fails, the message is sent.
• The partition function has two parameters: key and the number of available partitions, select a partition from the partition list and return the ID. The default partitioning policy is hash (key)%numpartitions. If key is null, select one randomly. You can customize the partition function with the parameter partitioner.class.
• Partition the message by using the partition function Kafka.producer.Partitioner class.
  
  • Interface Partitioner<t> {
  • int partition (T key, int numpartitions);
  • }

B.kafka Consumer API

There are two levels of the Consumer API. The lower level is kept connected with a specified broker and closes the connection after receiving the message, which is stateless, with offset for each read message.
The high-level API hides the details of the brokers connection and communicates with the server without having to care about the server-side architecture. You can also maintain your own consumption status, and you can specify subscription-specific topic, such as a whitelist blacklist or regular expression, by some criteria.

Low-level API

classSimpleconsumer {/*send a read request to a broker and get a message set*/ Publicbytebuffermessageset Fetch (fetchrequest request);/*send a read request to a broker and get a corresponding set*/ PublicMultifetchresponse Multifetch (list<fetchrequest>fetches);/*** The offsets* return value before the specified time is the offsets list, sorted in reverse order *@paramtime: times, milliseconds, * if specified as offsetrequest$. module$. Latiest_time (), get the latest offset.* if specified as offsetrequest$. module$. Earliest_time (), get the oldest offset.*/publiclong[] Getoffsetsbefore (String topic,intPartitionLongTimeintmaxnumoffsets);}

View Code

Low-level APIs are the basis for high-level API implementations and are for scenarios where there is a special need to maintain the consumption state, such as offline consumer such as Hadoop consumer.

High-level API

　

/*Create a connection*/consumerconnector connector=consumer.create (consumerconfig);InterfaceConsumerconnector {/*** This method can get a list of streams, each of which is an iteration of messageandmetadata, with Messageandmetadata can get messages and other meta data (currently topic) * INPUT:A Map of < Topic, #streams >* output:a Map of <topic, List of message streams>*/ PublicMap<string,list<kafkastream>> Createmessagestreams (map<string,int>topiccountmap);/*** You can also get a list of streams that contain an iteration of the message that conforms to Topicfiler, * a topicfilter is a regular expression that encapsulates a whitelist or blacklist. */ PublicList<kafkastream>Createmessagestreamsbyfilter (Topicfilter topicfilter,intnumstreams);/*to submit the current consumption to offset*/ Publiccommitoffsets ()/*Close Connection*/ Publicshutdown ()}

View Code

Kafka Getting Started