Kafka Performance Tuning

main principles and ideas of optimization

Kafka is a highly-throughput distributed messaging system and provides persistence. Its high performance has two important features: the use of disk continuous read and write performance is much higher than the characteristics of random reading and writing, concurrency, a topic split into multiple partition.

To give full play to the performance of Kafka, these two conditions need to be met

Kafka read-write units are partition, so splitting a topic into multiple partition can increase throughput. However, there is a premise that different partition need to be located on different disks (can be on the same machine). If multiple partition are located on the same disk, it means that multiple processes simultaneously read and write multiple files of a disk, allowing the operating system to frequently schedule disk reads and writes, which destroys the continuity of disk read and write.

In the Linkedlin test, each machine is loaded with 6 disks, and does not raid, is to make full use of multiple disk concurrent read and write, but also ensure that each disk continuous read and write characteristics.

On a specific configuration, you configure multiple directories for different disks into broker's log.dirs, for example
Log.dirs=/disk1/kafka-logs,/disk2/kafka-logs,/disk3/kafka-logs
Kafka will distribute the new partition to the least partition directory when the new partition is created, so it is generally not possible to set multiple directories for the same disk to Log.dirs

The consumer and partition within the same consumergroup must be guaranteed to be one-to-one consumer relationships at the same time

Arbitrary partition can only be consumed by one consumer within a consumer group at one time (in turn, a consumer may consume multiple partition at the same time) JVM parameter configuration

It is recommended to use the latest G1 instead of CMS as a garbage collector.
The recommended minimum version is JDK 1.7u51. The following are the JVM memory configuration parameters for the broker in this trial:

-xms30g-xmx30g-xx:permsize=48m-xx:maxpermsize=48m-xx:+useg1gc-xx:maxgcpausemillis=20-xx:i Nitiatingheapoccupancypercent=35

G1 compared to the advantages of CMS: G1 is a server-side garbage collector that balances throughput and responsiveness differently for memory partitioning, Eden, Survivor, old areas are no longer fixed, and using memory is more efficient. The memory fragmentation problem is effectively avoided by dividing the memory region by G1. G1 can specify the time that a GC can be used to suspend a thread (not guaranteed to be strictly adhered to). And CMS does not provide controllable options. The CMS merges the compressed memory only after FULLGC, while G1 combines the collection and the merge. CMS can only be used in old areas, when young is generally used in conjunction with the use of parnew, and G1 can unify two types of partition of the recovery algorithm.

G1 's application scenario: The JVM consumes a large memory (at least 4G) application itself frequently requests, frees up memory, resulting in large amounts of memory fragmentation. For a more sensitive application of GC time.

JVM parameters Detailed: http://blog.csdn.net/lizhitao/article/details/44677659 broker Parameter Configuration

Configuration optimization is to modify the parameter values in the Server.properties file

1. Network and IO operation thread Configuration optimization

# The maximum number of threads that the broker handles messages 
 num.network.threads=xxx 

 # Broker Number of threads processing disk IO 
 

Recommended configuration:

The number of threads used to receive and process network requests, which defaults to 3. Its internal implementation is to adopt selector model. Start a thread as a acceptor to establish the connection, and then with the start num.network.threads threads to take turns to read requests from the sockets, generally do not need to change, unless the upstream and downstream concurrent request volume is too large. General Num.network.threads mainly deal with network IO, read and write buffer data, basically no IO waiting, the number of configured threads for the CPU core number plus 1.

Num.io.threads mainly disk IO operations, the peak may be some IO waiting, so the configuration needs to be larger. The number of configured threads is twice times the CPU kernel, and the maximum is no more than 3 times times.

2. Log data file brush disk strategy
In order to significantly improve the producer write throughput, regular batch writes are required.
Recommended configuration:

# every time producer write 10,000 messages, brush the data to disk log.flush.interval.messages=10000

# 1 seconds per interval, brush data to disk
log.flush.interval.ms= 1000

3. Log Retention policy configuration

When the Kafka server is written to a large number of messages, will generate a lot of data files, and occupy a lot of disk space, if not cleaned up in time, may not be enough disk space, Kafka default is to retain 7 days.
Recommended configuration:

# Keep three days, also can be shorter 
log.retention.hours=72

# section file configuration 1GB, facilitate the rapid recovery of disk space, restart Kafka load will also be accelerated (if the file is too small, the number of files is more,
# Kafka is started with all data files under the single-threaded Scan directory (log.dir)
log.segment.bytes=1073741824

Tips Kafka authorities do not recommend that the Log.flush.interval.messages and log.flush.interval.ms of the broker end be forced to write the disk, that the reliability of the data should be guaranteed through the replica, while forcing flush data to disk will create a shadow of the overall performance Ring.

You can tune performance by adjusting/proc/sys/vm/dirty_background_ratio and/proc/sys/vm/dirty_ratio. Dirty page rate more than the first indicator will start Pdflush start flush Dirty pagecache. A dirty page rate exceeding the second metric blocks all write operations for flush. According to different business requirements can be appropriate to reduce dirty_background_ratio and improve dirty_ratio.

If the amount of topic data is small , consider reducing log.flush.interval.ms and log.flush.interval.messages to force the brush to write data, reducing the likelihood of inconsistencies caused by the cached data not being written.

4. Configure JMX Services
The default in Kafka server is to not start the JMX port, requiring the user to configure

[Lizhitao@root kafka_2.10-0.8.1]$ vim bin/kafka-run-class.sh

#最前面添加一行
jmx_port=8060

5. Replica Related configuration:

replica.lag.time.max.ms:10000

replica.lag.max.messages:4000

num.replica.fetchers:1
# A number of fetch threads are started on the replica to synchronize the corresponding data locally, and num.replica.fetchers This parameter is used to control the number of fetch threads.
#每个Partition启动的多个Fetcher, sharing offset ensures a one-to-one relationship between consumer and partition at the same time, and allows us to increase efficiency by increasing the number of fetch threads.


default.replication.factor:1
#这个参数指新创建一个topic时, the default number of replica
#Replica过少会影响数据的可用性, too much will be wasted storage resources, General advice in 2~3 is advisable.

6. Purgatory

fetch.purgatory.purge.interval.requests:1000
producer.purgatory.purge.interval.requests:1000

Let's start by introducing what this "purgatory" is for. One of the main tasks of broker is to receive and process request from the network. Some of these request can be answered immediately, and it is natural that these request will be answered directly. In addition, there is no way or request spontaneous requests for delayed replies (such as batch sent and received), Broker will put this request into the paurgatory, At the same time, each request to join the Purgatory will be added to the two monitoring queue: Watcherfor queue: To check whether the request is satisfied. Delayedqueue queue: Used to detect whether the request timed out.

The final state of request is only one, complete. The request is met and the timeout will eventually be unified as complete.

There are some defects in the current version of Purgatory design. When the request state is converted to complete, it is not immediately removed from the purgatory, but continues to occupy resources, so the memory build-up will eventually cause oom. This is usually only triggered when the topic flow is low. More detailed information can be consulted on extended readings, which do not unfold here.

In the actual use I also stepped on this pit, when the situation is a cluster of new topic, the initial topic data very little (low volume topic), resulting in that period of time in the early morning 3, 4 points will randomly have broker because Oom hang up. The location of the reason to adjust the configuration of the *.purgatory.purge.interval.requests as small as 100 to solve the problem.

Kafka's research and development team has begun to redesign purgatory to enable request to be removed from the purgatory as soon as complete. Other

Num.partitions:1
#分区数量

queued.max.requests:500
#这个参数是指定用于缓存网络请求的队列的最大容量, this queue will not receive new requests until the upper limit is reached. Generally will not become a bottleneck, unless I/O performance is too poor, then need to coordinate with num.io.threads and other configurations to adjust together.

compression.codec:none
#Message落地时是否采用以及采用何种压缩算法. Generally is the producer sent over the message directly saved, no longer change the compression mode.

in.insync.replicas:1
#这个参数只能在topic层级配置, specify at least how many replica to confirm in the ISR at each producer write operation, Commonly used with request.required.acks. Note that this parameter can significantly degrade throughput if set too high.

producer Optimization

buffer.memory:33554432 (32m)
#在Producer端用来存放尚未发送出去的Message的缓冲区大小. After the buffer is full, you can choose to block send or throw an exception, as determined by the Block.on.buffer.full configuration.

Compression.type:none
#默认发送不进行压缩, the recommended configuration of a suitable compression algorithm, can significantly reduce network pressure and broker storage pressure.


linger.ms:0
#Producer默认会把两次发送时间间隔内收集到的所有Requests进行一次聚合然后再发送 to increase throughput, The linger.ms is further, and this parameter adds some delay to each send to aggregate more messages.

batch.size:16384
#Producer会尝试去把发往同一个Partition的多个Requests进行合并, batch.size indicates the upper limit of the total size of the requests after a batch merger. If this value is set too small, it may cause all of the request to be batch.

acks:1
#这个配置可以设定发送消息后是否需要Broker端返回确认.

0: No need to confirm, the fastest speed. There is a risk of losing data.
1: Only need to leader to confirm, do not need ISR to confirm. is a compromise of efficiency and security.
all: Require all replica in the ISR to give receive acknowledgement, slowest and highest security, but since the ISR may shrink to include only one replica, setting the parameter to all does not necessarily prevent data loss.

Consumer Optimization

Num.consumer.fetchers:1
#启动Consumer的个数, the appropriate increase can increase the degree of concurrency.


fetch.min.bytes:1
#每次Fetch request at least how many bytes of data to get back. The

maximum length of time allowed to wait #在Fetch request gets at least the data before it reaches Fetch.min.bytes. Corresponds to the timeout time requested in the purgatory mentioned above.
fetch.wait.max.ms:100

The consumer Group enables both producer consumers and queue access modes. The Consumer API is divided into two kinds of high and low level. The former a heavy dependence on zookeeper, so poor performance and not free, but super worry. The second does not depend on zookeeper services, which have better performance in terms of both freedom and performance, but all exceptions (leader migrations, offset crossings, broker outages, etc.) and maintenance of offset need to be handled on their own. We can pay attention to the next release 0.9. Developers have also rewritten a set of consumer in Java. Combine the two sets of APIs and eliminate the reliance on zookeeper. It is said that performance has greatly improved OH ~ ~ list of all parameter configurations

Broker default parameters and configurable list of all parameters:
http://blog.csdn.net/lizhitao/article/details/25667831

Kafka principle, basic concept, broker,producer,consumer,topic all parameter configuration list
http://blog.csdn.net/suifeng3051/article/details/48053965 Reference

Http://bbs.umeng.com/thread-12479-1-1.html
http://www.jasongj.com/2015/01/02/Kafka%E6%B7%B1%E5%BA%A6%E8%A7%A3%E6%9E%90/

Official documents:
Http://kafka.apache.org/documentation.html#configuration