Kafka Design Analysis (v)-Kafka performance test method and benchmark report

Summary

This paper mainly introduces how to use Kafka's own performance test script and Kafka Manager to test Kafka performance, and how to use Kafka Manager to monitor Kafka's working status, and finally gives the Kafka performance test report.

Performance testing and cluster monitoring tools

Kafka provides a number of useful tools, such as Kafka Design Parsing (c)-Kafka high Availability (below) mentioned in the operation and maintenance tools--partition reassign tool,preferred Replica Leader Election Tool,replica verification tool,state Change Log Merge Tool. This article describes the performance testing tools provided by Kafka, metrics reporting tools, and Yahoo! Open source Kafka Manager.

Kafka Performance Test Script

• $KAFKA_HOME/bin/kafka-producer-perf-test.shThe script was designed to test the performance of the Kafka producer, mainly outputting 4 metrics, sending a total of messages in megabytes (MB), sending messages per second (Mb/second), sending a total of messages, and sending messages per second (Records/second). In addition to outputting test results to standard output, the script also provides CSV Reporter, which stores the results as a CSV file for easy use in other analysis tools
• $KAFKA_HOME/bin/kafka-consumer-perf-test.shThe script is used to test the performance of the Kafka consumer, and the test metrics are the same as the producer performance test script

Kafka Metrics

Kafka uses Yammer metrics to report metric information for both the server and the client. Yammer Metrics 3.1.0 offers 6 forms of Metrics collection--meters,gauges,counters,histograms,timers,health Checks. At the same time, Yammer metrics separates the collection of metric from reports (or releases) and can be freely combined as needed. Currently it supports Reporter with console reporter,jmx reporter,http reporter,csv reporter,slf4j reporter,ganglia reporter,graphite Reporter. Therefore, Kafka also supports the output of its metrics information through the above reporter.

Use Jconsole to view a single server metrics

using jconsole through JMX is without installing other tools (since the Kafka is already installed, you must have Java installed, And Jconsole is one of the simplest and most convenient ways to view Kafka server metrics in the case of a Java-brought tool). The
must first enable Kafka's JMX Reporter by setting a valid value for the environment variable jmx_port. such as export jmx_port=19797 . You can then use Jconsole to access a Kafka server by using the port set above to view its metrics information, as shown in.

One advantage of using jconsole is that there is no need to install additional tools, the shortcomings are obvious, the data display is not intuitive, the data organization is not friendly, more importantly, can not simultaneously monitor the entire cluster of metrics. In, under Kafka.cluster->partition->underreplicated->topic4, only 2 and 52 nodes, this is not because TOPIC4 only these two Partition data are in the replicated state. In fact, TOPIC4 has only these 2 partition on the broker, and the other partition on the other broker, so the JMX reporter through the server only sees the two partition.

View the metrics of the entire cluster through Kafka Manager

Kafka Manager is an open source Kafka management tool for Yahoo. It supports the following functions

• Manage multiple clusters
• easy to view cluster status
• Execute preferred replica election
• Bulk build and execute partition allocation scheme for multiple topic
• Create topic
• Delete topic (supports only 0.8.2 and above, and requires delete.topic.enable set to True"
• Add partition
• topic
• supports adding and viewing Logkafka

After installing Kafka Manager, it is convenient to add cluster, simply indicate the zookeeper list used by the cluster and indicate the Kafka version, as shown in.

Note here that adding Cluster here means adding an existing Kafka cluster into the monitoring list rather than deploying a new Kafka Cluster through Kafka Manager, unlike Cloudera Manager.

Kafka Benchmark

A core feature of Kafka is the high throughput rate, so the test in this paper is focused on the throughput rate of Kafka.
The test in this paper uses 6 virtual machines with red Hat 6.6 installed, 3 as broker, and 3 as producer or consumer. Each virtual machine is configured as follows

• Cpu:8 Vcpus, Intel (R) Xeon (r) CPU e5-2680 v2 @ 2.80ghz,2 sockets,4 cores per socket,1 Thread per core
• Memory: Up to GB
• Disk: Up to GB

Turn on Kafka JMX reporter and use 19797 ports to monitor throughput rates during performance testing with the JMX polling capabilities of Kafka-manager.

This article mainly tests the following four scenarios, the test is mainly the number of megabytes per second of data, how many messages per second.

Producer only

This set of tests does not use any consumer, only the broker and producer are started.

Producer number VS. Throughput

Experimental conditions: 3 broker,1 topic,6 partition, no replication, asynchronous mode, message payload 100 bytes
Test Project: Test throughput for each producer
Test objectives: such as Kafka design Analysis (a)-Kafka background and architecture introduction, multiple producer can send data to the same topic at the same time, before the broker load saturation, the more theoretically producer the number of messages received per second, the larger the cluster, and linearly increased. This experiment mainly verifies the characteristic. As a performance test, the experiment also monitors the CPU and memory usage of a single broker during testing.
Test Result: The total throughput rate when using different number of producer as shown

As can be seen, a single producer per second can successfully send about 1.28 million payload to 100 bytes of messages, and with the increase in the number of producer, the total amount of messages per second increased linearly, in line with the previous analysis.

During performance testing, the broker's CPU and memory usage are as shown.

It is known that, in the case of receiving about 1.17 million messages per second (3 producer a total of 3.5 million messages per second, average per broker receives about 1.17 million), the CPU usage of a broker is approximately 248% and memory usage is 601 MB.

Message Size VS. Throughput

Experimental conditions: 3 broker,1 topic,6 partition, no replication, asynchronous mode, 3 x producer
Test Project: Test the total cluster throughput when the message length is 10,20,40,60,80,100,150,200,400,800,1000,2000,5000,10000 bytes, respectively
Test Result: Total cluster throughput at different message lengths as shown

It is known that the longer the message, the fewer messages can be sent per second, and the greater the amount of messages (MB) that can be sent per second. In addition, each message, in addition to payload, also contains other metadata, so the amount of messages sent per second than the number of messages sent per second multiplied by 100 bytes, and the larger the payload, the smaller the metadata, the larger the volume of messages sent at the same time, The easier it is to get a higher message volume per second (MB/s). The payload used in other tests is 100 bytes, and the reason for using this short message (relatively short) is only to test the Kafka throughput rate in the case of a relatively poor comparison.

Partition number VS. Throughput

experimental Condition: 3 broker,1 topic, no replication, asynchronous mode, 3 producer, message payload 100 bytes
Test Project: Test throughput at 1 to 9 partition respectively
Test results: Total cluster throughput at different partition quantities as shown in

As we know, when the number of partition is smaller than the number of brokers (3), the greater the number of partition, the higher the throughput rate, and the linear increase. In all experiments in this article, only 3 brokers are started, and a partition can only exist on 1 brokers (regardless of replication. Even if there is replication, only its leader accepts read and write requests, so when a topic contains only 1 partition, only 1 brokers actually work for that topic. As mentioned in the previous article, Kafka will distribute all partition evenly across all brokers, so when there are only 2 partition, there will be 2 brokers for that topic service. 3 Partition the same time there will be 3 brokers for the topic service. In other words, when the number of partition is less than or equal to 3, the more partition represents the more broker for the topic service. As mentioned in the previous articles, the data on different brokers is inserted in parallel, which explains that when the number of partition is less than or equal to 3, the throughput rate increases linearly with the number of partition.
When the number of partition is more than the broker number, the total throughput has not improved or even decreased. The possible reason is that when the number of partition is 4 and 5 o'clock, the number of partition on different brokers is different, and producer sends the data evenly to each partition, which results in different load on each broker and cannot maximize cluster throughput. This is confirmed by the fact that the throughput is significantly higher than in other cases when the number of partition is an integer multiple of the broker number.

Replica number VS. Throughput

Experimental conditions: 3 broker,1 topic,6 partition, asynchronous mode, 3 producer, message payload 100 bytes
Test Project: Throughput rates for 1 to 3 replica respectively
Test results: as shown

As the number of replica increases, the throughput rate decreases. However, the throughput rate decreases and the nonlinearity decreases because multiple follower data replication is performed in parallel rather than serially.

　　

Consumer only

Experimental conditions: 3 broker,1 topic,6 partition, no replication, asynchronous mode, message payload 100 bytes
Test Project: Cluster total throughput rate for 1 to 3 consumer respectively
Test result: With a large number of messages in the cluster, the total cluster throughput when using 1 to 3 consumer as shown in

It is known that a single consumer can consume 3.06 million messages per second, which is much larger than the number of messages a single producer can consume per second, which ensures that messages can be processed in a reasonable configuration. And as the number of consumer increases, the total throughput of the cluster increases linearly.
According to the Kafka Design Analysis (four)-Kafka consumer design analysis, the multi-consumer consumption message partition as the allocation unit, when there are only 1 consumer, the consumer need to pull messages from 6 partition at the same time, The I/O of the consumer machine is the bottleneck of the whole consuming process, and when the number of consumer is increased to 2 to 3, multiple consumer pull messages from the cluster at the same time, taking full advantage of the throughput rate of the cluster.

Producer Consumer Pair

Experimental conditions: 3 broker,1 topic,6 partition, no replication, asynchronous mode, message payload 100 bytes
Test Project: Test the amount of messages consumer can consume when working with 1 producer and a consumer at the same time
Test Result: 1,215,613 Records/second

Kafka Design Analysis (v)-Kafka performance test method and benchmark report