Address: http://blog.csdn.net/honglei915/article/details/37564757
Kafka has made great efforts to improve efficiency. One of the main scenarios of Kafka is to process website activity logs. The throughput is very large and each page will produce many write operations. In terms of reading, it is assumed that each message is consumed only once, and the read volume is large. Kafka also tries its best to make read Operations lighter.
We have discussed the performance of the disk. In the case of linear read/write, there are two major problems affecting the disk performance: too many trivial I/O operations and too many bytes to copy. I/O problems occur between the client and the server, and also occur in persistent operations within the server.
Message Set)
To avoid these problems, Kafka establishes the concept of "Message Set" and organizes messages together as the processing unit. Processing a message in a message set improves the performance of a message set. The producer sends a message set to the server, instead of sending it one by one. The server appends the message set to the log file at a time, which reduces the trivial I/O operations. Consumer can also request a message set at a time.
Another performance optimization is in the aspect of byte copy. This is not a problem in the case of low load, but it still has a great impact in the case of high load. To avoid this problem, Kafka uses a standard binary message format that can be shared between producers, brokers, and producers without any changes.
Zero copy
The message log maintained by the broker is only some directory files, and the message set is written to the log file in a fixed queue format. This format is shared by producer and consumer, this allows Kafka to optimize the transmission of messages over the network. The modern Unix operating system provides a high-performance system function that sends data from the page cache to the socket. in Linux, this function is sendfile.
To better understand the advantages of sendfile, Let's first look at the data flow from the file to the socket:
- The operating system copies data from the file to the page cache in the kernel.
- Applications copy data from the page cache in their memory cache.
- The application writes data to the socket cache in the kernel.
- The operating system copies data from the socket cache to the NIC interface cache and sends the data to the network.
This is obviously inefficient, with four copies and two system calls. Sendfile directly caches data from the page cache and sends it to the NIC interface, which avoids duplicate copies and greatly optimizes the performance.
In a multi-consumers scenario, data is only copied to the page and cached once rather than repeatedly copied every time messages are consumed. This allows messages to be sent at a rate close to the network bandwidth. In this way, you can hardly see any read operations on the disk level, because data is directly sent to the network from the page cache.
This article details the application of sendfile and zero-copy in Java.
Data Compression
In many cases, the performance bottleneck is not the CPU or hard disk, but the network bandwidth. This is especially true for applications that need to transmit large amounts of data between data centers. Of course, users can compress their messages without Kafka support, but this will lead to a lower compression rate, because compared to compressing messages separately, only by compressing a large number of files can the best compression effect be achieved.
Kafka adopts end-to-end compression: Due to the concept of "Message Set", client messages can be compressed together and then sent to the server, and written to the log file in the compressed format, A compressed message is sent to the consumer. The message is compressed from the producer to the consumer. It is decompressed only when the consumer is used. Therefore, it is called "end-to-end compression ".
Kafka supports gzip and snappy compression protocols. For more details, see here.
