Detailed Kafka producer producer configuration

Basic Configuration

Metadata.broker.list:broker server cluster list, formatted as Host1:port1, Host2:port2 ...

Producer.type: Message Send type synchronous or asynchronous, default to synchronous

COMPRESSION.CODEC: Compressed format of message, default is none uncompressed, can also be gzip, snappy, lz4

SERIALIZER.CLASS: Message encryption format, default is Kafka.serializer.DefaultEncoder

Compressed.topics: The compression format of the theme, can be ' gzip ', ' snappy ', lz4

Asynchronous producer Configuration

Queue.buffering.max.ms: Maximum time for producers to cache data asynchronously, in milliseconds

Queue.buffering.max.messages: Producer's maximum capacity for asynchronous cache messages

Queue.enqueue.timeout.ms: Queue Timeout event. 0, if the queue is full, give up,-ve, if the queue is full, it will block forever, +ve, if the queue is full, it will block a period of time.

Batch.num.messages: Number of messages that a producer can batch process

Common configuration for Java client consumers

Bootstrap.servers:broker server cluster list, formatted as HOST1:PORT1, Host2:port2

Key.serializer: Defines the serialized interface, which is recommended for Org.apache.kafka.common.serialization.StringSerializer

Value.serializer: Class that implements the serialization interface, recommended for Org.apache.kafka.common.serialization.StringSerializer

ACKs: configuration can set whether the broker-side return confirmation is required after sending a message

0: No need to confirm, the fastest speed. There is a risk of data loss.

1: Only need leader to confirm, do not need ISR to confirm. is a way of compromising efficiency and safety.

All: Requires all replica in the ISR to receive acknowledgement, the slowest and most secure, but since the ISR may shrink to include only one replica, setting the parameter to all does not necessarily prevent data loss.

Buffer.memory: The producer's slow village capacity, if the record is sent faster than the transmission to the server, either the producer is blocked, or the configured Block.on.buffer.full buffer is full. Default size is 32M

Compression.type: All data compression formats generated by the producer, not compressed by default, can also be gzip, snappy, or lz4

linger.ms: Producer The default is to increase throughput by aggregating and then sending all the requests collected during the two send interval, and the linger.ms is further, and this parameter aggregates more messages by adding some delay to each send.

This article is from the "This person's IT World" blog, be sure to keep this source http://favccxx.blog.51cto.com/2890523/1767882

Detailed Kafka producer producer configuration