Kafka Main Configuration

Kafka provides a number of configuration parameters for Broker,producer and consumer. Understanding and understanding these configuration parameters is very important for us to use Kafka.

Official Address: Configuration

The configuration file server.properties in each Kafka broker must have the following properties configured by default:

1Broker.id=02port=90923num.network.threads=24Num.io.threads=85socket.send.buffer.bytes=10485766socket.receive.buffer.bytes=10485767socket.request.max.bytes=1048576008log.dirs=/tmp/kafka-logs9num.partitions=2Tenlog.retention.hours=168 Onelog.segment.bytes=536870912 Alog.retention.check.interval.ms=60000 -log.cleaner.enable=false   -zookeeper.connect=localhost:2181 thezookeeper.connection.timeout.ms=1000000

The following table lists some important configuration parameters, please refer to the official website for more configuration

Broker Configuration Parameters

Parameters	Default value	Describe
Broker.id	-1	Each boker has a unique ID as their name. When the IP address of the server changes, the broker.id does not change, it does not affect the consumers message condition
Port	9092	Broker Server service Port
Host.name	""	The host address of the broker, if set, is bound to this address, and if not, binds to all interfaces and sends one of them to ZK
Log.dirs	/tmp/kafka-logs	Kafka data storage address, multiple addresses are separated by commas, multiple directories distributed on different disks can improve read and write performance/data/kafka-logs-1,/data/kafka-logs-2
Message.max.bytes	1000012	Represents the maximum size of the message body, in bytes
Num.network.threads	3	The maximum number of threads that the broker processes messages, in general the number of CPU cores
Num.io.threads	8	Number of threads processing IO
Log.flush.interval.messages	Long.maxvalue	The maximum number of messages accumulated before the data is written to the hard disk and the consumer is available
log.flush.interval.ms	Long.maxvalue	The maximum time before the data is written to the hard disk
log.flush.scheduler.interval.ms	Long.maxvalue	Checks whether the data is to be written to the hard disk at a time interval.
Log.retention.hours	168 (24*7)	Control how long a log remains for an hour
Log.retention.bytes	-1	Control log file Maximum size
Log.cleaner.enable	False	Whether log cleaning
Log.cleanup.policy	Delete	Delete or Compat.
Log.segment.bytes	1073741824	Single Log segment file size
Log.roll.hours	168	The maximum time to start a new log file fragment
Background.threads	10	Background line Program
Num.partitions	1	Default number of partitions
Socket.send.buffer.bytes	102400	Socket So_sndbuff Parameters
Socket.receive.buffer.bytes	102400	Socket So_rcvbuff Parameters
Zookeeper.connect		Specifies the zookeeper connection string, formatted as Hostname:port/chroot. Chroot is a namespace
zookeeper.connection.timeout.ms	6000	Specify the maximum timeout period for client connection zookeeper
zookeeper.session.timeout.ms	6000	Session timeout time to connect ZK
zookeeper.sync.time.ms	2000	The longest time for ZK follower to lag behind ZK leader

Configuration parameters for high-level consumer

Parameters	Default value	Describe
GroupID	GroupID	A string that indicates the group to which a group of consumer
socket.timeout.ms	30000	Socket Timeout Time
Socket.buffersize	64*1024	Socket Receive Buffer
Fetch.size	300 * 1024	Controls the number of bytes of messages that are fetched in a request. This parameter is replaced by Fetch.message.max.bytes,fetch.min.bytes in 0.8.x.
backoff.increment.ms	1000	This parameter avoids repeating frequent pull data without new data. If you pull the empty data, you postpone the time
Queued.max.message.chunks	2	The consumer internal cache pulls back messages into a queue. This value controls the size of this queue
Auto.commit.enable	True	If True,consumer periodically writes offset for each partition to zookeeper
auto.commit.interval.ms	10000	How often to write offset on the zookeeper
Auto.offset.reset	Largest	If offset is returned, then smallest: automatically sets reset to the minimum offset. Largest: Automatically sets offset to the maximum offset. Other values are not allowed and an exception is thrown.
consumer.timeout.ms	-1	The default -1,consumer blocks indefinitely when no new messages are in the block. If you set a positive value, a timeout exception is thrown
Rebalance.retries.max	4	Maximum number of attempts when rebalance

Configuration parameters for Producer

Parameters	Default value	Describe
Producer.type	Sync	Specifies whether message sending is synchronous or asynchronous. Asynchronous ASYC batch send with Kafka.producer.AyncProducer, synchronous sync with Kafka.producer.SyncProducer
Metadata.broker.list	Boker List	Use this parameter to pass in Boker and partition static information, such as HOST1:PORT1,HOST2:PORT2, which can be part of all Boker
Compression.codec	Nocompressioncodec	Message compression, not compressed by default
Compressed.topics	Null	When compression is set, you can specify a specific topic compression, and all compression if unspecified
Message.send.max.retries	3	Maximum number of message send attempts
retry.backoff.ms	300	Each attempt to increase the extra interval of time
topic.metadata.refresh.interval.ms	600000	The time at which metadata is periodically fetched. When the partition is lost, leader is not available producer also proactively gets the metadata, if 0, the metadata is obtained each time the message is sent. Not recommended. If negative, the metadata is only obtained if the failure occurs.
queue.buffering.max.ms	5000	The maximum time for cached data in the producer queue, just for ASYC
Queue.buffering.max.message	10000	Producer The maximum number of cached messages, just for ASYC
queue.enqueue.timeout.ms	-1	0 when the queue is full, the negative value is the block when the queue is full, and the value is the time of the block when the queue is full, just for ASYC
Batch.num.messages	200	Number of messages, just for ASYC
Request.required.acks	0	0 means that the producer does not have to wait for leader confirmation, 1 represents the need to leader confirm that the local log is written to it and immediately confirms that-1 means that all backups are completed after confirmation. Just for Sync
request.timeout.ms	10000	Confirm Time-out

Kafka Main Configuration