Kafka [system configuration instructions]-server. Properties

Generally, the configuration of Kafka is as follows:

broker.id=0num.network.threads=2num.io.threads=8socket.send.buffer.bytes=1048576socket.receive.buffer.bytes=1048576socket.request.max.bytes=104857600log.dirs=/tmp/kafka-logsnum.partitions=2log.retention.hours=168log.segment.bytes=536870912log.retention.check.interval.ms=60000log.cleaner.enable=falsezookeeper.connect=localhost:2181zookeeper.connection.timeout.ms=1000000

All configuration parameter descriptions (explanations) in server. properties are listed as follows: parameter descriptions (explanations) Broker. ID = 0 each broker is uniquely represented in the cluster, and must be a positive number. When the IP address of the server changes, the broker. the message status log of consumers is not affected if the ID is not changed. dirs =/data/Kafka-logskafka data storage address, multiple addresses separated by commas/data/kafka-logs-1,/data/kafka-logs-2port = 9092 Broker Server Service port message. max. bytes = 6525000 indicates the maximum size of the message body. The unit is byte num. network. threads = the maximum number of threads for 4 brokers to process messages. Generally, you do not need to modify num. io. threads = the number of threads that 8 brokers process disk Io. The value should be greater than the number of hard disks. background. threads = 4 Number of threads processed by some background tasks, such as deletion of expired message files. In general, you do not need to modify queued. max. requests = 500 maximum number of request queues waiting for Io thread processing. If If the number of requests waiting for Io exceeds this value, it will stop accepting external messages. This is a self-protection mechanism. Host. if the host address of namebroker is set, it will be bound to this address. If not, it will be bound to all interfaces and one of them will be sent to ZK. Generally, no socket is set. send. buffer. bytes = 100 * 1024socket sending buffer, socket Tuning Parameter accept = 100 * 1024socket receiving buffer, socket Tuning Parameter so_rcvbuffsocket.request.max.bytes = 100*1024 * 1024socket maximum value of the request, prevents serveroom and message. max. bytes must be smaller than socket. request. max. bytes will be overwritten by the specified parameter at topic creation. segment. bytes = 1024*1024 * 1024topic partitions are stored in a pile of segment files. This control The size of each segment is overwritten by the specified parameter during topic creation. roll. hours = 24*7. segment. the size set by bytes will also force a new segment to be overwritten by the specified parameter during topic creation. cleanup. policy = delete: the log cleanup policies include: delete and compact are mainly used to process expired data, or the log file quota has reached the limit, which will be overwritten by the specified parameter during topic creation. retention. minutes = 3days the maximum data storage time exceeds this time according to log. cleanup. the policy set by the policy processes data, that is, how long the consumer can consume the data log. retention. bytes and log. retention. any minutes that meets the requirements will be deleted and will be overwritten by the specified parameter during topic creation. retention. byte S =-1 topic: Maximum file size of each partition. the size limit of one topic = number of partitions * log. Retention. bytes. -1 no log size limit. retention. bytes and log. retention. any minutes that meets the requirements will be deleted and will be overwritten by the specified parameter during topic creation. retention. check. interval. ms = 5minutes file size check cycle time, whether to punish log. cleanup. policy log. cleaner. enable = false whether to enable log compression. cleaner. threads = 2 log compression run the number of threads log. cleaner. io. max. bytes. per. second = the maximum size of logs processed during none log compression. cleaner. dedupe. buffer. size = 500*1024*1024 the cache space when logs are compressed and deduplicated. The larger the space, the better the log. cleaner. io. buffer. size = 512*1024 the size of the IO Block Used for log cleanup Generally, you do not need to modify the log. cleaner. io. buffer. load. factor = 0.9 the expansion factor of the hash table in log cleanup does not need to be modified. cleaner. backoff. ms = 15000 check whether Penalty Log cleanup interval log. cleaner. min. cleanable. ratio = 0.5 Frequency Control of log cleanup. A larger value means more efficient cleanup. There will also be some space waste and will be overwritten by the specified parameter during topic creation. cleaner. delete. retention. ms = 1 day indicates the maximum retention time of the compressed logs, which is also the maximum time for the client to consume messages. retention. the difference between minutes is that it controls uncompressed data and compressed data. Will be overwritten by the specified parameter at topic creation. index. size. max. bytes = 10*1024*1024 the size limit on the index file of the segment log will be overwritten by the specified parameter during topic creation. index. interval. bytes = 4096 after a fetch operation is executed, a certain amount of space is required to scan for the latest offset. A larger setting means faster scanning but better memory, generally, this parameter log is not required. flush. interval. messages = nonelog file "sync" to the number of messages accumulated before the disk, because disk Io operations are a slow operation, but also a necessary means of "data reliability, therefore, we need to make necessary trade-offs between "data reliability" and "performance. if this value is too large, it will lead to a long time (IO blocking) for each "fsync" operation. If this value is too small, it will lead to a large number of "fsync" operations, this also means that the overall client request has a certain latency. if the physical server fails, messages without fsync will be lost. log. flush. scheduler. interval. ms = 3000 check whether the time interval log needs to be solidified to the hard disk. flush. interval. ms = none it is insufficient to control the disk write time of the message only through interval. this parameter is used to control the interval of "fsync". If the message volume does not reach the threshold but the interval from the last Disk Synchronization reaches the threshold, it will also be triggered. log. delete. delay. ms = 60000 files are retained after being cleared in the index. Generally, you do not need to modify the log. flush. offset. checkpoint. interval. ms = 60000 control the time point of the last Hard Disk curing, so that you do not need to modify auto for data recovery. create. topics. enable = true: whether to allow automatic topic creation. If it is false, you need to use the command to create topicdefault. replication. whether to allow automatic topic creation. If it is false, you need to create topicnum by using the command. partitions = 1. The number of partitions for each topic. If this parameter is not specified during topic creation, it will be overwritten by the specified parameter during topic creation. The following is the leader in Kafka. The replicas configuration parameter is controller. socket. timeout. ms = 30000 when partition leader communicates with replicas, the socket timeout time controller. message. queue. size = 10 partition leader when synchronizing data with replicas, the queue size of the message is replica. lag. time. max. ms = explain replicas the longest wait time for the response partition leader. If this time is exceeded, replicas will be included in ISR (in-sync replicas) and considered dead, no more replica. lag. max. messages = 4000 if the follower lags behind the leader too much, it will be considered that the follower [or partition relicas] has expired # generally, when the follower communicates with the leader, because of network delay or disconnection, it will always lead to delayed message synchronization in replicas # if there are too many messages after the message, the leader will think that the follower network latency is large or the message throughput is limited, this replicas will be migrated # To other follower. # We recommend that you increase the number of brokers in an environment where the number of brokers is small or the network is insufficient. replica. socket. timeout. ms = 30 * 1000follower and leader socket timeout time replica. socket. receive. buffer. bytes = 64 * 1024leader: the socket cache size during replication replica. fetch. max. bytes = 1024 * 1024replicas the maximum size of data retrieved each time. fetch. wait. max. ms = 500replicas maximum waiting time for communication with the leader. If it fails, replica will be retried. fetch. min. bytes = the minimum data size of 1fetch. If the number of data that has not been synchronized in the leader is insufficient, it will be blocked until the conditional num is met. replica. fetchers = 1leader: Number of threads for replication. Increasing this value will increase the ioreplica of follower. high. watermark. checkpoint. interval. ms = 5000 each replica check determines whether the maximum water level is solidified at a frequency controlled. shutdown. enable = false: whether to allow the Controller to disable the broker. If it is set to true, all leaders on the broker will be closed and transferred to other brokercontrolled. shutdown. max. retries = 3 the number of attempts the controller has disabled controlled. shutdown. retry. backoff. ms = 5000 the time interval between each close attempt leader. imbalance. per. broker. percentage = 10leader unbalanced ratio. If this value is exceeded, the partition will be re-balanced. imbalance. check. interval. seconds = 300 check whether the leader is unbalanced at the offset interval. metadata. max. bytes client retains the maximum space size of the Offset information zookeeper parameter configuration in Kafka. connect = localhost: 2181zookeeper cluster address, which can be multiple. Use commas to separate hostname1: port1, hostname2: port2, hostname3: port3zookeeper. session. timeout. the maximum timeout time for Ms = 6000zookeeper is the heartbeat interval. If it is not reflected, it is considered dead and not too large zookeeper. connection. timeout. ms = 6000zookeeper connection timeout zookeeper. sync. time. ms = 2000zookeeper the actual synchronization between leader and follower in the Cluster

Kafka [system configuration instructions]-server. Properties