Kafka note Two topic operation, file parameter configuration _kafka

The following example I only started with a shb01, did not add 139

The general operation of the theme topic (Add a check), through the script kafka-topics.sh to execute

Create

[Root@shb01 bin]# kafka-topics.sh--create--topic Hello--zookeeper shb01:2181--partition 2--replication-factor 1

Created topic "Hello".

--partition 2 means partition

--replication-factor 1 represents the replica factor, previously said that the broker has no primary vice, but the partition has the primary vice, so the value of this parameter cannot be greater than the broker number is the number of Kafka service nodes. In my case, there is only one shb01 node, so if you specify 2, it will be an error.

Inquire

[Root@shb01 bin]# kafka-topics.sh--list--zookeeper shb01:2181

Hello

You can also specify topic

[Root@shb01 bin]# kafka-topics.sh--list--zookeepershb01:2181--topic Hello

Query Specifies topic details

[Root@shb01 bin]# kafka-topics.sh--describe--zookeeper shb01:2181--topic Hello

Topic:hello partitioncount:2 replicationfactor:1 configs:

Topic:hello partition:0 leader:0 replicas:0 isr:0

Topic:hello partition:1 leader:0 replicas:0 isr:0

I only have one broker,id for 0.

ID of the broker where the leader:0,partition Leader

Replicas:0,partition copy of the Brokerlist

Isr:0, representing the ID of the available broker

Query topic details

[Root@shb01 bin]# kafka-topics.sh--describe--zookeeper shb01:2181

Modified, can only increase the number of partition can not be reduced

[Root@shb01 bin]# kafka-topics.sh--alter--zookeeper shb01:2181---topic Hello--partition 3

[Root@shb01 bin]# kafka-topics.sh--describe--zookeeper shb01:2181--topic Hello

Topic:hello partitioncount:3 replicationfactor:1 configs:

Topic:hello partition:0 leader:0 replicas:0 isr:0

Topic:hello partition:1 leader:0 replicas:0 isr:0

Topic:hello Partition:2 leader:0 replicas:0 isr:0

Delete

[Root@shb01 bin]# kafka-topics.sh--delete--zookeeper shb01:2181--topic Hello

Delete is just tag deletion, this topic is still available and you need to modify the parameters in the Server.properties file

Delete.topic.enable=true Open Delete and restart Kafka, this parameter is not in the file by default is required to manually join

Producers, consumers

Kafka's own producers and consumers are only used to test

Producers

[Root@shb01 bin]# kafka-console-producer.sh--broker-list shb01:9092--topic topic_shb01

--broker-list represents a broker address and multiple addresses are separated by commas

--topic says to produce messages to that subject.

Consumers

[Root@shb01 bin]# kafka-console-consumer.sh--topic topic_shb01--zookeeper shb01:2181,192.168.79.139:2181-- From-beginning

--topic said that the theme of consumption

Topic\whitelist\blacklist:

1, the specific individual topic

[Root@shb01 bin]# kafka-console-consumer.sh--topic topic_shb01--zookeeper shb01:2181,192.168.79.139:2181-- From-beginning

2, multiple white list topic string [Comma separated].

--topic T1,t2 says consuming only t1,t2 themes

[root@shb01bin]# kafka-console-consumer.sh--whitelist t1,t2--zookeepershb01:2181,192.168.79.139:2181-- From-beginning

3, multiple blacklist topic string [Comma separated].

--topic T1,t2 says to consume all themes except the T1,t2 theme

[root@shb01bin]# kafka-console-consumer.sh--blacklist t1,t2--zookeepershb01:2181,192.168.79.139:2181-- From-beginning

--zookeeper represents the ZK address of the Kafka cluster

--from-beginning said that in the past, the start of the consumer production before the message can also be consumed

The last subject marked for deletion can also be consumed

File parameter configuration

Broker,server.propertie

1. Producer production sends a message that the broker cache data reaches a certain threshold or a certain amount of time will flush to disk so that the IO call can be reduced. There are two parameters in the server.properties in the Config directory to control the flush

# The number of messages to accept beforeforcing a flush of data to disk

#log. flush.interval.messages=10000

# The maximum amount of time a message cansit in a log before we force a flush

#log. flush.interval.ms=1000

2.kafka data will still exist in the disk after consumption, the consumer with zookeeper read data, usually the data will be stored for 168 hours, can be controlled by parameters in Server.properties

# The minimum age of a log file to beeligible for deletion

log.retention.hours=168

#log. retention.bytes=1073741824

The first one is 168 hours, and the second parameter is deleted by size

log.retention.check.interval.ms=300000

This is the interval between scans, to meet the above two conditions (looks like two select one) to perform the deletion

Producer, Producer.properties

1.Producer You can specify that messages are sent to specific partition. Specify a custom partition in the configuration file by Partitioner.class. Contains the package name Class name

Partitioner.class= Kafka. Producertest

2. Control the producer to send data asynchronously or synchronously using the Producer.type=sync parameter.

Asynchronous or synchronous send

Synchronization refers to the way in which the next data is sent after the sender sends the data and the receiver sends back the response.

Asynchronous refers to the way in which the sender sends the data, not the receiver sends back the response, and then the next data is communicated.

3. The producer uses the asynchronous method to increase the efficiency of the transmission, first caches the data on the disk and sends it out at once.

There are two parameters to control when to send a message, one at a time interval by another according to the size of the data

# maximum time, in milliseconds, forbuffering data on the producer queue

#queue. buffering.max.ms=

# The maximum size of the blocking queuefor buffering on the producer

#queue. buffering.max.messages=

Consumer, Consumer.properties

Each consumer in the Kafka belongs to a group and has a set ID

Consumption data

Group: A consumer can consume multiple partition, but a topic partition can only be consumed by a single consumer in a group. There are 4 consumers in group2 in the figure below but Topic2 only have 3 partitions at this time C4 is idle. At this time can add or subtract Topic2 partition number to increase the consumption capacity.

Between groups: consumption of the same data between each consumer group does not affect each other.

Business complexity increases topic, data volume increases the partition greatly. The more zoning means the more consumer the consuming power.

The message is the basic unit of communication stored in the topic partition, parttion at the storage level as a Append.log file, and the new messages are appended directly to the end of the file, each of which is in the location of the file called offset (offset) This information is stored in the zookeeper, so the consumer will need ZK coordination to read the message. The message has a type size offset of 3 attributes, where offset corresponds to the ID of the message in the partition

The following is a command to view offset, which I found in the data, not actually manipulated.

Bin/kafka-run-class.shkafka.tools.consumeroffsetchecker--zookeeper Zkserver1:2181,zkserver2:2181--group Consumer _demo_11

Kafka cannot specify offset for consumption Kafka cluster fault tolerance

After a broker hangs in the Kafka cluster, ZK will choose to provide services to other broker, and when the broker is repaired, ZK will have an automatic equalization strategy to make this broker again the leader of partition in topic.

Topic:hello partition:0 leader:0

Topic:hello Partition:1 Leader:2

Topic:hello Partition:2 leader:1

This parameter specifies an auto-equalization policy, and Kafka performs a balanced operation after 300 seconds

Auto.leader.rebalance.enable=true

Leader.imbalance.check.interval.seconds Default value: 300.

You can also manually perform

Bin/kafka-preferred-replica-election.sh--zookeeper zkserver1:2181,zkserver2:2181