Kafka Notes (i)

Each partition is a commit log and has an offset. The topic in the queue can be configured with an expiration time,

The consumer controls the position of the read Offset,consumer between the non-affected. Partitions are distributed, all requests are handled by leader, and if the number of replicas is configured, there is a corresponding follwer.

Consumer group: Message Queuing and broadcast are implemented.

Topic will be sent to one of the consumer instances of each consumer group.

Compared with the traditional publish-subscribe mechanism, the Kafka subscribe becomes a cluster.

Kafka the order in which messages are guaranteed. Each partition is consumed by only one consumer of the consumer group, guaranteeing the order.

Application fields:

Messages, site behavior tracking, metrics, log aggregation, stream processing, event sources, commit logs.

Environment configuration:

1, download the installation package, unzip.

2, edit the relevant configuration file, start ZK:

Bin/zookeeper-server-start.shconfig/zookeeper.properties

Start Kafkaserver:

Bin/kafka-server-start.shconfig/server.properties

3. Create a theme:

Bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--Topictest

Themes can be created manually, or they can be configured for automatic broker creation.

4. Test:

Bin/kafka-console-producer.sh--broker-list localhost:9092--topic Test

Bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic Test--from-beginning

5. Configure the cluster:

Copy server.properties, modify, use it to start kafkaserver. Can be simulated in a single machine.

Manual write producer test:

Start Kafkaserver

Start Consumer:kafka-console-consumer.sh--zookeeper xxx:2181--topic page_visits

Java-cp./testkafka-0.0.1-snapshot-jar-with-dependencies.jar Com.cuirong.kafka.test.TestProducer 10

(Maven executes assembly package all jar packages)

Manual Write Consumer test:

Most applications use the Advanced API, which is a low-level API when it comes to setting an initial offset when restarting consumer.

The advanced API hides a lot of detail, storing zookeeper the last offset of the specified partition, which is based on the consumer group.

The group name is global, and the consumer of the two group with the same name and different business logic runs at the same time.

multithreaded mode, if each thread processes multiple partitions, cannot guarantee the order.

Consumer Advanced API:

JAVA-CP./testkafka-0.0.1-snapshot-jar-with-dependencies.jarcom.cuirong.kafka.test.consumergroupexample xxx : 2181group1 page_visits 2

Consumer low-level API:

Demand:

1. Read a message multiple times

2. Read some data of a certain topic partition

3. The Management Service guarantees that messages are processed only once

Things to do:

1, control the offset in detail.

2. Specify the lead broker for the topic partition.

3. Switch broker manually.

Steps:

1, find out the activity of the broker, corresponding to the topic partition of the leader broker

2. Decide to copy the broker.

3. Data request definition

4. Access to Data

5, leader changes when the operation.

Java-cp./testkafka-0.0.1-snapshot-jar-with-dependencies.jarcom.cuirong.kafka.test.simpleexample 10page_visits 0 0 9092