Kafka (i)

Using Kafka latest Version 0.9

Kafka Configuration

1. Installation

First need to install Java, it is recommended to install JAVA8, otherwise there will be some inexplicable errors

Kafka_2.11-0.9.0.0.tgz

Tar-xzf kafka_2.11-0.9.0.0.tgz

For convenience, change the directory name

MV kafka_2.11-0.9.0.0.tgz Kafka

2. Configure Kafka service-side properties

Installed is a single node, the configuration of the cluster is very simple, you can look at other information

CD Config

Vim Server.properties

There are 2 key attributes that need to be modified

Advertised.host.name

Advertised.port

These 2 attributes need to be modified to the IP address or machine name, if the Kafka is deployed in the cloud, there may be other virtual IPs in the cloud, which need to be configured in these 2 addresses.

In this test environment, which is deployed in a virtual machine, connected by a route, you still need to configure these 2 properties to set Advertised.host.name to the IP address of the virtual machine.

If you do not configure Advertised.host.name, only configure Host.name, when you use the Java Client connection on another machine, it will be parsed to localhost.

3. Start the Kafka with the zookeeper

Bin/kafka-server-start.sh config/server.properties

4. Start Kafka

Bin/kafka-server-start.sh config/server.properties

Kafka Simple test

1. Create Topic

bin/kafka-topics.sh--create--zookeeper localhost:2181--replication-factor 1--partitions 1--topic test

2. View Topic

bin/kafka-topics.sh--list--zookeeper localhost:2181

3. Production News

Create a new console

bin/kafka-console-producer.sh--broker-list localhost:9092--topic test

Enter a string to return

4. Consumer News

Create a new console

bin/kafka-console-consumer.sh--zookeeper localhost:2181--topic test--from-beginning

In the console of the production message, continue typing the characters and enter, this will be consumed instantly

Kafka Java Client

1. maven File

<Dependency>            <groupId>Org.apache.kafka</groupId>            <Artifactid>Kafka-clients</Artifactid>            <version>0.9.0.0</version> </Dependency>

2. Produce code

Properties props =NewProperties (); Props.put ("Bootstrap.servers", "192.168.1.160:9092"); Props.put ("ACKs", "all"); Props.put ("Retries", 0); Props.put ("Batch.size", 16384); Props.put ("Linger.ms", 1); Props.put ("Buffer.memory", 33554432); Props.put ("Key.serializer", "Org.apache.kafka.common.serialization.StringSerializer"); Props.put ("Value.serializer", "Org.apache.kafka.common.serialization.StringSerializer"); Producer<string, string> producer =Neworg.apache.kafka.clients.producer.KafkaProducer (props);  for(inti = 0; I < 100; i++) Producer.send (NewProducerrecord<string, string> ("My-topic2", integer.tostring (i), integer.tostring (i));    Producer.close (); 

Bootstrap.servers is the address of Kafka broker

3. Consumer Code

Properties props =NewProperties (); Props.put ("Bootstrap.servers", "192.168.1.160:9092"); Props.put ("Group.id", "Test"); Props.put ("Enable.auto.commit", "true"); Props.put ("Auto.commit.interval.ms", "1000"); Props.put ("Session.timeout.ms", "30000"); Props.put ("Key.deserializer", "Org.apache.kafka.common.serialization.StringDeserializer"); Props.put ("Value.deserializer", "Org.apache.kafka.common.serialization.StringDeserializer"); Org.apache.kafka.clients.consumer.KafkaConsumer<string, string> consumer =NewOrg.apache.kafka.clients.consumer.kafkaconsumer<string, string>(props); //Consumer.subscribe (arrays.aslist ("foo", "Bar"));Consumer.subscribe (Arrays.aslist ("My-topic2"));  while(true) {consumerrecords<string, string> records = Consumer.poll (1000);  for(Consumerrecord<string, string>record:records) System.out.printf ("offset =%d, key =%s, value =%s", Record.offset (), Record.key (), Record.value ()); }

Consumer Group

Kafka Consumer can join a group, if the number of consumer in a group is greater than the partition number of the topic, then some consumer will not fetch the data. If the number of consumer is less than partition, some consumer will consume more of the interest than the other consumer .

If a consumer does not join any group, the consumer will consume all partition messages under the topic.

The best way to suggest a consumer corresponds to a partition

The Offset,consumer that stores each consumer consumer message in Kafka can choose Auto commit offset or manual commit

In the design, if there is a certain consistency in the consumer requirements, you can choose manual Commit, such as consumer downtime, restart or new consumer join, you can start from the partition offset record location to begin consumption, There is also a situation where the consumer is down, does not restart and no new consumer join, in 0.9.0. In version 0 of Kafka, there will be a consumer coordinator to handle consumer reblance, start reblance, Then the other consumer in the same group can also consume the partition message of the consumer subscription of the outage. But the premise of this design is that consumer must be designed to be stateless.

Messages passed by Kafka

Kafka passed the message is <key,value> reminds me of the map-reduce, using Kafka to quickly build a lightweight Mr Parallel analysis

Kafka's support for Hadoop is also very good.

Kafka (i)