Apache Kafka Client Development Demo

the blog is reproduced from: http://www.aboutyun.com/thread-9906-1-1.html

1. Dependency Packs <dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.1</version>
</dependency> Copy Code

2.producer Program Development Example
2.1 Producer Parameter Description
#指定kafka节点列表, for getting metadata, without having to specify all
metadata.broker.list=192.168.2.105:9092,192.168.2.106:9092
# Specifies the partition processing class. Default Kafka.producer.DefaultPartitioner, the table is hashed to the corresponding partition by key
#partitioner. Class=com.meituan.mafka.client.producer.customizepartitioner

# compression, default 0 means no compression, 1 for gzip compression, 2 for snappy compression. There will be headers in the compressed message to indicate the message compression type, so the message decompression is transparent at the consumer end without specifying.
Compression.codec=none

# Specifies the serialization processing class (Mafka Client API call Description-->3. Serialization convention wiki), default to Kafka.serializer.DefaultEncoder, or byte[]
Serializer.class=com.meituan.mafka.client.codec.mafkamessageencoder
# Serializer.class=kafka.serializer.defaultencoder
# Serializer.class=kafka.serializer.stringencoder
# If you want to compress the message, specify which topic to compress the message, the default empty, which means no compression.
#compressed. topics=

########### Request ACK ###############
# Producer The time to receive the message ACK. The default is 0.
# 0:producer will not wait for broker to send ACK
# 1: Send an ACK when leader receives a message
# 2: Send an ACK when all the follower sync the message successfully.
Request.required.acks=0
# The maximum time that broker allows to wait before sending an ACK to producer
# If the timeout occurs, broker will send an error ACK to producer. means the last message because of some kind of
# reason failed (e.g. follower failed to sync successfully)
request.timeout.ms=10000
########## End #####################


# synchronous or asynchronous Send message, default "Sync" Table synchronization, "async" table asynchronous. Asynchronous can increase delivery throughput,
# also means that the message will be in local buffer and sent in batches in a timely manner, but may also result in the loss of messages that are not sent past
Producer.type=sync
############## asynchronous forwarding (optional for the following four asynchronous parameters) ####################
# in Async mode, when the message is cached for more than this value, it will be sent to broker in bulk, default to 5000ms
# This value works in conjunction with Batch.num.messages.
queue.buffering.max.ms = 5000
# in Async mode, the maximum number of messages that the producer port allows for buffer
# in any case, producer cannot send the message to broker as soon as possible, leading to a massive deposition of messages at the producer end.
# at this point, if the message's number of bars reaches the threshold, it will cause producer end blocking or messages being discarded, default to 10000
queue.buffering.max.messages=20000
# If it is asynchronous, specify the amount of data sent each time, default is 200
batch.num.messages=500
# when the message is deposited on the producer end of the Queue.buffering.max.meesages, the number of bars reaches
# after blocking a certain time, the queue still has no enqueue (producer still does not send out any messages)
# at this point producer can continue to block or discard the message, this timeout value is used to control the time of "blocking"
#-1: No blocking timeout limit, messages will not be discarded
# 0: Empty queues immediately, messages are discarded
Queue.enqueue.timeout.ms=-1
################ End ###############

# The number of times a message is allowed to be producer when an error ACK is received or an ACK is not received
# because broker does not have a complete mechanism to avoid message duplication, so when network exceptions (such as ACK loss)
# may cause Broker to receive duplicate messages with the default value of 3.
Message.send.max.retries=3


# producer Refresh Topic Metada time interval, producer need to know partition leader location, and current topic situation
# So producer needs a mechanism to get the latest metadata, and when producer encounters a specific error, it refreshes immediately
# (such as topic failure, partition loss, leader failure, etc.), in addition to this parameter to configure additional refresh mechanism, the default value of 600000
topic.metadata.refresh.interval.ms=60000 Copy Code


Import java.util.*;

Import Kafka.javaapi.producer.Producer;
Import Kafka.producer.KeyedMessage;
Import Kafka.producer.ProducerConfig;

public class Testproducer {
public static void Main (string[] args) {
Long events = Long.parselong (Args[0]);
Random rnd = new Random ();

Properties Props = new properties ();
Props.put ("Metadata.broker.list", "192.168.2.105:9092");
Props.put ("Serializer.class", "Kafka.serializer.StringEncoder"); Default string Encoding message
Props.put ("Partitioner.class", "Example.producer.SimplePartitioner");
Props.put ("Request.required.acks", "1");

Producerconfig config = new Producerconfig (props);

producer<string, string> Producer = new producer<string, string> (config);

for (long nevents = 0; nevents < events; nevents++) {
Long runtime = new Date (). GetTime ();
String IP = "192.168.2." + rnd.nextint (255);
String msg = runtime + ", www.example.com," + IP;
keyedmessage<string, string> data = new keyedmessage<string, string> ("page_visits", IP, msg);
Producer.send (data);
}
Producer.close ();
}
}
Copy Code


2.2 Specifies the keyword key and sends the message to the specified partitions
Note: If you need to implement a custom partitions message delivery, you need to implement the Partitioner interface
public class Customizepartitioner implements Partitioner {
Public Customizepartitioner (verifiableproperties props) {

}
/**
* Return the partition index number
* When @param key SendMessage, the output Partkey
* @param total number of partitions in numpartitions topic
* @return
*/
@Override
public int partition (Object key, int numpartitions) {
System.out.println ("key:" + key + "Numpartitions:" + numpartitions);
String partkey = (string) key;
if ("Part2". Equals (Partkey))
return 2;
System.out.println ("Partkey:" + key);

........
........
return 0;
}
Copy Code