Roaming Kafka the client API of the actual combat chapter

Kafka Producer APIs

There are two types of legacy Procuder APIs: Kafka.producer.SyncProducer and Kafka.producer.async.AsyncProducer. They all implement the same interface:

[Java]View Plaincopy

2. Class Producer {
6. / * Send the message to the specified partition * /
8. public void Send (kafka.javaapi.producer.producerdata<k,v> producerdata);
12. / * Bulk send A batch of messages */
14. public void Send (java.util.list<kafka.javaapi.producer.producerdata<k,v>> producerdata);
18. / * Close producer * /
20. public void Close ();
24. }

The new Producer API provides the following features:

2. Multiple messages can be cached to the local queue and then asynchronously batched to the broker, which can be producer.type=async做到。 specified by some parameters by the size of the parameter cache: queue.time and batch.size . A Background thread (( kafka.producer.async.ProducerSendThread ) pulls the data from the queue and kafka.producer.EventHandler sends the message to the broker, or through parameter event.handler定制 handler, registers the processor at different stages of the producer-side processing of the data, such as a log trace of the process, or perform some monitoring. Simply implement the kafka.producer.async.CallbackHandler interface and configure it in callback.handler .
4. Write your own encoder to serialize the message, just implement the following interface. The default encoder is kafka.serializer.DefaultEncoder . [Java] view plaincopy
   1. Interface Encoder<t> {
   2. Public Message tomessage (T data);
   3. }
6. Provides zookeeper-based broker auto-sensing capability, which can be zk.connect implemented by parameters. If you do not use zookeeper, you can also use broker.list parameters to specify a static brokers list, so that messages will be sent randomly to a broker, and once the selected broker fails, the message will fail to send.
8. Through the partition function kafka.producer.Partitioner类对消息分区 . [Java] view plaincopy
   1. Interface Partitioner<t> {
   2. int partition (T key, int numpartitions);
   3. }
   The partition function has two parameters: key and the number of available partitions, select a partition from the partition list and return the ID. The default partitioning policy is hash(key)%numPartitions . If key is null, select one randomly. Partitioning functions can be customized by parameters partitioner.class .

A complete example of the new API is as follows:

[Java]View Plaincopy

2. Import java.util.*;
6. Import Kafka.javaapi.producer.Producer;
8. Import Kafka.producer.KeyedMessage;
10. Import Kafka.producer.ProducerConfig;
14. Public class Testproducer {
16. public static void Main (string[] args) {
18. Long events = Long.parselong (args[0]);
20. Random rnd = new Random ();
24. Properties props = new properties ();
26. Props.put ("Metadata.broker.list", "broker1:9092,broker2:9092");
28. Props.put ("Serializer.class", "Kafka.serializer.StringEncoder");
30. Props.put ("Partitioner.class", "Example.producer.SimplePartitioner");
32. Props.put ("Request.required.acks", "1");
36. Producerconfig config = new producerconfig (props);
40. producer<string, string> Producer = new producer<string, string> (config);
44. For (long nevents = 0; nevents < events; nevents++) {
46. Long runtime = new Date (). GetTime ();
48. String IP = "192.168. "   2. "+ rnd.nextint (255);
50. String msg = runtime + ", www.example.com," + IP;
52. keyedmessage<string, string> data = new keyedmessage<string, string> ("page_visits", IP, msg);
54. Producer.send (data);
56. }
58. Producer.close ();
60. }
62. }

The following is the partition function used:

[Java]View Plaincopy

2. Import Kafka.producer.Partitioner;
4. Import kafka.utils.VerifiableProperties;
8. Public class Simplepartitioner implements partitioner<string> {
10. Public Simplepartitioner (verifiableproperties props) {
14. }
18. public int partition (String key, int a_numpartitions) {
20. int partition = 0;
22. int offset = key.lastindexof ('. ');
24. if (offset > 0) {
26. Partition = Integer.parseint (key.substring (offset+1))% A_numpartitions;
28. }
30. return partition;
32. }
36. }

KafKa Consumer APIs

There are two levels of the Consumer API. The lower level is kept connected with a specified broker and closes the connection after receiving the message, which is stateless, with offset for each read message.

The high-level API hides the details of the brokers connection and communicates with the server without having to care about the server-side architecture. You can also maintain your own consumption status, and you can specify subscription-specific topic, such as a whitelist blacklist or regular expression, by some criteria.

Low-level API

[Java]View Plaincopy

2. Class Simpleconsumer {
6. / * Send a read request to a broker and get the message set * /
8. Public Bytebuffermessageset Fetch (fetchrequest request);
12. / * Send a read request to a broker and get a corresponding set * /
14. Public multifetchresponse Multifetch (list<fetchrequest> fetches);
18. /** 
20. * Get offsets before the specified time
22. * The return value is a offsets list, sorted in reverse order
24. * @param time: times, milliseconds,
26. * If specified as offsetrequest$. module$. Latiest_time (), get the latest offset.
28. * If specified as offsetrequest$. module$. Earliest_time (), get the oldest offset.
30. */
32. public long[] Getoffsetsbefore (String topic, int partition, long time, int maxnumoffsets);
34. }

Low-level APIs are the basis for high-level API implementations and are for scenarios where there is a special need to maintain the consumption state, such as offline consumer such as Hadoop consumer.

High Level API

[Java]View Plaincopy

2. /* Create connection */
4. Consumerconnector connector = consumer.create (consumerconfig);
8. Interface Consumerconnector {
12. /** 
14. * This method can get a list of streams, each of which is an iteration of messageandmetadata, with Messageandmetadata to get messages and other meta-data (currently topic)
16. * Input:a Map of <topic, #streams >
18. * Output:a Map of <topic, List of message streams>
20. */
22. Public map<string,list<kafkastream>> Createmessagestreams (map<string,int> topicCountMap);
26. /**     
28. * You can also get a list of streams that contain an iteration of the message that conforms to Topicfiler,
30. * A topicfilter is a regular expression that encapsulates a whitelist or blacklist.
32. */
34. Public list<kafkastream> Createmessagestreamsbyfilter (
36. Topicfilter topicfilter, int numstreams);
40. / * Submit the current consumption to offset * /
42. Public commitoffsets ()
46. / * Close connection * /
48. Public shutdown ()
50. }

The API revolves around an iterator implemented by Kafkastream, each of which represents a series of messages converging from one or more partitions and brokers, each of which is handled by a single thread, so the client can specify a few streams by parameter when it is created. A stream is a merge of multiple brokers on multiple partitions, but messages for each partition flow only to one stream.

Each call to Createmessagestreams will register consumer on topic so that load balancing between consumer and brokers is adjusted. The API encourages each invocation to create more topic streams to reduce this adjustment. The Createmessagestreamsbyfilter method registers the listener to perceive the new filter-compliant tipic.

Roaming Kafka the client API of the actual combat chapter