Kafka-Consumer Interface Analysis

1. Overview

In Kafka, there are two types of consumer APIs, one for high-level consumer APIs and the other for low-level consumer APIs. In the article "Advanced Consumer API", the API implementation of its advanced consumption is introduced. Let's introduce another consumer API today.

2. Content

After using Kafka's advanced consumer API, we know that it is a highly abstracted consumer API that is simple and convenient to use, but for some special needs we may need to use a second, more underlying API. So, first we need to know the role of low-level consumer APIs. It can help us to do those things:

• One message for multiple reads
• Consume only one part of a message during the process of Partition
• Add a thing management mechanism to ensure that messages are processed only once

Of course, there are some drawbacks in the use of the process, the contents are as follows:

• Value of Offset must be tracked in the program
• The lead Broker in the specified Topic Partition must be found
• Change of Broker must be handled

The idea steps for using its API are as follows:

• Find out which is the lead broker in the specified Topic Partition from all the active state brokers
• Find all the backup brokers in the specified Topic Partition
• Construct the request
• Send requests and query data
• Handling changes to the Leader Broker

3. Code implementation 3.1 Java Project

If you use the Java Project project to implement this part of the code, you need to add the relevant JAR file, whose contents include the following:

• Scala-xml_${version}-${version}.jar
• Scala-library-${version}.jar
• Metrics-core-${version}.jar
• Kafka-client-${version}.jar
• Kafka_${version}-${version}.jar

For Java project projects, you need to filter the JAR to add it yourself. Ensure the smooth execution of the code.

3.2 Maven Project

For Maven project, it is simple and convenient to add the corresponding dependency information in the Pom.xml file. Ask Maven to manage the corresponding dependent JAR files. The contents are as follows:

<Dependency>    <groupId>Org.apache.kafka</groupId>    <Artifactid>kafka_2.11</Artifactid>    <version>0.8.2.1</version>    <Exclusions>        <exclusion>            <groupId>Org.apache.zookeeper</groupId>            <Artifactid>Zookeeper</Artifactid>    </exclusion>    <exclusion>            <groupId>Log4j</groupId>            <Artifactid>Log4j</Artifactid>    </exclusion>    </Exclusions></Dependency>

This allows the corresponding dependency JAR file to be added to the Maven project.

3.3 Code Implementation

In the low-level consumption API, the implementation code looks like this:

/** * @Date Mar 2 * * @Author Dengjie * * @Note Simple Consumer API */public class Simplekafkaconsumer {private stat IC Logger log = Loggerfactory.getlogger (simplekafkaconsumer.class);p rivate list<string> m_replicabrokers = new Arraylist<string> ();p ublic Simplekafkaconsumer () {m_replicabrokers = new arraylist<string> ();} public static void Main (string[] args) {Simplekafkaconsumer example = new Simplekafkaconsumer ();//MAX read Numberlong max Reads = Systemconfig.getintproperty ("Kafka.read.max");/to subscribe to the topicstring topic = Systemconfig.getproperty ("Kafka.topic");//Find Partitionint partition = Systemconfig.getintproperty (" Kafka.partition ");//Broker node ' s iplist<string> seeds = new arraylist<string> (); String[] hosts = Systemconfig.getpropertyarray ("Kafka.server.host", ","); for (String host:hosts) {seeds.add (host);} int port = systemconfig.getintproperty ("Kafka.server.port"), try {example.run (maxreads, topic, partition, seeds, port);} CatCH (Exception e) {log.error ("Oops:" + e); E.printstacktrace ();}} public void run (long a_maxreads, String a_topic, int a_partition, list<string> a_seedbrokers, int a_port) throws Exce ption {//Get point topic Partition ' s metapartitionmetadata metadata = Findleader (A_seedbrokers, A_port, A_topic, A_partit ION), if (metadata = = null) {Log.info ("[Simplekafkaconsumer.run ()]-Can ' t find metadata for Topic and Partition. Exiting "); return;} if (metadata.leader () = = null) {Log.info ("[Simplekafkaconsumer.run ()]-Can ' t find leader for Topic and Partition. Exiting "); return;} String Leadbroker = Metadata.leader (). host (); String clientName = "Client_" + A_topic + "_" + a_partition; Simpleconsumer consumer = new Simpleconsumer (Leadbroker, A_port, 100000, * 1024x768, clientName); Long Readoffset = Getlasto Ffset (consumer, A_topic, A_partition, Kafka.api.OffsetRequest.EarliestTime (), clientName); int numerrors = 0;while (a_ Maxreads > 0) {if (consumer = = null) {consumer = new Simpleconsumer (Leadbroker, a_pORT, 100000, clientName);} Fetchrequest req = new Fetchrequestbuilder (). ClientId (ClientName). Addfetch (A_topic, A_partition, Readoffset, 100000). Build (); Fetchresponse fetchresponse = Consumer.fetch (req), if (Fetchresponse.haserror ()) {numerrors++;//Something went wrong! Short code = Fetchresponse.errorcode (A_topic, a_partition); Log.info ("[Simplekafkaconsumer.run ()]-Error fetching data From the Broker: "+ leadbroker+" Reason: "+ code"; if (Numerrors > 5) break;if (Code = = Errormapping.offsetoutofrangeco De ()) {//We asked for an invalid offset. For simple case ask for//the last element to Resetreadoffset = Getlastoffset (consumer, A_topic, A_partition, kafka.api.Of Fsetrequest.latesttime (), clientName); continue;} Consumer.close (); consumer = Null;leadbroker = Findnewleader (Leadbroker, A_topic, A_partition, a_port); continue;} Numerrors = 0;long Numread = 0;for (Messageandoffset messageAndOffset:fetchResponse.messageSet (A_topic, a_partition)) { Long Currentoffset = MESSAGEANDOFFSET.offset (), if (Currentoffset < Readoffset) {log.info ("[Simplekafkaconsumer.run ()]-Found an old offset:" + Currentoff Set + "expecting:" + readoffset); continue;} Readoffset = Messageandoffset.nextoffset (); Bytebuffer payload = Messageandoffset.message (). Payload (); byte[] bytes = new Byte[payload.limit ()];p ayload.get (bytes) ; System.out.println (String.valueof (Messageandoffset.offset ()) + ":" + New String (bytes, "UTF-8")); Message deal enternumread++;a_maxreads--;} if (Numread = = 0) {try {Thread.Sleep],} catch (Interruptedexception IE) {}}}if (consumer! = null) Consumer.close ();} public static long Getlastoffset (Simpleconsumer consumer, String topic, int partition, long whichtime,string clientName) { Topicandpartition topicandpartition = new Topicandpartition (topic, partition); Map<topicandpartition, partitionoffsetrequestinfo> requestinfo = new Hashmap<topicandpartition, Partitionoffsetrequestinfo> (); Requestinfo.put (Topicandpartition, New Partitionoffsetrequestinfo (whichTime, 1)); Kafka.javaapi.OffsetRequest request = new Kafka.javaapi.OffsetRequest (Requestinfo, Kafka.api.OffsetRequest.CurrentVersion (), clientName); Offsetresponse response = Consumer.getoffsetsbefore (Request) if (Response.haserror ()) {Log.info ("[Simplekafkaconsumer.getlastoffset ()]-Error fetching data Offset data the Broker. Reason: "+ response.errorcode (topic, partition)); return 0;} Long[] Offsets = response.offsets (topic, partition); return offsets[0];}             /** * @param a_oldleader * @param a_topic * @param a_partition * @param a_port * @return String * @throws Exception * Find Next leader Broker */private string Findnewleader (String A_oldleader, string a_topic, int a_partition, int a_ Port) throws Exception {for (int i = 0; i < 3; i++) {Boolean gotosleep = false; Partitionmetadata metadata = Findleader (M_replicabrokers, A_port, A_topic, a_partition); if (metadata = = null) {Gotosleep = true;} else if (metadata.leader () = = null) {Gotosleep = true;} else if (A_oldleader.equalsignoRecase (Metadata.leader (). Host ()) && i = = 0) {//first time through if the leader hasn ' t changed give//ZooKeeper a  Second to recover//second time, assume the broker does recover before failover,//or it was a non-broker issue//gotosleep = true;} else {return Metadata.leader (). host ();} if (gotosleep) {try {thread.sleep ()} catch (Interruptedexception IE) {}}}throw new Exception ("Unable to find new leads ER after Broker failure. Exiting ");} Private Partitionmetadata Findleader (list<string> a_seedbrokers, int a_port, String a_topic, int a_partition) { Partitionmetadata returnmetadata = null;loop:for (String seed:a_seedbrokers) {Simpleconsumer consumer = null;try {Consu Mer = new Simpleconsumer (seed, A_port, 100000, * 1024x768, "leaderlookup"); list<string> topics = Collections.singletonlist (a_topic); Topicmetadatarequest req = new Topicmetadatarequest (topics); Kafka.javaapi.TopicMetadataResponse resp = Consumer.send ( REQ); list<topicmetadata> MetaData = Resp.topicsmetAdata (); for (Topicmetadata Item:metadata) {for (Partitionmetadata Part:item.partitionsMetadata ()) {if (part.partition Id () = = a_partition) {returnmetadata = Part;break Loop;}}} catch (Exception e) {log.error ("error communicating with Broker [" + Seed + "] to find Leader for [" + A_topic + "," + a_p Artition + "] Reason:" + e);} finally {if (consumer! = null) Consumer.close ();}} if (returnmetadata! = null) {m_replicabrokers.clear (); for (Kafka.cluster.Broker Replica:returnMetaData.replicas ()) {m _replicabrokers.add (Replica.host ());}} return returnmetadata;}}

4. Summary

When using the Kafka low-level consumption API, to identify the business scenario we use, it is generally recommended that you use the Advanced consumption API unless you encounter special needs. In addition, in the use of the process, pay attention to the processing of Leader Broker, and the management of Offset.

5. Concluding remarks

This blog is to share with you here, if you study in the process of learning what is the problem, you can add groups to discuss or send e-mail to me, I will do my best to answer for you, with June encouragement!

Kafka-Consumer Interface Analysis