Kafka consumer Multi-threaded processing in the project

Source: Internet
Author: User
Tags thread class throwable

For Kafkaconsumer, it is not like kafkaproducer, not thread-safe, the state is maintained in the consumer, so the implementation should pay attention to the use of multi-threading, generally there are 2 ways to use: 1: Each consumer has its own thread, Consumer to pull data, and processing, this method is relatively simple, easy to implement, easy to process message 2: Consumer processor, create a thread pool, after the consumer pull the data, the thread pool to process the data, pull data and processing data decoupling, But data processing could destroy partition's message order from the Kafka document we can also find out about how consumer multi-threading is handled

Project Practice:

is the specific application of consumer in the project, although the thread pool is also used, but in fact it is the first way, the thread pool is only used to start the operation of consumer:


Consumergroup class: This corresponds to the consumer group, in the 1th step above will create a listener object, which will be passed into the Consumergroup object creation process, in CG will create a Runnableconsumer object list (list), that is, the 3rd step , the number of consumer objects in the list will correspond to the desired number of consumer in the group. Also create a thread pool object executor, where the number of thread pools is consistent with the number of consumer

Runnableconsumer class: This is a thread class, implements the Runnable interface, inside creates a Kafkaconsumer object, thread launcher executes the subscription to topic, and pulls the message

 Public classRunnableconsumer<k,v>ImplementsRunnable {PrivateConsumer<k,v>consumer; Private FinalIconsumerlistener<consumerrecords<k,v>>Listener; PrivateRunnableconsumer (FinalIconsumerlistener<consumerrecords<k,v>>Listener, Properties ... props) {         This. Consumer =NewKafkaconsumer<>(props, Keydeserclass, valuedeserclass);  This. Listener =Listener; }         Public voidrun () {Try{ consumer.subscribe (topics); while(True) {consumerrecords<K,V> records =NULL; Try {                    //Now handle any new record (s)Records = Consumer.poll (1000); if(Records! =NULL&& records.count () > 0) { listener. Notify (Records); }                } Catch(wakeupexception Wex) {logger.trace ("Got a wakeupexception. Doing nothing. Exception Details: ", Wex); }             }        } Catch(Throwable e) {//Ignore as we waking up from the poll, we need to cleanly shutdown our consumerLogger.error ("Getting non-recoverable throwable:", E); Throwe; } finally {            //todo:need to check on consumer closing, the any outstanding offset commits are done. //Otherwise we need to manually does it here.Processcommit (Syncmode.sync); Logger.info ("Trying to close Kafka consumer, consumergroup.isrunning: {}", Consumergroup. This. isrunning);        Consumer.close (); }    }}

Listener class: This is a listening class, used to actually process a topic message, first create a listener object, in the creation of CG, registered to the Runnerconsumer class, if the consumer pull to the message, the message is notified to the listener class to be specific processing, Different business needs to define different business listener classes

Modify to the second way

If you want to use the second way, the data processing from the consumer, you can modify the above listener to a thread class, in the consumer pull to the message, then remove the thread from the thread pool processing data, one of the biggest problems, is how to ensure that messages are processed sequentially, for example, if there are 2 messages in a partition, and when consumer poll to the message, it commits to 2 threads, which does not guarantee sequential processing and requires an additional thread synchronization mechanism. And because the data is not required to be processed in consumer, the performance of consumer is increased, and the data processing time-out, consumer rebalance and other potential problems are avoided.

Records = Consumer.poll (+); if null && records.count () > 0) {       executor.submit(new  Listener (Records));}




Kafka consumer Multi-threaded processing in the project

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.