Kafka Java consumer dynamically modifying topic subscriptions

some time ago in the Kafka QQ Group was asked about this--about how Java consumer dynamically modify topic subscription issues. It's really a good question to think about it, because if you simply hold the consumer instance in another thread and then call subscribe to modify it, the consumer side will inevitably throw an exception Concurrentmodificationexception:kafkaconsumer is isn'tsafe for multi-threaded access

Unlike Kafkaproducer, Kafkaconsumer is not thread-safe, so we cannot directly enable another thread to invoke consumer without a synchronous protection mechanism (except for wakeup). Therefore, there are two ways to achieve this requirement:

• Use a heavyweight synchorinzed mechanism for thread-safety
• Using the Java class Library's existing thread-safe data structures to implement

In the first way, any thread accessing the consumer must be equipped with the necessary synchronization protection, which is expensive and extremely error-prone. This article chooses the second approach, and we can use the Java-provided concurrentlinkedqueue to help us achieve this. The specific steps are:

1. Build Concurrentlinkedqueue objects to use for two threads (this is not Limited to two threads, but the most likely practical scenario for this requirement is the consumer main thread and a user thread for a background management class, which triggers the dynamic Modify subscription logic)
2. call Kafkaconsumer.poll (timeout) to consume the message continuously. Often people ask what is the timeout here for? Here's a unified answer: The timeout here gives the user the ability to perform other actions after consumer reads the message, such as regular logging. If your consumer does not have such a requirement, then call Kafkaconsumer.poll (1000) and Kafkaconsumer.poll ( Integer.max) without any distinction. In fact, we recommend that you use Kafkaconsumer.poll (Integer.max) + wakeup to respond to other backend logic !
3. After each poll try to explore Concurrentlinkedqueue there is nothing new (if there is a change in the list of subscriptions topic), respond to
4. Insert new subscription information into Concurrentlinkedqueue using another thread

The complete sample code is as follows:

public class Consumertest {public static void main (string[] args) {final concurrentlinkedqueue<string> s        Ubscribedtopics = new concurrentlinkedqueue<> ();  Create another test thread, pause for 10 seconds first, then change topic subscription Runnable Runnable = new Runnable () {@Override public void                Run () {try {thread.sleep (10000);                } catch (Interruptedexception e) {//Swallow it.            }//change to Subscription topic:btopic, Ctopic Subscribedtopics.addall (arrays.aslist ("Btopic", "ctopic"));        }        };        New Thread (runnable). Start ();        Properties Props = new properties ();        Props.put ("Bootstrap.servers", "localhost:9092");        Props.put ("Group.id", "my-group1");        Props.put ("Auto.offset.reset", "earliest");        Props.put ("Enable.auto.commit", "true");        Props.put ("auto.commit.interval.ms", "1000"); Props.put ("Key.deserializer", "Org.apachE.kafka.common.serialization.stringdeserializer ");        Props.put ("Value.deserializer", "Org.apache.kafka.common.serialization.StringDeserializer");        kafkaconsumer<string, string> consumer = new kafkaconsumer<> (props);        Initial subscription list: atopic, Btopic consumer.subscribe (arrays.aslist ("atopic", "btopic"));            while (true) {Consumer.poll (2000);//Consumer every 2 seconds there is a chance to poll the subscription status for changes//This example does not pay attention to the consumption of messages, so each time just print the results of the subscription!            System.out.println (Consumer.subscription ());                if (!subscribedtopics.isempty ()) {iterator<string> iter = Subscribedtopics.iterator ();                list<string> topics = new arraylist<> ();                while (Iter.hasnext ()) {Topics.add (Iter.next ());                } subscribedtopics.clear (); Consumer.subscribe (topics); Re-subscribe topic}}//This example is for testing purposes only and uses while (true), so there is no explicit shutdown of consumer//Consumer.closE (); }}

　　

The output is as follows:

[Atopic, Btopic]
[Atopic, Btopic]
[Atopic, Btopic]
[Ctopic, Btopic]
[Ctopic, Btopic]

This shows that the consumer in the case of no closure of the dynamic topic subscription changes. It is also necessary to say that dynamic changes are best not to call subscribe directly (topics), but to explicitly define Consumerrebalancelistener to avoid the confusion of displacement submissions.

Kafka Java consumer dynamically modifying topic subscriptions