NET solves the problem of multi-topic Kafka multi-threaded sending

Generally in the Kafka consumer can set up a number of themes, that in the same program needs to send Kafka different topics of the message, such as exceptions need to send to the exception topic, normal to send to the normal topic, this time you need to instantiate a number of topics, and then send each.

Use the Rdkafka component in net to do message processing, which is referenced in NuGet.

Initialize the producer in the program and create multiple topic

Private stringComtopic ="Topic1"; Private stringErrtopic ="Topic2"; Private stringKfkip ="192.168.80.32:9092"; Topic Topic=NULL; Topic Errtopic=NULL;  PublicExcuteflow () {Try{Producer Producer=NewProducer (KFKIP); Topic=producer.                Topic (Comtopic); Errtopic=producer.            Topic (Errtopic); }            Catch(Rdkafkaexception ex) {Loghelper.error ("Kafka initializing Kafka exceptions", ex); }            Catch(Exception ex) {Loghelper.error ("Kafka Initialization Exception", ex); }        }

Send one of the topics in the program:

Try            {                if(Topic! =NULL)                {                    byte[] Datas =Encoding.UTF8.GetBytes (Jsonhelper.tojson (Flowcommond)); Task<DeliveryReport> Deliveryreport =topic.                    Produce (datas); varunused = deliveryreport.continuewith (task ={loghelper.info ("content: {flowcommond.id} sent to partition: {task. Result.partition}, Offset is: {task. Result.offset}");                }); }                Else                {                    Throw NewException ("Send message to Kafka topic is empty"); }            }            Catch(Rdkafkaexception ex) {Loghelper.error ("Send Message to Kafka Kafka exception", ex); }            Catch(Exception ex) {Loghelper.error ("Send Message to Kafka exception", ex); }

Flowcommond the content of the object to be sent, formatted as a JSON string and sent again.

Treated as another topic.

Here, a thread is implemented to send multiple topics, and the following implements how multiple topics are sent in multiple threads.

In multi-threading, if each thread is new Producer (Kfkip), the Kafka connection will soon be filled.

So here is a singleton mode to solve this problem, each time to use the producer to check if there is already a producer instance, if there is a direct use without the generation.

    /// <summary>    ///the implementation of single-case pattern/// </summary>     Public classSingleproduct:producer {//define a static variable to hold an instance of the class        Private Staticsingleproduct uniqueinstance; //define an identity to ensure thread synchronization        Private Static ReadOnly ObjectLocker =New Object(); //define private constructors so that the outside world cannot create instances of that class        PrivateSingleproduct (stringBrokerlist):Base(brokerlist) {}/// <summary>        ///defining public methods provides a global access point, and you can also define public properties to provide global access points/// </summary>        /// <returns></returns>         Public Staticsingleproduct getinstance () {//when the first thread runs here, the locker object is "locked" at this point ,//When the second thread runs the method, it first detects that the locker object is a "lock" state, and the thread suspends waiting for the first line threads unlocked//after the lock statement finishes running (that is, after the thread finishes running) The object is "unlocked"            if(Uniqueinstance = =NULL)            {                Lock(locker) {//created if an instance of the class does not exist, or is returned directly                    if(Uniqueinstance = =NULL)                    {                        stringKfkip = system.configuration.configurationmanager.appsettings["Kfkip"]; Try{uniqueinstance=Newsingleproduct (KFKIP); Loghelper.error ("Singleton mode instantiation singleproduct"); }                        Catch(Rdkafkaexception ex) {Loghelper.error ("Singleton mode Kafka initialization of Kafka exception", ex); }                        Catch(Exception ex) {Loghelper.error ("Singleton Mode Kafka initialization exception", ex); }                    }                }            }            returnuniqueinstance; }    }

Then replace producer producer = new producer (Kfkip) in the initialized code, for producer producer = Singleproduct.getinstance ();

Ok! This completes the Multithreading multi-topic message sending.

NET solves the problem of multi-topic Kafka multi-threaded sending