Talking about concurrency: Producer consumer model

The use of producer and consumer patterns in concurrent programming can solve most concurrent problems. This model improves the overall processing speed of the program by balancing the working capability of the production process and the consuming thread.

Why to use producer and consumer models

In the online world, the producer is the thread that produces the data, and the consumer is the thread that consumes the data. In multithreaded development, if the producer processes quickly and the consumer is slow to process, then the producer must wait for the consumer to finish processing to continue producing the data. Similarly, if consumers have more processing power than producers, consumers must wait for the producers. In order to solve this problem, we introduced producer and consumer models.

What is the producer consumer model

The producer consumer model solves the problem of strong coupling between producers and consumers through a container. Producers and consumers do not communicate with each other directly, but by blocking the queue to communicate, so the producer after the production of data without waiting for consumers to deal with, directly thrown to the blocking queue, consumers do not find producers to data, but directly from the blocking queue, the blocking queue is equivalent to a buffer, Balance the processing power of producers and consumers.

This blocking queue is used to decouple producers and consumers. Look at most design patterns, will find a third party out to decouple, such as the factory model of the third party is the factory class, template mode of the third party is the template class. In the process of learning some design patterns, if you first find the model of the third party, can help us quickly become familiar with a design pattern.

The producer consumer model is in combat

I worked with my colleagues to use the producer and consumer models in Yuna tools developed in our spare time. First I introduce the next Yuna tools, in Alibaba many colleagues like to share technical articles through the mail, because by mail sharing is very convenient, students see good technical articles on the Internet, copy and paste sent to complete the share, but we found that technical articles can not precipitate down, For the new students can not see the previous sharing of technical articles, it is difficult to find the previously shared technical articles. To solve this problem, we developed the Yuna tool. Yuna named himself like a game in the Final Fantasy of the heroine.

First we applied for a mailbox that was dedicated to the collection of shared mail. For example, share@alibaba.com, students will share the article sent to this mailbox, it must be troublesome for students to copy to this mailbox every time, so our practice is to put this email address in the Department mailing list, so the students sharing just like before to the whole department to share articles on the line, Yuna tools by reading mail Clothing Mail of the mailbox in the service, to download all the shared emails, including attachments, pictures, and emails, we may download some articles from this mailbox, so we need to share a keyword, such as [domestic technology sharing], after downloading the message. Through the Confluence Web service interface, the article is inserted into the confluence, so that new colleagues can see the previously shared articles in Confluence, and Yuna tools can also automatically classify and archive articles.

In order to quickly line this feature, we spent three days in the spare time to quickly develop the Yuna1.0 version. In the 1.0 version I did not use the producer consumption pattern, but use single-threaded to handle, because at that time only need to deal with one of our department's mail, so the single thread is obvious enough, the whole process is executed serially. In a thread, the program extracts all the mail, converts it into an article object, adds all the articles, and finally deletes the extracted message. The code is as follows:

public void Extract () {
        logger.debug ("Start" + getextractorname () +). ");
        Extract Mail
        list<article> articles = Extractemail ();
        Add article for
        (Article article:articles) {
            addarticleorcomment (Article);
        }
        Empty mail
        cleanemail ();
        Logger.debug ("Finished" + getextractorname () +). ");
    }

Yuna tools are being promoted, more and more departments use this tool, the processing time is more and more slow, Yuna is taken every 5 minutes, and when the mail more time processing may take a few minutes, So I used the producer consumer model in the Yuna2.0 version to process the mail, and first the producer thread took a certain rule to extract the mail from the mail system, then put it in the blocking queue, and the consumer took the article out of the blocking queue and inserted it into the conflunce. The code is as follows:

public class Quickemailtowikiextractor extends Abstractextractor {private Threadpoolexecutor threadspool;

Private articleblockingqueue<exchangeemailshallowdto> Emailqueue;
        Public Quickemailtowikiextractor () {emailqueue= new articleblockingqueue<exchangeemailshallowdto> ();
        int corepoolsize = Runtime.getruntime (). Availableprocessors () * 2; Threadspool = new Threadpoolexecutor (corepoolsize, Corepoolsize, 10l, Timeunit.seconds, new linkedblocking
    
    Queue<runnable> (2000)); public void Extract () {Logger.debug ("start" + getextractorname () +).
        ");

        Long start = System.currenttimemillis ();

        Extract all messages into the queue new Extractemailtask (). Start ();

        Insert the article in the queue into the wiki inserttowiki ();
        Long end = System.currenttimemillis ();
        Double cost = (End-start)/1000;
    Logger.debug ("complete" + getextractorname () + ", Spend time:" + cost + "seconds"); /** * Insert the article in the queue into the WikI/private void Inserttowiki () {//login wiki, need to log in once each interval Confluenceservice.login (rulefactory.use

        R_name, Rulefactory.password);
            while (true) {//2 seconds to exit exchangeemailshallowdto email = emailqueue.poll (2, timeunit.seconds);
            if (email = = null) {break;
        } threadspool.submit (new Inserttowikitask (email)); }} protected List<article> Extractemail () {list<exchangeemailshallowdto> allemails = GetE
        Mailservice (). Queryallemails ();
        if (allemails = = null) {return null; for (Exchangeemailshallowdto exchangeemailshallowdto:allemails) {Emailqueue.offer (exchangeemailsh
        ALLOWDTO);
    return null;
        /** * Extract Mail Task * * @author TENGFEI.FANGTF */public class Extractemailtask extends Thread {
        public void Run () {extractemail ();
  }  }
}