Java Master-worker Pattern Implementation

Introduction to Master-worker Mode

Master-worker mode is a very classic common parallel computing mode, its core idea is the 2 class process collaboration work: Master process and worker process. Master is responsible for receiving client requests, assigning tasks, and the worker is responsible for handling the tasks specifically. When each worker finishes the task, the results are returned to master, which is collated and summarized by master. The benefit is the ability to break down a large job into several small jobs, which can be executed in parallel to improve the throughput of the system. such as the popular webSe rver, such as Nginx,apache HTTP has this master-worker mode of operation, offline distributed computing framework Hadoop Jobtracker and Tasktracker, Real-time Flow computing framework Strom Nimbus and supervisor are all related to this idea. Then let's analyze the implementation of Java Master-worker mode concretely.

Master-worker Mode Analysis

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/8B/12/wKiom1hDkfzB-5wbAAArs5hrPOc274.png "title=" Sogou _ 2016-12-04_11-48-34.png "alt=" Wkiom1hdkfzb-5wbaaars5hrpoc274.png "/>

We focus on the 2 roles of Master,worker.

Master

Master needs to accept task tasks submitted by the client, and the task is assigned to the worker for processing, so master needs a store to hold the task. What kind of storage collection do you use? First of all, you need to support concurrent collection classes because there may be competing tasks between multiple workers, so we need to consider the collection under the Java.util.concurrent package. It is possible to consider using non-blocking concurrentlinkedqueue.

Master needs to know the basic information of each woker clearly, such as whether each worker is finished, so the master side needs to save the worker's information, can use map storage.

Since the last worker will report the results of the operation, the master side needs to have a map that stores the results, which can be used to support concurrent concurrenthashmap.

Worker

The worker needs to hold a reference to the task set on the master side, because the worker needs to fetch the task from the inside.

As above, the worker needs to hold a reference to the storage result on the master side.

In summary, we can get the following:

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M00/8B/0F/wKioL1hDlqHwZeM3AABZMBWVf3Q051.png "title=" Sogou _ 2016-12-04_12-08-48.png "alt=" Wkiol1hdlqhwzem3aabzmbwvf3q051.png "/>

Can we further refine what the master/worker should provide?

Master:

1. By constructing a method to initialize the workers
   

2. The Submit (Task) method should be provided to accept tasks submitted by the client side.

3. Start () Let workers start processing the task

4. Provide Iscomplete () to determine the status of each worker and whether it has been processed

5. Provide GetResult () to return the results to the client
   

Worker:

1. A worker is essentially a runnable, providing a run ()

2. Handle () responsible for handling business logic

Java Master-worker Code Implementation

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M02/8B/13/wKiom1hDxaDg3Zb_AAAJ9yFjHPc824.png "title=" Sogou _ 2016-12-04_15-32-25.png "alt=" Wkiom1hdxadg3zb_aaaj9yfjhpc824.png "/>

Task

public class task {    private long id;     Private string name;    public task (Long id, string name)  {        this.id = id;         this.name = name;    }    public long  getid ()  {        return id;    }     public void setid (Long id)  {         this.id = id;    }    public String  GetName ()  {        return name;    }     public void setname (String name)  {         this.name = name;    }} 

Worker

Public class worker implements runnable {    private long  id;    private String name;    private  concurrentlinkedqueue<task> workqueue;    private concurrenthashmap< Long,object> results;    public void setworkqueue (ConcurrentLinkedQueue <task> workqueue)  {        this.workQueue =  Workqueue;    }    public void setresults (ConcurrentHashMap <long, object> results)  {        this.results  = results;    }    public worker (long id,  String name)  {        this.id = id;         this.name = name;    }     @Override      Public void run ()  {        while (true) {             task task = workqueue.poll ();             if (task == null) {                 break;             }             long start = system.currenttimemillis ();             long result = handle (Task);             this.results.put (Task.getid (), result);          &nbSp;  system.out.println (this.name +  " handle "  + task.getname ()  +   " success . result is "  + result +  " cost time  :   +  (System.currenttimemillis ()  - start);         }    }    /**     *  Responsible for handling specific business logic      *  @param  task     *  @return      */    private long handle (Task task)  {         //here is just the simulation, in the real environment may be the query database, perhaps the cache and so on          try {             Thread.Sleep (;        } catch ) (InterruptedException  e) &NBSP;{&NBSP;&NBSP;&NBSP;&NBSP;&NBsp;       e.printstacktrace ();         }        return new random (). NextLong ();     }}

Master

Public class master {    private concurrentlinkedqueue<task>  workQueue = new ConcurrentLinkedQueue<Task> ();    private  Map<long,thread> workers = new hashmap<long, thread> ();     private concurrenthashmap<long,object> results = new concurrenthashmap <Long, Object>     public master (int num) {         for (int i = 0 ; i < num ; i++) {             Worker worker = new  Worker (i, "worker-"  + i);             Worker.setresults (Results);             Worker.setworkqueue (workQueue);  &Nbsp;          workers.put (Long.valueOf (i), New Thread ( Worker));        }    }     Public void submit (task task) {        workqueue.add ( Task);     }    public void start () {         for  (Map.entry<long,thread> entry : workers.entryset ()) {             entry.getvalue (). Start ();         }    }    public boolean  iscomlepte () {        for (map.entry<long,thread> entry  : workers.entryset ()) {            if ( Entry.getvalue (). GetState () &NBsp;! = thread.state.terminated) {                 return false;            }         }        return  True;    }    public long getsumresult () {         long value = 0;         For (Map.entry<long,object> entry : results.entryset ()) {             value = value +  (Long) entry.getvalue ();         }        return value;     }}

Main

Public class main {    public static void main (String[]  args)  {        Master master = new  Master (        for); (int i = 0 ; i  < 10 ; i++) {            task  task = new task (i, "task-"  + i);             master.submit (Task);        }         long start = system.currenttimemillis ();         master.start ();         while (True) {             if (Master.iscomlepte ()) {         &nbSp;       system.out.println ("sum result is "  +  Master.getsumresult ()  +  " . cost time : "  +  ( System.currenttimemillis ()  - start));                 break;             }        }    }}

Run results

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M02/8B/0F/wKioL1hDxjzwX2K2AACBfC_nvdY147.png "title=" Sogou _ 2016-12-04_15-35-02.png "alt=" Wkiol1hdxjzwx2k2aacbfc_nvdy147.png "/>

Summarize

In a single-threaded time, processing a task requires 500ms, then processing 10 task needs 5S, if the use of master-worker this parallel model, can greatly shorten the calculation processing time.

This article is from the "Boundless Mind Infinite" blog, please be sure to keep this source http://zhangfengzhe.blog.51cto.com/8855103/1879323

Java Master-worker Pattern Implementation