Parallel programming mode-Master-Worker mode and masterworker Mode

Parallel programming mode-Master-Worker mode and masterworker Mode

• Introduction

The Master-Worker mode is a commonly used parallel design mode. Its core idea is that the system has two process protocols: Master process and Worker process. The Master process is responsible for receiving and allocating tasks, and the Worker process is responsible for processing subtasks. After the Worker processes process the sub-tasks, the results are returned to the Master process, which summarizes and summarizes the results to obtain the system results. The processing process is as follows:

The advantage of the Master-Worker mode is that it can break down large tasks into several small tasks and execute them concurrently, thus improving system performance. For the system requestor Client, once a task is submitted, the Master process immediately allocates the task and returns it immediately. It does not return the result after the system processes the complete task, the processing process is asynchronous.

• Master-Worker mode structure

　　The main structure of the Master-Worker mode is as follows:

As shown in, the Master process is the main process. It maintains a Worker process queue, A subtask queue, and a subresult set, the Worker Process in the Worker process constantly extracts the subtasks to be processed from the task queue, and puts the processing results of the subtasks into the subresult set.

In, Master: used to allocate tasks and merge the final results; Worker: used to actually process a task; client process: used to start the system and start the Master during scheduling.

• Master-Worker mode code implementation

　　Master code implementation:

1 public class Master {2 // task Queue 3 protected Queue <Object> workQueue = new concurrent1_queue <Object> (); 4 // worker Process Queue 5 protected Map <String, thread> threadMap = new HashMap <String, Thread> (); 6 // result set 7 protected Map <String, Object> resultMap = new HashMap <String, Object> (); 8 9 // whether all subtasks end 10 11 public boolean isComplete () {12 for (Map. entry <String, Thread> entry: threadMap. entrySet () {13 if (ent Ry. getValue (). getState ()! = Thread. state. TERMINATED) {14 return false; 15} 16} 17 return true; 18} 19 20 public Master (Worker worker, int countWorker) {21 worker. setResultMap (resultMap); 22 worker. setWorkQueue (workQueue); 23 for (int I = 0; I <countWorker; I ++) {24 threadMap. put (Integer. toString (I), new Thread (worker, Integer. toString (I); 25} 26} 27 28 // submit task 29 30 public void submit (Object obj) {31 workQueue. add (obj); 32 // System. out. println (obj. toString (); 33} 34 35 36 37 // return the subtask result set 38 public Map <String, Object> getResultMap () {39 return resultMap; 40} 41 42 // start running all worker processes and process 43 44 public void execute () {45 for (Map. entry <String, Thread> entry: threadMap. entrySet () {46 entry. getValue (). start (); 47} 48} 49 50}

Worker code implementation:

1 public class Worker implements Runnable {2 // task Queue 3 protected Queue <Object> workQueue; 4 // subtask result set 5 protected Map <String, object> resultMap = new HashMap <String, Object> (); 6 7 8 public void setWorkQueue (Queue <Object> workQueue) {9 this. workQueue = workQueue; 10} 11 public void setResultMap (Map <String, Object> resultMap) {12 this. resultMap = resultMap; 13} 14 15 public Object handle (Object input) {16 return input; 17} 18 @ Override19 public void run () {20 while (true) {21 Object input = workQueue. poll (); 22 23 if (null = input) break; 24 // process subtask 25 Object re = handle (input); 26 resultMap. put (Integer. toString (input. hashCode (), re); 27 // System. out. println (re. toString (); 28} 29} 30 31}

The Master-Worker mode is a method for parallel processing of serial tasks. The decomposed sub-tasks can be processed in parallel in the system. At the same time, if necessary, the Master process can calculate the final result based on some existing result sets without the completion of all sub-tasks.

Now, the above Master-Worker implementation is used to calculate 1-100 cubes. Computing is divided into 100 subtasks, and each subtask is only used to calculate a separate cube and. The Master generates a fixed number of workers to process these subtasks. Worker constantly extracts these computing cubes and subtasks from the task set and puts the computing results into the Master result set. The Master is responsible for accumulating all the Worker task results to generate the final cube and. During the entire computing process, the Worker and Master operations are completely asynchronous. The Master process can perform the sum operation without waiting for all Worker processes to be completed. That is to say, the Master can calculate the final result when obtaining the result set of a molecular task, thus improving the concurrency and throughput of the system.

The implementation of computing subtasks is as follows:

1 public class PlusWorker extends Worker {2 3     @Override4     public Object handle(Object input) {5         Integer i = (Integer) input;6         return i*i*i;7     }8     9 }

The client code is as follows:

1 public class Client {2 public static void main (String [] args) {3 Master m = new Master (new PlusWorker (), 5 ); // start five threads to process 4 for (int I = 0; I <100; I ++) {5 m. submit (I); 6} 7 m.exe cute (); 8 int re = 0; 9 Map <String, Object> resultMap = m. getResultMap (); 10 while (resultMap. size ()> 0 |! M. isComplete () {11 Set <String> keys = resultMap. keySet (); 12 String key = null; 13 for (String k: keys) {14 key = k; 15 break; 16} 17 Integer I = null; 18 if (key! = Null) {19 I = (Integer) resultMap. get (key); 20} 21 if (I! = Null) {22 re + = I; // parallel computing result set 23} 24 25 if (key! = Null) {26 resultMap. remove (key); // remove the calculated result 27} 28} 29 30 System. out. println (re); 31} 32}

Create five Worker threads and PlusWorker Worker instances through the Master node. After you submit 100 tasks, you can start to calculate the subtasks. These subtasks are completed by the five Worker threads. The Master does not wait until all the sub-tasks are computed. Then, it begins to access the sub-result set to compute the final result until all the data in the sub-result set is processed, the final result can be obtained only when all five active Worker threads are terminated.