Implementation mechanism of tiny parallel computing framework

This article, mainly introduces its realization mechanism. Of course, with my always point of view, so that the novice can also understand.
First look at the interface of the work:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21st 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68

Public interface work extends Serializable { /** * Return to work type, each job has a job type, contractor and workers can only handle the same type of work * * @return */ String GetType (); String getId (); /** * Return the work of the next steps, if there is, the description is compound work, if not, the description is simple to work * * @return */ Work getnextwork (); /** * Set up next steps work * * Follow-up work @param nextwork * @return return to follow-up work */ Work setnextwork (work nextwork); /** * Whether serialization is required * * @return True indicates that work is never lost, false means the container is lost when closed */ Boolean isneedserialize (); /** * Set if you need to serialize, if you want to use MQ, you need to set to need serialization * * @param needserialize True indicates that work is never lost, false means the container is lost when closed */ void Setneedserialize (Boolean needserialize); /** * Return to input Warehouse * * @return */ Warehouse Getinputwarehouse (); /** * Set Input Warehouse * * @param inputwarehouse */ void Setinputwarehouse (Warehouse inputwarehouse); /** * Set working status * * @param workstatus */ void Setworkstatus (Workstatus workstatus); /** * Get working status * * @return */ Workstatus Getworkstatus (); }

is not very simple, its implementation is equally simple:

1 2 3 4 5 6 7 8

public class Workdefault implements work { Private String ID; Private String type; Private Warehouse Inputwarehouse; Private Boolean needserialize = false; Private work nextstepwork = null; Private Workstatus workstatus = workstatus.waiting; }

is basically the Set,get method of the above attribute.
The worker's interface is as follows:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21st 22 23 24 25 26 27 28 29

/** * worker, used to do specific work * Created by Luoguo on 14-1-8. */ Public interface worker extends Parallelobject {    /**      * perform work      *      * @return      */     Warehouse work throws remoteexception;    /**      * Accept work      * even the same type of worker can pick up three or four of jobs, This gives the workers a certain amount of flexibility.      *      * @param work      * @ Return true to accept, false to not accept      */     Boolean acceptwork (work work) throws RemoteException;       /**      * return type      *      * @return      */     String GetType () throws remoteexception; }

Is it easy, too?
Here's a look at the foreman:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21st 22 23 24 25 26 27 28 29 30 31 32 33

/** * Contractor * Contractor is used to carry a group of workers and complete the corresponding tasks * Created by Luoguo on 14-1-8. */ Public interface Foreman extends Parallelobject { /** * Returns which type of work task to perform * * @return */ String GetType () throws remoteexception; /** * Start working to get the job done */ Warehouse work, list<worker> Workerlist throws IOException; /** * Set Work Consolidator * * @param workcombiner */ void Setworkcombiner (Workcombiner workcombiner); /** * Set up the work breakdown device * * @param worksplitter */ void Setworksplitter (Worksplitter worksplitter); }

The following look at the employment introduction, hehe, this is more complicated (method more)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21st 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124

/** * Job-Reference Institute * The job-center is the core of distributed processing, and all relevant elements of the work are linked through the staff. * Created by Luoguo on 14-1-8. */ Public interface Jobcenter { String work_queue = "WorkQueue"; String FOREMAN = "FOREMAN"; String worker = "worker"; Rmiserver Getrmiserver (); void Setrmiserver (Rmiserver rmiserver); /** * Registered Workers * * @param worker */ void Registerworker (worker worker) throws remoteexception; /** * Return to Work queue object * * @return */ WorkQueue Getworkqueue (); /** * Consumer Workers * * @param worker */ void Unregisterworker (worker worker) throws remoteexception; /** * Sign up for a job where you don't need immediate attention. So you don't have to wait and return and you can do other processing. * If there is a return result, you can asynchronously, asynchronously can be used in the way of subsequent work to specify * * @param work */ void Registerwork (Work work) throws IOException; /** * Cancellation of work, before the work is not assigned, can be removed from the job, if the work has been assigned, it cannot be * * @param work */ void Unregisterwork (Work work) throws remoteexception; /** * Returns the working status of the specified work * * @param work * @return */ Workstatus Getworkstatus (work work) throws remoteexception; /** * Perform a job that expects synchronization to get results or exceptions * If no suitable worker or contractor is processed, an exception will be thrown immediately. * * @param work */ Warehouse doWork (work work) throws IOException; /** * Register Contractor * * @param Foreman */ void Registerforeman (Foreman Foreman) throws RemoteException; /** * Logout Contractor * * @param Foreman */ void Unregisterforeman (Foreman Foreman) throws RemoteException; /** * Returns a list of workers with some type of idle and willing to accept work * * @return */ List<worker> getidleworkerlist (work work); /** * Return to all job listings * * @return */ List<work> getworklist () throws remoteexception; /** * Returns a work list of some kind of state * * @return */ List<work> getworklist (String type, Workstatus workstatus) throws RemoteException; /** * Returns the list of idle foremen who organize some kind of work * * @param type * @return */ List<foreman> Getidleforeman (String type); /** * Automatic matching, if a match is successful, trigger execution */ void Automatch () throws IOException; /** * Employment Briefing Closed * * @throws RemoteException */ void Stop () throws remoteexception; }

Talked about four important interfaces, now talk about the realization of ideas: workers, Contractor are stateless. The advantage is that no matter how much work is done, it can be dealt with, and the disadvantage is that there is no way of subsequent intervention.
In general, I still feel that there is no state better than a state.
Because at first my implementation is stateful, even allowing contractor to cancel a job, contractor the worker to cancel the work, but this can lead to unusually complex distributed state maintenance. But it's much easier to change to a stateless model.
In the Employment Institute, there are two important methods, one is DoWork, that is, to perform a job immediately, if the appropriate foreman and workers can not be found, the exception will be thrown, and the other is Automatch trigger the employment of the introduction of an automatic match. The reason why I did not open threads inside the Employment Institute is to provide more convenient control for the outside.
Therefore, its overall design idea is to open a career introduction, workers or foreman, work can be registered in, by the employment of the Institute to make a suitable virtual team to achieve the task. Isn't it easy to understand?
Here are all the interfaces and implementation classes:

155218_thit_1245989.jpg (16.92 KB, download number: 0)

Download attachments

2015-5-27 22:14 Upload

For those who do parallel development:
The Employment Institute, the workers, the foreman do not have to develop, the framework comes with enough. Developers can only develop workers and work breakdown Consolidator.
After workers inherit Abstractworker, only one method can be implemented. Work decomposition one method, work merge one method, others all to tiny parallel computation frame bar.
Summarize:
This framework decomposes the participants in parallel computing into the Employment Institute, the foreman, the worker and the job, the center of the Employment Institute and the cessation of the operation of the Employment Institute, which will not be able to perform parallel computations. Foreman, workers can be on any computer, the work can be added at any node, because the framework provides work serialization functionality, so long as the set of work needs serialization is true, this work will persist until it is completed, the use of this feature can easily implement simple MQ. At the same time for the convenience of business implementation, workers have abstract classes, it is recommended to directly inherit the abstract class implementation to close the business method.
Note: The current execution of the worker's cancellation or discontinuation of the service will result in the failure of the entire operation and the next execution, which is to be changed to consider the continuation of the work by other workers.
Today is more busy, write more hastily, if there is unclear can be asked below.

Implementation mechanism of tiny parallel computing framework