Distributed computing-(distributed + multi-process + multithreading + multi-coprocessor)

Let's start with a simple example:

Squared the 1-10000 each number

Servers Server:

Store tasks, results with two queues

Define two functions

To implement distributed inheritance Multiprocessing.managers.BaseManager

Multiprocessing.freeze_support () Open distributed support in the main function

Register two functions for client calls

Create manager, set IP address and open port, link password.

Use two queues to add tasks and receive results. With a function that has just been registered

Push 1-10000 into the queue,

Press the result into the queue

Finally finish shutting down the server

Client clients:

Also need to inherit Multiprocessing.managers.BaseManager

Define a process that processes one data while pressing the result into the result queue

Define a thread to process 10 data, turn on 10 processes

Define a process that drives 10 threads

Main function: Register two functions with client

With client creation Manager, set IP address and open port, link password.

Linked server

Functions that are registered with the client call, two queues

Set of four-layer loops: 10 processes, 100 threads, 1000 co-routines

Loop Process function

On the code:

Servers Server:

#Coding:utf-8ImportMultiprocessing#Distributed ProcessesImportMultiprocessing.managers#Distributed Process ManagerImportRandom,time#random number, timeImportQueue#QueueTask_queue=queue.queue ()#TaskResult_queue=queue.queue ()#ResultsdefReturn_task ():#return to task queue    returnTask_queuedefReturn_result ():#return result Queue    returnResult_queueclassQueuemanger (Multiprocessing.managers.BaseManager):#inheritance, process management shared data    Passif __name__=="__main__": Multiprocessing.freeze_support ()#turn on distributed supportQueuemanger.register ("Get_task", Callable=return_task)#The registration function is called to the clientQueuemanger.register ("Get_result", callable=Return_result) Manger=queuemanger (address= ("192.168.112.11", 8848), authkey="123456")#create a manager to set the address and passwordManger.start ()#OpenTask,result=manger.get_task (), Manger.get_result ()#Tasks, Results     forIinchRange (10000):        Print "Task Add Data", I task.put (i)Print "waitting for------"     forIinchRange (10000): Res=result.get (timeout=100)        Print "Get Data", Res manger.shutdown ()#shutting down the server

Client clients:

#Coding:utf-8ImportMultiprocessing#Distributed ProcessesImportMultiprocessing.managers#Distributed Process ManagerImportRandom,time#random number, timeImportQueue#QueueImportThreadingImportgeventImportGevent.monkeyclassQueuemanger (Multiprocessing.managers.BaseManager):#inheritance, process management shared data    PassdefGevetygo (num, result):#The process processes a single data    Printnum*num result.put (num*num)defThreadgo (Datalist,result):#thread processing 10 data, opening 10 coprocessortasklist=[]     forDatainchdatalist:tasklist.append (Gevent.spawn (Gevetygo, Data,result)) Gevent.joinall (tasklist)defProcessgo (Ddatalist,result):#[[1,2,3],[4,5,6]] process drives 10 threadsthreadlist=[]     forDataListinchDdatalist:mythread=threading. Thread (target=threadgo,args=(Datalist,result)) Mythread.start () threadlist.append (mythread) forMythreadinchThreadlist:mythread.join ()if __name__=="__main__": Queuemanger.register ("Get_task")#registering a function call serverQueuemanger.register ("Get_result") Manger=queuemanger (address= ("192.168.112.11", 8848), authkey="123456") Manger.connect ()#Linked servertask=manger.get_task () result=manger.get_result ()#Tasks, Results    # +    #10 Processes    #100 Threads    #1000 x co-process     forIinchRange (10): Cubelist= []#[[ [1],[2] ]         forJinchRange (10): Arealist= []             forKinchRange (10): LineList= []                 forLinchRange (10): Data=Task.get () linelist.append (data) arealist.append (linelist) Cubelist.appe nd (arealist) processlist= []         forMyarealistinchcubelist:process= multiprocessing. Process (Target=processgo, args=(myarealist, result)) Process.Start () processlist.append (process) forProcessinchProcesslist:process.join ()

Encountered pit: One months ago when the distribution of the time to write the IP address can not be opened, later changed the computer support = =.

If only on your own computer, write 127.0.0.1 can also run, if you also encounter the IP address how can not open the situation

Distributed computing-(distributed + multi-process + multithreading + multi-coprocessor)