Multi-process multiprocessing multithreaded threading thread pool and process pool Concurrent.futures

multiprocessing.procsess defining a functiondefFunc ():Passin the If__name__=="__main__": In the instantiation of P= Process (target=The function to be executed by the child process, args (parameters of the function and must be passed as a tuple)) P.start () turns the child process P.join () to the end of the child process, and the main process waits for the child process to finish before exiting P.terminate () ending a child process P.is_al Ive () to see if a process is still running properties p.name p.pid process name and process number P.deamon daemon, the main process's code is executed, the child process is finished, and the child process is set to open the child process before execution, must have run () Myprocs ESS (procsess):def __init__(Name,*args,**kwargs):Pass    defRun ():PassP= Myprocsess ('name')**do not appear in the child process input#Lock Semaphore Event Queue Joinablequeue object created in the main processmultiprocessing. Lock Instancing A way to protect data by only one process at a time L=Lock () L.acquire () lock key, other process cannot access l.release () and key release lock, after release, other process can take lock on data operation remaining process blocking wait multiprocessing . Semaphore instantiate the semaphore, the number of processes that are allowed to access simultaneously SEM= Semaphore (4Sem.acquire () gets the key while allowing 4 processes to access the sem.release () key, allowing other processes to access the multiprocessing. Event is a signal to control multiple processes, one signal can block all processes, or it can unblock all processes e=Event () E.is_set () to view the status of an event, default is set to False block, true to not block E.set () modifies the state of an event to Turee.clear () modifies the state of an event to flasee.wait ( ) is based on the state of the event to determine whether blocking, flase blocking, true does not block Multiprocessing.queue queue FIFO Q=Queue (number of queues) q.put () put one into the queue q.get () take one out of the queue Q.full () the queue is full Q.empty () The queue is empty multiprocessing. Joinablequeueq= Joinablequeue (20) Q.put () into the queue, Count+1Q.join ()#block until all data in a queue has been processedq.get () fetch message q.task_done ()#Count-1 after the get-to message has been processed, the queue is told after processing is finished#The queue can only hold the number of parameters set when the instantiation, the queue is not full will continue to put#use the. Task_done () after using Joinablequeue,get () so that the join on the put end of the queue will not block after the last call to Task_donemultiprocessing. Pipecon,pro= Pipe ()#data not secure, lockProcess (func,args=(Con,pro)) Con.recv () Con.close () pro.send (f) pro.close ()#Con,pro without send or recv close, leaving only one recv to fetch the data will throw a Eoferror exceptionmultiprocessing. Managerm=Manager () dic=m.dict ({'Count': 100}) P= Process (target=func,args=(dic,l))#Manager data sharing, different sub-processes on the data modification, the main process can take value, unsafe, lockmultiprocessing. Poolp= Pool (5) P.map (func,iterable)#default asynchronous execution of tasks, with close and join capabilitiesP.pool (5) P.apply_async (Func,args= (I,))#asynchronous calls and master processes are fully asynchronousP.close ()#End Process Pool Receive taskP.join ()#the end of task execution in the perceptual process poolRes= P.apply_async (func,args= (i,))#return ObjectObject. Get ()#the calculation of the blocking wait result Func object returns the resultret = P.map (func,iterable)#waits for all processes to finish executing, returning the results of the Func calculation to the listP.apply_async (Func,args= (I,), CALLBACK=FUNC2)#The callback function is executed in the main process#callback function, cannot receive parameters, can only pass the Func execution result as a parameter to the callback function

Multithreading Threading#multithreading and multi-process use are similar. #process is the smallest memory allocation unit#a thread is the smallest unit of operating system scheduling#threads are executed directly by the CPU, with at least one thread within the process, or multiple threads can be opened    #It takes much less time to open a thread than to open a process    #multiple threads have their own data stacks inside, data is not shared    #global variables are shared across multiple threads#Gil Lock (i.e. global interpreter lock)#The Python program under the CPython interpreter can only have one thread being executed by the CPU at the same time in multiple threads#High CPU: COMPUTE class---High CPU utilization#High IO: Crawl page 200 pages # QQ chat send recv # processing log file read File # Processing Web request # Read database write databaseThreading. Thread (target=wahaha,args=(i,)). Start ()Print(Threading.active_count ())Print(Threading.current_thread ())Print(Threading.enumerate ()) threading. Conditioncon= Condition ()#condition # A condition is created at the beginning of the default has a false status # False state will affect wait always waiting stateCon.acquire () con.wait ( )#wait for the keyCon.notify (num)#Making Keyscon.release ()#Destroy, notify (quantity), condition state set to notify (qty) true,wait () pass one time, decrease by one trueThreading. Timer#Timert = Timer (5,func). Start ()#non-blocking#just delay executing the Func function, opening the thread asynchronously

Import= queue. Queue ()        #  queues first -in and out q = queue. Lifoqueue ()    #  Stack advanced after q = queue. Priorityqueue () Q.put ((),'a'#  priority queue, The first tuple element is the priority, the smaller the number the higher the priority, the same priority, the message sorted by ASCLL

#concurrent.futures thread pool and process pool fromConcurrent.futuresImportThreadpoolexecutor,processpoolexecutorpool= Threadpoolexecutor (max_workers=5)#to set the number of processes in a process poolPool.map (Func,range (20))#= = For I in range: Submit (Func,i)#cannot get the return value, parameter *iterablesp =pool.submit (func,i)#The return object, P.result (), adds a callback function. Add_done_callback (Call_back), the callback function receives the parameter P object, and uses the P.result () to take the valuePool.shutdown ()#Close+join fromGeventImportMonkey;monkey.patch_all ()ImportGevent#The co-process is essentially a thread # print (Threading.current_thread (). GetName ()) # virtual threadG1 =Gevent.spawn (Eat) G2=Gevent.spawn (play) G1.join () G2.join ()#Gevent.joinall (G_LST) # for G in G_lst:g.join ()#task switching for processes and threads right operating system complete#The switch between the co-process tasks is completed by the program (code), only when encountering the IO operation that the co-module can recognize, the program will switch tasks and achieve the concurrency effect .

Multi-process multiprocessing multithreaded threading thread pool and process pool Concurrent.futures