"Python" multi-process processing

Source: Internet
Author: User

Try to learn Python's multi-process module, compared to multi-threading, the approximate difference is:

1. Multi-process processing faster

2. It is inconvenient to exchange data between various sub-processes of multiple processes

Multi-process invocation mode

Process basic use multicore ()

Process Pool Optimization process call Multicore_pool (), when using the process pool, the Operation Xu function has a return, and the basic usage of the process is not receive return

Process queue usage, most of the methods are consistent with Python's basic queue,

Q=mp. Queue () statement
Q.put () add
Q.get () release
Q.empty () judgment is not empty
"" "Created on Tue Dec 5 09:00:00 2017@author:hellcat" "" Import Timeimport NumPy as Npimport threading as Tdimport MULTIPR Ocessing as Mpdef Job (q): For I in range (3): Q.put (i) def multicore (): Q=mp. Queue () PS = [MP. Process (target=job,args= (q,)), MP.         Process (target=job,args= (q,))] # Basic Child process creation method for p in Ps:p.start () Time.sleep (3) for P in PS: P.join () # need to reconcile end while True:if q.empty ()! = True:print ( Q.get ()) Else:breakdef multithread (): Q=mp. Queue () PS = [TD. Thread (target=job,args= (q,)), TD.        Thread (target=job,args= (Q,))] # Basic Child thread creation method for p in Ps:p.start () Time.sleep (3) for P in PS: P.join () while True:if q.empty ()! = True:print (Q.get ()) Else:brea Kdef Job_pool (q): Res0 = 0 for I in range (Np.random.randint): Res0 + = i**3 return res0deF Multicore_pool (): Pool = MP.                          Pool (processes=2) # Initializes the process pool, you can specify the number of processes res1 = Pool.map (Job_pool,range (20)) # Map method Add multiple threads simultaneously in thread pool print (RES1) # return value for function res2 = [Pool.apply_async (Job _pool, (I,))) for I in range (20)] # Single add process in process pool print ([Res.get () for res in res2]) # Note at this point in each process's The return is not directly a return, and the Get method is required to get the return # for RES in Res2: # print (Res.get ()) if __name__== ' __main__ ': # Multicor E () # multithread () Multicore_pool ()
Process shared variables & process Locks
#!/usr/bin/env python3#-*-coding:utf-8-*-"" "Created on Tue Dec  5 10:15:58 2017@author:hellcat" "" Import Timeimport Multiprocessing as Mpdef Job (v,num,l):  # The shared variable instance needs to be passed in as a parameter    l.acquire ()  # process Lock Lock for    _ in range (10):        Time.sleep (0.1)        v.value + = num # Gets the value of the shared variable and operates        print (v.value)    l.release ()  # process lock Release def muticore ( ):    L = MP. Lock ()  # instantiates the process lock    v = MP. Value (' I ', 0)  # instantiates the shared variable    # p = MP. Pool (processes=2)        # P.map (Job, ((v,1,l), (v,3,l)))    P1 = MP. Process (target=job,args= (v,1,l))    P2 = MP. Process (target=job,args= (v,3,l))    P1.start ()    P2.start ()    p1.join ()    p2.join ()    if __name__ = = ' __main__ ':    muticore ()

Unlike multithreading, which uses global variables to share variables, multiple processes must use V = MP. The statement of Value (' I ', 0) declares the variable (which can actually be MP.) Array (' i ', [list,[[1,2]), note that this can only be 1-d]], and pass V as a parameter to the function, and use V.value to awaken its value when used inside the function.

A process lock prevents other processes from using shared variables when locked, so you can see that the output is executed 10 times +1 and then 10 times +3, without using process locks +1 and +3 without regular alternation:

Comment out the process lock:

1
4
5
8
9
12
13
16
17
20
21st
24
25
28
29
32
33
36
37
40

Use a process lock (the above code does not comment out the process lock):

1
2
3
4
5
6
7
8
9
10
13
16
19
22
25
28
31
34
37
40

"Python" multi-process processing

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.