Detailed introduction to multiple processes in Python (code example)

This article brings you a detailed description of the multi-process in Python (code example), there is a certain reference value, the need for friends can refer to, I hope to help you.

This section is about learning Python's multi-process.

One, multi-process and multi-threaded comparison

Multi-process Multiprocessing and multi-threaded threading similar, they are used in Python to 并行 calculate. But since threading, why would Python have a multiprocessing? The reason is very simple, is to make up Some of the disadvantages of threading, as mentioned in the threading tutorial GIL .

The use of multiprocessing is also very simple, if you have a certain understanding of threading friends, your enjoyment of the time is up. Because Python uses multiprocessing and threading in almost the same way. This makes it easier for us to get started. It's also easier to unleash the power of your computer's multicore system!

Second, add process

Import multiprocessing as Mpimport threading as Tddef Job (a,d):    print (' aaaaa ') T1 = TD. Thread (target=job,args=) P1 = MP. Process (target=job,args=) T1.start () P1.start () T1.join () P1.join ()

As you can see from the usage comparison code above, threads and processes are used in a similar way.

Use

You need to add a statement that defines the main function when you use it

If __name__== ' __main__ ':

Full application code:

#-*-Coding:utf-8-*-"" "@author: Corwien@file:process_test.py@time:18/8/26 01:12" "" Import multiprocessing as Mpdef job (A, D):    print A, dif __name__ = = ' __main__ ':    p1 = MP. Process (Target=job, args= (1, 2))    P1.start ()    p1.join ()

Operating environment in the terminal environment, it is possible that other editing tools will appear without printing results after the run finishes, and the result of printing in terminal is:

➜  baselearn python./process/process_test.py1 2➜  Baselearn

Third, the storage process output queue

The function of the queue is to put the results of each kernel or thread in the team, wait until each thread or core has finished running and then fetch the result from the queue and continue loading the operation. The reason is simple, a function called by multithreading cannot have a return value, so using a queue to store the results of multiple thread operations

process_queue.py

#-*-Coding:utf-8-*-"" "@author: Corwien@file:process_queue.py@time:18/8/26 01:12" "" Import multiprocessing as mp# define a A function called by multithreading, Q is like a queue that holds the result of each function run Def job (q):    res = 0 for    i in range (+):        res + = i + i**2 + i**3    q.put (res )   #queueif __name__ = = ' __main__ ':    q = MP. Queue ()    P1 = MP. Process (Target=job, args= (Q,))    P2 = MP. Process (Target=job, args= (Q,))    # Start, connect two threads respectively    P1.start ()    P2.start ()    p1.join    () P2.join ()    # The above is divided into two batches, so there are two batches of output, the results are saved separately    res1 = Q.get ()    res2 = Q.get ()    print Res1,res2

Print out the results:

➜python./process/process_queue.py249833583000 249833583000

Iv. Process Pool

进程池Is that we put the things we want to run into the pool Python会自行解决多进程的问题 .

1. Importing multi-process modules

First import multiprocessing and definejob()

Import multiprocessing as Mpdef job (x):    return x*x

2. Process pool () and map ()

Then we define aPool

Pool = MP. Pool ()

With the pool, you can have the pool correspond to a function, we throw the data into the pool, and the pool returns the value returned by the function. The difference from the previous one is that the Pool Process的 function that threw the pool has a return value and Process no return value .

The next step is to map() get the result, in which map() you need to put the function and the value that needs to be iterated, and then it is automatically assigned to the CPU core, returning the result

res = Pool.map (Job, Range (10))

Let's run a little bit.

Def multicore ():    pool = MP. Pool ()    res = Pool.map (job, Range)    print (res)    if __name__ = = ' __main__ ':    multicore ()

Complete the Code:

#-*-Coding:utf-8-*-"" "@author: Corwien@file:process_queue.py@time:18/8/26 01:12" "" Import multiprocessing as Mpdef Jo B (x):    return x*x  # Note The function here has a return value of Def multicore ():    pool = MP. Pool ()    res = Pool.map (job, Range)    print (res)    if __name__ = = ' __main__ ':    multicore ()

Execution Result:

➜  Baselearn python./process/process_pool.py[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

3. Number of custom cores

How do we know Pool if we've actually called multiple cores? We can increase the number of iterations and then turn on the CPU load to see how the CPU is running.

Turn on CPU load (MAC): Activity Monitor > CPU > CPU Load (click on it)

Pool default size is the number of cores in the CPU, we can also Pool processes customize the required number of cores by passing in parameters in

Def multicore ():    pool = MP. Pool (processes=3) # Defines the number of CPU cores as 3    res = POOL.MAP (job, Range)    print (res)

4, Apply_async ()

PoolBesides map() , there are ways to return the results, that is apply_async() .

apply_async()Only one value can be passed in, it will only be put into one core for operation, but the incoming value should be aware that it is iterative, so you need to add a comma after passing in the value, and you need to get the return value with the Get () method

Def multicore ():    pool = MP. Pool ()     res = Pool.map (job, Range)    print (res)    res = Pool.apply_async (Job, (2,))    # Get results    with get Print (Res.get ())

Operation result;

[0, 1, 4, 9,  +,- # map () 4 # Apply_async ()

Summarize

• PoolThe default call is the number of cores of the CPU, the incoming processes parameter can be customized CPU cores

• map()Put the iteration parameter, return multiple results

• apply_async()Can only be put into a set of parameters, and return a result, if you want to get the effect of map () need to pass the iteration

V. Shared memory

This section we learn how to define shared memory. 只有用共享内存才能让CPU之间有交流.

Shared Value

We can store it Value in a shared memory table by using the data.

Import multiprocessing as Mpvalue1 = MP. Value (' I ', 0) value2 = MP. Value (' d ', 3.14)

where d and i parameters are used to set the data type, d represents a double-fine floating-point type Double, which i represents a signed 整型 .

Type Code	C Type	Python Type	Minimum size in bytes
'b'	Signed Char	Int	1
'B'	unsigned char	Int	1
'u'	Py_unicode	Unicode character	2
'h'	Signed Short	Int	2
'H'	unsigned short	Int	2
'i'	Signed int	Int	2
'I'	unsigned int	Int	2
'l'	Signed Long	Int	4
'L'	unsigned long	Int	4
'q'	Signed Long Long	Int	8
'Q'	unsigned long long	Int	8
'f'	Float	Float	4
'd'	Double	Float	8

Shared Array

In Python, there is mutiprocessing also a Array class that can interact with shared memory to enable sharing of data between processes .

Array = MP. Array (' i ', [1, 2, 3, 4])

Here is Array different from the numpy, it can only be 一维 , can not be multidimensional. Similarly Value , you need to define the data form, otherwise you will get an error. We will give an example of how these two are used in the latter section.

Wrong form

Array = MP. Array (' i ', [[1, 2], [3, 4]]) # 2-dimensional list "" "Typeerror:an Integer is required" ""

Six, Process lock lock

No process lock

Let's see what happens when there is no process lock.

#-*-Coding:utf-8-*-"" "@author: Corwien@file:process_no_lock.py@time:18/8/26 09:22" "" Import multiprocessing as Mpimport Timedef Job (V, num):    for _ in range (5):        time.sleep (0.5) # Pause for 0.5 seconds to make the output more visible        v.value + = num  # v.val UE Gets the shared variable value        print (v.value) def multicore ():    v = MP. Value (' I ', 0)  # defines the shared variable    p1 = MP. Process (Target=job, args= (V, 1))    P2 = MP. Process (Target=job, args= (V, 4)) # Set a different number to see how to rob Memory    P1.start ()    P2.start () p1.join ()    p2.join () if __name__ = = ' __main__ ':    multicore ()

In the above code, we define a shared variable v that can be manipulated by two processes. In Job () we want to v output a cumulative result every 0.1 seconds num , but p1 p2 set a different cumulative value in two processes. So let's see if there is a conflict between the two processes.

Results Print:

➜  baselearn python./process/process_no_lock.py1599131317171818➜  Baselearn

We can see that process 1 and process 2 are used with each other 抢 共享内存v .

Plus process lock

In order to solve the problem of sharing resources between the different processes mentioned above, we can solve them by adding process lock.

First you need to define a process lock

L = MP. Lock () # defines a process lock

The process lock information is then passed into each process

P1 = MP. Process (Target=job, args= (v,1,l)) # requires lock to be passed into P2 = MP. Process (Target=job, args= (v,3,l))

The job() use of a process lock is set in, guaranteeing the exclusive of the lock content of a process at run time

def job (V, NUM, l):    L.acquire () # Lock    for _ in range (5):        Time.sleep (0.1)         v.value + = num # V.value get shared memory 
  
   print (V.value)    l.release () # Release
  

All code:

#-*-Coding:utf-8-*-"" "@author: Corwien@file:process_lock.py@time:18/8/26 09:22" "" Import multiprocessing as Mpimport Timedef Job (V, NUM, l):    L.acquire () # Lock for-in    range (5):        time.sleep (0.5) # Pause for 0.5 seconds to make the output more visible        V.value + = n Um  # V.value Gets the shared variable value        print (v.value)    l.release () # Release Def multicore ():    L = MP. Lock () # defines a process lock    V = MP. Value (' I ', 0)  # defines the shared variable    p1 = MP. Process (Target=job, args= (V, 1, L)) # requires the lock to be passed into    P2 = MP. Process (Target=job, args= (V, 4, L)) # Set a different number to see how to rob Memory    P1.start ()    P2.start ()    p1.join ()    P2.join () if __name__ = = ' __main__ ':    multicore ()

Run it and let's see if there's a situation where the resources will be preempted:

Results Print:

➜  baselearn python./process/process_lock.py12345913172125

Clearly, the process lock ensures that the process p1 is fully operational before the process is p2 run