The problem of sharing variables between threads in Python multi-process programming

This article is mainly about exploring the sharing of variables between Python's multiple-process programming threads, and multi-process programming is an important knowledge in Python's advanced learning, and friends who need it can refer to

1, Questions:

Some students in the group posted the following code, asked why the list last printed is null value?

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27-28

From multiprocessing import Process, manager import OS Manager = Manager () vip_list = [] #vip_list = Manager.list () de F TestFunc (cc): Vip_list.append (cc) print ' Process ID: ', os.getpid () if __name__ = = ' __main__ ': threads = [] for LL in Range: t = Process (Target=testfunc, args= (ll,)) T.daemon = True threads.append (t) for I in range (len (threads)): THR Eads[i].start () for J in Range (len (threads)): Threads[j].join () print "------------------------" print ' Process ID: ', Os.getpid () Print vip_list

In fact, if you understand Python's multithreaded model, the GIL problem, and then understand the multithreading, the principle of multiple processes, the above questions are not difficult to answer, but if you do not know it does not matter, run the above code you know what is the problem.

?

1 2 3 4 5 6 7 8 9 10 11 12 13-14

Python aa.py process id:632 process id:635 process id:637 process id:633 process id:636 process id:634 process Id:6 Process id:638 Process id:641 process id:640------------------------process id:619 []

Open the 6th line comment and you will see the following results:

?

1 2 3 4 5 6 7 8 9 10 11 12-13

Process id:32074 Process id:32073 process id:32072 process id:32078 process id:32076 process id:32071 process id:32 077 process id:32079 Process id:32075 process id:32080------------------------process id:32066 [3, 2, 1, 7, 5, 0, 6, 8, 4, 9]

2, Python multi-process sharing variables in several ways:

(1) Shared Memory:

Data can be stored in a shared memory map using Value or Array. For example, the following code

Http://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16-17

From multiprocessing import Process, Value, Array def f (N, a): N.value = 3.1415927 to I in range (Len (a)): a[i] =-a[i] if __name__ = = ' __main__ ': num = Value (' d ', 0.0) arr = Array (' i ', range) P = Process (target=f, args= (num, arr)) p. Start () p.join () Print num.value print arr[:]

Results:

?

1 2	3.1415927 [0,-1,-2,-3,-4,-5,-6,-7,-8,-9]

(2) Server Process:

A manager object returned by manager () controls a server process which holds Python objects and allows the other processes to manipulate them using proxies.

A Manager returned by manager () would support types list, dict, Namespace, Lock, Rlock, Semaphore, Boundedsemaphore, Condit Ion, Event, Queue, Value and Array.

Code see the example at the beginning.

Http://docs.python.org/2/library/multiprocessing.html#managers

3, the problem of multiple processes is far more than that: Data synchronization

Read a simple piece of code: a simple counter:

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 The 25 26

From multiprocessing import Process, manager import OS Manager = Manager () sum = Manager. Value (' tmp ', 0) def testfunc (cc): Sum.value + = cc if __name__ = ' __main__ ': threads = [] for ll in range (m): t =   Process (Target=testfunc, args= (1,)) T.daemon = True threads.append (t) for I in range (len (threads)): Threads[i].start () For j in Range (len (threads)): Threads[j].join () print "------------------------" print ' Process ID: ', os.getpid () print Sum.value

Results:

?

1 2 3

------------------------Process id:17378 97

Perhaps you would ask: WTF? In fact, this problem in the multi-threaded era exists, but in the era of multiple processes and a repeat of the Cup: lock!

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

From multiprocessing import Process, manager, lock import OS lock = Lock () Manager = Manager () sum = Manager. Value (' tmp ', 0) def testfunc (CC, lock): with Lock:sum.value + = cc if __name__ = ' __main__ ': threads = [] for L L in range (m): T = Process (Target=testfunc, args= (1, lock)) T.daemon = True threads.append (t) for I in Range (Len (threa DS): Threads[i].start () for J in Range (len (threads)): Threads[j].join () print "------------------------" print "Proce SS ID: ', os.getpid () print Sum.value

What is the performance of this code? Run and watch, or increase the number of cycles try ...

4, the final proposal:

Note this usually sharing data between processes May is the best choice, because of all synchronization issues; An approach involving actors exchanging messages is usually seen as a better choice. Also Python Documentation:as mentioned above, when doing concurrent the IT is programming the best to usually using Sha Red state as far as possible. This is particularly true when using multiple processes. However, if you really did need to use some shared data then multiprocessing provides a couple of ways of doing.

5, refer:

Http://stackoverflow.com/questions/14124588/python-multiprocessing-shared-memory

http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing/

Http://docs.python.org/2/library/multiprocessing.html#multiprocessing.sharedctypes.synchronized