Python multi-process and multi-threaded

Multi-process

The Unix/linux operating system provides a fork() system call, which is very special. A normal function call, called once, is returned once, but fork() called once, and returned two times, because the operating system automatically replicates the current process (called the parent process) with a copy (called a child process), and then returns within the parent and child processes, respectively.

To write multiple processes on the Windows platform, you need to introduce multiprocessing modules.

 fromMultiprocessingImportProcessImportOSdefRun_proc (name):Print('Child PID%s%s/but parent is%s'%(name, Os.getpid (), Os.getppid ()))#the fork-out child process can get the parent process ID through the Getppid () functionif __name__=='__main__':    Print('parent PID is%s'%os.getpid ()) P= Process (Target=run_proc, args= ('Test',))#The multiprocessing module provides a process class to represent a processing object, where the class is instantiated, the execution function and the function arguments are passed inP.start ()#To start a child process instance    the P.join () #join () method can wait for the child process to end before continuing to run, typically for inter-process synchronization.     Print('Child End')

Program results:

 is 4736 are 4736 childend

The join () function means that the child process and the main process are finished simultaneously

If you comment out P.join () This line of code

The output is as follows:

 is 8152 are 8152

The child process ends prematurely, leading to a different step.

Process pooling can create child processes in a large number of ways

 fromMultiprocessingImportPoolImportOS, time, RandomdefLong_time_task (name):Print('Run Task%s (%s) ...'%(name, Os.getpid ())) Start=time.time () time.sleep (Random.random ()* 3) End=time.time ()Print('Task%s runs%0.2f seconds.'% (name, (End-start )))if __name__=='__main__':    Print('Parent process%s.'%os.getpid ()) P= Pool (4)     forIinchRange (5): P.apply_async (long_time_task, args=(i,))Print('waiting-subprocesses done ...') the P.close () #close () function means that a new process is not allowed to join P.join () after the process pool has been addedPrint('All subprocesses is done .')#The Pool (4) represents the current CPU as 4 cores, so a task4 that can make 4 sub-processes at the same time is waiting

Execution Result:

Parent process 5460 for all subprocesses done ... Run Task 0 (95441 (78082 (81763 (43363 runs 1.674 (43361.95< c10>2 runs 1.954 runs 0.311 runs 2.52 seconds. All subprocesses is done.

Apply_async () asynchronous functions, passing in execution functions and function arguments. Leave the child process to the CPU at the same time for each child process.

Inter-process communication

ProcessThere is definitely a need for communication, and the operating system provides many mechanisms for communicating between processes. The Python multiprocessing module wraps the underlying mechanism, providing, and Queue Pipes so on, a variety of ways to exchange data.

QueueFor example, we create two sub-processes in the parent process, one to Queue write the data, and one to Queue read the data from the inside:

 fromMultiprocessingImportProcess, QueueImportOS, time, Random#code to write the data Process execution:defWrite (q):Print('Process to write:%s parent PID is:%s'%(Os.getppid (), Os.getpid ())) forValueinch['A','B','C']:        Print('Put%s to queue ...'%value) q.put (value)#Write QueueTime.sleep (Random.random ())#read the code that the data process executes:defRead (q):Print('Process to read:%s parent PID is:%s'%(Os.getpid (), Os.getppid ())) whileTrue:value=q.get (True)Print('Get%s from queue.'%value)if __name__=='__main__':    #The parent process creates a queue instance and passes it to each child process:Q =Queue () PW= Process (Target=write, args=(q,)) PR= Process (Target=read, args=(q,)) Pw.start ()#Start child process PW, write:Pr.start ()#start child process PR, read:Pw.join ()#wait for PW to end:Pr.terminate ()#PR process is a dead loop, can not wait for its end, can only forcibly terminate:

Execution Result:

Process to write:6264 parent PID is:1740
Put A to queue ...
Process to read:5296 parent PID is:1740
Get A from queue.
Put B to queue ...
Get B from the queue.
Put C to queue ...
Get C from queue.

Multithreading

Python's standard library provides two modules: _thread and threading , is the _thread low-level module, threading is the Advanced module, the _thread encapsulation. In most cases, we only need to use threading this advanced module.

ImportThreadingImport TimeImportOS#code executed by the new thread:defLoop ():Print('thread%s is running ...'% Threading.current_thread (). Name)#the Threading.current_thread () function can get the name of the current thread of executionn =0 whileN < 5: N= n + 1Print('Thread%s >>>%s'%(Threading.current_thread (). Name, N)) Time.sleep (1)    Print('thread%s ended.'%Threading.current_thread (). Name)Print('Main pid is%s'%os.getpid ())Print('thread%s is running ...'%Threading.current_thread (). Name) T= Threading. Thread (Target=loop, name='Loopthread') #在实例中传入执行函数t. Start () T.join ()Print('thread%s ended.'% Threading.current_thread (). Name)

Execution Result:

 is 8380are>>> 1>>> 2   >>> 3>>> 4>>> 5thread Loopthread ended.thread mainthread ended.

Thread Lock Lock

The biggest difference between multithreading and multi-process is that, in many processes, the same variable, each one copy is in each process, does not affect each other, and in many threads, all variables are shared by all threads, so any one variable can be modified by any one thread, so, The biggest danger of sharing data between threads is that multiple threads change a variable at the same time, and the content is scrambled.

My understanding is that each process will request its own memory, while the process of thread existence will only request memory once, as long as all thread variables that exist within the same process are shared.

This example is a problem with thread sharing variables

ImportThreading#Suppose this is your bank deposit:Balance =0defchange_it (n):#The result should be 0 if you save it first:    GlobalBalance Balance= Balance +N Balance= Balance-Ndefrun_thread (n): forIinchRange (100000): Change_it (n) T1= Threading. Thread (Target=run_thread, args= (5,)) T2= Threading. Thread (Target=run_thread, args= (8,)) T1.start () T2.start () T1.join () T2.join ( )Print(balance)

Each run results in a different output: 0,8,--8 and so on, with the expected balance constant of 0

The reason is that T1 and T2 threads are executed alternately, and variable sharing results in variability.

From the code, T1 is executed before T2, and the result is that the scheduling of threads is actually determined by the CPU.

Workaround:

The key to resolving thread variable sharing is to lock the function when it is created and called by the thread instance. T1 This thread uses the end to release the lock, T2 then obtains the lock release lock ....

ImportThreading#Suppose this is your bank deposit:Balance =0lock= Threading. Lock ()#Create a thread lock objectdefchange_it (n):#The result should be 0 if you save it first:    GlobalBalance Balance= Balance +N Balance= Balance-Ndefrun_thread (n): forIinchRange (100000): Lock.acquire ()#Request Lock        Try: Change_it (n)finally: Lock.release ()#Release LockT1= Threading. Thread (Target=run_thread, args= (5,)) T2= Threading. Thread (Target=run_thread, args= (8,)) T1.start () T2.start () T1.join () T2.join ( )Print(balance)

The thread lock does not work on a function or on a thread, because the function is actually executed by the thread instance.

The advantage of the lock is to ensure that a certain section of the key code can only be performed by a single thread from start to finish, the disadvantage is also many, the first is to prevent multi-threaded concurrent execution, a piece of code containing the lock can actually only be executed in single-threaded mode, the efficiency is greatly reduced. Second, because multiple locks can exist, different threads hold different locks, and attempting to acquire a lock held by the other side can cause deadlocks, cause multiple threads to hang all, neither execute nor end, only by the operating system to force termination.

No matter how much your CPU core, Python does not really do multi-threaded concurrency.

Python because of the use of the global interpretation of the Lock (GIL), the code can not concurrently run on multiple cores concurrently, that is, Python multithreading can not be concurrent, many people will find the use of multithreading to improve their own Python code, the operation of the program has decreased efficiency. This article describes the global Interpretation lock (GIL) in Python. The author argues that this is the most troubling question in Python.

This article is very clear: the most frustrating python question

I heard that the go language concurrency is very powerful ... But the grammar is too difficult qwer ...

Python multi-process and multi-threaded