April 27 Python Learning summary GIL, process pool, thread pool, synchronous, asynchronous, blocking, non-blocking

One, GIL: Global interpreter Lock

1. GIL: Global Interpreter Lock

Gil is essentially a mutex, which is sandwiched between the interpreter,

All threads within the same process need to grab the Gil lock before executing the interpreter code

2. Gil's Advantages and disadvantages:

Pros: Ensure thread safety for CPython interpreter memory management

Disadvantage: All threads in the same process can only have one execution at a time, and it is said that multithreading of the CPython interpreter cannot be implemented in parallel

Second, Gil and multithreading

With Gil's presence, at the same moment only one thread in the same process is executed

Heard here, some students immediately questioned: The process can take advantage of multicore, but the overhead, and Python's multithreaded overhead, but can not take advantage of multicore advantage, that is, Python is useless, PHP is the most awesome language?

To solve this problem, we need to agree on several points:

#1. Is the CPU used for computing, or is it used for I/O? #2. Multi-CPU means that multiple cores can be computed in parallel, so multi-core boosts compute performance. Once each CPU encounters I/O blocking, it still needs to wait, so it's useless to check I/O operations more

A worker is equivalent to the CPU, at this time the calculation is equivalent to workers in the work, I/O blocking is equivalent to work for workers to provide the necessary raw materials, workers work in the process if there is no raw materials, the workers need to work to stop the process until the arrival of raw materials.

If your factory is doing most of the tasks of preparing raw materials (I/O intensive), then you have more workers, the meaning is not enough, it is not as much as a person, in the course of materials to let workers to do other work,

Conversely, if your plant has a full range of raw materials, the more workers it is, the more efficient it is.

Conclusion:

For computing, the more CPU, the better, but for I/O, no more CPU is useless

Of course, to run a program, with the increase in CPU performance will certainly be improved (regardless of the increase in size, there is always improved), this is because a program is basically not pure computing or pure I/O, so we can only compare to see whether a program is computationally intensive or I/o-intensive, Further analysis of the Python multithreading in the end there is no useful

#分析: We have four tasks to deal with, the processing method must be to play a concurrency effect, the solution can be: Scenario One: Open four process Scenario two: a process, open four threads # single core case, the analysis results: If four tasks are computationally intensive, no multicore to parallel computing, Scenario one increases the cost of creating a process, and the scheme wins if four tasks are I/O intensive, the cost of the scenario one creation process is large, and the process is much less than the thread, and the scenario is faster than the threads, the result of the analysis: if four tasks are computationally intensive, multicore means parallel computing, In Python a process in the same time only one thread execution is not multi-core, scheme one win if four tasks are I/O intensive, no more cores can not solve the I/O problem, the scheme wins #结论: Now the computer is basically multicore, Python's efficiency in multi-threading for computationally intensive tasks does not bring much performance gains, or even better serial (no large switching), but there is a significant increase in the efficiency of IO-intensive tasks.

　　

Multithreaded Performance Testing

　

1  fromMultiprocessingImportProcess2  fromThreadingImportThread3 ImportOs,time4 defWork ():5res=06      forIinchRange (100000000):7res*=I8 9 Ten if __name__=='__main__': OneL=[] A     Print(Os.cpu_count ())#This machine is 4 cores -start=time.time () -      forIinchRange (4): theP=process (Target=work)#time-consuming 5s -P=thread (Target=work)#time-Consuming 18s - l.append (P) - P.start () +      forPinchL: - P.join () +stop=time.time () A     Print('run time is%s'% (Stop-start))

computationally Intensive: high-efficiency multi-process

1  fromMultiprocessingImportProcess2  fromThreadingImportThread3 ImportThreading4 ImportOs,time5 defWork ():6Time.sleep (2)7     Print('===>')8 9 if __name__=='__main__':TenL=[] One     Print(Os.cpu_count ())#This machine is 4 cores Astart=time.time () -      forIinchRange (400): -         #p=process (target=work) #耗时12s多, most of the time spent on the creation process theP=thread (Target=work)#time-consuming 2s - l.append (P) - P.start () -      forPinchL: + P.join () -stop=time.time () +     Print('run time is%s'% (Stop-start))

I/O intensive: high-efficiency multithreading

Third, process pool and thread pool

Process Pool vs thread pool

Why use a pool: pools to limit the number of concurrent tasks and limit our computers to perform tasks concurrently in a way that is affordable to them

What time does the pool load? Process: Concurrent tasks are computationally intensive

When to install threads in a pool: concurrent tasks are IO intensive

Process Pool　　

 fromConcurrent.futuresImportProcesspoolexecutor,threadpoolexecutorImportTime,os,randomdefTask (x):Print('pick -up from%s'%os.getpid ()) Time.sleep (Random.randint (2,5))    returnX**2if __name__=='__main__': P=processpoolexecutor ()#The number of processes that are turned on by default is the number of CPU cores    #Alex, Wupech, Yang Li, Wu Chen Taro, Zhang San     forIinchRange (20): P.submit (task,i)

Thread pool

 fromConcurrent.futuresImportProcesspoolexecutor,threadpoolexecutorImportTime,os,randomdefTask (x):Print('pick -up from%s'%x) time.sleep (Random.randint (2,5))    returnX**2if __name__=='__main__': P=threadpoolexecutor (4)#the number of threads that are turned on by default is the CPU's core count    #Alex, Wupech, Yang Li, Wu Chen Taro, Zhang San     forIinchRange (20): P.submit (task,i)

Iv. synchronous, asynchronous, blocking, non-blocking

1, blocking and non-blocking refers to the two operating states of the program

Blocking: Blocking occurs when an IO is encountered, the program stops in place once it encounters a blocking operation, and immediately releases CPU resources

Non-blocking (ready state or Run State): No IO operation, or some means to let the program even encounter IO operation will not stop in place, perform other operations, and strive to occupy as much CPU

2, synchronous and asynchronous refers to the two ways of submitting a task:

Synchronous invocation: After the task is submitted, wait in place until the task has finished running and get the return value of the task before continuing to execute the next line of code

Asynchronous invocation: After the task is submitted, do not wait in situ, directly execute the next line of code, the result?

asynchronous invocation

 fromConcurrent.futuresImportProcesspoolexecutor,threadpoolexecutorImportTime,os,randomdefTask (x):Print('pick -up from%s'%x) time.sleep (Random.randint (1,3))    returnX**2if __name__=='__main__':    #Asynchronous InvocationP=threadpoolexecutor (4)#the number of threads that are turned on by default is the CPU's core count    #Alex, Wupech, Yang Li, Wu Chen Taro, Zhang Sanobj_l=[]     forIinchRange (10): obj=p.submit (task,i) obj_l.append (obj)#p.close ()    #P.join ()P.shutdown (wait=True)Print(obj_l[3].result ())Print('Master')

　　

Synchronous invocation

 fromConcurrent.futuresImportProcesspoolexecutor,threadpoolexecutorImportTime,os,randomdefTask (x):Print('pick -up from%s'%x) time.sleep (Random.randint (1,3))    returnX**2if __name__=='__main__':    #Synchronous InvocationP=threadpoolexecutor (4)#the number of threads that are turned on by default is the CPU's core count    #Alex, Wupech, Yang Li, Wu Chen Taro, Zhang San     forIinchRange (10): Res=P.submit (task,i). Result ()Print('Master')

April 27 Python Learning summary GIL, process pool, thread pool, synchronous, asynchronous, blocking, non-blocking