On the multithreading Gil of Python

Source: https://www.zhihu.com/question/23474039/answer/269526476

Before introducing the threads in Python, let's make it clear that multithreading in Python is fake multithreading! Why say that, we first define a concept, global interpreter lock (GIL). The execution of Python code is controlled by a Python virtual machine (interpreter). Python is designed to be considered in the main loop, while only one thread is executing, like running multiple processes in a single-CPU system, where multiple programs can be stored in memory, but at any given time, only one program runs on the CPU. Similarly, although the Python interpreter can run multiple threads, only one thread runs in the interpreter. Access to the Python virtual machine is controlled by the global Interpreter Lock (GIL), which guarantees that only one thread is running at the same time. In a multithreaded environment, Python virtual machines are executed in the following manner. 1. Set the Gil. 2. Switch to a thread to execute. 3. Run. 4. Set the thread to sleep. 5. Unlock the Gil. 6. Repeat the above steps again. For all I/O-oriented programs that invoke the built-in operating system C code, the Gil is freed before this I/O call to allow other threads to run while the thread waits for I/O. If a thread does not use many I/O operations, it will occupy the processor and Gil on its own time slice. In other words, I/O intensive Python programs take advantage of multithreading more than computationally intensive python programs. We all know, for example, that I have a 4-core CPU, so that in the unit time each core can only run one thread, then the time slice rotation switch. But Python is different, it doesn't matter how many cores you have, multiple cores per unit of time can only run one thread, and then the time slice is rotated. It looks amazing? But this is the Gil's ghost. The Gil lock must be obtained before any Python thread executes, and then, every 100 bytecode is executed, the interpreter automatically releases the Gil lock, allowing other threads to execute. This Gil global lock actually locks the execution code of all threads, so multithreading can only be performed alternately in Python, even if 100 threads run on a 100-core CPU, only 1 cores are used. Usually the interpreter we use is an officially implemented CPython, to really take advantage of multicore, unless you rewrite a non-Gil interpreter.

We might as well do an experiment:

#Coding=utf-8 fromMultiprocessingImportPool fromThreadingImportThread fromMultiprocessingImportProcessdefLoop (): whileTrue:Passif __name__=='__main__':     forIinchRange (3): T= Thread (target=loop) T.start () whileTrue:Pass

My Computer is 4 cores, so I opened 4 threads and looked at CPU resource share:

We found that CPU utilization was not full, roughly equivalent to the single core level.

And what if we become a process?

Let's Change the code:

#Coding=utf-8 fromMultiprocessingImportPool fromThreadingImportThread fromMultiprocessingImportProcessdefLoop (): whileTrue:Passif __name__=='__main__':     forIinchRange (3): T= Process (target=loop) T.start () whileTrue:Pass

The result went directly to 100%, stating that the process can be made using multicore!

To verify that this is the Gil in Python, I tried to write the same code in Java, open the thread, and we looked at:

class testthread {public    static void Main (string[] args) {        for (int i = 0; i < 3; i++) {            New Thread (New Runnable () {                @Override public                void Run () {                    while  (True) {                    }                }            }). Start ();        }          while (True) {        }     }}

This shows that the multithreading in Java can take advantage of multicore, which is the real multi-threaded! And in Python, multithreading can only take advantage of the single core, which is fake multithreading!

Is that so? We have no way of using multicore in Python? Of course! Just a multi-process is a solution, there is a call C language of the link library. For all I/O-oriented programs that invoke the built-in operating system C code, the Gil is freed before this I/O call to allow other threads to run while the thread waits for I/O. We can write some computationally intensive tasks in C and then load the. So link library content into Python, because executing the C code, the Gil lock will be released, so that every kernel can run a thread for the purpose!

There may be small partners who don't understand what compute-intensive tasks are, and what are I/O intensive tasks?

Compute-intensive tasks are characterized by a large number of computations that consume CPU resources, such as PI, HD decoding video, and so on, all relying on the computing power of the CPU. This computationally intensive task can be accomplished with multitasking, but the more tasks, the more time it takes to switch tasks, the less efficient the CPU is to perform the task, so the most efficient use of the CPU should be equal to the number of cores in the CPU.

Compute-intensive tasks are critical to the efficiency of your code because they consume CPU resources primarily. Scripting languages like Python are inefficient and are completely unsuitable for compute-intensive tasks. For computationally intensive tasks, it is best to write in C.

The second type of task is IO-intensive, the tasks involved in network, disk IO are IO-intensive tasks, which are characterized by low CPU consumption and most of the time the task is waiting for the IO operation to complete (because IO is much slower than CPU and memory speed). For IO-intensive tasks, the more tasks you have, the higher the CPU efficiency, but there is a limit. Most of the tasks that are common are IO-intensive tasks, such as Web applications.

IO-intensive task execution, 99% of the time spent on the IO, the time spent on the CPU is very small, so the fast-running C language to replace the very low-speed scripting language with Python, completely unable to improve operational efficiency. For IO-intensive tasks, the most appropriate language is the most efficient (least code) language, the scripting language is preferred, and the C language is the worst.

In summary, Python multithreading is equivalent to single core multi-threading, multithreading has two advantages: CPU parallel, IO parallel, single core multithreading equivalent to self-broken arm. So, in Python, you can use multi-threading, but don't expect to make efficient use of multicore. If you must use multi-core through multithreading, that can only be achieved through C extension, but this will lose the Python simple and easy to use features. However, there is no need to worry too much that Python cannot use multithreading to achieve multicore tasks, but it can achieve multi-core tasks through multiple processes. Multiple Python processes have separate Gil locks that do not affect each other.

Links: https://www.zhihu.com/question/23474039/answer/269526476
Source: Know

On the multithreading Gil of Python