The Python GIL series on the Gil__python of Python

1. Previously wrote a "through an example to know Python Gil" article, feel a little more than the meaning of

2. The example was extended to further analyze the impact of Gil on Python programs

2.1 First Look at the example:

From threading import Thread to threading import Event as tevent from multiprocessing import Process from multiproces Sing import Event as pevent from Timeit import Timer def countdown (n,event): while n > 0:n = 1 E Vent.set () def io_op (n,event): F = open (' Test.txt ', ' W ') while not Event.is_set (): F.write (' Hello,world ') F.close () def t1 (): count=100000000 event = tevent () Thread1 = Thread (target=countdown,args= (COUNT , event)) Thread1.start () thread1.join () def t2 (): count=100000000 event = tevent () Thread1 = Thr EAD (target=countdown,args= (count//2,event)) Thread2 = Thread (target=countdown,args= (count//2,event)) Thread1.sta RT (); Thread2.start () Thread1.join (); Thread2.join () def t3 (): count=100000000 event = pevent () p1 = Process (target=countdown,args= (count//2,eve NT)) P2 = Process (target=countdown,args= (count//2,event)) P1.start (); P2.start () P1.join(); P2.join () def t4 (): count=100000000 event = tevent () Thread1 = Thread (target=countdown,args= (count,event) ) Thread2 = Thread (target=io_op,args= (count,event)) Thread1.start (); Thread2.start () Thread1.join (); Thread2.join () def T5 (): count=100000000 event = pevent () p1 = Process (target=countdown,args= (count,event ) P2 = Process (target=io_op,args= (count,event)) P1.start (); P2.start () P1.join (); P2.join () if __name__ = = ' __main__ ': t = Timer (t1) print (' Countdown in one thread:%f '% (T.timeit (1),)) T = Timer (T2) print (' Countdown use two thread:%f '% (T.timeit (1),)) T = timer (T3) print (' Countdown use two Proces S:%f '% (T.timeit (1),)) T = Timer (T4) print (' Countdown in one thread with IO op in another '% (thread:%f (1)
 ,) T = Timer (T5) print (' countdown in one process with IO op in another process:%f '% (T.timeit (1))

2.2 To see the output again:

2.2.1 First look at multi-core CPU disable other CPUs, run only one cpu,windows system, Python2.7.6 The result of running:

Countdown in one thread: ', 5.9650638561501195

countdown use two thread: ', 5.8188333656781595 countdown use

two Pr Ocess ', 6.197559396296269

countdown in one thread with IO op in another thread: ', 11.369204522553051

countdown in One process with IO op in another process: ', 11.79234388645473

2.2.2 Then look at the results of the four nuclear cpu,windows system, Python2.7.6 running:

Countdown in one thread:6.479085

countdown use two thread:24.266131

countdown use two

Process4.360930 Countdown in one thread with IO op in another thread:29.967870

countdown in one process with IO op in another process: 6.478644

2.2.3 then look at the four-core 64-bit CPU,WIDONWS system, the results of the Python3.4 run:

Countdown in one thread:12.333187

countdown use two thread:19.358091

countdown use two

process:7.105101 Countdown in one thread with IO op in another thread:10.443203

countdown in one process with IO op in another process: 18.682883

For the sake of comparison, let's go to the previous picture:

1), Single thread

2), two threads

3), the second process

4), CPU compute thread +I/O thread

5), CPU computing process +I/O process

2.3 Summary:

2.3.1 on a single core CPU, everything is fine.

Single-threaded and multi-threaded operations are nearly as efficient.

Multiple processes are consistent with multithreading, and multiple processes are slightly slower, possibly due to more time-consuming process switching.

When CPU and IO are mixed, the performance of multiple processes and multithreading is consistent

2.3.2 But when it comes to multi-core CPUs, the distinction between multithreading and multiple processes is compromised:

Using multithreading, the same amount of computation is performed, and the CPU is four times times slower than a single thread (6.479085:24.266131)

If it is multi-threaded CPU computing and IO mixed operation, the situation becomes worse (29.96787 seconds), the time here is still good, if more open programs, other programs are performing IO operation, the time will be more.

In the case of multiple processes, everything is still fine.

2.3.3 on the Python3.4, because of the Gil made a lot of optimizations, the running efficiency of multithreading has improved greatly, I/O operations on the CPU calculation of the impact is relatively small, but not as large as the 3.2 version, but the overall speed of operation is twice times slower than the Python2.7.

3. From the above example, the situation is very bad, python in the case of multi-core CPU, thread seems to be useless, but do not be so pessimistic. Let's take a look at the next example:

4. We optimized the original example and moved the countdown to C code for processing:

4.1 First look at the code:

4.1.1 Utility.pyx

DEF countdown (int n): with

    Nogil: while

        n > 0:

            N-= 1

4.1.2 setup.py

From Distutils.core Import Setup from

distutils.extension import extension to

cython.build import cythonize< C6/>ext = Extension ("Utility", 

                    define_macros = [(' Major_version ', ' 1 '),

                                     (' minor_version ', ' 0 ')],

                    sources  = ["Utility.pyx",])

setup (

    name = ' callback ',

    Version = ' 1.0 ',

    description = ' This is a callback demo Package ',

    author = ',

    author_email = ' shi19@163.com ',

    url = ',

    long_description =

    ', ext_ Modules=cythonize ([ext,])



 

4.1.3 count.py

From threading import Thread to threading import Event as tevent from multiprocessing import Process from multiproces

        Sing import Event as pevent Import utility from Timeit import Timer def countdown (n,event): For I in range (100): Utility.countdown (n) event.set () def io_op (n,event): F = open (' Test.txt ', ' W ') while not event.is_s ET (): F.write (' Hello,world ') f.close () def t1 (): count=100000000 event = tevent () Thread1 =

    Thread (target=countdown,args= (count,event)) Thread1.start () thread1.join () def t2 (): count=100000000 event = tevent () thread1 = Thread (target=countdown,args= (count//2,event)) Thread2 = Thread (Target=countdown,args = (count//2,event)) Thread1.start (); Thread2.start () Thread1.join (); Thread2.join () def t3 (): count=100000000 event = pevent () p1 = Process (target=countdown,args= (count//2,eve

    NT)) P2 = Process (target=countdown,args= (count//2,event))P1.start (); P2.start () P1.join (); P2.join () def t4 (): count=100000000 event = tevent () Thread1 = Thread (target=countdown,args= (count,event) ) Thread2 = Thread (target=io_op,args= (count,event)) Thread1.start (); Thread2.start () Thread1.join (); Thread2.join () def T5 (): count=100000000 event = pevent () p1 = Process (target=countdown,args= (count,event ) P2 = Process (target=io_op,args= (count,event)) P1.start (); P2.start () P1.join (); P2.join () if __name__ = = ' __main__ ': t = Timer (t1) print (' Countdown in one thread:%f '% (T.timeit (1),)) T = Timer (T2) print (' Countdown use two thread:%f '% (T.timeit (1),)) T = timer (T3) print (' Countdown use two Proces S:%f '% (T.timeit (1),)) T = Timer (T4) print (' Countdown in one thread with IO op in another '% (thread:%f (1)
 ,) T = Timer (T5) print (' countdown in one process with IO op in another process:%f '% (T.timeit (1))

4.2 Some notes:

4.2.1 Utility.pyx is a Cython script that enables mixed programming of Python and C with Cython, and can eventually generate a C file. With Nogil, which means that the Gil is released when the while loop is executed, because the next calculation does not involve the operation of the Python object, and it is safe to boldly remove the Gil's yoke.

4.2.2 Setup.py is a utility.pyx compilation script that executes the Python setup.py build_ext- InPlace can generate UTILITY.PYD dynamic library under Windows, the dynamic library generated under Linux is called utility.so, and the extension is referenced by import utility in Python code.

4.2.3 count.py, the modified test program, please note that countdown, here will be more than the original code calls 100 times countdown.

4.3 After running to get output:

Countdown in one thread:16.968686

countdown use two thread:9.333422

countdown use two

process:9.620321 Countdown in one thread with IO op in another thread:17.754015

countdown in one process with IO op in another process: 17.867098

4.4 Well, the world has become very beautiful, please remember that the above output is countdown than the original call 100 times times the output, you can see the operation of the digital calculation to C code will get how performance improvement.

5. Well, finally, let's make a summary:

The 5.1 Python Gil's effect on performance in a single core situation can be negligible, almost without.

5.2 Python The thread performance is really bad because of the presence of the Gil in multi-core CPUs, but the process is not affected by Gil.

5.3 Python built-in data classes are not suitable for a large number of mathematical calculations, of course, this is not just a python problem, the other fully object-oriented language has this problem, to do a lot of mathematical calculations to be used to move the code to C + + to implement, so as not only to remove the Gil Impact, The performance can be increased by dozens of times times more than a hundredfold, or an extension such as numpy can make a significant increase in performance when performing scientific calculations.

5.4 Python slow is actually slow in digital computing, think about it, if each number is an object, in the calculation of the time is unavoidable constantly for the object to apply for memory, free memory, speed is sure to slow down.

5.5 However, Python's operation of data structures is very efficient, such as Python built-in powerful dict,str,list and other classes, not to boast, the speed of its processing can really be comparable with C, because their implementation itself is implemented in C. We were told at the beginning of the program: data structure + algorithm = program, the truth may only be in the language of Python will have more immediate experience.

5.6 In using Python to develop programs, you have to spend some time on performance optimization, the process is simple: Use the Cprofile class to find out more time-consuming operations, and then move it to c to implement, in addition, if the use of multi-core CPU, you must use the thread carefully, As far as possible using process to replace thread, through the analysis of Gil in this paper, the performance optimization will provide a good help. In fact, Python's performance optimization process is also a challenging and very fulfilling part of program development.

5.7 However, remember that it is not premature to optimize the program, that premature optimization is the source of sin---Donald Knuth. Early development should focus on the implementation of the function and the readability and maintainability of the code.

5.8 Finally, I would like to conclude with a sentence: All said that love a person must love his (her) all, including his (her) shortcomings, to people so, to things.

Sample code

Finish