Use the process pool to circumvent Python GIL restrictions, and the pool to circumvent pythongil

Use the process pool to circumvent Python GIL restrictions, and the pool to circumvent pythongil

Operating System: CentOS7.3.1611 _ x64

Python version: 2.7.5

Problem description

Python GIL will affect CPU-intensive programs. If Python is fully used for programming, how can we avoid GIL restrictions?

Solution

Use a process pool in multiple threads to avoid GIL restrictions. The details are as follows:

1. Use the multiprocessing module to create a process pool;

2. Assign computing tasks to different threads;

3. Submit the task to the previously created process pool in the task thread;

 

Whenever a thread needs to execute a cpu-intensive task, the task is submitted to the process pool, and the process pool will hand over the task to the Python interpreter running in another process.

When the thread waits for the result, GIL is released, and the computing task is executed in another separate Python interpreter, which is no longer limited by GIL.

Using this solution in a multi-core system makes it easy to use all the CPU cores.

 

Suppose there is such a worker function:

def worker(arr):    s = 0    for n in arr :        arrTmp = range(1,n+1)        if len(arrTmp) == 0 : continue        rtmp = 1        for i in arrTmp :            rtmp *= i        s += rtmp    return s

Complete code: https://github.com/mike-zhang/pyExamples/blob/master/gilAvoid/gilAvoidTest1/taskCommon.py

Common Single-Process Task implementation:

def main():    s = 0    tStart,tStop = 1,1000    for i in range(1,100):        #t = worker(range(tStart,tStop))        t = worker(range(1,1000))        s += t        tStart = tStop        tStop += 1000    print("len : ",len(str(s)))    print(s%10000)

Complete code: https://github.com/mike-zhang/pyExamples/blob/master/gilAvoid/gilAvoidTest1/t1_normal.py

The running effect is as follows:

(py27env) [mike@localhost test]$ time python t1_normal.py('len : ', 2567)987real    0m17.919suser    0m17.915ssys     0m0.003s

Use process pool implementation:

def wokerThread(start,stop):    #r = gPool.apply(worker,(range(start,stop),))    r = gPool.apply(worker,(range(1,1000),))    q.put(r)def main():    s = 0    thrdArr = []    tStart,tStop = 1,1000    for i in range(1,gCount+1):        thrd = threading.Thread(target=wokerThread,args=(tStart,tStop))        thrdArr.append(thrd)        tStart = tStop        tStop += 1000    for t in thrdArr :        t.daemon = True        t.start()    while not q.full(): time.sleep(0.1)    while not q.empty(): s += q.get()    print("len : ",len(str(s)))    print(s%10000)

Complete code: https://github.com/mike-zhang/pyExamples/blob/master/gilAvoid/gilAvoidTest1/t2_mp.py

The running effect is as follows:

(py27env) [mike@localhost test]$ time python t2_mp.pyqueue full('len : ', 2567)987real    0m4.917suser    0m18.356ssys     0m0.146s

We can see that the above method can avoid GIL restrictions (the test machine is 4-core i5), and the program speed is significantly improved.

Okay, that's all. I hope it will help you.

Github address:

Bytes

Please add