Python multi-process pools multiprocessing pool

1. Background

Due to the need to write a Python program, a regular, large number of Htttp requests are sent, and the results are processed.
Refer to the other code with the process pool and record it.

2. Multi-Process vs multithreading

• In a C + + program, a single module typically 单进程 launches dozens of, hundreds of threads, giving full play to machine performance. (currently C++11 has Std::thread programming multithreading is very convenient, can refer to my previous blog)
• In the shell script, it is performed in the 多进程 background. ({...} &, can refer to my previous blog to implement the Shell concurrency Processing Task)

• Python scripts have multiple threads and multiple processes. Due to the existence of the Python global unlock-lock Gil , it is generally recommended that CPU-intensive use multiple processes to give full play to multicore advantages, and I/O intensive can be multi-threaded.

  Although Python fully supports multithreaded programming, the C-language implementation of the interpreter is not thread-safe in fully parallel execution.
  In fact, the interpreter is protected by a global interpreter lock, which ensures that only one Python thread executes at any time.
  The biggest problem with the Gil is that Python's multithreaded programs do not take advantage of multicore CPUs (such as a computationally intensive program that uses multiple threads that only runs on a single CPU).

3. Multiprocessing Pool Use Example

Calling the Join () method on the pool object waits for all child processes to complete, and must call Close () before calling join (), so that it no longer accepts the new process.

#coding=utf-8import loggingimport timefrom multiprocessing import Poollogging.basicConfig(level=logging.INFO, filename='logger.log')class Point:    def __init__(self, x = 0, y= 0):        self.x = x        self.y = y    def __str__(self):        return "(%d, %d)" % (self.x, self.y)def fun1(point):    point.x = point.x + 3    point.y = point.y + 3    time.sleep(1)    return pointdef fun2(x):    time.sleep(1)    logging.info(time.ctime() + ", fun2 input x:" + str(x))    return x * xif __name__ == '__main__':    pool = Pool(4)    #test1    mylist = [x for x in range(10)]    ret = pool.map(fun2, mylist)    print ret    #test2    mydata = [Point(x, y) for x in range(3) for y in range(2)]    res = pool.map(fun1, mydata)    for i in res:        print str(i)    #end    pool.close()        pool.join()    print "end"

4. Reference

• Python multi-process multiprocessing. The pool class is detailed
• Python multithreading and multi-process programming summary
• Global lock problem for Python

Python multi-process pools multiprocessing pool