My Python growth path---the eighth Day---Python Foundation---March 5, 2016 (Sunny)

Multi-Process Multiprocessing module

The multiprocessing module provides a process class to represent a processing object

123456789101112131415161718192021222324252627282930

#!/usr/bin/env python3# coding:utf-8‘‘‘Created on: 2016年3月5日@author: 张晓宇Email: [email protected]Version: 1.0Description: 多进程演示程序Help:‘‘‘from multiprocessing import Processimport osdef run_proc(name):    # 子进程要执行的函数    print(‘Run child process %s (%s)...‘ % (name, os.getpid())) # os.getpid()表示获得当前进程的pid if __name__==‘__main__‘:    print(‘Parent process %s.‘ % os.getpid()) # 打印父进程的pid    p = Process(target=run_proc, args=(‘test‘,)) # 创建进程对象，参数结构和多线程一样    print(‘Child process will start.‘)    p.start() # 启动子进程    p.join() # 阻塞等待子进程执行完毕    print(‘Child process end.‘)

Inter-process Communication queue

Memory between different processes is not shared, so multi-process can not be as multi-threaded through global variables (of course, global variables are not advocated), so only through the queue, multi-process module also comes with a queue, using the same method and threading in the same way

Pipe

Pipeline, which can be understood as a bridge between two processes

123456789101112131415161718192021222324252627

#!/usr/bin/env python3# coding:utf-8‘‘‘Created on: 2016年3月5日@author: 张晓宇 Email: [email protected]Version: 1.0Description: 管道演示程序Help:‘‘‘from multiprocessing import Process, Pipedef f(conn):    conn.send([42, None, ‘hello‘]) # 网管道里传递数据    conn.close()if __name__ == ‘__main__‘:    parent_conn, child_conn = Pipe() # 一个是父进程的管道对象，一个是子进程的对象，自己成往里面send，父进程对象recv，有点像socket    p = Process(target=f, args=(child_conn,)) # 把管道对象作为参数传递给子进程    p.start()    print(parent_conn.recv())   # 接收管道里的数据并打印出来    p.join()

Execution results

1	[42, None, ‘hello‘]

Some people will say that since you can go to the child process to execute the function passed parameters, directly through this parameter to pass the data passed by the child process, such as can be used as a list of variable data types (string and numeric types, such as immutable type of data, do not want to think, the unified process can not do) why the pipeline or queue

123456789101112131415161718192021222324252627282930

#!/usr/bin/env python3# coding:utf-8‘‘‘Created on: 2016年3月5日@author: 张晓宇Email: [email protected]Version: 1.0Description: 管道演示程序Help:‘‘‘from multiprocessing import Process, Pipedef f(conn, strinfo):    conn.send([42, None, ‘hello‘]) # 网管道里传递数据    conn.close() # 关闭管道    strinfo.append(‘child‘) if __name__ == ‘__main__‘:    parent_conn, child_conn = Pipe() # 一个是父进程的管道对象，一个是子进程的对象，自己成往里面send，父进程对象recv，有点像socket    strinfo = [‘parent‘]    p = Process(target=f, args=(child_conn, strinfo)) # 把管道对象作为参数传递给子进程    p.start()    print(parent_conn.recv())   # 接收管道里的数据并打印出来    print(strinfo)    p.join()

Execution results

12	[42, None, ‘hello‘][‘parent‘]

From the execution results can be seen, the value of the Strinfo has not changed, because, the process started when the memory space is re-partition, equal to the strinfo in the child process copy, and the parent process is not the strinfo of a half-penny relationship, so there must be a pipeline queue, etc.

Process Pool

A process sequence is maintained internally by the process pool, and when used, a process is fetched in the process pool, and if the process pool sequence does not have a available process, it waits until a process is available

The pool module has two common ways to start a process

Apply and Apply_assync, literally understand that Apply_assync is asynchronous, is actually apply_assync support to pass a function as a parameter in, when the process function is finished, you can return a value, this value, is automatically passed as a parameter to the function passed in, and executes the function, which we call callback (callback)

123456789101112131415161718192021222324252627282930313233343536373839404142434445

#!/usr/bin/env python# coding:utf-8‘‘‘Created on: 2016年3月5日@author: 张晓宇Email: [email protected]Version: 1.0 Description: 进程池演示程序Help:‘‘‘frommultiprocessing import Pool, freeze_supportimport timedef Foo(i):    ‘‘‘    子进程执行的函数    :param i:     :return:     ‘‘‘    time.sleep(2)    return i+100def Bar(arg):    ‘‘‘    子进程回调函数    :param arg:    :return:    ‘‘‘    print(‘-->exec done:‘,arg)if __name__ == ‘__main__‘: # 这个在windows环境中绝对不能省略否则会报错    freeze_support()    pool = Pool(5) # 创建进程池对象     for i in range(10):        pool.apply_async(func=Foo, args=(i,), callback=Bar)        # pool.apply(func=Foo, args=(i,))    print(‘end‘)    pool.close()    pool.join()#进程池中进程执行完毕后再关闭，如果注释，那么程序直接关闭。

Execution results

1234567891011

end-->exec done: 100-->exec done: 101-->exec done: 102-->exec done: 103-->exec done: 104-->exec done: 105-->exec done: 106-->exec done: 107-->exec done: 108-->exec done: 109

My Python growth path---the eighth Day---Python Foundation---March 5, 2016 (Sunny)