Example of how to use Python multi-process concurrency (multiprocessing)

Example of how to use Python multi-process concurrency (multiprocessing)

This example describes how to use Python multi-process concurrency (multiprocessing. Share it with you for your reference. The specific analysis is as follows:

Due to restrictions of Python design (I am talking about CPython ). A maximum of one CPU core can be used.

Python provides a very useful multi-process package multiprocessing. You only need to define a function, and Python will do everything else for you. With this package, you can easily complete the conversion from a single process to concurrent execution.

1. Create a single process

If we create a small number of processes, we can:

?

1 2 3 4 5 6 7 8 9 10 11

Import multiprocessing Import time Def func (msg ): For I in xrange (3 ): Print msg Time. sleep (1) If _ name _ = "_ main __": P = multiprocessing. Process (target = func, args = ("hello ",)) P. start () P. join () Print "Sub-process done ."

2. Use the process pool

Yes, you are not mistaken, not the thread pool. It allows you to run a full range of multi-core CPUs, and the usage is very simple.

Note that you must use apply_async. If async is dropped, the version will become blocked.

Processes = 4 is the maximum number of concurrent processes.

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Import multiprocessing Import time Def func (msg ): For I in xrange (3 ): Print msg Time. sleep (1) If _ name _ = "_ main __": Pool = multiprocessing. Pool (processes = 4) For I in xrange (10 ): Msg = "hello % d" % (I) Pool. apply_async (func, (msg ,)) Pool. close () Pool. join () Print "Sub-process (es) done ."

3. Use the Pool and follow the results

In more cases, we not only need to execute multiple processes, but also pay attention to the execution results of each process, as shown below:

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Import multiprocessing Import time Def func (msg ): For I in xrange (3 ): Print msg Time. sleep (1) Return "done" + msg If _ name _ = "_ main __": Pool = multiprocessing. Pool (processes = 4) Result = [] For I in xrange (10 ): Msg = "hello % d" % (I) Result. append (pool. apply_async (func, (msg ,))) Pool. close () Pool. join () For res in result: Print res. get () Print "Sub-process (es) done ."

I hope this article will help you with Python programming.