Introduction to Python standard library multi-process (multiprocessing package), python multi-process

Introduction to Python standard library multi-process (multiprocessing package), python multi-process

After learning about Python multi-process, we can continue to explore more advanced tools in the multiprocessing package. These tools make it easier for us to implement multi-process.

Process pool

A Process Pool can create multiple processes. These processes are like standing soldiers preparing to execute tasks (programs ). A process pool can accommodate multiple on-standby soldiers.

"Process pool of three processes"

For example, the following program:

Copy codeThe Code is as follows:
Import multiprocessing as mul
Def f (x ):
Return x ** 2
Pool = mul. Pool (5)
Rel = pool. map (f, [1, 3, 4, 5, 6, 7, 8, 9, 10])
Print (rel)

We created a Process Pool that allows five processes ). Every process running in the Pool executes the f () function. We use the map () method to apply the f () function to each element of the table. This is similar to the map () function of built-in, but five processes are used for parallel processing. If an element still needs to be processed after the process finishes running, the process will be used to re-run the f () function. In addition to the map () method, the Pool also has the following common methods.

Apply_async (func, args) extracts a process from the process pool to execute func. args is the func parameter. It returns an AsyncResult object. You can call the get () method on this object to obtain the result.

Close () process pool no longer creates new process

Join () all processes in the wait process pool. You must call the close () method for the Pool to join.

Exercise

There is a file named download.txt below.

Copy codeThe Code is as follows:
Www.sina.com.cn
Www.163.com
Www.iciba.com
Www.cnblogs.com
Www.qq.com
Www.douban.com

Use a process pool containing three processes to download the homepage of the website in the file. (You can use subprocess to call download tools such as wget or curl to execute specific download tasks)

Shared resources

We have mentioned in Python that we should avoid sharing resources among multiple processes as much as possible. Multi-Process Resource Sharing will inevitably lead to competition between processes. This competition will cause race condition, and our results may be affected by the uncertainty of competition. But if necessary, we can still do this through the shared memory and Manager object.

Share "Resources"

Shared Memory

In Linux inter-process communication, we have already talked about the principle of shared memory. Here is an example of using Python:

Copy codeThe Code is as follows:
# Modified from official documentation
Import multiprocessing
Def f (n, ):
N. value = 3.14
A [0] = 5
Num = multiprocessing. Value ('d, 0.0)
Arr = multiprocessing. Array ('I', range (10 ))
P = multiprocessing. Process (target = f, args = (num, arr ))
P. start ()
P. join ()
Print num. value
Print arr [:]

Here, we only have the Process represented by the main Process and the Process object. We create shared memory in the memory space of the main process, that is, two objects, Value and Array. The object Value is set to double precision (d) and initialized to 0.0. Array is similar to an Array in C and has a fixed type (I, that is, an integer ). In the Process, we modified the Value and Array objects. Return to the main program and print out the results. The main program also saw the changes to the two objects, indicating that the resources are indeed shared between the two processes.

Manager

The Manager object is similar to the server-client communication between the server and the customer. It is similar to our activities on the Internet. We use a process as the server and create a Manager to truly store resources. Other processes can access the Manager by passing parameters or by address. After a connection is established, the resources on the server can be operated. As permitted by the firewall, we can apply the Manager to multiple computers, thus imitating a real network situation. In the following example, the use of Manager is similar to shared memory, but more object types can be shared.

Copy codeThe Code is as follows:
Import multiprocessing
Def f (x, arr, l ):
X. value = 3.14
Arr [0] = 5
L. append ('hello ')
Server = multiprocessing. Manager ()
X = server. Value ('d, 0.0)
Arr = server. Array ('I', range (10 ))
L = server. list ()
Proc = multiprocessing. Process (target = f, args = (x, arr, l ))
Proc. start ()
Proc. join ()
Print (x. value)
Print (arr)
Print (l)

Manager uses the list () method to share tables. In fact, you can use dict () to share the dictionary and Lock () to share threading. Lock (note that we share threading. Lock instead of mutiprocessing. Lock of the process. The latter itself has implemented process sharing. In this way, the Manager allows us to share more diverse objects.

Here we will not go into detail on the Remote Application of Manager. If you have the opportunity, you will further explore network applications.

Summary

Pool
Shared memory, Manager