Python standard library multi-process (multiprocessing package) Introduction _python

After a preliminary understanding of the Python multi process, we can continue to explore the more advanced tools in the multiprocessing package. These tools can make it easier for us to implement multiple processes.

Process Pool

Process pool can create multiple processes. These processes are like soldiers on standby, ready to perform tasks (procedures). A pool of processes can hold more than one standby soldier.

"Process pool for three processes"

For example, the following program:

Copy Code code as follows:

Import multiprocessing as Mul
def f (x):
Return x**2
Pool = Mul. Pool (5)
rel = Pool.map (f,[1,2,3,4,5,6,7,8,9,10])
Print (REL)

We created a process pool that allows 5 processes. Each process running by pool executes the f () function. We use the map () method to effect the F () function on each element of the table. This is similar to the built-in map () function, except that it is handled in parallel with 5 processes. If the process is finished and there are elements that need to be processed, the process is used to rerun the F () function. In addition to the map () method, pool has the following common methods.

Apply_async (Func,args) takes a process from the process pool and executes Func,args as Func parameters. It returns a AsyncResult object that you can call the Get () method for the result.

The close () process pool no longer creates a new process

Join () The entire process in the pool of the process. The close () method must be called on pool before join.

Practice

There is a file download.txt below.

Copy Code code as follows:

www.sina.com.cn
Www.163.com
Www.iciba.com
Www.cnblogs.com
Www.qq.com
Www.douban.com

Download the first page of the Web site in the file using a process pool containing 3 processes. (You can use the subprocess call wget or curl download tools to perform specific download tasks)

Shared resources

As we have initially mentioned in the Python multi process, we should try to avoid multiple processes sharing resources. Multi-process shared resources will inevitably lead to competition between processes. And this kind of competition can cause race condition, our result may be affected by the uncertainty of competition. But we can still do this with shared memory and manager objects if we need to.

Share "Resources"

Shared memory

In the case of Linux interprocess communication, we've talked about the principle of shared memory, and here's an example of a Python implementation: memory

Copy Code code as follows:

# Modified from official documentation
Import multiprocessing
def f (N, a):
N.value = 3.14
A[0] = 5
num = multiprocessing. Value (' d ', 0.0)
arr = multiprocessing. Array (' I ', Range (10))
p = multiprocessing. Process (target=f, args= (num, arr))
P.start ()
P.join ()
Print Num.value
Print arr[:]

Here we actually have only the processes represented by the main process and the Process object. We create shared memory in the main process's memory space, which is the value and array two objects. The object value is set to a double-precision number (d) and is initialized to 0.0. The array is similar to the one in C and has a fixed type (I, that is, an integer). In process processes, we modify the value and array objects. Back to the main program, print out the results, the main program also saw two objects changed, indicating that resources are indeed shared between two processes.

Manager

The manager object is similar to the communication between the server and the customer (server-client), similar to our activities on the Internet. We use a process as a server to build the manager to really store resources. Other processes can be passed through parameters or access to the manager based on the address, and after the connection is established, the resources on the server are manipulated. With the firewall allowed, we can completely apply the manager to multiple computers, thus mimicking a real-world network situation. In the following example, our use of the manager is similar to shared memory, but it is possible to share a richer object type.

Copy Code code as follows:

Import multiprocessing
def f (x, arr, L):
X.value = 3.14
Arr[0] = 5
L.append (' Hello ')
Server = multiprocessing. Manager ()
x = Server. Value (' d ', 0.0)
arr = Server. Array (' I ', Range (10))
L = server.list ()
proc = multiprocessing. Process (Target=f, args= (x, arr, L))
Proc.start ()
Proc.join ()
Print (X.value)
Print (arr)
Print (L)

Manager uses the list () method to provide a way to share tables. You can actually use Dict () to share the dictionary, Lock () to share the threading. Lock (Note that we are sharing the threading.) Lock, not the mutiprocessing of the process. Lock. The latter itself has implemented process sharing) and so on. This manager allows us to share more diverse objects.

We do not delve into the manager's application in remote situations here. If there is a chance, it will be further explored in the network application.

Summarize

Pool
Shared Memory, Manager