Python standard library 11 multi-process Exploration (multiprocessing package)

Vamei Source: Http://www.cnblogs.com/vamei Welcome reprint, Please also keep this statement. Thank you!

After a preliminary understanding of Python's many processes, we can continue to explore more advanced tools in the multiprocessing package. These tools allow us to implement multiple processes more conveniently.

Process Pool

The process pool can create multiple processes. These processes are like soldiers on standby, ready to perform tasks (procedures). A process pool can accommodate multiple soldiers on standby.

"Process pool of three processes"

For example, the following program:

Import multiprocessing as Muldef f (x):    return x**2pool = mul. Pool (5print (rel)     

We have created a process pool that allows 5 processes. Each process running by the pool executes the f () function. We use the map () method to effect the F () function on each element of the table. This is similar to built-in's map () function, except that it is handled in parallel with 5 processes . If there are elements that need to be processed after the process has finished running, then the process is used to rerun the F () function. In addition to the map () method, the pool has the following common methods.

Apply_async (Func,args) takes a process from the process pool to execute the parameter Func,args to Func. It returns an AsyncResult object that you can call the get () method to get the result.

The close () process pool no longer creates a new process

Join () Wait for all processes in the process pool. You must first call the close () method on the pool to join.

Practice

There is a file download.txt below.

Www.sina.com.cnwww.163.comwww.iciba.comwww.cnblogs.comwww.qq.comwww.douban.com

Use the process pool containing 3 processes to download the first page of the Web site in a file. (You can use subprocess to call wget or curl and other download tools to perform specific download tasks)

shared Resources

As we have already mentioned in the Python multi-process, we should try to avoid multi-process shared resources. Multi-process shared resources will inevitably lead to competition between processes. And this competition will cause race condition, our results may be affected by the uncertainty of competition. But if you want, we can still do this through shared memory and manager objects.

Share "Resources"

Shared memory

In the Linux interprocess communication, we have described the principle of shared memory, which gives examples of Python implementations:

# modified from official Documentationimport Multiprocessingdef F (n, a): n.value   = 3.14 A[0] = 5num   = multiprocessing. Value ( ' d ", 0.0) arr = multiprocessing. Array ( ' i)) P = multiprocessing. Process (target=f, Args= Num.valueprint arr[:]    

Here we actually only have the process represented by the main and process objects. We create shared memory in the memory space of the main process, which is the value and array two objects. The object value is set to the double-precision number (d) and initialized to 0.0. The array is similar to the arrays in C, with a fixed type (I, which is an integer). In the process, we modified the value and the array object. Back to the main program, print out the results, the main program also saw a change of two objects, indicating that the resources are indeed shared between two processes.

Manager

The manager object is similar to the communication between the server and the customer (server-client), similar to our activity on the Internet. We use a process as a server to build the manager to really store the resources. Other processes can be passed through parameters or access the manager based on the address, and after establishing the connection, manipulate the resources on the server. With the firewall permitting, we can fully apply the manager to multiple computers, mimicking a real-world network scenario. In the following example, our use of the manager is similar to shared memory, but we can share a richer object type.

ImportMultiprocessingDefF (x, arr, l): X.value = 3.14Arr[0] = 5L.append ( ' hello ) server = multiprocessing. Manager () x = server. Value ( ' d ", 0.0) arr = server. Array ( ' i) L = Server.list () proc = multiprocessing. Process (target=f, Args= (x, arr, L)) Proc.start () Proc.join () print (X.value) print< Span style= "color: #000000;" > (arr) print (L)          

The manager uses the list () method to provide a way to share tables. You can actually use dict () to share the dictionary,Lock () to share the threading. Lock (Note that we are sharing the threading. Lock, not the mutiprocessing of the process. Lock. The latter itself has realized process sharing) and so on. This allows the manager to allow us to share more diverse objects.

We do not delve into the manager's application in remote situations. If there is a chance, it will be explored further in the network application.

Summary

Pool

Shared Memory, Manager

Python standard library 11 multi-process Exploration (multiprocessing package)