Python multi-process concurrent operations in the process Pool instance, pythonpool

Python multi-process concurrent operations in the process Pool instance, pythonpool

When using Python for system management, especially operating multiple file directories at the same time, or remotely controlling multiple hosts, parallel operations can save a lot of time. When the number of objects to be operated is small, you can directly use the Process in multiprocessing to dynamically generate multiple processes. A dozen processes are fine, but if they are hundreds or thousands of targets, manually limiting the number of processes is too cumbersome. At this time, it is time for the process Pool to play a role.

The Pool can provide a specified number of processes for users to call. When a new request is submitted to the pool, if the Pool is not full, A new process will be created to execute the request. However, if the number of processes in the pool has reached the maximum value, the request will wait until the process in the pool ends, to create a new process. Here is a simple example:

#!/usr/bin/env python#coding=utf-8"""Author: SquallLast modified: 2011-10-18 16:50Filename: pool.pyDescription: a simple sample for pool class"""from multiprocessing import Poolfrom time import sleepdef f(x):  for i in range(10):    print '%s --- %s ' % (i, x)    sleep(1)def main():  pool = Pool(processes=3)  # set the processes max number 3  for i in range(11,20):    result = pool.apply_async(f, (i,))  pool.close()  pool.join()  if result.successful():    print 'successful'if __name__ == "__main__":  main()

First, create a 3 process pool, then pass f (I) to it in sequence, run the script, and use ps aux | grep pool. view the Process status in py and you will find that only three processes can be executed at most. Pool. apply_async () is used to submit target requests to the process pool. pool. join () is used to wait for the worker Process in the process pool to complete execution and prevent the master process from ending before the worker process ends. But pool. join () must be used after pool. close () or pool. terminate. The difference between close () and terminate () Is that close () will wait until the worker Process in the pool finishes executing and then close the pool, while terminate () will directly close. Result. successful () indicates the invocation status of the entire call. If there is still a worker that has not been executed, an AssertionError exception is thrown.

Using the Pool in multiprocessing can easily process hundreds or thousands of parallel operations at the same time, greatly reducing the complexity of the script.

----------------------------------

Python multi-process concurrency (multiprocessing)

Due to restrictions of Python design (I am talking about CPython ). A maximum of one CPU core can be used.

Python provides a very useful multi-process package multiprocessing. You only need to define a function, and Python will do everything else for you. With this package, you can easily complete the conversion from a single process to concurrent execution.

1. Create a single process

If we create a small number of processes, we can:

import multiprocessingimport timedef func(msg):for i in xrange(3):print msgtime.sleep(1)if __name__ == "__main__":p = multiprocessing.Process(target=func, args=("hello", ))</ p.start()p.join()print "Sub-process done."

2. Use the process pool

Yes, you are not mistaken, not the thread pool. It allows you to run a full range of multi-core CPUs, and the usage is very simple.

Note that you must use apply_async. If async is dropped, the version will become blocked.

Processes = 4 is the maximum number of concurrent processes.

importmultiprocessingimporttime deffunc(msg):  foriinxrange(3):    printmsg    time.sleep(1) if__name__=="__main__":  pool=multiprocessing.Pool(processes=4)  foriinxrange(10):    msg="hello %d"%(i)    pool.apply_async(func,(msg,))  pool.close()  pool.join()  print"Sub-process(es) done." 

3. Use the Pool and follow the results

In more cases, we not only need to execute multiple processes, but also pay attention to the execution results of each process, as shown below:

import multiprocessingimport timedef func(msg):for i in xrange(3):print msgtime.sleep(1)return "done " + msgif __name__ == "__main__":pool = multiprocessing.Pool(processes=4)result = []for i in xrange(10):msg = "hello %d" %(i)result.append(pool.apply_async(func, (msg, )))pool.close()pool.join()for res in result:print res.get()print "Sub-process(es) done."

2014.12.25 update

According to feedback from comments from netizens, running in Windows may crash (a lot of new Windows and processes are enabled) and can be solved through the following calls:

multiprocessing.freeze_support()

Simple worker multiprocessing. Pool

Multi-task model design is a complicated logic, but python has various convenient class libraries for multi-task processing, and does not need to tangle the operation details between processes or threads. For example, multiprocessing. Pool is one of them.

The official example is also very simple.

from multiprocessing import Pooldef f(x):  return x*xif __name__ == '__main__':  pool = Pool(processes=4)       # start 4 worker processes  result = pool.apply_async(f, [10])  # evaluate "f(10)" asynchronously  print result.get(timeout=1)      # prints "100" unless your computer is *very* slow  print pool.map(f, range(10))     # prints "[0, 1, 4,..., 81]"

I didn't give much detailed explanation. I have a piece of code at hand. I need to request hundreds of URLs and parse the html page to obtain some information. The single-thread for loop is very inefficient. So I saw this module, to implement multi-task analysis, refer to the code below:

from multiprocessing import Pooldef analyse_url(url):  #do something with this url  return analysis_resultif __name__ == '__main__':  pool = Pool(processes=10)  result = pool.map(analyse_url, url_list)

It is indeed much faster than the previous single-thread for loop url_list list to request analyse_url one by one, but the problem is that once the pool. if map is not executed, ctrl-c will interrupt the program, and the program will be abnormal and will never exit. Refer to stackoverflow's post and change it to the following code:

#result = pool.map(analyse_url, url_list)result = pool.map_async(analyse_url, url_list).get(120)

Now the problem is solved perfectly.

The above example of the process Pool in the Python multi-process concurrent operations is all the content shared by the editor. I hope to give you a reference, and I hope you can provide more support to the customer center.