Process pool and thread pool in Python

0.concurrent.futures Library

Reference: https://docs.python.org/3/library/concurrent.futures.html

Before we used multithreading (threading) and multi-process (multiprocessing) to complete the general requirements:    start, Jon and other steps can not be saved, complex need to use 1-2 queues.    as the requirements become more complex, without good design and abstraction of this part of the functional hierarchy, the more code is more difficult to debug. is    there any good way to abstract these steps and let us not pay attention to these details and travel light? The answer is: Yes, starting from Python3.2 a called Concurrent.futures is included in the standard library, while in Python2 it belongs to the third-party futures library and needs to be installed manually: Pip install futuresthe Concurrent.futures module provides a high-level interface for asynchronously executing callables. The asynchronous execution can be is performed by threads using threadpoolexecutor or seperate processes using Processpool Executor. Both implement the same interface, which is defined by the abstract Executor class.

1. Process Pool-The case of serial execution:

Import math,timeprimes = [    112272535095293,    112582705942171,    112272535095293,    115280095190773,    115797848077099,    1099726899285419]def is_prime (n):    if n% 2 = = 0:        return False    sqrt_n = Int ( Math.floor (MATH.SQRT (n)))) for    I in range (3, Sqrt_n + 1, 2):        if n% i = = 0:            return False    return truedef Main (): For    num in PRIMES:        print ('%d is prime:%s '% (num, is_prime (num))) if __name__ = = ' __main__ ':    start_ Time = Time.time ()    main ()    end_time = Time.time ()    print (' Run time is%s '% (end_time-start_time))--- Results---112272535095293 is prime:true112582705942171 are prime:true112272535095293 are prime:true115280095190773 is prime:t rue115797848077099 is prime:true1099726899285419 are prime:falserun time is 3.9570000171661377

-Use multiprocessing. The situation of pool:

Import math,timefrom multiprocessing Import poolprimes = [112272535095293, 112582705942171, 112272535095293, 115280095190773, 115797848077099, 1099726899285419]def is_prime (n): if n% 2 = = 0:return False sqrt_n = Int (Math.floor (MATH.SQRT (n)))) for I in range (3, Sqrt_n + 1, 2): if n% i = = 0:return False retu RN Truedef Main (): Pool = Pool () res_l = [] for prime in primes:res = Pool.apply_async (Func=is_prime,args = (prime,)) Res_l.append (res) pool.close () Pool.join () for number, Prime in Zip (PRIMES, res_l): Prin  T ('%d is prime:%s '% (number, prime.get ())) if __name__ = = ' __main__ ': start_time = Time.time () main () End_time = Time.time () print (' Run time is%s '% (end_time-start_time))---result---112272535095293 is prime:true112582705942171 is PR IME:TRUE112272535095293 is prime:true115280095190773 are prime:true115797848077099 are prime:true1099726899285419 is PRI Me:falserun Time is 2.687000036239624 

-Using process Pools Concurrent.futures.ProcessPoolExecutor of the situation:

-Reference: http://pythonhosted.org/futures/#concurrent. futures.processpoolexecutor

Processpoolexecutor uses the multiprocessing module, which allows it to side-step the Global interpreter Lock but also mea NS that is picklable objects can be executed and Returned.class Concurrent.futures.ProcessPoolExecutor (max_workers= None)    executes calls asynchronously using a pool of at the most max_workers processes.     If Max_workers is None or not given then as many worker processes would be created as the machine has processors.

　　　　-Processpoolexecutor essentially also calls the multiprocessing module

Import math,timefrom Concurrent Import futuresprimes = [    112272535095293,    112582705942171,    112272535095293,    115280095190773,    115797848077099,    1099726899285419]def is_prime (n):    if n% 2 = = 0:        return False    sqrt_n = Int (Math.floor (MATH.SQRT (n)))    for I in range (3, Sqrt_n + 1, 2):        if n% i = = 0:
   return False    return truedef main (): With    futures. Processpoolexecutor () as executor:        for number, Prime in Zip (PRIMES, Executor.map (Is_prime, PRIMES)):            print (' %d is prime:%s '% (number, prime)) if __name__ = = ' __main__ ':    start_time = Time.time ()    main ()    end_time = Ti Me.time ()    print (' Run time is%s '% (end_time-start_time))---result---112272535095293 is prime:true112582705942171 is PRIME:TRUE112272535095293 is prime:true115280095190773 are prime:true115797848077099 are prime:true1099726899285419 is P Rime:falserun Time is 2.482999801635742

2. Thread pool

　　-Reference: http://pythonhosted.org/futures/#threadpoolexecutor-objects

The Threadpoolexecutor class is an Executor subclass, uses a pool of threads to execute calls Asynchronously.class con Current.futures.ThreadPoolExecutor (max_workers)    executes calls asynchronously using at pool of the most max_workers Threads.

-Serial execution of the case:

Import Urllib.requestimport timeurls = [    ' http://www.foxnews.com/',    ' https://www.stanford.edu/',    ' http ://www.mit.edu/',    ' https://www.python.org/',    ' https://www.yahoo.com/',    ' http://www.ox.ac.uk/']def Load_url (URL, timeout):    return Urllib.request.urlopen (URL, timeout=timeout). Read () Start_time = Time.time () for  URL in URLs:    print ('%r page was%d bytes '% (URL, len (Load_url (url,60)))) End_time = Time.time () print ("Run time is%s"% (End_time-start_time)) ---result---' http://www.foxnews.com/' page is 71131 bytes ' https://www.stanford.edu/' page is 68595 bytes '/http www.mit.edu/' page is 21405 bytes ' https://www.python.org/' page was 47701 bytes ' https://www.yahoo.com/' page is 434510 byte S ' http://www.ox.ac.uk/' page is 93411 bytesrun time is 5.068000078201294

-use multithreading in the case:

Import urllib.requestimport timefrom Threading Import threadurls = [    ' http://www.foxnews.com/',    ' https:// www.stanford.edu/',    ' http://www.mit.edu/',    ' https://www.python.org/',    ' https://www.yahoo.com/',    ' http://www.ox.ac.uk/']def load_url (URL, timeout):    res = urllib.request.urlopen (URL, timeout=timeout). Read ()    print ('%r page is%d bytes '% (URL, len (res))] t_l = []start_time = Time.time () for URL in URLs:    t = Thread (targe T=load_url,args= (url,60,))    t_l.append (t)    T.start () for T in t_l:    t.join () End_time = Time.time () print (" Run time is%s "% (end_time-start_time))---result---' http://www.mit.edu/' page is 21403 bytes ' http://www.foxnews.com/' page i s 71735 bytes ' https://www.python.org/' page is 47701 bytes ' https://www.stanford.edu/' page is 69130 bytes '/http www.ox.ac.uk/' page is 93411 bytes ' https://www.yahoo.com/' page was 446715 bytesrun time is 2.6540000438690186

-Using the thread pool Concurrent.futures.ThreadPoolExecutor:

From concurrent import futuresimport urllib.requestimport timeurls = [' http://www.foxnews.com/', ' https://www.stanf ord.edu/', ' http://www.mit.edu/', ' https://www.python.org/', ' https://www.yahoo.com/', ' http://www.ox.ac.uk/'] def load_url (URL, timeout): Return Urllib.request.urlopen (URL, timeout=timeout). Read () Start_time = Time.time () with FUT Ures. Threadpoolexecutor (max_workers=5) as Executor:future_to_url = Dict ((executor.submit (Load_url, URL, $), url) for URL I n URLs) for the future in Futures.as_completed (future_to_url): url = future_to_url[future] if Future.exceptio N () is not none:print ('%r generated an exception:%s '% (Url,future.exception ())) Else:prin T ('%r page is%d bytes '% (URL, len (Future.result ()))) End_time = Time.time () print ("Run time is%s"% (End_time-start_time) )---result---' http://www.mit.edu/' page is 21405 bytes ' http://www.foxnews.com/' page is 71197 bytes ' https://www.python.org /' page is 47701 bytes ' http://www.ox.ac.uk/' page is 93411 bytes ' https://www.yahoo.com/' page was 444854 bytes ' https://www.stanford.edu/' page is 68595 bytesrun Time is 2.497999906539917

　　

Note: Because of network instability factors, run time is only used as a reference value;

Process pool and thread pool in Python