Python中並發future模組的介紹（代碼）

本篇文章給大家帶來的內容是關於Python中並發future模組的介紹（代碼） ，有一定的參考價值，有需要的朋友可以參考一下，希望對你有所協助。

concurrent.futures模組

該模組主要特色在於ThreadPoolExecutor 和 ProcessPoolExecutor 類，這兩個類都繼承自concurrent.futures._base.Executor類，它們實現的介面能分別在不同的線程或進程中執行可調用的對象，它們都在內部維護著一個背景工作執行緒或者進程池。

ThreadPoolExecutor 和 ProcessPoolExecutor 類是進階類，大部分情況下只要學會使用即可，無需關注其實現細節。

####ProcessPoolExecutor 類

>class ThreadPoolExecutor(concurrent.futures._base.Executor)>|  This is an abstract base class for concrete asynchronous executors.>|  Method resolution order:>|      ThreadPoolExecutor |      concurrent.futures._base.Executor |      builtins.object | |  Methods defined here: | |  init(self, max_workers=None, thread_name_prefix='') |      Initializes a new ThreadPoolExecutor instance. | |      Args: |          max_workers: The maximum number of threads that can be used to |              execute the given calls. |          thread_name_prefix: An optional name prefix to give our threads. | |  shutdown(self, wait=True) |      Clean-up the resources associated with the Executor. | |      It is safe to call this method several times. Otherwise, no other |      methods can be called after this one. | |      Args: |          wait: If True then shutdown will not return until all running |              futures have finished executing and the resources used by the |              executor have been reclaimed. | |  submit(self, fn, *args, **kwargs) |      Submits a callable to be executed with the given arguments. | |      Schedules the callable to be executed as fn(*args, **kwargs) and returns |      a Future instance representing the execution of the callable. | |      Returns: |          A Future representing the given call. | |  ---------------------------------------------------------------------- |  Methods inherited from concurrent.futures._base.Executor: | |  enter(self) | |  exit(self, exc_type, exc_val, exc_tb) | |  map(self, fn, *iterables, timeout=None, chunksize=1) |      Returns an iterator equivalent to map(fn, iter). | |      Args: |          fn: A callable that will take as many arguments as there are |              passed iterables. |          timeout: The maximum number of seconds to wait. If None, then there |              is no limit on the wait time. |          chunksize: The size of the chunks the iterable will be broken into |              before being passed to a child process. This argument is only |              used by ProcessPoolExecutor; it is ignored by |              ThreadPoolExecutor. | |      Returns: |          An iterator equivalent to: map(func, *iterables) but the calls may |          be evaluated out-of-order. | |      Raises: |          TimeoutError: If the entire result iterator could not be generated |              before the given timeout. |          Exception: If fn(*args) raises for any values.

初始化可以指定一個最大進程數作為其參數 max_workers 的值，該值一般無需指定，預設為當前運行機器的核心數，可以由os.cpu_count()擷取；類中含有方法：

1. map()方法，與python內建方法map() 功能類似，也就是映射，參數為：

• 一個可調用函數 fn

• 一個迭代器 iterables

• 逾時時間長度 timeout

• 塊數chuncksize 如果大於1, 迭代器會被分塊處理

---->> 該函數有一個特性：其返回結果與調用開始的順序是一致的；在調用過程中不會產生阻塞，也就是說可能前者被調用執行結束之前，後者被調用已經執行結束了。

如果一定要擷取到所有結果後再處理，可以選擇採用submit()方法和futures.as_completed函數結合使用。

1. shutdown()方法，清理所有與當前執行器（executor)相關的資源

2. submit() 方法，提交一個可調用對象給fn使用

3. 從concurrent.futures._base.Executor繼承了__enter__() 和 __exit__()方法，這意味著ProcessPoolExecutor 對象可以用於with 語句。

from concurrent import futureswith futures.ProcessPoolExecutor(max_works=3) as executor:     executor.map()

ThreadPoolExecutor類

class ThreadPoolExecutor(concurrent.futures._base.Executor) |  This is an abstract base class for concrete asynchronous executors. | |  Method resolution order: |      ThreadPoolExecutor |      concurrent.futures._base.Executor |      builtins.object | |  Methods defined here: | |  init(self, max_workers=None, thread_name_prefix='') |      Initializes a new ThreadPoolExecutor instance. | |      Args: |          max_workers: The maximum number of threads that can be used to |              execute the given calls. |          thread_name_prefix: An optional name prefix to give our threads. | |  shutdown(self, wait=True) |      Clean-up the resources associated with the Executor. | |      It is safe to call this method several times. Otherwise, no other |      methods can be called after this one. | |      Args: |          wait: If True then shutdown will not return until all running |              futures have finished executing and the resources used by the |              executor have been reclaimed. | |  submit(self, fn, *args, **kwargs) |      Submits a callable to be executed with the given arguments. | |      Schedules the callable to be executed as fn(*args, **kwargs) and returns |      a Future instance representing the execution of the callable. | |      Returns: |          A Future representing the given call. | |  ---------------------------------------------------------------------- |  Methods inherited from concurrent.futures._base.Executor: | |  enter(self) | |  exit(self, exc_type, exc_val, exc_tb) | |  map(self, fn, *iterables, timeout=None, chunksize=1) |      Returns an iterator equivalent to map(fn, iter). | |      Args: |          fn: A callable that will take as many arguments as there are |              passed iterables. |          timeout: The maximum number of seconds to wait. If None, then there |              is no limit on the wait time. |          chunksize: The size of the chunks the iterable will be broken into |              before being passed to a child process. This argument is only |              used by ProcessPoolExecutor; it is ignored by |              ThreadPoolExecutor. | |      Returns: |          An iterator equivalent to: map(func, *iterables) but the calls may |          be evaluated out-of-order. | |      Raises: |          TimeoutError: If the entire result iterator could not be generated |              before the given timeout. |          Exception: If fn(*args) raises for any values.

與ProcessPoolExecutor 類十分相似，只不過一個是處理進程，一個是處理線程，可根據實際需要選擇。

樣本

from time import sleep, strftimefrom concurrent import futuresdef display(*args):    print(strftime('[%H:%M:%S]'), end="")    print(*args)def loiter(n):    msg = '{}loiter({}): doing nothing for {}s'    display(msg.format('\t'*n, n, n))    sleep(n)    msg = '{}loiter({}): done.'    display(msg.format('\t'*n, n))    return n*10def main():    display('Script starting')    executor = futures.ThreadPoolExecutor(max_workers=3)    results = executor.map(loiter, range(5))    display('results:', results)    display('Waiting for inpidual results:')    for i, result in enumerate(results):        display('result {} : {}'.format(i, result))if __name__ == '__main__':    main()

運行結果：

[20:32:12]Script starting[20:32:12]loiter(0): doing nothing for 0s[20:32:12]loiter(0): done.[20:32:12]      loiter(1): doing nothing for 1s[20:32:12]              loiter(2): doing nothing for 2s[20:32:12]results: <generator object Executor.map.<locals>.result_iterator at 0x00000246DB21BC50>[20:32:12]Waiting for inpidual results:[20:32:12]                      loiter(3): doing nothing for 3s[20:32:12]result 0 : 0[20:32:13]      loiter(1): done.[20:32:13]                              loiter(4): doing nothing for 4s[20:32:13]result 1 : 10[20:32:14]              loiter(2): done.[20:32:14]result 2 : 20[20:32:15]                      loiter(3): done.[20:32:15]result 3 : 30[20:32:17]                              loiter(4): done.[20:32:17]result 4 : 40

不同機器運行結果可能不同。

樣本中設定max_workers=3，所以代碼一開始運行，則有三個對象（0,1,2）被執行loiter() 操作; 三秒後，對象0運行結束，得到結果result 0之後，對象3才開始被執行，同理，對象4的執行時間在對象1執行結果result 1列印結束之後。