Concurrent.futures of Python concurrent modules (i)

Python3.2 started, the standard library provides us with the Concurrent.futures module, which provides Threadpoolexecutor and Processpoolexecutor two classes, which enables further threading and multiprocessing Abstraction, which provides direct support for writing thread pool/process pools, he belongs to the upper layer of encapsulation, and for users, there is no need to consider so many things.

Official reference: https://pythonhosted.org/futures/

1.Executor

Exectuor is the base module, which is an abstract class whose subclasses are divided into Threadpoolexecutor and processpoolexecutor, which are used to create thread pools and process pools, respectively.

The following methods are provided:

executor.submit (FN, *args, **kwargs)

fn: For functions that need to be executed asynchronously
Args,kwargs: Parameters passed to the function
Just take a look at this example of the official website:

 withThreadpoolexecutor (max_workers=1) asExecutor
Future = Executor.submit (POW,323,1235)
Print (Future.result ())

We use the Submit method to add a task (POW function) to the thread pool, and submit returns a future object. Where the result method of the Future.result () function is to get the results returned by the call. If you do not finish, you will wait. Here we use the WITH operator, so that when the task executes, the SHUTDOWN function is executed automatically without writing the related release code.
A concrete approach to more future is explained in the later part of the future.

Executor.map (FN, *args, **kwargs)

Map (func, *iterables, Timeout=none)
This map function is similar to Python's own map function, except that the map function of the concurrent module executes asynchronously after the parameter is obtained from the iterator. And, each asynchronous operation, can use timeout parameter to set timeout time, timeout value can be int or float type, if Operation timeout, will raisestimeouterror. If the timeout parameter is not specified, no time is set.

Func: For functions that need to be executed asynchronously
Iterables: Can be an object that can iterate.
Timeout: Sets the time-out for each asynchronous operation

 fromConcurrent.futuresImportThreadpoolexecutor
ImportRequests
URLS = [' http://www.163.com ',' https://www.baidu.com/',' https://github.com/']
 def load_url(URL):
req= requests.get (URL, timeout= -)
Print'%r page is%d bytes '% (URL, len (req.content)))
Executor = Threadpoolexecutor (max_workers=3)
Executor.map (Load_url,urls)
Print' main thread end ')

The submit function and the map function, depending on the need, choose one to use.

Executor.shutdown (wait=true)

This function is used to release system resources after an asynchronous execution operation. Executor implements enter__ and __exit so that its objects can use the with operator.
Here you can use the WITH context keyword instead, as in the first submit example above.

2.Future Objects

The Submit function returns the future object, and the future provides a way to track the execution state of the task, which can be created by the Executor.submit () method. You should not create it directly except for testing.

Cancel (): Attempt to cancel the call. Cannot be canceled if the call is currently executing. This method will return false, otherwise the call will be canceled and the method will return true

Cancelled (): If the call is successfully canceled returns true

Running (): cannot be canceled if it is currently being executed return true

Done (): If the call is successfully canceled or completed running returns true

Result (Timeout = None): Gets the results returned by the call. If it's not done, we'll wait.

Exception (Timeout=none): Capturing exceptions during program execution

Add_done_callback (FN): binds FN to the future object. When the future object is canceled or finished running, the FN function is called

3.wait Method

The wait method then returns a tuple (tuple) with two sets (set), one completed (completed) and the other uncompleted (unfinished). One advantage of using the wait method is that you get greater freedom, which receives three parameters first_completed, First_exception, and All_complete, which are set to all_completed by default.

If the default all_completed is used, the program blocks until all the tasks in the thread pool are complete, and then the main thread is executed:

#!/usr/bin/env python
# Encoding:utf-8
 fromConcurrent.futuresImportthreadpoolexecutor,wait,as_completed
ImportRequests
URLS = [' http://www.163.com ',' https://www.baidu.com/',' https://github.com/']
 def load_url(URL):
req = Requests.get (URL, timeout= -)
Print'%r page is%d bytes '% (URL, len (req.content)))
Executor = Threadpoolexecutor (max_workers=3)
F_list = []
 forUrlinchURLS:
Future = Executor.submit (Load_url,url)
F_list.append (future)
Print (Wait (f_list))
Print' main thread end ')

If the first_completed parameter is used, the program does not wait for all tasks in the thread pool to complete.

 fromConcurrent.futuresImportthreadpoolexecutor,wait,as_completed
ImportRequests
URLS = [' http://www.163.com ',' https://www.baidu.com/',' https://github.com/']
 def load_url(URL):
Req=requests.get (URL, timeout= -)
Print'%r page is%d bytes '% (URL, len (req.content)))
Executor = Threadpoolexecutor (max_workers=3)
F_list = []
 forUrlinchURLS:
Future = Executor.submit (Load_url,url)
F_list.append (future)
Print (Wait (f_list,return_when=' first_completed '))
Print' main thread end ')

The basic use of the module is the above. Follow-up will do some expansion or case.

Concurrent.futures of Python concurrent modules (i)