Detailed description of the asyncio module in python and pythonasyncio

Detailed description of the asyncio module in python and pythonasyncio

I have always been interested in the asyncio library. After all, this is a highly recommended module on the official website to implement high concurrency. python also introduced the concept of coroutine in python 3.4. Through this sort, we can better understand the use of this module.

What does asyncio do?

Asynchronous Network Operation concurrency coroutine

In the python3.0 era, Asynchronous Network modules in the standard library: In the select (very underlying) python3.0 era, third-party Asynchronous Network Libraries: Tornado python3.4 era, asyncio: Support for TCP and sub-Processes

Now asyncio, with a lot of modules already supported: aiohttp, aiodns, aioredis and so on https://github.com/aio-libs here list the supported content, and in the continuous update

Of course, so far, not only asyncio has implemented coroutine, but tornado and gevent have both implemented similar functions.

Description of some asyncio keywords:

Event_loop event loop: The program starts an infinite loop and registers some functions to the event loop. When an event is met, the corresponding coroutine function is called.

Coroutine: coroutine object refers to a function defined by the async keyword. Its call does not execute the function immediately, but returns a coroutine object. The coroutine object needs to be registered to the event loop, which is called cyclically by the event.

Task: A coroutine object is a native function that can be suspended. The task further encapsulates the coroutine, including the various statuses of the task.

Future: the result of a task that will be executed or has not been executed. It is essentially different from a task.

Async/await Keyword: python3.5 is used to define the key word of the coroutine. async defines a coroutine. await is used to suspend the blocked asynchronous call interface.

After reading the above keywords, you may turn your head away. In fact, at the beginning, I learned that there was a conflict with the asyncio module, and I didn't know why. This also caused a long time, this module is basically not focused on and used by myself, but as python encounters various performance problems at work, I will tell myself that I should study this module well.

Define a coroutine

Import timeimport asyncionow = lambda: time. time () async def do_some_work (x): print ("waiting:", x) start = now () # Here is a coroutine object, at this time, the do_some_work function does not execute coroutine = do_some_work (2) print (coroutine) # create an event looploop = asyncio. get_event_loop () # Add the coroutine to the looploop of the event loop. run_until_complete (coroutine) print ("Time:", now ()-start)

In the above section, we define a coroutine using the async keyword. Of course, coroutine cannot be run directly and must be added to the event loop.

Asyncio. get_event_loop: creates an event loop, uses run_until_complete to register the coroutine to the event loop, and starts the event loop.

Create a task

The coroutine object cannot be run directly. When registering an event loop, the run_until_complete method encapsulates the coroutine as a task object. the task object is a subclass of the Future class. It stores the status after the coroutine runs and is used to obtain the coroutine result in the Future.

import asyncioimport timenow = lambda: time.time()async def do_some_work(x):  print("waiting:", x)start = now()coroutine = do_some_work(2)loop = asyncio.get_event_loop()task = loop.create_task(coroutine)print(task)loop.run_until_complete(task)print(task)print("Time:",now()-start)

Result:

<Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex2.py:13>>waiting: 2<Task finished coro=<do_some_work() done, defined at /app/py_code/study_asyncio/simple_ex2.py:13> result=None>Time: 0.0003514289855957031

After a task is created, it is in pending state before the task is added to the event loop. After the task is completed, the status is finished.

You can also use asyncio. ensure_future (coroutine) to create a task by using loop. create_task (coroutine ).

Explanation of the two Commands: https://docs.python.org/3/library/asyncio-task.html.

asyncio.ensure_future(coro_or_future, *, loop=None)¶Schedule the execution of a coroutine object: wrap it in a future. Return a Task object.If the argument is a Future, it is returned directly.

Https://docs.python.org/3/library/asyncio-eventloop.html

AbstractEventLoop.create_task(coro)Schedule the execution of a coroutine object: wrap it in a future. Return a Task object.Third-party event loops can use their own subclass of Task for interoperability. In this case, the result type is a subclass of Task.This method was added in Python 3.4.2. Use the async() function to support also older Python versions.

Binding callback

Bind the callback to obtain the execution result when the task is executed. The final parameter of the callback is the future object, through which the coroutine return value can be obtained.

import timeimport asyncionow = lambda : time.time()async def do_some_work(x):  print("waiting:",x)  return "Done after {}s".format(x)def callback(future):  print("callback:",future.result())start = now()coroutine = do_some_work(2)loop = asyncio.get_event_loop()task = asyncio.ensure_future(coroutine)print(task)task.add_done_callback(callback)print(task)loop.run_until_complete(task)print("Time:", now()-start)

Result:

<Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex3.py:13>><Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex3.py:13> cb=[callback() at /app/py_code/study_asyncio/simple_ex3.py:18]>waiting: 2callback: Done after 2sTime: 0.00039196014404296875

The add_done_callback method is used to add a callback function to a task. When the task (or coroutine) is completed, the callback function is called. The result of coroutine execution is obtained through the parameter future. Here, the task we created and the future object in the callback are actually the same object.

Blocking and await

Async can be used to define the coroutine object. await can be used to suspend time-consuming operations. Just like yield in the generator, the function gives control. When a coroutine encounters await, the event loop will suspend the coroutine and execute another coroutine until other coroutine is also suspended or executed, and then the next coroutine is executed.

Time-consuming operations are generally IO operations, such as network requests and file reading. We use the asyncio. sleep function to simulate IO operations. The purpose of coroutine is to make these IO operations asynchronous.

Import asyncioimport timenow = lambda: time. time () async def do_some_work (x): print ("waiting:", x) # await is followed by calling the time-consuming operation await asyncio. sleep (x) return "Done after {} s ". format (x) start = now () coroutine = do_some_work (2) loop = asyncio. get_event_loop () task = asyncio. ensure_future (coroutine) loop. run_until_complete (task) print ("Task ret:", task. result () print ("Time:", now ()-start)

In await asyncio. sleep (x), because sleep is used to simulate blocking or time-consuming operations, this will give control. That is, when a function is blocked, the await method is used to control the coroutine so that the loop can call other coroutine.

Concurrency and Parallelism

Concurrency refers to a system with multiple activities at the same time.

Parallelism is worth using concurrency to make a system run faster. Parallelism can be used in multiple abstract layers of the operating system.

Therefore, concurrency usually means that multiple tasks need to be executed at the same time, while parallelism means that multiple tasks are executed at the same time.

The following example is very vivid:

In the case of concurrency, a teacher assists different people in their homework at the same time. In parallel, several teachers assist multiple students in their homework at the same time. Simply put, if one person eats three steamed buns at the same time or three others eat one at the same time, it is a task to eat one steamed bread.

import asyncioimport timenow = lambda :time.time()async def do_some_work(x):  print("Waiting:",x)  await asyncio.sleep(x)  return "Done after {}s".format(x)start = now()coroutine1 = do_some_work(1)coroutine2 = do_some_work(2)coroutine3 = do_some_work(4)tasks = [  asyncio.ensure_future(coroutine1),  asyncio.ensure_future(coroutine2),  asyncio.ensure_future(coroutine3)]loop = asyncio.get_event_loop()loop.run_until_complete(asyncio.wait(tasks))for task in tasks:  print("Task ret:",task.result())print("Time:",now()-start)

Running result:

Waiting: 1Waiting: 2Waiting: 4Task ret: Done after 1sTask ret: Done after 2sTask ret: Done after 4sTime: 4.004154920578003

The total time is about 4s. 4s congestion time, enough for the first two coroutines to be completed. If it is a synchronization sequence task, it takes at least 7 s. Now we use aysncio to implement concurrency. Asyncio. wait (tasks) can also use asyncio. gather (* tasks). The former accepts a list of tasks, and the latter receives a bunch of tasks.

Description on the official asyncio. gather and asyncio. wait websites:

Https://docs.python.org/3/library/asyncio-task.html

Return a future aggregating results from the given coroutine objects or futures.All futures must share the same event loop. If all the tasks are done successfully, the returned future's result is the list of results (in the order of the original sequence, not necessarily the order of results arrival). If return_exceptions is true, exceptions in the tasks are treated the same as successful results, and gathered in the result list; otherwise, the first raised exception will be immediately propagated to the returned future.

Https://docs.python.org/3/library/asyncio-task.html

Wait for the Futures and coroutine objects given by the sequence futures to complete. Coroutines will be wrapped in Tasks. Returns two sets of Future: (done, pending).The sequence futures must not be empty.timeout can be used to control the maximum number of seconds to wait before returning. timeout can be an int or float. If timeout is not specified or None, there is no limit to the wait time.return_when indicates when this function should return.

Coroutine nesting

Async can be used to define coroutine. coroutine is used for time-consuming io operations. We can also encapsulate more io operation processes to implement nested coroutine, that is to say, await has another coroutine in one process, so it is connected.

import asyncioimport timenow = lambda: time.time()async def do_some_work(x):  print("waiting:",x)  await asyncio.sleep(x)  return "Done after {}s".format(x)async def main():  coroutine1 = do_some_work(1)  coroutine2 = do_some_work(2)  coroutine3 = do_some_work(4)  tasks = [    asyncio.ensure_future(coroutine1),    asyncio.ensure_future(coroutine2),    asyncio.ensure_future(coroutine3)  ]  dones, pendings = await asyncio.wait(tasks)  for task in dones:    print("Task ret:", task.result())  # results = await asyncio.gather(*tasks)  # for result in results:  #   print("Task ret:",result)start = now()loop = asyncio.get_event_loop()loop.run_until_complete(main())print("Time:", now()-start)

If we put the following in the above Code:

  dones, pendings = await asyncio.wait(tasks)  for task in dones:    print("Task ret:", task.result())

Replace:

  results = await asyncio.gather(*tasks)  for result in results:    print("Task ret:",result)

The result list is displayed.

If the await content is directly returned instead of the processing result in the main coroutine function, then the outermost run_until_complete will return the result of the main coroutine. Change the above Code:

import asyncioimport timenow = lambda: time.time()async def do_some_work(x):  print("waiting:",x)  await asyncio.sleep(x)  return "Done after {}s".format(x)async def main():  coroutine1 = do_some_work(1)  coroutine2 = do_some_work(2)  coroutine3 = do_some_work(4)  tasks = [    asyncio.ensure_future(coroutine1),    asyncio.ensure_future(coroutine2),    asyncio.ensure_future(coroutine3)  ]  return await asyncio.gather(*tasks)start = now()loop = asyncio.get_event_loop()results = loop.run_until_complete(main())for result in results:  print("Task ret:",result)print("Time:", now()-start)

Or use asyncio. wait to suspend the coroutine.

Change the code:

import asyncioimport timenow = lambda: time.time()async def do_some_work(x):  print("waiting:",x)  await asyncio.sleep(x)  return "Done after {}s".format(x)async def main():  coroutine1 = do_some_work(1)  coroutine2 = do_some_work(2)  coroutine3 = do_some_work(4)  tasks = [    asyncio.ensure_future(coroutine1),    asyncio.ensure_future(coroutine2),    asyncio.ensure_future(coroutine3)  ]  return await asyncio.wait(tasks)start = now()loop = asyncio.get_event_loop()done,pending = loop.run_until_complete(main())for task in done:  print("Task ret:",task.result())print("Time:", now()-start)

You can also use the as_completed method of asyncio.

import asyncioimport timenow = lambda: time.time()async def do_some_work(x):  print("waiting:",x)  await asyncio.sleep(x)  return "Done after {}s".format(x)async def main():  coroutine1 = do_some_work(1)  coroutine2 = do_some_work(2)  coroutine3 = do_some_work(4)  tasks = [    asyncio.ensure_future(coroutine1),    asyncio.ensure_future(coroutine2),    asyncio.ensure_future(coroutine3)  ]  for task in asyncio.as_completed(tasks):    result = await task    print("Task ret: {}".format(result))start = now()loop = asyncio.get_event_loop()loop.run_until_complete(main())print("Time:", now()-start)

We can also see from the above that the call and combination of coroutine is flexible, mainly reflected in the processing of results: How to return, how to suspend

Stopovers

The future object has several statuses:

Pending Running Done Cacelled

When you create a future, the task is pending, and the execution of the event loop call is running. After the call is completed, the task is canceled first if you need to stop the event loop. You can use asyncio. Task to obtain the task of the event loop.

import asyncioimport timenow = lambda :time.time()async def do_some_work(x):  print("Waiting:",x)  await asyncio.sleep(x)  return "Done after {}s".format(x)coroutine1 =do_some_work(1)coroutine2 =do_some_work(2)coroutine3 =do_some_work(2)tasks = [  asyncio.ensure_future(coroutine1),  asyncio.ensure_future(coroutine2),  asyncio.ensure_future(coroutine3),]start = now()loop = asyncio.get_event_loop()try:  loop.run_until_complete(asyncio.wait(tasks))except KeyboardInterrupt as e:  print(asyncio.Task.all_tasks())  for task in asyncio.Task.all_tasks():    print(task.cancel())  loop.stop()  loop.run_forever()finally:  loop.close()print("Time:",now()-start)

After the event loop is started, ctrl + c will immediately trigger the execution exception KeyBorardInterrupt of run_until_complete. Then, use asyncio. Task to cancel the future. The output is as follows:

Waiting: 1Waiting: 2Waiting: 2^C{<Task finished coro=<do_some_work() done, defined at /app/py_code/study_asyncio/simple_ex10.py:13> result='Done after 1s'>, <Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex10.py:15> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/local/lib/python3.5/asyncio/tasks.py:428]>, <Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex10.py:15> wait_for=<Future pending cb=[Task._wakeup()]> cb=[_wait.<locals>._on_completion() at /usr/local/lib/python3.5/asyncio/tasks.py:428]>, <Task pending coro=<wait() running at /usr/local/lib/python3.5/asyncio/tasks.py:361> wait_for=<Future pending cb=[Task._wakeup()]>>}FalseTrueTrueTrueTime: 1.0707225799560547

True indicates that the cannel is successful. After loop stop, you must enable the event loop again and close it. Otherwise, an exception is thrown.

Loop tasks, one by one cancel is a solution, but as we have encapsulated the list of tasks in the main function above, the main function calls events cyclically. At this time, the main function is equivalent to the most out-of-the-box task, so you can process the packaged main function.

Event loops of different threads

Most of the time, our event loop is used to register the coroutine, and some coroutine needs to be dynamically added to the event loop. A simple method is to use multiple threads. The current thread creates an event loop, and then creates a new thread to start the event loop in the new thread. The current thread is not blocked.

import asynciofrom threading import Threadimport timenow = lambda :time.time()def start_loop(loop):  asyncio.set_event_loop(loop)  loop.run_forever()def more_work(x):  print('More work {}'.format(x))  time.sleep(x)  print('Finished more work {}'.format(x))start = now()new_loop = asyncio.new_event_loop()t = Thread(target=start_loop, args=(new_loop,))t.start()print('TIME: {}'.format(time.time() - start))new_loop.call_soon_threadsafe(more_work, 6)new_loop.call_soon_threadsafe(more_work, 3)

After the above Code is started, the current thread will not be blocked. In the new thread, the more_work method registered by the call_soon_threadsafe method will be executed in order. The latter is due to time. the sleep operation is synchronous blocking, so it takes about 6 + 3 to run more_work.

New thread coroutine

import asyncioimport timefrom threading import Threadnow = lambda :time.time()def start_loop(loop):  asyncio.set_event_loop(loop)  loop.run_forever()async def do_some_work(x):  print('Waiting {}'.format(x))  await asyncio.sleep(x)  print('Done after {}s'.format(x))def more_work(x):  print('More work {}'.format(x))  time.sleep(x)  print('Finished more work {}'.format(x))start = now()new_loop = asyncio.new_event_loop()t = Thread(target=start_loop, args=(new_loop,))t.start()print('TIME: {}'.format(time.time() - start))asyncio.run_coroutine_threadsafe(do_some_work(6), new_loop)asyncio.run_coroutine_threadsafe(do_some_work(4), new_loop)

In the preceding example, create a new_loop in the main thread and enable an infinite event loop in another child thread. The main thread registers the coroutine object through run_coroutine_threadsafe. In this way, the concurrent operations of the event loop can be performed in the Child thread, and the main thread will not be blocked. The total execution time is about 6 seconds.