A detailed description of how Python handles concurrency using the Asyncio package

This article mainly for you in detail the Python use Asyncio packet processing concurrency related data, with a certain reference value, interested in small partners can refer to

Blocking I/O and Gil

The CPython interpreter itself is not thread-safe, so there is a global interpreter lock (GIL) that allows only one thread to execute Python bytecode at a time. Therefore, a Python process typically cannot use multiple CPU cores at the same time.

However, all functions in the standard library that perform blocking I/O operations will release the Gil when waiting for the operating system to return results. This means that multiple threads can be used at this level in the Python language, and I/O intensive Python programs can benefit: When a Python thread waits for a network response, the blocking I/O function releases the GIL and then runs a thread.

Asyncio

This package implements concurrency using the event loop-driven coprocessor. The yield from expression is heavily used by Asyncio and is therefore incompatible with the old version of Python.

The Asyncio package uses a "co-process" that is a more rigorous definition. The process for Asyncio API must use yield from in the definition body, not yield. In addition, the process for Asyncio is to be driven by the caller and called by the caller through yield from;

Example 1

Import threadingimport asyncio@asyncio.coroutinedef Hello ():  print (' Start hello ', Threading.currentthread ())  yield from Asyncio.sleep (5)  print (' End Hello ', Threading.currentthread ()) @asyncio. Coroutinedef World ():  print (' Start World ', Threading.currentthread ())  yield from Asyncio.sleep (3)  print ("End World", Threading.currentthread ()) # Get Eventloop:loop = Asyncio.get_event_loop () tasks = [Hello (), World ()]# Execution Coroutineloop.run_until_complete (asyncio.wait (Tasks)) Loop.close ()

@asyncio. Coroutine to mark the generator function as the type of the co-process.
Asyncio.sleep (3) creates a co-process that completes after 3 seconds.
Loop.run_until_complete (future), runs until the future is complete, and if the argument is Coroutine object, it needs to be wrapped with the ensure_future () function.
Loop.close () Close event loop

Example 2

Import asyncio@asyncio.coroutinedef Worker (text):  function run by "" "  :p Aram Text:  : return:  " "" I = 0  while True:    the print (text, i)    try:      yield from Asyncio.sleep (. 1)    except Asyncio. Cancellederror:      break    i + = 1@asyncio.coroutinedef Client (text, io_used):  Worker_fu = asyncio.ensure_ Future (Worker (text))  # Pretending to wait for I/O for a period of time  yield from Asyncio.sleep (io_used)  # end run coprocessor  worker_fu.cancel ()  return ' done ' loop = Asyncio.get_event_loop () tasks = [Client (' Xiaozhe ', 3), client (' zzzz ', 5)]result = Loop.run_ Until_complete (asyncio.wait (Tasks)) Loop.close () print (' Answer: ', result)

Explain:

1. Asyncio.ensure_future (coro_or_future, *, Loop=none): Plan to schedule an execution of a Coroutine object, returning a Asyncio. Task object.
2. Worker_fu.cancel (): Cancels execution of a co-process, throws Cancellederror exception.
3. Asyncio.wait (): The parameter of the process is an iterative object consisting of a period or a co-process; Wait will wrap each of the threads into a Task object, respectively.

Asyncio. The Task object is associated with threading. Comparison of Thread objects

Asyncio. The Task object is almost the same as threading. The Thread object is equivalent.
A Task object is used to drive a thread object that invokes a callable object.
The Task object is not instantiated by itself, but is obtained by path to the asyncio.ensure_future (...) function or the Loop.create_task (...) method.
The fetched Task object has been scheduled to run, and the Thread instance must call the Start method and explicitly tell it to run.
If you want to terminate a task, you can use the Task.cancel () instance method to throw an Cancellederror exception inside the process.

Security comparison of threads and processes

If you have done important programming with threads, because the scheduler can break threads at any time. The retention lock must be remembered to protect important parts of the program from being interrupted during execution to prevent the data from being in an invalid state.

The co-process is fully protected by default to prevent interruptions. We have to explicitly output to get the rest of the program to run. To the coprocessor, without having to retain the lock and synchronize the operations between multiple threads, the coprocessor itself synchronizes because only one of the processes is running at any time. To surrender control, you can use yield or yield from to return control to the dispatch program. This is why it is possible to safely cancel the process: by definition, the co-process can only be canceled at the paused yield, so it is possible to handle cancellederror exceptions and perform cleanup operations.

Future (the term object)

In general, you should not create a period object, but only by the concurrency framework (concurrent.futures or Asyncio) instantiation. The reason is simple: The event indicates what will happen, and the only way to make sure something happens is that the time of execution has been scheduled.

Asyncio. Future

In the Asyncio package, the Baseeventloop.create_task (...) method receives a co-process, schedules its run time, and then returns a Asyncio. Task instances--also asyncio. An instance of the future class, because the Task is a subclass of the future and is used to wrap the threads.

Asyncio.ensure_future (coro_or_future, *, Loop=none)

This function unifies the covariance and the period objects: the first parameter can be either one of the two. If it is a future or Task object, it is returned intact. If it is a co-process, the async function calls the Loop.create_task (...) method to create the Task object. The loop= keyword parameter is optional for passing in the event loop, and if not passed in, the async function gets the loop object by calling the Asyncio.get_event_loop () function.

Baseeventloop.create_task (Coro)

This method schedules the execution time of the association and returns a Asyncio. The Task object.

Multiple functions in the Asyncio package automatically wrap the parameters specified in the Asyncio. A Task object, such as the Baseeventloop.run_until_complete (...) method.

asyncio.as_completed

In order to integrate the progress bar, we can use the as_completed generator function; Fortunately, the Asyncio package provides the appropriate version of this generator function.

Using Asyncio and Aiohttp packages

From Python 3.4, the Asyncio package only supports TCP and UDP directly. If you want to use HTTP or other protocols, then third-party package aiohttp.

Cc_list = [' China ', ' USA '] @asyncio. Coroutinedef get_flag (cc):  url = ' {}/{cc}/{cc}.gif '. Format (Base_url, cc= Cc.lower ())  resp = yield from aiohttp.request (' GET ', url)  image = Yield from resp.read ()  return Image@asyncio.coroutinedef Download_one (name):   image = Yield from Get_flag (name)   Save_flag (image, Name.lower () + '. gif ')  return nameloop = Asyncio.get_event_loop () Wait_coro = Asyncio.wait ([Download_one (cc) for CC in sorted (cc _list)]) res, _ = Loop.run_until_complete (Wait_coro) loop.close ()

When using the Asyncio package, we write asynchronous code that contains the Asyncio itself, the delegation generator, which ultimately delegates responsibility to the Asyncio package or to the coprocessor in a third-party library such as Aiohttp. This approach is equivalent to a pipeline that allows the Asyncio event loop (through the coprocessor we write) to drive library functions that perform low-level asynchronous I/O operations.

Avoid blocking calls

There are two ways to avoid blocking calls that abort the entire application process:
1. Run each blocking operation in a separate thread
2. Convert each blocking operation to a non-blocking asynchronous call using

Multiple threads are possible, but each operating system thread (Python uses this thread) consumes megabytes of memory (depending on the type of operating system). We can't afford to handle thousands of connections and use one thread per connection.

The use of generators as writers Association is another way of programming asynchronously. For an event loop, the call callback is almost as effective as calling the. Send () method on a paused thread. Each paused process consumes memory, but is less than the amount of memory consumed by the thread.

Why is the above script quick?

In the above script, when the Loop.run_until_complete method is called, the event loop drives each Download_one, and when it runs to the first yield from expression, that expression drives each get_flag, and then the get_ When the flag association runs inside the first yield from expression, call Aiohttp.request (...). Function. These calls are not blocked, so all requests begin in fraction seconds.

Asyncio infrastructure Once the first response is received, the event loop sends the response to the get_flag that waits for the result. When the response is received, Get_flag forwards to the next yield from expression, invokes the Resp.read () method, and returns control back to the main loop. Other responses will be returned in succession. After all get_ flag processes have obtained the result, the delegate generator Download_one Restore and save the image file.

Async and await

To simplify and better identify asynchronous IO, new syntax async and await are introduced from Python 3.5 to make Coroutine's code more concise and readable.

Async and await are the new syntax for Coroutine, and to use the new syntax, you only need to do a two-step simple substitution.
1. Replace the @asyncio.coroutine with async
2. Replace yield from await

For example:

@asyncio. Coroutinedef Hello ():  print ("Hello world!")  R = Yield from Asyncio.sleep (1)  print ("Hello again!")

Equivalent to

Async def hello ():  print ("Hello world!")  R = await Asyncio.sleep (1)  print ("Hello again!")

Site Request Instance

import asyncioimport aiohttpurls = [' http://www.163.com/', ' http://www.sina.com.cn/', ' HTTPS ://www.hupu.com/', ' http://www.php.cn/']async def get_url_data (U): "" reads data from the URL:p Aram u:: Return: "" "Print (' Runni Ng ', u) async with aiohttp. Clientsession () as Session:async with Session.get (U) as Resp:print (U, resp.status, type (Resp.text ())) # PRI NT (await Resp.text ()) return Resp.headersasync def request_url (U): "" "Main dispatch function:p Aram U:: Return:" "" res = await get  _url_data (u) return resloop = Asyncio.get_event_loop () task_lists = Asyncio.wait ([Request_url (U) for u in URLs]) All_res, _ = Loop.run_until_complete (task_lists) loop.close () print (all_res)