Python uses the asyncio package to process concurrency details, pythonasyncio

Python uses the asyncio package to process concurrency details, pythonasyncio

Blocked I/O and GIL

CPython interpreter itself is not thread-safe, so there is a global interpreter lock (GIL) that allows only one thread to execute Python bytecode at a time. Therefore, a Python process generally cannot use multiple CPU cores at the same time.

However, all functions in the standard library that execute blocking I/O operations will release GIL while waiting for the operating system to return results. This means that multithreading can be used at the Python level, and I/O-intensive Python programs can benefit from this: a Python thread waits for network response, the blocked I/O function releases GIL and runs another thread.

Asyncio

This package uses the event loop-driven coroutine to implement concurrency. Asyncio uses a large number of yield from expressions, so it is incompatible with the earlier version of Python.

The "coroutine" used by the asyncio package is strictly defined. The coroutine suitable for asyncio APIs must use yield from instead of yield in the definition body. In addition, the coroutine suitable for asyncio should be driven by the caller and called by the caller through yield from;

Example 1

Import threadingimport asyncio@asyncio.coroutinedef hello (): print ('start hello', threading. currentThread () yield from asyncio. sleep (5) print ('end hello', threading. currentThread () @ asyncio. coroutinedef world (): print ('start world', threading. currentThread () yield from asyncio. sleep (3) print ('end world', threading. currentThread () # Get EventLoop: loop = asyncio. get_event_loop () tasks = [hello (), world ()] # execute coroutineloop. run_until_complete (asyncio. wait (tasks) loop. close ()

@ Asyncio. coroutine mark the generator function as the coroutine type.
Asyncio. sleep (3) creates a coroutine that is completed 3 seconds later.
Loop. run_until_complete (future), run until future is complete; if the parameter is coroutine object, use the ensure_future () function to wrap it.
Loop. close () close the event loop

Example 2

Import asyncio@asyncio.coroutinedef worker (text): "coroutine running function: param text: return:" I = 0 while True: print (text, I) try: yield from asyncio. sleep (. 1) snapshot t asyncio. cancelledError: break I + = 1@asyncio.coroutinedef client (text, io_used): worker_fu = asyncio. ensure_future (worker (text) # Pretend to wait for I/O for a while yield from asyncio. sleep (io_used) # end the running coroutine worker_fu.cancel () return 'done' loop = asyncio. get_event_loop () tasks = [client ('xiaozhes', 3), client ('zzzz ', 5)] result = loop. run_until_complete (asyncio. wait (tasks) loop. close () print ('answer: ', result)

Explanation:

1. asyncio. ensure_future (coro_or_future, *, loop = None): schedules the execution of a coroutine object and returns an asyncio. Task object.
2. worker_fu.cancel (): cancels the execution of a coroutine and throws a CancelledError exception.
3. asyncio. wait (): The coroutine parameter is an iterative object consisting of a phase object or coroutine. wait packages each coroutine into a Task object.

Comparison between asyncio. Task object and threading. Thread object

The asyncio. Task object is similar to the threading. Thread object.
The Task object is used to drive the coroutine, And the Thread object is used to call callable objects.
Task objects cannot be instantiated by themselves, but are passed to asyncio. ensure_future (...) Function or loop. create_task (...) Method.
The obtained Task object has been scheduled to run. For a Thread instance, the start method must be called to explicitly instruct it to run.
To terminate a Task, you can use the Task. cancel () instance method to throw a CancelledError exception within the coroutine.

Security comparison between threads and coroutine

If the thread is used for important programming, because the scheduler can interrupt the thread at any time. Remember to keep the lock in mind to protect the important part of the program, prevent the interruption of multi-step operations during execution, and prevent the data from being invalid.

By default, coroutine provides comprehensive protection to prevent interruption. We must make explicit output to run the rest of the program. For coroutine, The coroutine itself will be synchronized without retaining the lock and performing synchronization between multiple threads, because there is only one coroutine running at any time. You can use yield or yield from to return control to the scheduler. This is why the coroutine can be safely canceled: as defined, the coroutine can only be canceled at the paused yield. Therefore, the CancelledError exception can be handled and the cleanup operation can be performed.

Future)

Generally, you should not create a period object, but only instantiate it by the concurrent. futures or asyncio. The reason is simple: a period object indicates what will eventually happen, and the only way to determine whether something will happen is that the execution time has been scheduled.

Asyncio. Future

In the asyncio package, BaseEventLoop. create_task (...) The method receives a coroutine, schedules its running time, and returns an asyncio. Task instance, which is also an instance of the asyncio. Future class, because Task is a subclass of the Future and used to wrap the coroutine.

Asyncio. ensure_future (coro_or_future, *, loop = None)

This function unifies the coroutine and Phase Object: the first parameter can be either of them. If it is a Future or Task object, it will be returned intact. If it is a coroutine, The async function will call loop. create_task (...) Method To create a Task object. The loop = keyword parameter is optional and is used to pass in the event loop. If the parameter is not input, The async function obtains the loop object by calling the asyncio. get_event_loop () function.

BaseEventLoop. create_task (coro)

This method schedules the coroutine execution time and returns an asyncio. Task object.

Multiple Functions in the asyncio package automatically wrap the coroutine specified by parameters in the asyncio. Task object, such as BaseEventLoop. run_until_complete (...) Method.

Asyncio. as_completed

To integrate the progress bar, we can use the as_completed generator function. Fortunately, the asyncio package provides the corresponding version of this generator function.

Use asyncio and aiohttp packets

From Python 3.4, the asyncio package only supports TCP and UDP. If you want to use HTTP or other protocols, you need to use a third-party package aiohttp.

cc_list = ['China', 'USA']@asyncio.coroutinedef get_flag(cc):  url = '{}/{cc}/{cc}.gif'.format(BASE_URL, cc=cc.lower())  resp = yield from aiohttp.request('GET', url)  image = yield from resp.read()  return image@asyncio.coroutinedef download_one(name):   image = yield from get_flag(name)   save_flag(image, name.lower() + '.gif')  return nameloop = asyncio.get_event_loop() wait_coro = asyncio.wait([download_one(cc) for cc in sorted(cc_list)]) res, _ = loop.run_until_complete(wait_coro) loop.close()

When using the asyncio package, the asynchronous code we write contains the coroutine (that is, the delegate generator) driven by the asyncio itself, and the generator finally delegates the responsibility to the asyncio package or a third-party library (such as aiohttp). This processing method is equivalent to setting up a pipeline to enable the asyncio event loop (through the coroutine we have compiled) to drive the library functions that execute low-level asynchronous I/O operations.

Avoid blocking calls

There are two ways to avoid blocking calls to stop the entire application process:
1. run each blocking operation in a separate thread
2. convert each blocking operation into a non-blocking asynchronous call

Multiple Threads are acceptable, but each operating system thread (which Python uses) consumes up to MB of memory (the specific amount depends on the operating system type ). If we want to process thousands of connections and each connection uses one thread, we cannot afford it.

Using the generator as a coroutine is another way of asynchronous programming. For the event loop, calling callback is similar to calling the. send () method on the paused coroutine. The coroutine of each pause consumes memory, but it is smaller than the memory consumed by the thread.

Why is the above script very fast?

In the above script, call loop. when the run_until_complete method is used, the event loop will drive the download_one coroutine. When the first yield from expression is run, the expression will drive the get_flag coroutine, then, when the first yield from expression is run in the get_flag coroutine, aiohttp is called. request (...) Function. These calls are not blocked, so all requests start in a few seconds.

After the infrastructure of asyncio receives the first response, the event loop sends the response to the get_flag coroutine waiting for the result. After the response is obtained, get_flag runs forward to the next yield from expression, calls the resp. read () method, and returns the control to the main loop. Other responses will be returned one after another. After all get _ flag coroutines obtain the result, delegate the generator download_one to restore and save the image file.

Async and await

To simplify and better identify asynchronous IO, a new syntax async and await has been introduced from Python 3.5 to make the coroutine code simpler and easier to read.

Async and await Are New syntaxes for coroutine. To use the New syntaxes, you only need to replace them in two simple steps.
1. Replace @ asyncio. coroutine with async
2. Replace yield from with await

For example:

@asyncio.coroutinedef hello():  print("Hello world!")  r = yield from asyncio.sleep(1)  print("Hello again!")

Equivalent

async def hello():  print("Hello world!")  r = await asyncio.sleep(1)  print("Hello again!")

Website request instance

Import asyncioimport aiohttpurls = ['HTTP: // www.163.com/', 'HTTP: // www.sina.com.cn/', 'https: // www.hupu.com/', 'HTTP: // www.csdn.net/'{async def get_url_data (u): "reads url data: param u: return:" "print ('running', u) async with aiohttp. clientSession () as session: async with session. get (u) as resp: print (u, resp. status, type (resp. text () # print (await resp. text () return resp. headersasync def request_url (u): "main scheduling function: param u: return:" "res = await get_url_data (u) return resloop = asyncio. get_event_loop () task_lists = asyncio. wait ([request_url (u) for u in urls]) all_res, _ = loop. run_until_complete (task_lists) loop. close () print (all_res)

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.