Python co-process

Disclaimer: I have all the works, prohibited for marketing purposes of any reprint!

Objective

A long time ago I heard that Python's async/await is very powerful, but it has not been used until now and has been using multithreaded models to solve various problems. Recently saw the Go next door is very hot, so decided to take the time to study the Python process related content, and finally after looking through a underwear of the material has some understanding.

Up: Everything starts with the generator

In the past, in Python development, if concurrent programming was required, it was usually done using a multithreaded/multi-process model. Because of the GIL's existence, multithreading is not very friendly to computationally intensive tasks, and for IO-intensive tasks, thread scheduling can be done while waiting for Io, giving the Gil a "false concurrency".

Of course, for IO-intensive tasks, the other option is the co-process, which is actually running in a single thread, avoiding thread context switching in a multithreaded model, and reducing the overhead. To understand the process, async/await, and Asyncio, we start with the oldest generators.

Recalling the history of Python, the concept of generator was proposed for the first time in Pep 255, when the Python version was Python2.2. We all know the range () function, and now consider that we're going to write a range () function of our own, which is probably the most straightforward and easiest way to think about it:

When you want to create a small sequence, such as creating a list from 0 to 100, there seems to be no problem. But if you want to create a list of sizes from 0 to 999999999, you have to create a complete list with a length of 999999999, which is very memory intensive. So there is the generator, and the generator to rewrite this function, it will look like this:

When the function execution encounters yield, it pauses execution. This simply maintains memory space in memory where an integer can be stored.

Cheng: The Birth of the Association

It may not have anything to do with the process, but in fact it's already a prototype of the Python process, so let's look at the definition of the process on the wiki:

Coroutines is computer program components The generalize subroutines for non-preemptive multitasking, by allowing

Multiple entry points for suspending and resuming execution at certain locations.

From some point of view, the process is actually a function that can be paused and can resume execution. Then the yield can be paused, and if there is a way to send some value back to the paused function after the pause, then Python has a "co-process". So in pep 342, we added the "Send something back to the already paused generator" method, which is send (), and is implemented in Python2.5. Using this feature, we continue to rewrite

Range () function:

In this way, the whole generator seems to have entered the stable state, but in Python3.3, the situation has changed. In PEP 380, the yield from is added for Python3.3, which allows you to return any value from the iterator (using an iterator here, because the generator is also an iterator), or you can refactor the generator, let's take a look at this example:

This feature also allows generators to be concatenated, allowing data to be passed through multiple generators. As history has developed, the emergence of the association seems to have been one step closer, or it is more appropriate to say asynchronous programming. The Asyncio library was added to the Python3.4, allowing Python to get the characteristics of the event loop (the contents of the event loop are not described here). The Asyncio + Generator has reached the conditions of asynchronous programming, and in Python3.4 we can implement an asynchronous model like this:

Here the Asyncio.coroutine decorator is used to mark this function as a co-process, because Asyncio requires all generators to be writers association to be decorated by asyncio.coroutine. In this code, the time loop starts two countdown (), and they execute until the yield from Asyncio.sleep () is encountered, and a asyncio is paused. The future object is returned to the event loop. The event loop will monitor the Asyncio. The future object, once its execution is completed, will return the result of this future execution to the process that was just paused because of the future, and continue with the original process.

From this specific example, the abstract point of view, in fact, the process has become:

1. You can do it for any asyncio. Future object to yield from, the future object to the event loop;

2. The execution of the suspension will wait for the future to complete;

3. Once the future gets to the event loop and executes all the code;

4. The event loop senses that the future execution is complete, and the original paused process obtains the return value of the future object through the Send () method and continues execution;

Turn: from yield to await

Finally to the most exciting place, in Python3.5, added the Types.coroutine decorator as well as async def and await. Let's take a look at how the Python3.4 and Python3.5 define a co-process function:

It seems simpler to define the Python3.5, but in fact the difference between the generator and the process becomes more pronounced. Here we first point out two points of attention:

1. Await can only be used in the function of async def;

2. Yield from cannot be used in the function of async def;

In addition, the acceptable object for yield from and await is slightly different, and the object to be accepted by await must be a awaitable object. What is a Awaitable object, is an object that implements the __await () __ method, and this method must return an iterator that is not a co-process. Satisfies these two conditions, only then is a Awaitable object, certainly the association process itself is also the Awaitable object, because Collections.abc.Coroutine inherits Collections.abc.Awaitable. In other words, there are two types of objects that can be accepted after an await: the co-and Awaitable objects, and, of course, the Awaitable object.

In Python3.6, this feature continues to flourish and can now use yield and await in the same function body, and in addition to this, you can use the async for or await syntax in a list derivation.

End

The history of the entire process has been reviewed, and there is some understanding of the process in Python!

Python co-process