Study and summary of Python coroutine [go]

Brief introduction

Because some openstack-related things need to be researched recently, there are a number of Python-specific features that are encountered while reading some related code, such as generator, coroutine, and some related libraries such as Eventlet, Greenlet. There are a number of third-party class libraries referenced in OpenStack, and these features and class libraries look more complex. If you need to be familiar with the implementation of some of the features in OpenStack, you need to have a good understanding of the underlying things that are involved. Here is a summary of the characteristics of coroutine and its use.

Definition and use of coroutine

In a previous article on generator, I mentioned how to define and use generator. At that time we used yield value or a similar syntax to list comprehension to define generator. The generator we return is actually a collection of data that can be constantly obtained, so the code that uses them generally is a loop. In this case, we feel like we are defining a set of pipelines first and then extracting them in the code when we really need them.

In fact, coroutine and generator are still very related, let's look at the following code:

def grep (pattern):    print "Looking for%s" percent pattern while    True: line        = (yield)        if pattern in line:            pri NT Line,

This part of the code looks like generator, but it's different. Here is a line = (yield) statement. In generator, we need yield value to return the value. You can also print out the values in the back, depending on the value obtained. The code we use them is as follows:

if __name__ = = ' __main__ ':    g = grep ("python")    G.next ()    g.send ("Yeah, but no, but Yeah, but no")    g.send ("A Series of Tubes ")    g.send (" Python generators rock! ")

Here, we define the method grep and then call a Send method. The output of the code execution is as follows:

Looking for Pythonpython generators rock!

In conjunction with the previous code, it looks as if the yield section is followed by the character that contains the Python string. And none of them will handle it. It seems that this yield seems to have some mystery, it does not seem simple.

In fact, we're using line = (yield), which defines a coroutine. What is Coroutine? Coroutine can be said to be a way to realize collaborative programming. It can set up points with multiple entry points and recovery execution, which can enable the transfer of some execution processes. This part of the concept seems a bit difficult to understand. Let's take the previous section of the code as an example to analyze the concepts we want to close.

In our code, the portion of yield that is defined is equivalent to waiting for data to be received. So when there's no data coming, it's the equivalent of being blocked, waiting there. In order to trigger this part, we use G.next () to initialize the code first. The data is then sent to it by means of the G.send ("") method. In this way, yield returns the parameters of the Send method. Then we can proceed to the Loop section to deal with it. In this view, Coroutine is more of a data consumer role. Each time it is waiting for the data to come over, and then return through the yield portion, and then deal with it. Otherwise, it will be there.

Several applications of Coroutine

Because Coroutine is equivalent to a consumer of data, we can do one of these applications. First, a producer prepares some data and then sends the data to a coroutine to handle it. Let's take a look at a file processing example:

Import timedef Follow (thefile, target):    Thefile.seek (0,2)      # Go to the end of the file and    True: line         = t Hefile.readline ()         If not line:             Time.sleep (0.1)    # Sleep briefly             continue         target.send (line)

Here is the process of reading a file and then sending the contents of the file to target via target. So what we're going to do here is pass target as a parameter to the follow method. Of course, target must be a coroutine here to receive and process this data. Target is defined as follows:

@coroutinedef printer ():    while True: line         = (yield)         print line,

The yield is used here, and then the received data can be printed directly. Of course, there is also a notable place to use a @coroutine decorator. Because we need to call Target.next () or Target.send (None) before each use of coroutine to initialize it. So we can easily forget this step when we use it, one way is to define a decorator such as good one, and then each time this decorator is added to ensure that this step is executed. @coroutine decorator is defined as follows:

def coroutine (func):    def start (*args,**kwargs):        cr = Func (*args,**kwargs)        cr.next ()        return CR    return start

For this part of the decorator understanding if there is a problem, you can refer to my decorator related to this article. We end up using the code in the previous section as follows:

if __name__ = = ' __main__ ':    f = open ("Access-log")    Follow (F,printer ())

Of course, this example is mainly about using a coroutine to process a message that passes through. If we want to construct something similar to pipeline, we can use a coroutine as a part of data processing, or as a part of the transmission, for example, look at the following code:

@coroutinedef grep (pattern,target): While    True: line        = (yield)           # Receive A line        if pattern            in line: Target.send (line)    # Send to next stage

Here we define the grep method after receiving the data, the equivalent of making a judgment, if the pattern in the middle of the incoming line, then the data is passed to the next coroutine processing. This is the realization of a pipeline embryonic. Of course, in addition to the example of this transmission, we can also pass the message to multiple coroutine. For these examples we can refer to some of the following information to do detailed analysis.

About Control delivery

The introduction of Coroutine on the wiki, citing the classic Producer-consumer problem. Previously, for this problem, more on the multi-threaded producer-consumer problem processing. In that scenario, we need to implement the processing of the queue elements through locks or certain mutex variables. In fact, it is also an ideal way to solve this problem with coroutine. The pseudo-code given on the wiki is this:

Coroutine produce    Loop while Q isn't full            create some new items add the items to            Q yield to        consume

Coroutine consume    Loop while        Q isn't empty            remove some items from Q with the            items yield to        produce

The most interesting part of the code is that we can do it in a Python coroutine way. For example, we can use the number of elements in the queue as arguments to pass. As a consumer, you can write in this way:

@coroutinedef Consumer (target): While    True:        items = (yield)        if items > 0:            Remove item from queue            Items-= 1        target.send (items)

As producer can also be written in a similar way. We will not repeat it here. Of course, there's one more thing to note, though, is that the transfer of control is not involved in the transformation of the thread, and all of this is actually done in the same process. However, this method of control transfer has more efficient applications in many places, and it has many applications for Event loop and distribution, non-blocking IO access. I will also make further analysis of some of their applications in a number of subsequent articles.

Summarize

Coroutine is a means of switching between multiple tasks, which is equivalent to passing one of the current execution results to another process. Compared with generator, it is more like a "push" mode. Because we want to use a coroutine, it is necessary to have other process send data come over. Because the yield process is somewhat similar to the concept of interruption in an operating system, it is equivalent to pausing the current execution of a process and then jumping to another process. This process is quite different from the subroutine calls we've traditionally implemented through stacks, so it's a little difficult to understand on the surface.

reference materials

Http://en.wikipedia.org/wiki/Coroutine

http://www.dabeaz.com/coroutines/

From:http://www.myexception.cn/perl-python/1673253.html

Study and summary of Python coroutine [go]