Python Generator and yield

List derivation and builder expressions

When we create a list, we create an object that we can iterate over:

>>> squares=[n*n for N in range (3)]>>> for I in Squares:print i014

This creates a list of operations that are common, called list inference. But iterators like lists such as STR, file, and so on are handy, but one thing is that they are stored in memory, which can be cumbersome if the values are large.

Unlike a generator expression, it performs calculations that contain the same, but iterative, build results as the list. Its syntax is the same as a list derivation, except that brackets are used instead of brackets:

>>> squares= (n*n for N in range (3)) >>> for I in Squares:print i014

The generator expression does not create a sequence of objects, does not read all the values into memory, but creates a generator object (Generator) that iterates through and generates values as required.

So, is there any other way to generate the generator?

Example: Fibonacci sequence

For example, there is a requirement to generate the first 10 bits of the Fibonacci sequence, which we can write:

def fib (n):    result=[]    a=1    b=1    result.append (a) for    I in range (n-1):        a,b=b,a+b        Result.append (a)    return resultif __name__== ' __main__ ':    print fib (10)

When the numbers are small, the function works fine, but when the numbers are large, the problem comes, and obviously generating a thousands of-tens of thousands of-length list is not a good idea.

In this way, the requirement becomes: Write a function that can generate an iterative object, or let the function return a value at a time instead of returning all the values at once.

This seems to be contrary to our common sense, when we call a normal Python function, it is usually executed from the first line of the function, ending with a return statement, an exception, or the end of the function (which can be considered an implicit return of none):

def fib (n):    a=1    b=1 for    i in range (n-1):        a,b=b,a+b        return AIF __name__== ' __main__ ':    Print fib >>> 1    #返回第一个值时就卡住了

Once a function returns control to the caller, it means that it is all over. All the work done in the function and the data saved in the local variables will be lost. When you call this function again, everything will be created from scratch. The function has only one chance to return the result, so all results must be returned at once. Usually we all think so. But what if they are not? Please see the magic Yield:

def fib (n):    a=1    yield a    b=1 for    i in range (n-1):        a,b=b,a+b        yield AIF __name__== ' __main__ ': For    i in fib:        print i>>> 112358132134

Generator generator

The definition of a generator in Python is simple, and a function that uses the yield keyword can be called a generator, which generates a sequence of values:

def countdown (N): While    n>0:        yield n        n-=1if __name__== ' __main__ ': For    I in Countdown (TEN):        Print I

The generator function returns the generator. It is important to note that the generator is a special kind of iterator. As an iterator, the generator must define some methods, one of which is __next__ (). As with iterators, we can use the next () function (Python3 is __next__ ()) to get the next value:

>>> C=countdown (Ten) >>> c.next () 10>>> C.next () 9

Whenever the generator is called, it returns a value to the caller. Use yield inside the generator to do this. The simplest way to remember what yield actually did is to use it as a special return for the generator function. When you call next (), the generator function continuously executes the statement until yield is encountered, at which point the "state" of the generator function is frozen, the values of all variables are preserved, and the next line of code to be executed is recorded until another call to next () continues to execute the statement after yield.

Next () cannot be executed indefinitely, and when the iteration ends, a stopiteration exception is thrown. If you want to end the generator at the end of the iteration, you can use the close () method.

>>> c.next () 1>>> c.next () stopiteration>>> C=countdown (Ten) >>> C.next () 10> >> c.close () >>> c.next () stopiteration

Co-and yield expressions

The yield statement also has a more powerful function, as a statement appears to the right of the assignment operator, accepts a value, or generates a value at the same time and accepts a value.

Def recv ():    print ' ready ' while    True:        n=yield        print ' Go%s '%n>>> c=recv () >>> C.next () ready>>> c.send (1) Go 1>>> c.send (2) Go 2

A function that uses the yield statement in this way is called a co-process. In this example, the initial call to next () is necessary so that the process can execute the statement that leads to the first yield expression. Here the coprocessor hangs, waiting for the related generator object Send () method to send a value to it. The value passed to send () is returned by the yield expression in the association.

The run of the process is generally indefinite, and the method close () can be used to close it explicitly.

If a value is provided in the yield expression, the coprocessor can use the yield statement to receive and emit the return value at the same time.

Def split_line ():    print ' ready to split '    Result=none while    True:        line=yield result        result= Line.split () >>> s=split_line () >>> S.next () ready to split>>> S.send (' 1 2 3 ') [' 1 ', ' 2 ', ' 3 '] >>> S.send (' a b C ') [' A ', ' B ', ' C ']

Note : It is important to understand the sequencing in this example. The first next () method allows the coprocessor to execute to yield result, which returns the value of result none. In the next send () call, the received value is placed in line and split into result. The return value of the Send () method is the value of the next yield statement. That is, the Send () method can pass a value to the yield expression, but its return value comes from the next yield expression, not the yield expression that receives the value passed by Send ().

If you want to use the Send () method to open the execution of the process, you must first send a value of none, because there is no yield statement to accept the value, otherwise it throws an exception.

>>> s=split_line () >>> s.send (' 1 2 3 ') Typeerror:can ' t send non-none value to a just-started generator> ;>> s=split_line () >>> s.send (None) ready to split

Using the generator and the co-process

At first glance, it doesn't seem obvious how to use generators and co-routines to solve real-world problems. However, generators and co-processes are particularly useful in solving certain problems in system, network, and distributed computing. In fact, yield has become one of the most powerful keywords in python.

For example, to create a pipeline that processes files:

Import Os,sysdef Default_next (func):    def start (*args,**kwargs):        f=func (*args,**kwargs)        f.next ()        Return F    return start@default_nextdef Find_files (target):    Topdir=yield while    True: for        path,dirname , FileList in Os.walk (topdir): for            filename in filelist:                target.send (Os.path.join (path,filename)) @default_ Nextdef opener (target): While    True:        name=yield        f=open (name)        target.send (f)    @default_ Nextdef catch (target): While    True: F=yield for line in        F:            target.send (line)            @default_ Nextdef printer (): While    True:        line=yield        Print Line

Then, you can create a data flow processing pipeline by connecting these processes together:

Finder=find_files (Opener (catch (printer))) Finder.send (Toppath)

The execution of the program is driven entirely by sending the data to the first find_files (), which is always active until it explicitly calls Close ().

In short, the generator has a very powerful function. The co-process can be used to implement some form of concurrency. In some types of applications, a multi-threaded, Greenlet, collaborative user space can be implemented with a task scheduler and some generators or processes. Yield's power will be true in the process, collaborative multitasking (cooperative multitasking), and asynchronous Io.