Yield usage in Python

16.yield Use

List derivation and builder expressions

When we create a list, we create an object that we can iterate over:

>>> squares=[n*n forn in range(3)]>>> for i in squares:    printi014

This creates a list of operations that are common, called list inference. But iterators like lists such as STR, file, and so on are handy, but one thing is that they are stored in memory, which can be cumbersome if the values are large.

Unlike a generator expression, it performs calculations that contain the same, but iterative, build results as the list. Its syntax is the same as a list derivation, except that brackets are used instead of brackets:

>>> squares=(n*n forn in range(3))>>> for i in squares:    printi014

The generator expression does not create a sequence of objects, does not read all the values into memory, but creates a generator object (Generator) that iterates through and generates values as required. So, is there any other way to generate the generator?

Example: Fibonacci sequence The Fibonacci sequence refers to a sequence of 1, 1, 2, 3, 5, 8, 13, 21, 34 ...

0 is the No. 0 item, not the first.

This sequence starts with the second item and each item is equal to the sum of the first two items.

For example, there is a requirement to generate the first 10 bits of the Fibonacci sequence, which we can write:

deffib(n):    result=[]    a=1    b=1    result.append(a)    fori inrange(n-1):        a,b=b,a+b        result.append(a)    returnresultif__name__==‘__main__‘:    printfib(10)

When the numbers are small, the function works fine, but when the numbers are large, the problem comes, and obviously generating a thousands of-tens of thousands of-length list is not a good idea. In this way, the requirement becomes: Write a function that can generate an iterative object, or let the function return a value at a time instead of returning all the values at once.

This seems to be contrary to our common sense, when we call a normal Python function, it is usually executed from the first line of the function, ending with a return statement, an exception, or the end of the function (which can be considered an implicit return of none):

def   fib (n):      a = 1      b = 1      for  , i  in   range (n - 1 ):          A-b = B,a + b          return   a if   __name__ = = ' __main__ ' :      print   fib ( ten ) >>> 1      #返回第一个值时就卡住了

Once a function returns control to the caller, it means that it is all over. All the work done in the function and the data saved in the local variables will be lost. When you call this function again, everything will be created from scratch. The function has only one chance to return the result, so all results must be returned at once. Usually we all think so. However, this is not the case:

deffib(n):    a=1    yielda    b=1    fori inrange(n-1):        a,b=b,a+b        yieldaif__name__==‘__main__‘:    fori infib(10):        printi>>>112358132134

Generator generator

The definition of a generator in Python is simple, and a function that uses the yield keyword can be called a generator, which generates a sequence of values:

defcountdown(n):    while n>0:        yield n        n-=1if __name__==‘__main__‘:    for i in countdown(10):        printi

The generator function returns the generator. Note that the generator is a special kind of iterator. As an iterator, the generator must define some methods, one of which is __next__ (). Like an iterator, we can use the next () function (Python3 is __next__ ()) To get the next value:

>>> c=countdown(10)>>> c.next()10>>> c.next()9

Whenever the generator is called, it returns a value to the caller. Use yield within the generator to do this. To remember what yield actually did, the simplest way is to treat it as a special return for the generator function. When you call Next (), The generator function continuously executes the statement until yield is encountered, at which point the "state" of the generator function is frozen, the values of all variables are preserved, and the next line of code to be executed is recorded until another call to next () continues to execute the statement after yield. Next () cannot be executed indefinitely, and when the iteration ends, a stopiteration exception is thrown. If you want to end the generator, you can use the close () method when the iteration is not over.

>>> c.next()1>>> c.next()StopIteration>>> c=countdown(10)>>> c.next()10>>> c.close()>>> c.next()StopIteration

What is the co-process and yield expression Association?

The principle of the process is very simple, to make an analogy can be understood: suppose there are 10 people to eat in the cafeteria, the canteen is poorer, only a dozen rice windows and a dozen rice aunt, then dozen Rice can only one one in line. These 10 people appetite is very big, everyone points 5 dishes, but these 10 people like to hesitate, a la carte when each point of a dish and then want to the next dishes what, so the people in the back are very anxious ah.

This has been standing is not a thing, so the aunt of the dish to see someone hesitated 5 seconds later will Roar, let him line to the end of the team, let others first dozen dishes, and so his turn when he also almost want to eat something. This is really a good way, but there is a disadvantage, that is, the aunt will wait for each person 5 seconds, if the person in 5 seconds did not make a decision to eat what, in fact, this 5 seconds wasted. A person to order a dish is a waste of 5 seconds, 10 people each point 5 dishes can be wasted more "dishes are cool to". What about that? Aunt again said: We are students, students will be conscious, I will not take the initiative to let you line to the end, if you think you will hesitate, take the initiative to point directly to a dish on the back, and so on the next line to the time also almost want to eat what. This method is effective, the first thing you want to order after the food is not the next dish to eat what, but they will not hesitate, if you would hesitate to go directly to the back of the queue, if not then order. In this way the efficiency of the whole team is naturally high.

In this example, the roar of the queue aunt is our CPU interrupt, which is used to toggle the context. Every student who plays a meal is a task. And everyone decides whether or not to let the window of this behavior, in fact, we are the core of the process of thinking.

A process is a function that can pause and resume execution in various predefined locations of code, which avoids meaningless scheduling, thereby improving code performance. And the subroutine is a special kind of cooperative program, it only has a single entrance, through callback to complete execution. The Python process "existing generator-based and new-proposed processes" is not a general-purpose process because they can only transfer control to the caller when execution is paused, rather than transferring control to other threads as usual. Supported by the event loop, the coprocessor can be used for asynchronous processing, especially in I/O.

Let's take a look at the processes and threads first.

The future allows the computer to handle multiple tasks at the same time, the operating system has a process concept, and within the process, the basic can be considered that the current system only one process is running, the operating system is a very good package. Inter-process switching is done with an operating system. The problem with the process is that inter-process switching consumes very large computer resources, and the cost of applying for a new process is very high. So then there was the thread, which generated a much lower cost and switching consumption than the process, and the communication between the threads was very convenient. The thread problem is: 1) there is a thread safety problem, the problem is very difficult to locate. 2) limit on the number of threads within the process. 3) as the amount of concurrency increases, the cost of thread generation and switching becomes expensive.

One solution to concurrency is IO multiplexing, which is really very efficient, but the code complexity is also very high: it breaks a process into a single node, scattered in multiple places, and is very detrimental to development and maintenance (this is the solution we often use). OK, look at how the process solves these problems: 1) The resultant cost of the co-process is lower. is actually a piece of memory that records the stack information of the previous call. You can even further reduce the size of the process by controlling the hierarchy of function calls. To generate a co-process, you only need to request a piece of memory and assign a value. 2) switch faster. The basic is the speed of the memory copy. 3) There is no thread safety issue. There can be multiple concurrent processes within a process, but only one is active, and the programmer is programmed to control the activation and hibernation of the threads, rather than the kernel. This will not be a thread-safe issue. 4) Better readability. The service interface or IO interface you invoke is asynchronous relative to Io multiplexing, but your code is smooth (sequential) and not scrambled by async and callbacks. The co-process is also asynchronous, but it encapsulates asynchronous events and callbacks to form a similar remote invocation interface.

The python implementation of the process:

Yield can be implemented in a co-process. In addition, there are many third-party versions, such as Greenlet.

What can a co-process do?

1) Description Logic: I mainly use the co-process to describe logic. A process may require multiple interfaces to be called, many of which are asynchronous. This can be a bit more difficult to describe. Using threads can solve some problems, but the complexity increases. 2) Increased concurrency: Mainly used in IO-intensive applications. Gevent is a framework for dealing with concurrency on a greenlet basis, and the difference is that the events and interfaces here are IO interfaces.

Defects:

Cannot use multi-core. However, it can be solved by the process + co-path.

The yield statement also has a more powerful function, as a statement appears to the right of the assignment operator, accepts a value, or generates a value at the same time and accepts a value.

def   recv (): code>     print   ' ready '      while   True :           n = yield           print   ' Go%s ' % n >>> c = recv () >>> C. Next () ready >>> c.send ( 1 ) go  1 >>> c.send ( 2 ) go  2

A function that uses the yield statement in this way is called a co-process. In this example, the initial call to next () is necessary so that the process can execute the statement that leads to the first yield expression. Here the coprocessor hangs, waiting for the related generator object Send () method to send a value to it. The value passed to send () is returned by the yield expression in the association.

The run of the process is generally indefinite, and the method close () can be used to close it explicitly.

If a value is provided in the yield expression, the coprocessor can use the yield statement to receive and emit the return value at the same time.

defsplit_line():    print‘ready to split‘    result=None    whileTrue:        line=yield result        result=line.split()>>> s=split_line()>>> s.next()ready to split>>> s.send(‘1 2 3‘)[‘1‘, ‘2‘, ‘3‘]>>> s.send(‘a b c‘)[‘a‘, ‘b‘, ‘c‘]

Note : It is important to understand the sequencing in this example. The first next () method allows the coprocessor to execute to yield result, which returns the value of result none. In the next send () call, the received value is placed in line and split into result. The return value of the Send () method is the value of the next yield statement. That is, the Send () method can pass a value to the yield expression, but its return value comes from the next yield expression, not the yield expression that receives the value passed by Send ().

If you want to use the Send () method to open the execution of the process, you must first send a value of none, because there is no yield statement to accept the value, otherwise it throws an exception.

>>> s=split_line()>>> s.send(‘1 2 3‘)TypeError: can‘t send non-Nonevalue to a just-started generator>>> s=split_line()>>> s.send(None)ready to split

Using the generator and the co-process

At first glance, it doesn't seem obvious how to use generators and co-routines to solve real-world problems. However, generators and co-processes are particularly useful in solving certain problems in system, network, and distributed computing. In fact, yield has become one of the most powerful keywords in python.

For example, to create a pipeline that processes files:

importos,sysdefdefault_next(func):    defstart(*args,**kwargs):        f=func(*args,**kwargs)        f.next()        returnf    returnstart@default_nextdeffind_files(target):    topdir=yield    whileTrue:        forpath,dirname,filelist inos.walk(topdir):            forfilename infilelist:                target.send(os.path.join(path,filename))@default_nextdefopener(target):    whileTrue:        name=yield        f=open(name)        target.send(f)    @default_nextdefcatch(target):    whileTrue:        f=yield        for line inf:            target.send(line)            @default_nextdefprinter():    whileTrue:        line=yield        printline

Then, you can create a data flow processing pipeline by connecting these processes together:

finder=find_files(opener(catch(printer())))finder.send(toppath)

The execution of the program is driven entirely by sending the data to the first find_files (), which is always active until it explicitly calls Close ().

In short, the generator has a very powerful function. The co-process can be used to implement some form of concurrency. In some types of applications, a multi-threaded, Greenlet, collaborative user space can be implemented with a task scheduler and some generators or processes. Yield's power will be true in the process, collaborative multitasking (cooperative multitasking), and asynchronous Io.

Yield usage in Python