Generator and yield in Python

This article mainly introduces Generator and yield in Python. This article describes list derivation and Generator expressions, Fibonacci series, Generator, coroutine and yield expressions, and Generator and coroutine usage, for more information, see List derivation and generator expression

When we create a list, we create an object that can be iterated:

The code is as follows:

>>> Squares = [n * n for n in range (3)]
>>> For I in squares:
Print I

0
1
4

This kind of list creation operation is very common, called list derivation. However, the list iterators, such as str and file, are easy to use, but they are stored in the memory. if the value is large, it will be very troublesome.

The generator expression is different. it executes the same computation as the list, but iteratively generates results. Its syntax is the same as list derivation, but it should replace brackets with parentheses:

The code is as follows:

>>> Squares = (n * n for n in range (3 ))
>>> For I in squares:
Print I

0
1
4

Generator expressions do not create objects in sequence form, and do not read all values into the memory. Instead, they create a Generator object (Generator) that is iterated and generates values as needed ).

Is there any other way to generate a generator?

Example: Fibonacci series

For example, to generate the first 10 digits of the Fibonacci series, we can write as follows:

The code is as follows:

Def fib (n ):
Result = []
A = 1
B = 1
Result. append ()
For I in range (n-1 ):
A, B = B, a + B
Result. append ()
Return result
If _ name __= = '_ main __':
Print fib (10)

When there are few numbers, the function runs well, but when there are many numbers, the problem arises. it is not a good idea to generate a list of thousands or tens of thousands of lengths.

In this way, the requirement becomes: write a function that can generate iteratable objects, or do not let the function return all values at a time, but return a value at a time.

This seems to be contrary to our common sense. when we call a common Python function, it is generally executed from the first line of code of the function, it ends with the return statement, exception, or function (which can be considered as an implicit return of None ):

The code is as follows:

Def fib (n ):
A = 1
B = 1
For I in range (n-1 ):
A, B = B, a + B
Return
If _ name __= = '_ main __':
Print fib (10)
>>>
1 # gets stuck when the first value is returned

Once the function returns the control to the caller, it means the operation is complete. All the work done in the function and the data stored in the local variable will be lost. When you call this function again, everything will be created from scratch. A function has only one chance to return results, so all results must be returned at a time. We usually think so. But what if they are not? See the magic yield:

The code is as follows:

Def fib (n ):
A = 1
Yield
B = 1
For I in range (n-1 ):
A, B = B, a + B
Yield
If _ name __= = '_ main __':
For I in fib (10 ):
Print I
>>>
1
1
2
3
5
8
13
21
34

Generator

The definition of a generator in python is very simple. a function using the yield keyword can be called a generator, which generates a sequence of values:

The code is as follows:

Def countdown (n ):
While n> 0:
Yield n
N-= 1
If _ name __= = '_ main __':
For I in countdown (10 ):
Print I

Generator function return generator. Note that the generator is a special iterator. As an iterator, the generator must define some methods, one of which is _ next __(). Like the iterator, we can use the next () function (Python3 is _ next _ () to obtain the next value:

The code is as follows:

>>> C = countdown (10)
>>> C. next ()
10
>>> C. next ()
9

Whenever the generator is called, it returns a value to the caller. Use yield in the generator to complete this action. To remember what yield did, the simplest way is to treat it as a special return for the generator function. When next () is called, the generator function continues to execute statements until yield is encountered. at this time, the "state" of the generator function will be frozen, and the values of all variables will be retained, the location of the code to be executed in the next line will also be recorded until the next () command is called again to continue executing the statement after yield.

Next () cannot be executed infinitely. when iteration ends, a StopIteration exception is thrown. If you want to end the generator when iteration is not over, you can use the close () method.

The code is as follows:

>>> C. next ()
1
>>> C. next ()
StopIteration
>>> C = countdown (10)
>>> C. next ()
10
>>> C. close ()
>>> C. next ()
StopIteration

Coroutine and yield expressions

Yield statements have more powerful functions. as a statement that appears on the right of the value assignment operator, it accepts a value, or generates a value at the same time and accepts a value.

The code is as follows:

Def recv ():
Print 'ready'
While True:
N = yield
Print 'go % s' % n
>>> C = recv ()
>>> C. next ()
Ready
>>> C. send (1)
Go 1
>>> C. send (2)
Go 2

The functions that use the yield statement in this way are called coroutines. In this example, the initial call to next () is essential, so that the coroutine can execute the statement that can lead to the first yield expression. Here, the coroutine will be suspended and wait for the relevant generator object to send a value to it by the send () method. The value passed to send () is returned by the yield expression in the coroutine.

The coroutine generally runs indefinitely. you can use close () to explicitly close it.

If a value is provided in the yield expression, the coroutine can use the yield statement to receive and send return values simultaneously.

The code is as follows:

Def split_line ():
Print 'ready to split'
Result = None
While True:
Line = yield result
Result = line. split ()
>>> S = split_line ()
>>> S. next ()
Ready to split
>>> S. send ('1 2 3 ')
['1', '2', '3']
>>> S. send ('a B c ')
['A', 'B', 'C']

Note: it is important to understand the order in this example. The first next () method allows the coroutine to execute the yield result, which returns the value of result None. In the next send () call, the received value is placed in line and split into result. The return value of the send () method is the value of the next yield statement. That is to say, the send () method can pass a value to the yield expression, but its return value comes from the next yield expression, rather than the yield expression that receives the value passed by send.

If you want to use the send () method to enable coroutine execution, you must first send a None value, because there is no yield statement to accept the value at this time, otherwise an exception will be thrown.

The code is as follows:

>>> S = split_line ()
>>> S. send ('1 2 3 ')
TypeError: can't send non-None value to a just-started generator
>>> S = split_line ()
>>> S. send (None)
Ready to split

Use the generator and coroutine

At first glance, it does not seem obvious how to use generators and coroutines to solve practical problems. However, generators and coroutines are particularly useful in solving system, network, and distributed computing problems. In fact, yield has become one of the most powerful keywords in Python.

For example, to create a file processing pipeline:

The code is as follows:

Import OS, sys
Def default_next (func ):
Def start (* args, ** kwargs ):
F = func (* args, ** kwargs)
F. next ()
Return f
Return start
@ Default_next
Def find_files (target ):
Topdir = yield
While True:
For path, dirname, filelist in OS. walk (topdir ):
For filename in filelist:
Target. send (OS. path. join (path, filename ))

@ Default_next
Def opener (target ):
While True:
Name = yield
F = open (name)
Target. send (f)

@ Default_next
Def catch (target ):
While True:
F = yield
For line in f:
Target. send (line)

@ Default_next
Def printer ():
While True:
Line = yield
Print line

Connect these coroutines to create a data stream processing pipeline:

The code is as follows:

Finder = find_files (opener (catch (printer ())))
Finder. send (toppath)

The execution of the program is completely driven by sending data to the first coroutine find_files (). The coroutine pipeline will remain active forever until it explicitly calls close ().

In short, the generator is very powerful. Coroutine can be used to achieve some form of concurrency. In some types of applications, you can use a Task Scheduler and some generators or coroutines to Implement Collaborative User space multithreading, that is, greenlet. Yield's power will be truly reflected in coroutine, cooperative multitasking, and asynchronous IO.