Python function Programming Guide (4): Generator

The generator is an iterator, and it is not just an iterator. However, there are not many uses outside the iterator, so we can say it out loud: the generator provides a convenient way to customize the iterator.

This is the last article in the functional programming guide. It seems that it took a week to write it well ......

For reprint, please specify the original author and original address :)

4. Generator 4.1. Introduction to Generator

First, be sure that the generator is an iterator. The generator has the next method and has the same behavior as the iterator, which means that the generator can also be used in a python for loop. In addition, the special syntax support for generators makes it much easier to compile a generator than to customize a conventional iterator, so the generator is also one of the most common features.

Since Python 2.5, [pep 342: Implementing a collaborative program through an enhanced generator] adds more features to the generator, which means that the generator can do more work. This part will be introduced later.

4.2. Generator function 4.2.1. Use generator Function Definition Generator

How to obtain a generator? First, let's look at a short piece of code:

>>> def get_0_1_2():...   yield 0...   yield 1...   yield 2...>>> get_0_1_2<function get_0_1_2 at 0x00B2CB70>

We defined a function get_0_1_2, and we can see that this is indeed a function type. However, unlike general functions, the get_0_1_2 function uses the keyword yield, which makes get_0_1_2 a generator function. The features of generator functions are as follows:

1. Call the generator function to return a generator;

   >>> generator = get_0_1_2()>>> generator<generator object get_0_1_2 at 0x00B1C7D8>

2. When the next method of the generator is called for the first time, the generator starts to execute the generator function (instead of building the generator) until the execution (suspension) is paused in the case of yield ), the yield parameter is used as the return value of the next method;

   >>> generator.next()0

3. Then, each time the next method of the generator is called, the generator will resume the execution of the generator function from the position where the execution was paused last time until the execution of the generator function is paused again when yield is encountered, and the same is true, the yield parameter will be used as the return value of the next method;

   >>> generator.next()1>>> generator.next()2

4. If the generator function ends when the next method is called (an empty Return Statement is encountered or it reaches the end of the function body ), the next method call will throw a stopiteration exception (that is, the termination condition of the for loop );

   >>> generator.next()Traceback (most recent call last):  File "<stdin>", line 1, in <module>StopIteration

5. Every time a generator function is paused, all the variables in the function will be stored in the generator and restored when the function is resumed, similar to a closure, even if a generator function returns a generator, the stored variables are independent of each other.
   
   Variables are not used in our small example, so another generator is defined here to demonstrate this feature:

   >>> def fibonacci():...   a = b = 1...   yield a...   yield b...   while True:...     a, b = b, a+b...     yield b...>>> for num in fibonacci():...   if num > 100: break...   print num,...1 1 2 3 5 8 13 21 34 55 89

   Don't be surprised when you see while true. Because the generator can be suspended, the computation is delayed and the infinite loop does not matter. In this example, we define a generator to obtain the Fibonacci series.

4.2.2. FAQs about generator Functions

Next we will discuss some interesting topics about generators.

1. In your example, the generator function does not have any parameters. Can the generator function contain parameters?

   Yes, of course. It also supports all the parameter forms of the function. You need to know that the generator function is also a function :)

   >>> def counter(start=0):...   while True:...     yield start...     start += 1...

   This is a counter starting with a specified number.

2. Since the generator function is also a function, can it use return to output the return value?
   
   If this is not the case, the generator function already has the default return value-generator, and you cannot return another value. However, it can end with an empty return statement. If you insist on specifying the return value for it, python will give a syntax error exception at the defined position, as shown in the following code:

   >>> def i_wanna_return():...   yield None...   return None...  File "<stdin>", line 3SyntaxError: 'return' with argument inside generator

3. Well, the people need to ensure that the resources are released, and yield needs to be in try... finally. Will this be the case? (I just want to play with you) I have returned yield in finally!
   
   Python will execute the code in finally when it really leaves try... finally, and I'm sorry to tell you that the pause is not counted! So you can guess the ending!

   >>> def play_u():...   try:...     yield 1...     yield 2...     yield 3...   finally:...     yield 0...>>> for val in play_u(): print val,...1 2 3 0

   * This is different from return. Return is the true exit code block, so the finally clause is executed immediately at return.

   * In addition, "yield in the try block with a finally clause" is defined in pep 342, which means that only Python 2.5 and later versions support this syntax, A syntax error occurs in versions earlier than Python 2.4.

4. What if I need to integrate the iteration of another generator in the generator iteration process? It's so silly and naive to write it as below ..

   >>> def sub_generator():...   yield 1...   yield 2...   for val in counter(10): yield val...

   In this case, the syntax improvement has been defined in [pep 380: Delegate to the sub-generator syntax]. It is said that it will be implemented in Python 3.3 and may be fed back to 2.x. After implementation, you can write it like this:

   >>> def sub_generator():...   yield 1...   yield 2...   yield from counter(10)  File "<stdin>", line 4    yield from counter(10)             ^SyntaxError: invalid syntax

   What are syntax errors? Let's be naive now ~

More questions? Please reply to this article :)

4.3. coroutine)

A collaborative program (coroutine) generally refers to such a function:

• They have different local variables and command pointers, but still share global variables;
• It can be conveniently suspended and restored, and there are multiple entry points and exit points;
• Multiple cooperative programs run collaboratively. For example, the execution of a requires the result of B to continue.

The feature of coroutine determines that only one coprocessor is running at a time (regardless of multithreading ). Thanks to this, the coroutine can directly pass objects without considering resource locks, or directly wake up other coroutine without active sleep, just like a thread with built-in locks. In scenarios that meet the characteristics of coroutine, it is easier to use coroutine than to use threads.

On the other hand, coroutine cannot be concurrent, but it also limits its application scenarios to a very narrow range. This feature allows coroutine to be more compared with conventional functions, instead of with the thread. Of course, threads are much more complex than coroutine threads and have more powerful functions. Therefore, I suggest you master the threads firmly: Python thread guide

In this section, I will not list examples of coroutine. The following describes the methods.

Python 2.5 enhances the generator to implement other features of the coroutine. In this version, the generator adds the following methods:

1. Send (value ):

   Send is another method to restore the generator except next. In Python 2.5, the yield statement is changed to the yield expression, which means that yield can now have a value, which is when the send method of the generator is called to resume execution, the parameter that calls the send method.

   >>> def repeater():...   n = 0...   while True:...     n = (yield n)...>>> r = repeater()>>> r.next()0>>> r.send(10)10

   * Before sending a non-none value, the generator must be in the suspended state; otherwise, an exception will be thrown. However, the unstarted generator can still use none as the parameter to call send.

   * If you use the next recovery generator, the yield expression value will be none.

2. Close ():

   This method is used to disable the generator. If you call next or send again after the generator is closed, a stopiteration exception is thrown.

3. Throw (type, value = none, traceback = none ):

   This method is used to throw an exception within the generator (where the generator is currently suspended or defined when it is not started.

* Don't regret the absence of coroutine examples. The most common use of coroutine is actually generator.

4.4. An interesting Library: Pipe

In this section, I would like to brief you on pipe. Pipe is not a python built-in library, if you install easy_install, you can install it directly, otherwise you need to download it yourself: http://pypi.python.org/pypi/pipe

This library is introduced because it shows us a very new way to use the iterator and generator: stream. Pipe regards iterated data as a stream. Similar to Linux, pipe uses '|' to transmit data streams and defines a series of "stream processing" functions to accept and process data streams, and finally output the data stream again or summarize the data stream to get a result. Let's look at some examples.

First, it is very simple to use add to sum:

>>> from pipe import *>>> range(5) | add10

The where clause is used to obtain the number of parity values, which is similar to the built-in filter function to filter out qualified elements:

>>> range(5) | where(lambda x: x % 2 == 0) | add6

Do you still remember the Fibonacci sequence generator we defined? Find all the even numbers less than 10000 in the number of columns and use take_while. functions with the same name as itertools have similar functions, intercepting elements until the condition is not true:

>>> fib = fibonacci>>> fib() | where(lambda x: x % 2 == 0)\...       | take_while(lambda x: x < 10000)\...       | add3382

To apply a function to an element, you can use select, which is similar to the built-in Function Map. To obtain a list, you can use as_list:

>>> fib() | select(lambda x: x ** 2) | take_while(lambda x: x < 100) | as_list[1, 1, 4, 9, 25, 64]

PIPE also includes more stream processing functions. You can even define a stream processing function by yourself. You only need to define a generator function and add the modifier pipe. The following defines a stream processing function that gets elements until the index does not meet the conditions:

>>> @Pipe... def take_while_idx(iterable, predicate):...   for idx, x in enumerate(iterable):...     if predicate(idx): yield x...     else: return...

Use this stream processing function to obtain the first 10 digits of Fib:

>>> fib() | take_while_idx(lambda x: x < 10) | as_list[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

More functions are not introduced here. You can view the pipe source files. For a total of less than 600 lines of files, 300 of them are documents, which contain a large number of examples.

PIPE is very simple to implement. It uses the pipe decorator to aggregate common generator functions (or return the functions of the iterator) the proxy can be used on a common class instance that implements the _ ror _ method, but this idea is really interesting.

The full text of the functional programming guide is all over here. I hope this series of articles will help you. I hope you can see some programming methods outside of Structured Programming and use them in the right place :)

Tomorrow, I will sort out a directory for you to view and list some articles for your reference. Unfortunately, these articles are almost all in English. Please study English hard --#