Detailed yield and generators generator

The generator and yield keywords may be one of the most powerful and difficult-to-understand concepts in Python (perhaps none), but it does not prevent yield from becoming the most powerful keyword in python, which is very difficult for beginners to understand, and to come to an article written by a foreign bull on yield. , allowing you to quickly understand yield. The article is a bit long, please read patiently, some examples in the process, gradually, so that you do not feel boring.

Generator

A generator is a function that consists of one or more yield expressions, each of which is an iterator (but the iterator is not necessarily a generator).

If a function contains the yield keyword, the function becomes a generator.

Instead of returning all the results at once, the generator returns the result each time it encounters the yield keyword, preserving the current running state of the function and waiting for the next call.

Since the generator is also an iterator, it should support the next method to get the next value. ( You can also use the. __next__ () property, which is. Next () in Python2)

Co-routines and subroutines

When we call a normal Python function, we typically start with the first line of the function, ending with a return statement, an exception, or the end of the function (which can be considered an implicit return of none). Once the function returns control to the caller, it means that it is all over. All the work done in the function and the data saved in the local variables will be lost. When you call this function again, everything will be created from scratch.

This is a very standard process for the functions discussed in computer programming. Such a function can only return a value, but sometimes it can be helpful to create a function that produces a sequence. To do this, this function needs to be able to "save its own work".

As I said, being able to "produce a sequence" is because our function does not return as usual. Return implies that the function is returning control of the execution code to the place where the function was called. The implied meaning of "yield" is that the transfer of control is temporary and voluntary, and our function will regain control in the future.

In Python, a "function" with this ability is called a generator, which is very useful. The initial introduction of the generator (and the yield statement) is intended to allow programmers to write more easily the code that produces the sequence of values. Previously, to implement something like a random number generator, you needed to implement a class or a module that kept track of the state between each invocation while generating the data. Once the generator is introduced, this becomes very simple.

To better understand the problem that the generator solves, let's look at an example. In the process of understanding this example, always remember the problem we need to solve: the sequence of generated values.

Note: Outside of Python, the simplest generator should be something called coroutines. In this article, I will use this term. Keep in mind that in the Python concept, the process referred to here is the generator. The formal terminology of Python is the generator; The process is only easy to discuss and is not formally defined at the language level.

Example: Interesting primes

Suppose your boss lets you write a function, the input parameter is a list of int, and returns a result that can iterate with the number of primes 1.

Remember that iterators (iterable) are just one of the abilities of an object to return a particular member at a time.

You must think "This is very simple" and then quickly write the following code:

def get_primes (input_list):    result_list = list () for    element in Input_list:        if Is_prime (element):            Result_list.append ()    return result_list# or better ... def get_primes (input_list):    return (element for element in Input_list if Is_prime (Element) # Below is an implementation of is_prime ... def is_prime (number):    If number > 1:        if number = = 2:
  
   return True        If number% 2 = = 0:            return False for current in        range (3, int (math.sqrt (number) + 1), 2):            I F number% Current = = 0:                 return false        return True    return False
  

The implementation of the above is_prime completely satisfies the requirement, so we told the boss that it was done. She said that our function worked properly and that was exactly what she wanted.

Handling Infinite Sequences

Oh, is that so? A few days later, the boss came to tell us that she had some minor problems: she was going to use our get_primes function for a large list with numbers. In fact, this list is very large, just creating this list will run out of all of the system's memory. To do this, she wants to be able to take a start parameter when calling the Get_primes function and return all primes that are larger than this parameter (perhaps she wants to resolve Project Euler problem 10).

Let's take a look at this new demand, and it's clear that simply modifying get_primes is not possible. Naturally, we cannot return a list containing all the primes from start to infinity (although there are many useful applications that can be used to manipulate an infinite sequence). It seems that the likelihood of dealing with the problem with ordinary functions is slim.

Before we give up, let's identify the core barrier that prevents us from writing functions that meet the new needs of the boss. By thinking, we get the conclusion that the function has only one chance to return the result, and therefore must return all results at once. It seems pointless to draw such a conclusion: "That's not how the function works", as we all usually think. However, do not learn, do not ask not know, "if they are not so?" ”

Imagine if Get_primes could simply return the next value instead of returning all the values at once, what can we do? We no longer need to create a list. Without a list, there is no memory problem. Because the boss told us that she only had to traverse the results, she would not know the difference between our implementations.

Unfortunately, this seems unlikely. Even if we have a magical function that allows us to traverse from N to infinity, we also get stuck after returning the first value:

def get_primes (start): for    element in Magical_infinite_range (start):        if Is_prime (element):            return Element

Assume this to call Get_primes:

Def solve_number_10 ():    # She *is* working on Project Euler #10, I knew it!    Total = 2 for    next_prime in Get_primes (3):        if Next_prime < 2000000: Total            + = Next_prime        Else:            Print (total)            return

Obviously, in get_primes, the input equals 3 and is returned on line 4th of the function. Unlike direct return, what we need is a value that can be prepared for the next request on exit.

But the function can't do that. When the function returns, it means that it is all done. We guarantee that the function can be called again, but we can't guarantee that, "Well, this time from line 4th on the last exit, instead of the regular start from the first line." The function has only one single entry: The 1th line of code for the function.

Into the generator

Such problems are so common that python specifically joins a structure to solve it: generators. A generator will "generate" a value. Creating a generator is almost as simple as the principle of a generator function.

The definition of a generator function is much like a normal function, except that it uses the yield keyword instead of return when it wants to generate a value. If the body of a def contains yield, the function will automatically become a generator (even if it contains a return). In addition to the above, there is no extra step in creating a generator.

The generator function returns an iterator to the generator. This is probably the last time you see the term "generator iterator," because they are often referred to as "generators." It is important to note that the generator is a special kind of iterator. As an iterator, the generator must define some methods, one of which is __next__ ()"Note: in Python2: Next () method". As with iterators, we can use the next () function to get the next value.

To get the next value from the generator, we use the next () function, just like with an iterator.

(Next () will worry about how to call the generator's __next__ () method). Since the generator is an iterator, it can be used in a for loop.

Whenever the generator is called, it returns a value to the caller. Use yield within the generator to do this (for example, yield 7). The simplest way to remember what yield actually did is to use it as a special return (plus a little bit of magic) for the generator function. **

Yield is the return (plus a little bit of magic) that is dedicated to the generator.

The following is a simple generator function:

>>> def simple_generator_function ():>>>    yield 1>>>    yield 2>>>    Yield 3

Here are two simple ways to use it:

>>> for value in Simple_generator_function ():>>>     Print (value) 123>>> Our_generator = Simple_generator_function () >>> Next (Our_generator) 1>>> next (Our_generator) 2>>> Next ( Our_generator) 3

Magic?

So where's the magic part? I'm glad you asked that question! When a generator function calls yield, the "state" of the generator function is frozen, the values of all the variables are preserved, and the next line of code to execute is also recorded until you call next again. Once next () is called again, the generator function starts where it last left off. If you never call next (), the state of yield preservation is ignored.

Let's rewrite the Get_primes () function, this time we'll write it a generator. Note that we no longer need the Magical_infinite_range function. Using a simple while loop, we created our own infinite string of columns.

def get_primes (number): While    True:        if Is_prime (number):            yield number number        + = 1

If the generator function calls return or executes to the end of the function, a stopiteration exception occurs. This informs the caller of next () that the generator has no next value (this is the behavior of the ordinary iterator). This is why this while loop appears in our Get_primes () function. Without this while, the generator function executes to the end of the function, triggering the stopiteration exception when we call next again. Once the generator values have been exhausted, then calling next () will cause an error, so you can only use each generator once. The following code is incorrect:

>>> our_generator = simple_generator_function () >>> for value in Our_generator:>>>     Print (value) >>> # Our generator does not have the next value ...>>> print (Next (our_generator)) Traceback (most recent call last):  File "<ipython-input-13-7e48a609051a>", line 1, in <module>    next (our_generator) stopiteration >>> # However, we can always create another generator >>> # just call the generator function again >>> new_generator = Simple_generator_function () >>> print (Next (new_generator)) # working OK 1

Therefore, this while loop is used to ensure that the generator function is never executed at the end of the function. Whenever you call next () This generator generates a value. This is a common method of dealing with infinite sequences (which is also very common).

Execution process

Let's go back to the place where we called Get_primes: solve_number_10.

Def solve_number_10 ():    # She *is* working on Project Euler #10, I knew it!    Total = 2 for    next_prime in Get_primes (3):        if Next_prime < 2000000: Total            + = Next_prime        Else:            Print (total)            return

Let's take a look at the call to Get_primes in the solve_number_10 for loop and see how the first few elements were created to help us understand. When a for loop requests the first value from Get_primes, we enter Get_primes, which is no different from entering the normal function.

The while loop into the third row

Stop at If condition judgment (3 is prime number)

Return 3 and execution control to solve_number_10 by yield

Next, go back to Insolve_number_10:

The For loop gets the return value 3

The For loop assigns it to the Next_prime

Total Plus Next_prime

A For loop requests the next value from the Get_primes

This time, when we entered Get_primes, we did not execute from the beginning, we continued from line 5th, where we left last.

def get_primes (number): While    True:        if Is_prime (number):            yield number number        + = 1 # <<<< <<<<<<

Most crucially, number also retains the value we had when we last called yield (for example, 3). Remember that yield passes the value to the caller of next () and also saves the "state" of the generator function. Next, Number adds to 4, returns to the start of the while loop, and continues to increase until the next prime number (5) is obtained. Once again we return the value of number through yield to the for loop of solve_number_10. This cycle is executed until the For loop ends (the number of primes is greater than 2,000,000).

Summarize

Key points:

Generator is used to generate a series of values.

Yield is like the return result of the generator function.

Yield the only other thing to do is to save the state of a generator function

Generator is a special type of iterator (iterator)

Similar to iterators, we can get the next value from generator by using next ()

Ignoring some values by implicitly invoking next ()