Principle of Python Generator

The function call meets the principle of "last in, first out", that is to say, the last function called should return first, and the recursive function call is a classic example. Obviously, the stack segment in memory that processes data in a "last-in-first-out" manner is the most suitable carrier for implementing function calls. In a compiled programming language, after a function is called, the function's parameters, return address, register value, and other data It will be pushed onto the stack. After the function body is executed, the above data will be popped from the stack. This also means that once a called function is executed, its life cycle ends.
In an interpreted language like Python, function calls are also stack dependent. As I said before, Python's standard interpreter is written in C. The interpreter uses a C function called PyEval_EvalFrameEx to execute the python program. For a function in Python, the interpreter accepts a Python stack frame object and executes python bytecode in the context of this stack frame.
Let's take a look at such an example:

import dis

def foo():
    bar()

def bar():
    pass

print(dis.dis(foo))

We can use the dis module to view the bytecode of the Python program. The following is the bytecode of the function foo():

0 LOAD_GLOBAL 0 (bar)
2 CALL_FUNCTION 0
4 POP_TOP
6 LOAD_CONST 0 (None)
8 RETURN_VALUE

The foo function loads bar onto the stack and calls it, then pops the return value from the stack, and finally loads and returns None. When PyEval_EvalFrameEx encounters the CALL_FUNCTION bytecode, it creates a new python stack frame and then uses this The new frame is used as a parameter to recursively call PyEval_EvalFrameEx to execute bar. However, one thing to note is that the Python interpreter is an ordinary C program, so its stack frame is an ordinary stack. But the Python stack frame it operates is allocated on the heap, so the Python stack frame can survive outside its call. And it can be saved explicitly.
This is the technical foundation of the Python generator. The following is a generator:

def gen():
    yield 1
    yield 2

g = gen()
next(g)

print(type(g))

The result returned is:

<class'generator'>

All generators generated by calling gen() point to the same code object, but each has its own stack frame. This stack frame does not exist on the actual stack, it is waiting to be used on the heap memory.

The stack frame has a "last instruction" pointer that points to the most recently executed instruction. At the beginning, the last instruction pointer is -1, which means that the generator has not yet started, which is why the next example has the next(g) line of code. The generator can be resumed by any function at any time, because its stack frame is actually not on the stack but on the heap. The position of the generator in the calling hierarchy is not fixed, nor does it need to follow the first-in-first-out order that regular functions follow. Because of these characteristics, generators can be used not only to generate iterable objects, but also to achieve multi-task collaboration.