Python yield learning

The yield function is similar to return, but the difference is that it returns a generator.

Generator
A generator is a function that consists of one or more yield expressions, each of which is an iterator (but the iterator is not necessarily a generator).

If a function contains the yield keyword, the function becomes a generator.

Instead of returning all the results at once, the generator returns the result each time it encounters the yield keyword, preserving the current running state of the function and waiting for the next call.

Since the generator is also an iterator, it should support the next method to get the next value.

Basic operations

# Create generators with ' yield '
def func ():
For I in xrange (10);
Yield I

# Create a builder from a list
[I for I in Xrange (10)]
# Call the following
>>> f = func ()
>>> F # At this point the generator is not running yet
<generator object Func at 0x7fe01a853820>
>>> F.next () # when i=0 encountered the yield keyword and returned directly
0
>>> F.next () # Continue the last execution position and go to the next layer loop
1
...
>>> F.next ()
9
>>> F.next () # When the last loop is executed, the yield statement is ended, and the stopiteration exception is generated
Traceback (most recent):
File "<stdin>", line 1, in <module>
Stopiteration
>>>
In addition to the next function, the generator also supports the Send function. The function can pass parameters to the generator.

>>> def func ():
... n = 0
... while 1:
... n = yield n #可以通过send函数向n赋值
...
>>> f = func ()
>>> F.next () # By default n is 0
0
>>> f.send (1) #n赋值1
1
>>> F.send (2)
2
>>>
Application

The most classic example, generates an infinite sequence.

The general workaround is to generate a large list of requirements that need to be kept in memory, and obviously memory limits the problem.

def get_primes (start):
for element in Magical_infinite_range (start):
If Is_prime (Element):
return element
If you use the generator, you do not need to return the entire list, only one data is returned each time, which avoids the memory limit problem.

def get_primes (number):
While True:
If Is_prime (number):
Yield number
Number + = 1
Generator Source Analysis
The generator source code is objects/genobject.c.

Call stack

Before explaining the generator, you need to explain how the Python virtual machine is called.

The Python virtual machine has a stack frame call stack, where the stack frame is pyframeobject, located in Include/frameobject.h.

typedef struct _FRAME {
Pyobject_var_head
struct _frame *f_back;/* Previous frame, or NULL */
Pycodeobject *f_code; /* Code segment */
Pyobject *f_builtins;/* builtin symbol table (pydictobject) */
Pyobject *f_globals;/* Global Symbol table (pydictobject) */
Pyobject *f_locals;/* Local symbol table (any mapping) */
Pyobject **f_valuestack; /* points after the last local */
/* Next free slots in F_valuestack. Frame creation sets to F_valuestack.
Frame evaluation usually NULLs it, but a Frame this yields sets it
to the current stack top. */
Pyobject **f_ Stacktop;
Pyobject *f_trace;/* Trace function */

/* If An exception was raised in this frame, the next three be used to
* Record the exception info (If any) origin Ally in the thread state. See
* Comments before Set_exc_info ()--it's not obvious.
* Invariant:if _type was NULL, then so was _value and _t Raceback.
* Desired Invariant:all Three is NULL, or all three is non-null. That
* One isn ' t currently true, but "should is".
*/
Pyobject *f_exc_type, *f_exc_value, *f_exc_traceback;

Pythreadstate *f_tstate;
int f_lasti; /* Last instruction if called */
/* Call Pyframe_getlinenumber () instead of reading this field
Directly. As of 2.3 F_lineno is only valid when tracing is
Active (i.e. when F_trace is set). At and times we use
Pycode_addr2line to calculate the
Bytecode index. */
int F_lineno; /* Current line number */
int f_iblock; /* Index in F_blockstack */
Pytryblock F_blockstack[co_maxblocks]; /* for try and loop blocks */
Pyobject *f_localsplus[1]; /* Locals+stack, dynamically sized */
} Pyframeobject;
The stack frame holds the information and context that gives the code, which contains information such as the last executed instruction, the global and local namespaces, the exception state, and so on. F_valueblock saves the data, and B_blockstack preserves the exception and loop control methods.

Give an example to illustrate that

def foo ():
x = 1
def bar (y):
z = y + 2 # <---(3) ... and the interpreter are here.
Return Z
return bar (x) # <---(2) ... which is returning a call to bar ...
Foo () # <---(1) We ' re in the middle of a call to Foo ...
So, the corresponding call stack is as follows, a py file, a class, a function is a block of code, the counterpart of a frame, save the context and bytecode instructions.

C---------------------------
A | Bar Frame | Block stack: []
l | (Newest) | Data stack: [1, 2]
L---------------------------
| Foo Frame | Block stack: []
s | | Data stack: [<function foo.<locals>.bar at 0x10d389680>, 1]
T---------------------------
A | Main (module) Frame | Block stack: []
C | (oldest) | Data stack: [<function foo at 0x10d3540e0>]
K---------------------------
Each stack frame has its own data stack and block stack, and the independent data stack and block stack allow the interpreter to break and restore the stack frame (the generator formally exploits this).

Python code is first compiled into bytecode and then executed by a Python virtual machine. In general, a Python statement corresponds to multiple bytecode (because each bytecode corresponds to a C statement instead of a machine instruction, code performance cannot be judged by the number of bytecode).

Call the DIS module to parse the byte code,

From dis import dis

Dis (foo)

5 0 Load_const 1 (1) # Load constant 1
3 Store_fast 0 (x) # x is assigned a value of 1

6 6 Load_const 2 (<code object bar at 0x7f3cdee3a030, file ' t1.py ', line 6>) # Load constant 2
9 Make_function 0 # Create function
1 Store_fast (BAR)

9 Load_fast 1 (BAR)
Load_fast 0 (x)
Call_function 1 # Call function
Return_value
which

The first code line number of the behavior;
Second behavior offset address;
The third act bytecode instruction;
The four behavior instruction parameter;
Explanation of the behavior parameter five.
Generator Source Analysis

By the above understanding of the call stack, it is easy to understand the concrete implementation of the generator.

The generator source code is located in Object/genobject.c.

Creation of generators

Pyobject *
Pygen_new (Pyframeobject *f)
{
Pygenobject *gen = pyobject_gc_new (Pygenobject, &pygen_type); # Create Builder Object
if (Gen = = NULL) {
Py_decref (f);
return NULL;
}
Gen->gi_frame = f; # Give code blocks
Py_incref (F->f_code); # Reference Count +1
Gen->gi_code = (Pyobject *) (F->f_code);
gen->gi_running = 0; # 0 represents execution, which is the initial state of the generator
Gen->gi_weakreflist = NULL;
_pyobject_gc_track (gen); # GC Tracking
Return (Pyobject *) Gen;
}
Send and Next

Next and the Send function, as follows

Static Pyobject *
Gen_iternext (Pygenobject *gen)
{
Return Gen_send_ex (gen, NULL, 0);
}

Static Pyobject *
Gen_send (Pygenobject *gen, Pyobject *arg)
{
Return gen_send_ex (gen, arg, 0);
}
As you can see from the code above, both send and next are called the same function gen_send_ex, with the difference between having parameters.

Static Pyobject *
GEN_SEND_EX (Pygenobject *gen, pyobject *arg, int exc)
{
Pythreadstate *tstate = Pythreadstate_get ();
Pyframeobject *f = gen->gi_frame;
Pyobject *result;

if (gen->gi_running) {# To determine if the generator is already running
Pyerr_setstring (Pyexc_valueerror,
"Generator already executing");
return NULL;
}
if (F==null | | f->f_stacktop = = NULL) {# throws a Stopiteration exception if the code block is empty or the call stack is empty
/* Only set exception if called from Send () */
if (Arg &&!exc)
Pyerr_setnone (pyexc_stopiteration);
return NULL;
}

if (F->f_lasti = =-1) {# F_lasti=1 represents the first execution
if (arg && arg! = py_none) {# First execution is not allowed with parameters
Pyerr_setstring (Pyexc_typeerror,
"Can ' t send Non-none value to a"
"Just-started generator");
return NULL;
}
} else {
/* Push arg onto the frame ' s value stack */
result = arg? Arg:py_none;
Py_incref (result); # The parameter reference count +1
* (f->f_stacktop++) = result; # parametric Compression stack
}

/* Generators always return to their more recent caller, not
* Necessarily their creator. */
F->f_tstate = tstate;
Py_xincref (Tstate->frame);
ASSERT (F->f_back = = NULL);
F->f_back = tstate->frame;

gen->gi_running = 1; # Modify Generator Execution status
result = Pyeval_evalframeex (f, exc); # Execute byte code
gen->gi_running = 0; # Revert to the non-executed state

/* Don ' t keep the reference to f_back any longer than necessary. It
* May keep a chain of frames alive or it could create a reference
* Cycle. */
ASSERT (F->f_back = = Tstate->frame);
Py_clear (F->f_back);
/* Clear the borrowed reference to the thread state */
F->f_tstate = NULL;

/* If The generator just returned (as opposed to yielding), signal
* That's generator is exhausted. */
if (result = = Py_none && F->f_stacktop = = NULL) {
Py_decref (result);
result = NULL;
/* Set exception if not called by Gen_iternext () */
if (ARG)
Pyerr_setnone (pyexc_stopiteration);
}

if (!result | | f->f_stacktop = = NULL) {
/* Generator can ' t be rerun, so release the frame */
Py_decref (f);
Gen->gi_frame = NULL;
}

return result;
}
Byte-code execution

The function of the Pyeval_evalframeex function is to execute a byte code and return the result.

# The main process is as follows
for (;;) {
switch (opcode) {# opcode is the opcode, which corresponds to various operations
Case NOP:
Goto Fast_next_opcode;
...
...
Case Yield_value: # if the opcode is YIELD
retval = POP ();
F->f_stacktop = Stack_pointer;
why = Why_yield;
Goto Fast_yield; # Use Goto to jump out of the loop
}
}

Fast_yield:
...
return vetval; # return results
As an example, f_back the offset of the last executed instruction on the previous frame,f_lasti,

Import Sys
From dis import dis

def func ():
f = sys._getframe (0)
Print F.f_lasti
Print F.f_back
Yield 1

Print F.f_lasti
Print F.f_back
Yield 2

A = func ()
Dis (func)
A.next ()
A.next ()
The results are as follows, where the third line of English is the opcode, corresponding to the above opcode, each switch is selected between different opcode.

6 0 Load_global 0 (SYS)
3 load_attr 1 (_getframe)
6 Load_const 1 (0)
9 Call_function 1
0 Store_fast (f)

7 Load_fast 0 (f)
Load_attr 2 (F_lasti)
Print_item
Print_newline

8 Load_fast 0 (f)
Load_attr 3 (F_back)
Print_item
Print_newline

9 to Load_const 2 (1)
Yield_value # At this time the operation code is yield_value, jump directly to the above Goto statement, at this time F_lasti is the current instruction, F_back is the current frame
Pop_top

Load_fast 0 (f)
2 load_attr (F_lasti)
Print_item
Print_newline

Load_fast 0 (f)
Load_attr 3 (F_back)
Print_item
Wuyi Print_newline

Load_const 3 (2)
Yield_value
Pop_top
Load_const 0 (None)
Return_value
18
<frame object at 0x7fa75fcebc20> #和下面的frame相同, which belongs to the same frame, that is, within the same function (namespace), the frame is the same.
39
<frame Object at 0x7fa75fcebc20>

Python yield learning