A detailed explanation of the usage of yield in Python

List resolution:

[Expr for Iter_var in iterable if COND_EXPR]

Builder expression:

(Expr for Iter_var in iterable if cond_expr)

The biggest generator is not that it doesn't return a real array

rows = [1,2,3,17]

DEF cols ():
Yield 56
Yield 2
Yield 1

ABC = cols ()

For I in ABC:
Print I

Print rows
Print ABC

Only when it is called does it actually become a real value, so it doesn't put all the values in memory, so you can only loop it once

Yield is this magical thing, like return, he returns something, but he returns a generator instead of executing immediately

The intuitive point is that, for example,

You ask and, ask for a very large array inside of all the array of the and, and then the memory is not big enough, you certainly cannot beg, because [1,2.3,4 ...] It's too big to be kept inside.

But if you use the generator to generate an array of the same, (1,2,3, ...), you can.

rows = [x for x in range (10000000000)]

rows = (x for x in range (10000000000))

(is to explain, representing 100000000000 very large)

Because the generator is to generate one, delete one, and then the next, all do not take up too much memory

Build device

A generator is a function consisting of one or more yield expressions, each of which is an iterator (but the iterator is not necessarily a builder).

If a function contains the yield keyword, the function becomes a generator.

Instead of returning all the results at once, the generator returns the corresponding result each time the yield keyword is encountered, and retains the current running state of the function, waiting for the next call.

Because the generator is also an iterator, it should support the next method to get the next value.

Basic operations

# Create a builder from ' yield '
def func ():
For I in xrange (10);
Yield I

# Create a builder from a list
[I for I in Xrange (10)]
# calls are as follows
>>> f = func ()
>>> F # The generator is not running at this time
<generator object Func at 0x7fe01a853820>
>>> F.next () # when i=0, encounter yield keyword, return directly
0
>>> F.next () # Continue to the last execution position, go to the next level loop
1
...
>>> F.next ()
9
>>> F.next () # When the last loop is completed, the yield statement is ended and the stopiteration exception is generated
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Stopiteration
>>>
In addition to the next function, the generator also supports the Send function. This function can pass parameters to the builder.

>>> def func ():
.. N = 0
. While 1:
.. N = yield n #可以通过send函数向n赋值
...
>>> f = func ()
>>> F.next () # By default n is 0
0
>>> f.send (1) #n赋值1
1
>>> F.send (2)
2
>>>
Application

The most classic example is the generation of infinite sequences.

The general solution is to generate a very large list of requirements that need to be kept in memory, and it is obvious that memory limits the problem.

def get_primes (start):
for element in Magical_infinite_range (start):
If Is_prime (Element):
return element
If you use a generator you do not need to return the entire list, each time you simply return one data, avoiding memory limitations.

def get_primes (number):
While True:
If Is_prime (number):
Yield number
Number = 1
Generator Source Analysis

The source of the generator is objects/genobject.c.

Call stack

Before you explain the generator, you need to explain the invocation principle of the Python virtual machine.

The Python virtual machine has a stack frame call stack, where the stack frame is pyframeobject, located in Include/frameobject.h.

typedef struct _FRAME {


Pyobject_var_head


struct _frame *f_back; /* Previous frame, or NULL */


Pycodeobject *f_code; /* Code Segment * *


Pyobject *f_builtins; /* Builtin symbol table (pydictobject) * *


Pyobject *f_globals; /* Global symbol table (pydictobject) * *


Pyobject *f_locals; /* Local symbol table (any mapping) * *


Pyobject **f_valuestack; /* points after the last local * *


/* Next free slot in F_valuestack. Frame creation sets to F_valuestack.


Frame evaluation usually NULLs it, but a frame that yields sets it


To the current stack top. */


Pyobject **f_stacktop;


Pyobject *f_trace; /* Trace function */

/* If An exception was raised in this frame, the next three are used to
* The record of the exception info (if any) is originally in the thread state. The
* Comments before Set_exc_info ()--it ' s not obvious.
* Invariant:if _type is NULL, then so are _value and _traceback.
* Desired Invariant:all Three are NULL, or all three are. That
* One isn ' t currently true, but "should be".
*/
Pyobject *f_exc_type, *f_exc_value, *f_exc_traceback;

Pythreadstate *f_tstate;


int f_lasti; /* Last instruction if called * *


/* Call Pyframe_getlinenumber () instead of the reading this field


directly. As of 2.3 F_lineno is only valid when tracing is


Active (i.e. when F_trace is set). At the other times we use


Pycode_addr2line to calculate the


Bytecode index. */


int F_lineno; /* Current line number * *


int f_iblock; /* Index in F_blockstack * *


Pytryblock F_blockstack[co_maxblocks]; /* for try and loop blocks * *


Pyobject *f_localsplus[1]; * Locals+stack, dynamically sized * *


} Pyframeobject;


The stack frame holds the information and context for the code, including the last command, the global and local namespaces, the exception status, and so on. F_valueblock saves the data and B_blockstack preserves the exception and the loop control method.

Let me give you an example to illustrate that

def foo ():
x = 1
def bar (y):
z = y + 2 # <---(3) ... and the interpreter is here.
Return Z
return bar (x) # <---(2) ... which is returning-a call to bar ...
Foo () # <---(1) We ' re in the middle of-a call to Foo ...
So, the corresponding call stack is as follows, a py file, a class, a function is a code block, the counterpart is a frame, and the context environment and bytecode directives are stored.

C---------------------------


A | Bar Frame | -&gt; block stack: []


l | (Newest) | -&gt; Data stack: [1, 2]


L---------------------------


| Foo Frame | -&gt; block stack: []


s | | -&gt; data stack: [&lt;function foo.&lt;locals&gt;.bar at 0x10d389680&gt;, 1]


T---------------------------


A | Main (module) Frame | -&gt; block stack: []


C | (oldest) | -&gt; data stack: [&lt;function foo at 0x10d3540e0&gt;]


K---------------------------


Each stack frame has its own data stack and block stack, and separate data stacks and block stacks allow the interpreter to interrupt and restore the stack frame (the generator uses this).

The Python code is first compiled into bytecode, and then executed by the Python virtual machine. In general, a Python statement corresponds to more than one byte code (because each byte code corresponds to a C statement, not a machine instruction, so the code performance cannot be judged by the number of bytecode).

Call the DIS module to parse the byte code,

From dis import dis

Dis (foo)

5 0 Load_const 1 (1) # Load constant 1
3 Store_fast 0 (x) # X assigned to 1

  6           6 load_const                2 (<code object bar at 0x7f3cdee3a030, file "t1.py", Line 6>) # Load Constants 2
              9 Make_ function            0 # Create function
              store_fast                1 (BAR)

9 Load_fast 1 (BAR)
0 (x) load_fast
Call_function 1 # Call function
Return_value
which

The first behavior code line number;
The second behavior offset address;
Third act byte Code instruction;
The four-act instruction parameter;
The behavior parameter explanation.
Generator Source Analysis

From the above understanding of the call stack, it is easy to understand the specific implementation of the generator.

The source of the generator is located in Object/genobject.c.

Builder creation

Pyobject *
Pygen_new (Pyframeobject *f)
{
Pygenobject *gen = pyobject_gc_new (Pygenobject, &pygen_type); # Create Builder Object
if (Gen = NULL) {
Py_decref (f);
return NULL;
}
Gen->gi_frame = f; # Give code blocks
Py_incref (F->f_code); # Reference Count +1
Gen->gi_code = (Pyobject *) (F->f_code);
gen->gi_running = 0; # 0 is represented as execution, which is the initial state of the generator
Gen->gi_weakreflist = NULL;
_pyobject_gc_track (gen); # GC Tracking
Return (Pyobject *) Gen;
}
Send and Next

Next and the Send function, as follows

Static Pyobject *
Gen_iternext (Pygenobject *gen)
{
Return Gen_send_ex (gen, NULL, 0);
}

Static Pyobject *
Gen_send (Pygenobject *gen, Pyobject *arg)
{
Return gen_send_ex (gen, arg, 0);
}
As you can see from the above code, both send and next are called the same function gen_send_ex, and the difference is whether you have parameters.

Static Pyobject *
GEN_SEND_EX (Pygenobject *gen, pyobject *arg, int exc)
{
Pythreadstate *tstate = Pythreadstate_get ();
Pyframeobject *f = gen->gi_frame;
Pyobject *result;

if (gen->gi_running) {# Determines whether the generator is already running
Pyerr_setstring (Pyexc_valueerror,
"Generator already executing");
return NULL;
}
if (F==null | | f->f_stacktop = = NULL) {# Throw the stopiteration exception if the code block is empty or the call stack is empty
/* Only set exception if called from Send () * *
if (Arg &&!exc)
Pyerr_setnone (pyexc_stopiteration);
return NULL;
}

if (F-&gt;f_lasti = = 1) {# F_lasti=1 represents the first execution


if (Arg &amp;&amp; arg!= py_none) {# First execution does not allow parameters


Pyerr_setstring (Pyexc_typeerror,


"Can ' t send Non-none value to a"


"Just-started generator");


return NULL;


}


} else {


/* Push arg onto the frame ' s value stack * *


result = arg? Arg:py_none;


Py_incref (result); # This parameter reference count +1


* (f-&gt;f_stacktop++) = result; # parameter pressure stack


}

/* Generators always return to their most recent caller
* Necessarily their creator. */
F->f_tstate = tstate;
Py_xincref (Tstate->frame);
ASSERT (F->f_back = NULL);
F->f_back = tstate->frame;

gen->gi_running = 1; # Modify Generator Execution status
result = Pyeval_evalframeex (f, exc); # Execute byte code
gen->gi_running = 0; # Revert to not performing state

/* Don ' t keep the reference to f_back any longer than necessary. It
* May keep a chain of frames alive or it could create a reference
* Cycle. */
ASSERT (F->f_back = = Tstate->frame);
Py_clear (F->f_back);
/* Clear the borrowed reference to the thread state * *
F->f_tstate = NULL;

/* If The generator just returned (as opposed to yielding), signal
* That's the generator is exhausted. */
if (result = = Py_none && f->f_stacktop = NULL) {
Py_decref (result);
result = NULL;
/* Set exception if not called by Gen_iternext () * *
if (ARG)
Pyerr_setnone (pyexc_stopiteration);
}

if (!result | | f->f_stacktop = = NULL) {
/* Generator can ' t be rerun and so release the frame * *
Py_decref (f);
Gen->gi_frame = NULL;
}

return result;
}
Byte-code execution

The function of the Pyeval_evalframeex function is to execute the byte code and return the result.

# The main process is as follows,
for (;;) {
switch (opcode) {# opcode is opcode, corresponding to various operations
Case NOP:
Goto Fast_next_opcode;
...
...
Case Yield_value: # if the opcode is YIELD
retval = POP ();
F->f_stacktop = Stack_pointer;
why = Why_yield;
Goto Fast_yield; # Use Goto to jump out of the loop
}
}

Fast_yield:
...
return vetval; # return results
For example, f_back the offset of the last instruction that was executed on the previous frame,f_lasti,

Import Sys
From dis import dis

def func ():
f = sys._getframe (0)
Print F.f_lasti
Print F.f_back
Yield 1

Print F.f_lasti
Print F.f_back
Yield 2

A = func ()
Dis (func)
A.next ()
A.next ()
The results are as follows, where the third line of English is the opcode, corresponding to the above opcode, each switch is between the different opcode to choose between.

  6           0 load_global               0 (SYS)
               3 load_attr                 1 (_getframe)
               6 load_const                1 (0)
              9 call_function             1
              store_fast                0 (f)

  7          load_fast                 0 (f)
              load_attr                 2 (f_lasti)
             print_item         
              print_newline      

  8          load_fast                 0 (f)
              load_attr                 3 (f_back)
             print_item         
              print_newline      

  9          to load_const                2 (1)
              yield_value     # At this time the operation code for Yield_value, directly jump the above goto statement, at this time f_lasti for the current instruction, f_ Back is the current frame
             pop_top             

 11          load_fast                 0 (f)
              load_attr                 2 (f_lasti)
             print_item         
              print_newline      

 12          load_fast                 0 (f)
              load_attr                 3 (f_back)
             print_item         
              Wuyi print_newline      

 13          load_const                3 (2)
              yield_value        
              pop_top             
             57 load_const               0 (None)
             return_value        

<frame object at 0x7fa75fcebc20> #和下面的frame相同, belonging to the same frame, that is, within the same function (namespace), Frame is the same one.

<frame object at 0x7fa75fcebc20>