Python closures and decorators for virtual machine function mechanisms (vii)

Access to local variables in functions

After parsing the function parameters, let's take a look at how the local variables of the function are implemented in Python. As mentioned earlier, the function parameter is also a local variable. Therefore, the implementation mechanism of local variables is exactly the same as that of function parameters. What does this "same" mean?

We have previously dissected some of the instructions of a Python virtual machine, and if you want to access a variable, you should use the LOAD_NAME directive to retrieve the variable value corresponding to the variable name in the three namespaces of local, global, and builtin. Then, when the function is called, the Python virtual machine creates a new Pyframeobject object through Pyframe_new, and the critical local object is not created

During the analysis of the Python virtual machine mechanism, we learned that when a script is called directly, f_locals and f_globals are actually the same object in the corresponding Pycodeobject object of the script. So when the function is executed, read and write to the variable must not be reflected on the f_globals, otherwise, a function can not be executed, outside the function of the function to know the local variables? So, how does a function read and write to a local variable? Let's take a function with a local variable to explain the bytecode instruction with the DIS module.

>>> def f (A, B): ...     c = a + b     ... Print (c) ... >>> import dis>>> Dis.dis (f)  2           0 load_fast                0 (a)              3 load_fast                1 (b)              6 Binary_add                        7 store_fast               2 (c)  3          load_fast                2 (c)             Print_item                       Print_ NEWLINE                    load_const               0 (None)             

　　

Here, we do not find that there are load_name or store_name such as the namespace of the read and write instructions, instead of load_fast and store_fast, this is not to confirm that we said before, the implementation of local variables and function parameters of the implementation mechanism is exactly the same, Local variables are read and written on F_localsplus.

Why is the local namespace not used in the implementation of the function? This is because the local variables in the function are always fixed, so at compile time it is possible to determine the location of the memory space used by the local variables and to determine how the bytecode instructions accessing the local variables should access the memory. With this information, Python can use static methods to implement local variables without resorting to the technique of dynamically locating pydictobject objects. After all, function calls are too common, and static methods can greatly improve the efficiency of function execution.

Nested functions

In Python, there is a core concept called namespaces, and the result of execution of a piece of code depends not only on the symbols in the code, but on the semantics of the symbols in the code, which is determined by the namespace. Namespaces are maintained dynamically by Python virtual machines at runtime, but sometimes we want namespaces to be static. In other words, we want the code to be unaffected by the change in namespace, and always maintain consistent behavior and results. What is the point of doing this?

If we want to set a benchmark, and then compare many values to this value, the simplest way is to write a function:

>>> def compare (base, value): ...     return value > Base ... >>> compare (5) false>>> Compare (Ten) True

　　

We compare 10 as the benchmark, compared with 5 and 20, but find that each time a function is called, it must pass one more 10. As a result, Python provides nested functions

>>> base = 1>>> def get_compare (base): ...     def real_compare (value): ...         return value > Base     ... Return real_compare ... >>> compare_with_10 = Get_compare (Ten) >>> compare_with_10 (5) false>> > compare_with_10 () True

　　　　

As in the above code, we only set a baseline value once. Thereafter, each time the comparison operation, although the actual function called Real_compare's local namespace does not have a base, and the Get_compare function has "base = 1" In the global namespace, but the result of the function call is displayed, Real_ Compare is based on the 10 we've passed in as base, not the base book of the Get_compare function.

That is, when real_compare this function as the return value is passed to Compare_with_10, a namespace has been tightly bound with real_compare, in the execution of Real_compare code, This namespace has been restored again, and this is the method of static namespaces. The result of this namespace and function bundling is called a closure. As we saw earlier, Pyfunctionobject is a Python virtual machine that is specially prepared for the parcel Bytecode directive, the global namespace, the default parameter values, can find its location in the Pyfunctionobject, again, Closures in Python are also implemented by Pyfunctionobject objects.

The cornerstone of implementing closures

Let's take a look at Pycodeobject, Pyfunctionobject, Pyframeobject, the properties of the closure related to the objects we are familiar with. Closures are often created using nested functions, and in Pycodeobject, the properties associated with nested functions are Co_freevars, co_cellvars. The two specific meanings are as follows:

• Co_freevars: Typically a tupple that holds a collection of variable names used in nested scopes
• Co_cellvars: Typically a tupple that holds a collection of variable names in the outer scope that is used

Consider the following code:

# Cat demo4.py def get_func ():    a = 1    value = "inner"    def inner_func ():        print (value)    return Inner_ Func

　　

Obviously, the code above compiles 3 Pycodeobject objects, two of them, one corresponding to the function Get_func, and one corresponding to the function Inner_func. Then co_cellvars in the Pycodeobject object corresponding to Get_func should contain the string "value" because it is used in its nested scope (the scope of the Inner_func). Similarly, the co_freevars in the Pycodeobject object corresponding to the function Inner_func should also have the string "value". Here, let's confirm:

>>> Source = open ("demo4.py"). Read () >>> CO = compile (source, "demo4.py", "exec") >>> Co.co_ Consts (<code object get_func at 0x7efc8c8c1b70, file ' demo4.py ', line 1>, None)

　　

demo4.py corresponding to the pycodeobject, co_consts this tuple the first object is get_func corresponding pycodeobject, we take it out, and then look at the Co_cellvars

>>> Get_func_co = co.co_consts[0]>>> get_func_co.co_name ' get_func ' >>> get_func_co.co_ Cellvars (' value ',)

　　

As can be known from the previous demo4.py, although the variable value is removed in Get_func, there is also a variable a, but a does not use nested functions inside the function, so co_cellvars only has the symbolic value, no symbol a. As we all know, pycodeobject can be nested pycodeobject, since Get_func's Pycodeobject object is taken out, we might as well take out inner_func corresponding Pycodeobject, but before this, We have to look first, Inner_func pycodeobject, in the get_func corresponding pycodeobject co_consts which position

>>> get_func_co.co_consts (None, 1, ' inner ', <code object inner_func at 0x7efc8c8c1cd8, file "demo4.py", line 5 >) >>> inner_func_co = get_func_co.co_consts[3]>>> inner_func_co.co_freevars (' value ',)

　　

As you can see, the inner_func corresponds to the pycodeobject in which the Co_freevars is indeed a signed value

These are the properties associated with closures in Pycodeobject. Next, let's look at the objects in the Pyframeobject object that are related to the closure property. In fact, there's only one object and closure related, and that's our old friend F_localsplus.

In the Pyframe_new function, there is a piece of code:

Ncells = Pytuple_get_size (code->co_cellvars); nfrees = Pytuple_get_size (code->co_freevars); extras = Code->co _stacksize + code->co_nlocals + ncells + nfrees;

　　

Extras is the size of the memory that F_localsplus points to, which belongs to the runtime stack, local variables, cell objects (Co_cellvars), and free objects (Co_freevars). Below, show the layout of the F_localsplus:

Figure 1-1 Full memory layout for F_localsplus

Implementation of closures

After introducing the cornerstone of the closure, we can begin to trace the implementation of the closure package. But we seem to have forgotten one thing, previously said that the object related to closures, in addition to Pycodeobject, Pyframeobject, there is a pyfunctionobject. It doesn't matter, we'll soon learn about the Pyfunctionobject and closures that cannot be said. But first, let's take a look at the demo5.py compiled bytecode directive:

In fact, demo5.py compared to demo4.py, only the Get_func function in the variable A to remove

# cat Demo5.pydef Get_func ():    value = "inner"    def inner_func ():        print (value)    return Inner_funcshow_ Value = Get_func () show_value ()

　　

With demo5.py, we can analyze Pycodeobject objects by layer, what the closure instructions look like.

First, we get the demo5.py corresponding Pycodeobject object

>>> Source = open ("demo5.py"). Read () >>> CO = compile (source, "demo5.py", "exec") >>> Import Dis>>> Dis.dis (CO)  1           0 load_const               0 (<code object get_func at 0x255d120, file ' demo5.py ', line 1& gt;)              3 make_function            0              6 store_name               0 (get_func)           9 load_name                0 (get_func)             Call_ FUNCTION            0             store_name               1 (show_value)          load_name                1 (show_value)             Call_ FUNCTION            0             pop_top                          load_const               1 (None)             Return_value  

　　

We are already familiar with the instruction sequence of demo5.py, and the emphasis is not on this, but on the sequence of instructions in Get_func and Inner_func.

Get_func instruction Sequence

>>> co.co_consts (<code object get_func at 0x255d120, file ' demo5.py ', line 1>, None) >>> get_func_ CO = co.co_consts[0]>>> get_func_co.co_flags3>>> Dis.dis (get_func_co)  2           0 LOAD_CONST               1 (' inner ')              3 store_deref              0 (value)  4           6 load_closure             0 (value)              9 build_tuple              1             12 Load_const               2 (<code object Inner_func at 0x7efc8c8c1918, file "demo5.py", line 4>)             make_closure             0             store_fast               0 (inner_func)  7          load_fast                0 (Inner_func)             

　　　　

Inner_func instruction Sequence

>>> get_func_co.co_consts (None, ' inner ', <code object inner_func at 0x7efc8c8c1918, file "demo5.py", line 4&G t;) >>> Inner_func_co = get_func_co.co_consts[2]>>> inner_func_co.co_flags19>>> Dis.dis ( INNER_FUNC_CO)  5           0 load_deref               0 (value)              3 Print_item                        4 print_newline                     5 load_const               0 ( None)              8 Return_value  

　　

In Fast_function, we have previously introduced a fast channel, but this fast channel is not any function can enter, before entering to meet a number of conditions, one of the conditions is co->co_flags = = (Co_optimized | Co_newlocals | Co_nofree), i.e. Co_flags is 67. Unfortunately Get_func and Inner_func Co_flags are not 67, destined only obediently walk Pyeval_evalcodeex

If the length of the current Pycodeobject Co_cellvars is not 0, it will go to the branch of the code below, and the Python virtual opportunity will copy things from Co_cellvars to the newly created Pyframeobject f_ as it does with the default parameters. In Localsplus

Ceval.c

pyobject *pyeval_evalcodeex (pycodeobject *co, Pyobject *globals, PyObject * Locals, Pyobject **args, int argcount, pyobject **kws, int kwcount, pyobject **defs, int defcount, Pyobject *closure) {... if (Pytuple_get_size (co->co_cellvars)) {int I, J, Nargs, Found;char *cellname, *argname; Pyobject *c;nargs = co->co_argcount;if (Co->co_flags & Co_varargs) nargs++;if (Co->co_flags & CO_ Varkeywords) nargs++;for (i = 0; i < pytuple_get_size (co->co_cellvars); ++i) {//[1]: Gets the symbolic name that is shared by the nested function Cellname = Pystring_as_string (Pytuple_get_item (Co->co_cellvars, i)) found = 0;for (j = 0; J < Nargs; J + +) {argname = Pystring_as _string (Pytuple_get_item (Co->co_varnames, j)); if (strcmp (cellname, argname) = = 0) {c = Pycell_new (Getlocal (j)); if (c = = NULL) goto fail; Getlocal (co->co_nlocals + i) = C;found = 1;break;}} Handles the default parameter if the nested function shares the outer function (found = = 0) {c = pycell_new (null); if (c = = NULL) goto fail; SETLOCAL (co->co_nlocals + i, c);}}} ...} 

　　

At [1] of the above code, the Python virtual machine obtains the symbolic name referenced by the inner nested function, in our case, the string "value" is obtained. The found here is whether the symbol referenced by the inner nested function has been identified with a value binding, or has a constraint relationship with an object. This identifier can be 1 only if the inner nested function refers to a parameter with a default value for the outer function. For our example, found must be 0. Because the get_func corresponds to the pycodeobject, Co_varnames cannot find the symbol "value". So the Python virtual machine next creates the Cell object--pycellobject

Cellobject.c

typedef struct {Pyobject_headpyobject *ob_ref;/* Content of the cell or NULL when empty */} Pycellobject;

　　

This object is very simple, just maintain a ob_ref, point to a Pyobject object, let's take a look at Pycellobject's creation code

Cellobject.c

Pyobject *pycell_new (Pyobject *obj) {pycellobject *op;op = (Pycellobject *) pyobject_gc_new (PyCellObject, &PyCell_ Type); if (op = = NULL) return null;op->ob_ref = obj; Py_xincref (obj); _pyobject_gc_track (OP); return (Pyobject *) op;}

　　

In our example, the creation of the Pycellobject object maintained by ob_ref points to null, which means that it is not yet known what the symbolic value is, so when will it be known? When value = "inner" This assignment statement executes. The Cell object is then copied into the f_localsplus of the newly created Pyframeobject object. The value of the note is that the object is copied to the location of Co_co_nlocals + i. Description in N_localsplus, the location of the cell object is after the local variable, which fully conforms to the memory layout shown in Figure 1-1

When dealing with co_cellvars, there is a strange place where, in the process of creating Pycellobject objects, the Cellname at code [1] is completely ignored. In fact, this and the previous analysis of the Python function mechanism to the local variable symbol access from the lookup of Dict to the index of the list is a reason. During the execution of the Get_func function, access to the cell variable of value will be done through the index-based access F_localsplus, as there is no need to know cellname at all. This cellname is actually generated when dealing with the default parameters of the inner nested function referencing the outer function.

After the cell object has been processed, the Python virtual machine will enter Pyeval_evalframeex to formally begin the call procedure for the function Get_func, where we post the Get_func bytecode instruction sequence

>>> Dis.dis (Get_func_co)  2           0 load_const               1 (' inner ')              3 store_deref              0 (value)  4           6 load_closure             0 (value)              9 build_tuple              1             load_const               2 (<code object Inner_func at 0x7efc8c8c1918, file "demo5.py", line 4>)             make_closure             0             store_fast               0 (inner_func)  7          load_fast                0 (Inner_func)             

　　

First execute the "0 load_const 1" command to press the Pystringobject object "inner" into the runtime stack, and then the Python virtual machine starts executing a new bytecode directive for US--store_deref

Ceval.c

Pyobject * Pyeval_evalframeex (pyframeobject *f, int throwflag) {... freevars = f->f_localsplus + co->co_nlocals; ...}

　　

Ceval.c

Case store_deref:w = POP (); x = Freevars[oparg]; Pycell_set (x, W); Py_decref (w); continue;

　　

The Pystringobject object "inner" is ejected from the runtime stack, and the Pycellobject object is obtained from F_localsplus. Originally, Store_deref is to set the Ob_ref in the Pycellobject object

Cellobject.c

#define Pycell_set (OP, V) (((Pycellobject *) (OP))->ob_ref = v) int Pycell_set (Pyobject *op, Pyobject *obj) {if (! Pycell_check (OP)) {Pyerr_badinternalcall (); return-1;} Py_xdecref (((pycellobject*) op)->ob_ref); Py_xincref (obj); Pycell_set (OP, obj); return 0;}

　　

As a result, the f_localsplus changed, 1-2

Figure 1-2 Pyframeobject object that sets the Get_func function after the Cell object

Now in the GET_FUNC environment we know that the value symbol corresponds to a Pystringobject object, but the purpose of the closure is to freeze the constraint so that it can be used when the nested function inner_func is called. This time, you need to use the object of Pyfunctionobject. When executing the def inner_func () expression in demo5.py, the Python virtual machine plugs the constraint (value, "inner") into the Pyfunctionobject

Ceval.c

Case load_closure:x = Freevars[oparg]; Py_incref (x); PUSH (x); if (x! = NULL) Continue;break;

　　

The "6 load_closure 0" instruction takes the Pycellobject object that was just placed and presses it into the runtime stack, then executes the "9 Build_tuple 1" instruction to package the Pycellobject object into a tupple, obviously, this tupple You can place multiple Pycellobject objects. However, our example has only one Pycellobject object

The Python virtual machine then presses the inner_func corresponding Pycodeobject object into the runtime stack by executing a "Load_const 2" instruction, and then completes the constraint and Pycodeobjec with a "make_closure 0" instruction Binding of T

Ceval.c

Case make_closure:{v = POP ();//Get Pycodeobject object x = Pyfunction_new (v, f->f_globals);//Bind global namespace Py_decref (v); if ( X! = NULL) {v = POP ();//Get Tupple, which contains a collection of Pycellobject objects err = pyfunction_setclosure (x, v);//Binding constraint set Py_decref (v);} Handle parameters that have default values if (x = null && oparg > 0) {v = pytuple_new (Oparg), if (v = = null) {py_decref (x); x = Null;break;} while (--oparg >= 0) {w = POP (); Pytuple_set_item (V, Oparg, W);} Err = pyfunction_setdefaults (x, v); Py_decref (v);} PUSH (x); break;}

　　

The last "Store_fast 0" directive corresponding to the expression "Def inner_func ()" Places the Pyfunctionobject object created in F_localsplus. In this way, F_localsplus has changed again.

Figure 1-3 Setting the Pyframeobject object in the Get_func function after the functions object

At the end of the Get_func, the newly created Pyfunctionobject object is returned to the previous stack frame as a return value, and is pressed into the stack frame's runtime stack

Using closures (closure)

Closures are created in Get_func, and for closures, they are in Inner_func. Co_flags is included in co_nested in the pycodeobject corresponding to Inner_func when performing the corresponding call_function of "show_value ()", so Fast_ The function still cannot pass the fast channel verification, or must enter to the Pyeval_evalcodeex

Here, let's take a look at the byte-code instruction sequence of the Inner_func.

>>> Dis.dis (Inner_func_co)  5           0 load_deref               0 (value)              3 Print_item                        4 print_newline                     5 Load_const               0 (None)              

　　

As we've seen, inner_func corresponds to Pycodeobject, which has a symbolic name that references an external scope, and in Pyeval_evalcodeex, this co_freevars is processed

Ceval.c

Pyobject *pyeval_evalcodeex (Pycodeobject *co, Pyobject *globals, Pyobject *locals,  pyobject **args, int argcount, Pyobject **kws, int kwcount,  pyobject **defs, int defcount, Pyobject *closure) {... if (pytuple_get_size (co->co_ Freevars)) {int i;for (i = 0; i < pytuple_get_size (co->co_freevars); ++i) {Pyobject *o = Pytuple_get_item (closure, i); Py_incref (o); Freevars[pytuple_get_size (Co->co_cellvars) + i] = o;}} ......}

　　

Where the closure variable is passed in as the last function parameter, let's see what's coming in fast_function

Funcobject.c#define Pyfunction_get_closure (Func) (((Pyfunctionobject *) func), func_closure)//ceval.cstatic Pyobject *fast_function (Pyobject *func, pyobject ***pp_stack, int n, int na, int nk) {... return Pyeval_evalcodeex (CO, global S, (Pyobject *) NULL, (*pp_stack)-N, Na, (*pp_stack)-2 * nk, NK, D, ND, Pyfunction_get_closure (func));}

　　

The original is in the Pyfunctionobject object is bound with the Pycodeobject object is filled with the Pycellobject object tupple, so in Pyeval_evalcodeex, The action is to put a Pycellobject object in the tupple into the corresponding position in F_localsplus. After processing the closure, the f_localsplus1-4 in the corresponding pyframeobject of Inner_func is shown

Figure 1-4 Pyframeobject object that sets the Inner_func function after the Cell object

So, in the Inner_func call process, when referencing the symbol to the outer scope, it must be the value of the symbol corresponding to the free variable area in F_localsplus. This is the first bytecode instruction "0 load_deref 0" corresponding to the "Print (value)" Expression in the Inner_func function

Ceval.c

Case load_deref:x = freevars[oparg];//Gets the Pycellobject object w = pycell_get (x);//Gets the object that Pycellobject.ob_obj points to if (w! = NULL) { PUSH (w); continue;} ......

　　

Adorner (Decorator)

On the basis of closure technology, Python implements the adorner (Decorator), which looks at the following example:

# Cat demo6.py def should_say (FN):    def Say (*args):        print ("Say something ...")        fn (*args)    return Say@should_saydef func ():    print ("in Func") func () # python2.5 demo6.py say something...in func

　　

In fact, we can achieve the same effect without decorator at all, only minor changes to demo6.py.

# Cat demo7.py def should_say (FN):    def Say (*args):        print ("Say something ...")        fn (*args)    return saydef Func ():    print ("in func") func = Should_say (func) func () # python2.5 demo7.py say something...in func

　　

You will find that the output of demo6.py and demo7.py is the same. In fact, based on the above analysis of the closure, the adorner's behavior is well understood, the adorner just uses one function to wrap another function, similar to the form of "Func = Should_say (func)". Now, let's take a look at some of the compilation results in demo6.py and demo7.py

demo6.py byte code instruction sequence

@should_saydef func ():    //Byte code instruction sequence 9 load_name                0 (Should_say) load_const               1 (<code object func at 0x255d3f0, file "demo6.py", line 9>) make_function            018 call_function 121            store_name               1 (func)    Print ("in Func")

　　

demo7.py byte code instruction sequence

def func ()://Byte code instruction sequence 9 Load_const               1 (<code object func at 0x255d558, file "demo7.py", line 9>) Make_function
   015 store_name               1 (func)    print ("in func") func = Should_say (func)//bytecode instruction sequence load_name                0 (Should_say) 21 Load_name                1 (func) call_function            127 store_name               1 (func)

　　

In demo7.py, the two bytecode instructions "Store_name 1" and "Load_name 1" are inverse each other and can be deleted. As a result, the demo7.py compiled bytecode instruction sequence and the demo6.py compiled bytecode instruction sequence are identical except for the location of "Load_name 0".

Python closures and decorators for virtual machine function mechanisms (vii)