Python Source code profiling note 6-function mechanism

Python's function mechanism is a very important part, many times with Python script, that is, a few functions to solve the problem, do not need to like Java to get a class or something.

This book address: Http://www.jianshu.com/p/d00108741a18

1 Function Object Pyfunctionobject

The Pyfunctionobject object is defined as follows:

typedef struct{Pyobject_head Pyobject *func_code;/ * A Code Object * /Pyobject *func_globals;/* A dictionary (Other mappings won ' t do) */Pyobject *func_defaults;/ * NULL or a tuple * /Pyobject *func_closure;/ * NULL or a tuple of cell objects * /Pyobject *func_doc;/ * The __doc__ attribute, can be anything * /Pyobject *func_name;/ * The __name__ attribute, a String object * /Pyobject *func_dict;/ * The __dict__ attribute, a dict or NULL * /Pyobject *func_weakreflist;/ * List of weak references * /Pyobject *func_module;/ * The __module__ attribute, can be anything * /} Pyfunctionobject;

Let's talk about some important variables in Pyfunctionobject, Func_code, Func_globals. Where Func_code is a function object corresponding to the Pycodeobject, and Func_globals is the function of the global namespace, in fact, this value is passed from the previous layer pyframeobject. Func_defaults is the default value of the stored function, the following analysis of the function parameters will be mentioned, Func_closure is related to the closure, also mentioned later.

###func.pydef f():  print"Function"f()

As the above example func.py, the file compiled after the corresponding 2 PyCodeObject objects, one is the func.py itself, one is the function f. The Pyfunctionobject is def f(): generated by instructions when the bytecode is executed MAKE_FUNCTION . When you create a Pyfunctionobject object, the Pycodeobject object and the current Pyframeobject object that corresponds to the function f are passed in as arguments, and ultimately the Func_code and func_ in Pyfunctionobject are assigned The Globals field. When the function is called, the PyFunctionObject object is passed into the fast_function function, and the new stack frame object is eventually built based on the Func_code and Func_globals fields of the Pyfunctionobject object PyFrameObject , and then called PyEval_EvalFrameEx Executes the function bytecode in the new stack frame. Where the Pyeval_evalframeex function was mentioned in the previous Python execution principle, the function mentioned at the time was PyEval_EvalCodeEx actually creating a new stack frame object Pyframeobject and then executing the Pyeval_evalframeex function.

2 Function Call stack frame

A function call is associated with a stack frame, and the stack frame of each called function points to the PyFrameObject calling function through the f_back pointer. While the Local,global and Builtin namespaces, the local namespace is new for new stack frames, and the global namespace is passed from Pyfunctionobject when the Pyframeobject is created. The Builtin namespace is the shared caller stack frame (if the stack frame is the initial stack frame, the Builtin dictionary is first used to set the F_builtins field for Pyframeobject).

Here you can review the stack frame relationship of function calls in C. As the following code, the corresponding stack frame structure. When the function is called, the function arguments are pressed into the stack frame of the current function, each function has its own stack frame, and since ESP changes, other functions will index the function parameters via EBP.

//函数调用栈帧测试代码func.cint bar(intint d){    int e = c + d;    return e;}int foo(intint b){    return bar(a, b);}int main(void){    foo(23);    return0;}

So how do you simulate function arguments in Python? It is known from the C function call procedure that function arguments are pressed into the stack frame of the calling function before the function call, and the called function takes arguments based on EBP. Here we first review the structure of the Pyframeobject object, function call and pyframeobject are inextricably linked.

typedef struct_frame {Pyobject_var_headstruct_frame *f_back;/ * Previous frame, or NULL * /Pycodeobject *f_code;/ * Code segment * /Pyobject *f_builtins;/* Builtin symbol table (pydictobject) */Pyobject *f_globals;/* Global symbol table (pydictobject) */Pyobject *f_locals;/* Local symbol table (any mapping) */Pyobject **f_valuestack;/* points after the last local * /Pyobject **f_stacktop; Pyobject *f_trace;/ * Trace function * /Pyobject *f_exc_type, *f_exc_value, *f_exc_traceback; Pythreadstate *f_tstate;intF_lasti;/ * Last instruction if called * /    intF_lineno;/* Current line number * /    intF_iblock;/ * Index in F_blockstack * /Pytryblock F_blockstack[co_maxblocks];/ * for try and loop blocks * /Pyobject *f_localsplus[1];/ * Locals+stack, dynamically sized * /} Pyframeobject;

In the Pyframeobject object, F_valuestack points to the bottom of the runtime stack, and f_stacktop points to the top of the stack, and f_stacktop changes when the function parameter is pressed into the runtime stack, which is a bit like the EBP and ESP in C. F_localsplus is pointing 局部变量+Cell对象+Free对象+运行时栈 to its memory layout as shown in 2. Where the Cell object and the free object are used in closures, and later on, the local variables and runtime stacks are mainly described here. Before invoking a function, Python presses the Function object, function arguments into the runtime stack of the current stack frame, and when the function executes, creates a new Pyframeobject stack frame object and copies the function parameters to the space where the new stack frame stores the local variables (that is, f_ Localsplus execution of the memory), then the code that Pyeval_evalframeex executes the called function is called.

Look at the following func2.py, this example can be used to look at the function call flow. The example code and corresponding byte code are as follows.

#func2. PY def F(name, age):Age + =5    Print '%s is%s old '% (name, age) F (' SSJ ', -)# #字节码in [1]:ImportDisin [2]: Source = open (' func2.py '). Read () in [3]: CO = compile (source,' func2.py ',' EXEC ') in [4]: Dis.dis (CO)1           0Load_const0(<code object F at0x10776faf8, file"func2.py", line1>)3Make_function0              6Store_name0(f)4           9Load_name0(f) ALoad_const1(' SSJ ') theLoad_const2( -) -Call_function2              +Pop_top ALoad_const3(None) -Return_value in [5]: Dis.dis (co.co_consts[0])2           0Load_fast1(age)3Load_const1(5)6Inplace_add7Store_fast1(age) ...

As you can see def f(name, age): before, bytecode is the MAKE_FUNCTION creation of a Pyfunctionobject object by instruction and stored in the local namespace, the corresponding symbol is the function name F, if the functions have default parameters, in the MAKE_FUNCTION directive will also set the default parameters to func_defaults Field. When the function is ready to be called, it is the function object and function parameter execution function, and the function arguments are stacked before the function is called through the CALL_FUNCTION instruction. Before calling Pyeval_evalframeex to execute the function code, after creating a new stack frame, the function arguments are copied into the local variable space pointed to by F_localsplus before the function f call code is actually executed. When the function f is executed, the age parameter is pressed into the stack and added 5, and then stored to the second field in F_localsplus (the first is the name string "SSJ"). The function parameter position changes as shown.

3 Function Execution namespace

Or look at the example given in the first section func.py, the corresponding byte code is as follows, in fact, the definition function def f(): is the function corresponding to the pycodeobject and stack frame corresponding f_globals build Pyfunctionobject object, and then through Store_ The name Directive associates the Pyfunctionobject object with the function name F and stores it in the local namespace. The corresponding pycodeobject of function f can be obtained and viewed by co.co_consts[0].

inch[1]: Source = open (' func.py '). Read ()inch[2]: Import Disinch[3]: CO = Dis.dis (source,' func.py ',' EXEC ')1           0Load_const0(<code object F at0x1107688a0, file"func.py", line1>)3Make_function0              6Store_name0(f)4           9Load_name0(f) ACall_function0              thePop_top -Load_const1(None) +Return_valueinch[Ten]: Dis.dis (co.co_consts[0])2           0Load_const1(' Function ')3Print_item4Print_newline5Load_const0(None)8Return_value

It was previously said that Python is divided into Local,global,builtin namespaces, which are slightly different when the function is executed. The global namespace we can see is passed pyfunctionobject from the previous stack frame, while the local namespace is assigned null, that is, the function does not use the local namespace, so the problem is, How are those local variables in the function accessed? In fact, in the function local variables are accessed through the load_fast instruction (the next section of this instruction is analyzed), that is, it accesses the f_localsplus memory space, does not need to dynamically find f_locals this pydictobject, static method can provide efficiency.

4 function parameters

In Python, function parameters are divided into positional parameters, key parameters, extended position parameters, and extended key parameters. The positional parameter is the parameter in our previous example, and the key parameter is the value of the parameter specified in the call function. The extended position parameter and the extended key parameter format are similar *lst and **kwargs . The position parameter can also set the default value, if there is a default value, the default value is assigned MAKE_FUNCTION to func_defaults in the instruction.

The use of these parameters can be seen in the following example. The extended position parameter is stored inside Python through a tuple object, regardless of the final pass of several parameters. The extension key parameter is stored inside Python through a Dictionary object. For def f(a, b, *lst): functions like this, if the function is called, the argument is f(1,2,3,4) , in fact, in the Pycodeobject object co_argcount=2, co_nlocals=3 . Co_argcount is the number of positional parameters, while Co_nlocals is the number of local variables, including positional parameters.

# #params1. py positional parameters and key parameters def F(A, b):    PrintA, BF (1,2) F (b=2, a=1)# #params2. PY positional parameters, extended position parameters, extension key parameters def F(value, *lst, **kwargs):    PrintValue#-1    PrintLst# (+)    PrintKwargs# {' A ': 3, ' B ': 4}F (-1,1,2, a=3, b=4)# #params3. py location parameter default value def F(lst = []):Lst.append (3)PrintLST f ()#打印 [3]F ()#打印 [3,3]

The last thing to mention is that the default value of the function parameter is set when the function is defined. As params3.py in the example shows, if you specify a parameter default value, and the function is called without overriding the default value, the problem is prone to occur. To solve this problem, you can add a judgment in the function F if lst: lst = [] .

5 Closures and decorators

As mentioned earlier, there are two fields in Pycodeobject that are related to closures, namely Co_cellvars and Co_freevars. Where Co_cellvars is usually a tuple, which holds the set of variable names used in nested scopes, and co_freevars is usually a tuple, which holds the set of variable names in the outer scope. As the following closure example, there are three Pycodeobject objects, closure.py itself, function get_func and inner_func distribution corresponding to a pycodeobject. The Co_cellvars value in Get_func Pycodeobject is the tuple (' value '), and the pycodeobject of Inner_func Co_freevars is also the variable name value.

 #closure. py closure  def  get_func   () :  value =  "inner"  def  inner_func   () :  print  value return  inner_funcshow_value = Get_func () show_value ()  

You can see the Get_func and Inner_func of the Pyframeobject memory layout, you can get a general understanding of the closure mechanism. In fact, the Pyfunctionobject object of the inner nested function inner_func is stored in the local variable of the outer function, and the Func_closure field in Pyfunctionobject is a tuple object that stores pycellobject. When executing Inner_func, the Pycellobject object stored in Func_closure is copied to the Pyframeobject free object of Inner_func, which is the storage space behind the cell object, so that the inner _func is referenced to value by Freevars (Note that this freevars is not pycodeobject in Inner_func co_freevars, but rather the corresponding memory area in Pyframeobject, Although their content is consistent).

Adorners are based on closures, can be a function, method, class processing, to achieve some additional functions, in the actual coding will be used frequently, such as checking whether the user is logged in, check the input parameters, etc., you can use the adorner to reduce redundant code. Here is an example of an adorner:

 #decorator. py Adorner  def  wrapper   (FN) :  def  _wrapper   () :  print   ' wrapper '  fn () return  _wrapper @wrapper  def   Func   () :  print   ' real func '  if  __name__ = " __main_ _ ": Func ()  #输出 ' wrapper ' real func '  

For more adorners, see this article in Vamei python depth 05 decorator

6 References

• The main examples and principles of Python source code are reference to this book
• Python Quick Tutorials
• Song Jinsong "Linux C language One-stop programming"

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Python Source code profiling note 6-function mechanism