Python source code profiling Note 6-function mechanism, python source code profiling 6-
The function mechanism of Python is a very important part. Many times, writing scripts using python is just a few simple functions to solve the problem. You don't need to get a class or something like java.
The book address: http://www.jianshu.com/p/d00108741a18
1. function object: PyFunctionObject
The PyFunctionObject object is defined as follows:
typedef struct { PyObject_HEAD PyObject *func_code; /* A code object */ PyObject *func_globals; /* A dictionary (other mappings won't do) */ PyObject *func_defaults; /* NULL or a tuple */ PyObject *func_closure; /* NULL or a tuple of cell objects */ PyObject *func_doc; /* The __doc__ attribute, can be anything */ PyObject *func_name; /* The __name__ attribute, a string object */ PyObject *func_dict; /* The __dict__ attribute, a dict or NULL */ PyObject *func_weakreflist; /* List of weak references */ PyObject *func_module; /* The __module__ attribute, can be anything */} PyFunctionObject;
Let's talk about several important variables in PyFunctionObject: func_code and func_globals. Among them, func_code is the PyCodeObject corresponding to the function object, while func_globals is the global namespace of the function. In fact, this value is passed from the previous layer of PyFrameObject. Func_defaults is the default value for storing functions. It will be mentioned later when analyzing function parameters that func_closure is related to closures and will be mentioned later.
###func.pydef f(): print "Function"f()
As in the above example func. py, the file is compiled and corresponds to 2PyCodeObjectObject, one is func. py itself, and the other is function f. PyFunctionObject is executing the bytecode.def f():PassMAKE_FUNCTIONCommand generation. When you create a PyFunctionObject object, the PyCodeObject object corresponding to function f and the current PyFrameObject object are passed as parameters, which are assigned to the func_code and func_globals fields in PyFunctionObject. When you call a functionPyFunctionObjectObject passedfast_functionIn the function, a new stack frame object is constructed based on the func_code and func_globals fields of the PyFunctionObject object.PyFrameObjectAnd then callPyEval_EvalFrameExExecute the function bytecode in the new stack frame. The PyEval_EvalFrameEx function mentioned in the previous Python execution principle.PyEval_EvalCodeExThe function actually creates a new stack frame object PyFrameObject and then executes the PyEval_EvalFrameEx function.
2. function call stack frame
Function call uses stack frames to establish association. Each stack frame of the called FunctionPyFrameObjectThe f_back pointer is used to point to the called function. The local, global, and builtin namespaces are brand new for new stack frames, while the global namespace is transmitted from PyFunctionObject when a PyFrameObject is created. The builtin namespace shares the caller's stack frame (if the stack frame is an initial stack frame, the builtin dictionary is obtained first to set the f_builtins field of PyFrameObject ).
Here we will review the stack frame relationship of function calls in C language. The following code corresponds to the stack frame structure. When calling a function, the function parameter is first pushed into the stack frame of the current function. Each function has its own stack frame. Because esp changes, therefore, other functions use ebp to index function parameters.
// Function call stack frame test code func. cint bar (int c, int d) {int e = c + d; return e;} int foo (int a, int B) {return bar (a, B );} int main (void) {foo (2, 3); return 0 ;}
In python, how does one simulate the passing of function parameters? From the C function call process, we can know that before a function is called, the function parameters are first pushed to the stack frame of the function to be called, and the called function gets the parameters based on ebp. Here we will first review the structure of the PyFrameObject object. function calls are closely related to PyFrameObject.
typedef struct _frame { PyObject_VAR_HEAD struct _frame *f_back; /* previous frame, or NULL */ PyCodeObject *f_code; /* code segment */ PyObject *f_builtins; /* builtin symbol table (PyDictObject) */ PyObject *f_globals; /* global symbol table (PyDictObject) */ PyObject *f_locals; /* local symbol table (any mapping) */ PyObject **f_valuestack; /* points after the last local */ PyObject **f_stacktop; PyObject *f_trace; /* Trace function */ PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; int f_lasti; /* Last instruction if called */ int f_lineno; /* Current line number */ int f_iblock; /* index in f_blockstack */ PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */ PyObject *f_localsplus[1]; /* locals+stack, dynamically sized */} PyFrameObject;
In the PyFrameObject object, f_valuestack points to the stack bottom of the runtime stack, while f_stacktop points to the top of the stack. When a function parameter is added to the runtime stack, f_stacktop changes, these two variables are a bit similar to the ebp and esp in C. F_localsplus indicatesLocal variable + Cell Object + Free object + runtime StackThe memory layout is shown in figure 2. The cell Object and free object are used in the closure. Let's look at them later. Here we mainly talk about local variables and runtime stacks. Before calling a function, python pushes function objects and function parameters to the runtime stack of the current stack frame. When executing the function, python creates a new PyFrameObject stack frame object, then copy the function parameters to the space in which the new stack frame stores local variables (that is, the memory executed by f_localsplus ), then, PyEval_EvalFrameEx is called to execute the code of the called function.
Let's take a look at func2.py below. Through this example, we can look at the function call process. The sample code and the corresponding bytecode are as follows.
# Func2.pydef f (name, age): age + = 5 print '% s is % s old' % (name, age) f ('ssj', 18) # bytecode In [1]: import disIn [2]: source = open ('func2. py '). read () In [3]: co = compile (source, 'func2. py', 'exec ') In [4]: dis. dis (co) 1 0 LOAD_CONST 0 (<code object f at 0x10776faf8, file "func2.py", line 1>) 3 MAKE_FUNCTION 0 6 STORE_NAME 0 (f) 4 9 LOAD_NAME 0 (f) 12 LOAD_CONST 1 ('ssj') 15 LOAD_CONST 2 (18) 18 CALL_FUNCTION 2 21 POP_TOP 22 LOAD_CONST 3 (None) 25 RETURN_VALUE In [5]: dis. dis (co. co_consts [0]) 2 0 LOAD_FAST 1 (age) 3 LOAD_CONST 1 (5) 6 INPLACE_ADD 7 STORE_FAST 1 (age )......
We can see thatdef f(name, age):As mentioned earlier, bytecode isMAKE_FUNCTIONCommand to create a PyFunctionObject object and store it in the local namespace. The corresponding symbol is function name f. If a function has a default parameterMAKE_FUNCTIONThe command also sets the default parametersfunc_defaultsField. When you prepare to call a function, the function parameter is first pushed to the stack when the function object and function parameters are executed.CALL_FUNCTIONCommand to call a function. Before calling PyEval_EvalFrameEx to execute the function code, after creating a new stack frame, the function parameter is copied to the local variable space pointed to by f_localsplus, then the code for calling the function f is actually executed. When the f function is executed, the age parameter is pushed to the stack, followed by 5, and then stored in the second field of f_localsplus (the first field is the name string "ssj "). Shows the function parameter location change.
3. namespace during function execution
Let's take a look at the example func. py given in section 1. Its corresponding bytecode is as follows. In fact, the function is defined.def f():The PyFunctionObject object is constructed using the f_globals corresponding to the function and the stack frame, and then the PyFunctionObject object is associated with the function name f through the STORE_NAME command and stored in the local namespace. The PyCodeObject corresponding to function f can be obtained and viewed through co. co_consts [0.
In [1]: source = open('func.py').read()In [2]: import disIn [3]: co = dis.dis(source, 'func.py', 'exec') 1 0 LOAD_CONST 0 (<code object f at 0x1107688a0, file "func.py", line 1>) 3 MAKE_FUNCTION 0 6 STORE_NAME 0 (f) 4 9 LOAD_NAME 0 (f) 12 CALL_FUNCTION 0 15 POP_TOP 16 LOAD_CONST 1 (None) 19 RETURN_VALUE In [10]: dis.dis(co.co_consts[0]) 2 0 LOAD_CONST 1 ('Function') 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE
As mentioned before, python is divided into local, global, and builtin namespaces. The namespace for function execution is slightly different. The global namespace is passed through PyFunctionObject from the previous stack frame, while the local namespace is assigned NULL, that is, the local namespace is not used in the function, so the question is, how are the local variables in the function accessed? In fact, the local variables in the function are accessed through the LOAD_FAST command (which will be analyzed in the next section). That is to say, It accesses the memory space of f_localsplus, you do not need to dynamically search for the f_locals PyDictObject. Static methods provide efficiency.
4. Function Parameters
In Python, function parameters are divided into location parameters, key parameters, extended location parameters, and extended key parameters. The location parameter is the parameter in the previous example, while the key parameter is the value of the specified parameter in the call function. The format of the extended location parameter and the extended key parameter is similar.*lstAnd**kwargs. You can also set the default value for the location parameter. If there is a default value, the default value isMAKE_FUNCTIONThe command is assigned to func_defaults.
The following example shows the usage of these parameters. Extension location parameters are stored in python through a single tuples, no matter how many parameters are passed. Extension key parameters are stored in python through a dictionary object. For imagesdef f(a, b, *lst):For such a function, if the parameter isf(1,2,3,4)In fact, in the PyCodeObject objectco_argcount=2, co_nlocals=3. Co_argcount is the number of location parameters, while co_nlocals is the number of local variables, including location parameters.
# Params1.py position parameter and key parameter def f (a, B): print a, bf (1, 2) f (B = 2, a = 1) # params2.py location parameter, extended location parameter, extended key parameter def f (value, * lst, ** kwargs): print value #-1 print lst # (1, 2) print kwargs # {'A': 3, 'B': 4} f (-1, 1, 2, a = 3, B = 4) # default def f (lst = []): lst. append (3) print lst f () # print [3] f () # print [3, 3]
The last thing to mention is that the default value of the function parameter is set when the function is defined. As shown in params3.py In the example, if the default value of the parameter is specified and the default value is not overwritten when the function is called, the problem may occur. To solve this problem, add a judgment in function f.if lst: lst = [].
5 closure and decorator
As mentioned before, PyCodeObject has two fields related to closures: co_cellvars and co_freevars. Co_cellvars is usually a set of variable names in the nested scope, while co_freevars is usually a set of variable names in the outer scope. In the example of the closure below, there are three PyCodeObject objects, closure. py itself, and the functions get_func and inner_func correspond to a PyCodeObject. The co_cellvars value in the PyCodeObject of get_func is a tuples ('value',). At the same time, the content stored in the co_freevars of the PyCodeObject of inner_func is also the variable name value.
# Closure. py closure def get_func (): value = "inner" def inner_func (): print value return inner_funcshow_value = get_func () show_value ()
Let's take a look at the memory layout of PyFrameObject in get_func and inner_func to get a general idea of the closure mechanism. In fact, the PyFunctionObject object of the inner_func nested function is stored in the local variables of the outer function. The func_closure field in the PyFunctionObject is a tuple object storing PyCellObject. When inner_func is executed, the PyCellObject object stored in func_closure is copied to the free object of PyFrameObject in inner_func, that is, the block storage space next to the cell object, in this way, the value is referenced through freevars in inner_func (note that this freevars is not the co_freevars in the PyCodeObject of inner_func, but the corresponding memory area in the PyFrameObject, although their content is consistent ).
The decorator is implemented based on closures. It can process a function, method, or class to implement some additional functions, which are often used in actual encoding, such as checking whether a user logs on, check the input parameters, and you can use the decorator to reduce redundant code. The following is an example of a decorator:
# Decorator. py decorator def wrapper (fn): def _ wrapper (): print 'wrapper' fn () return _ wrapper @ wrapperdef func (): print 'real func' if _ name _ = "_ main _": func () # output 'wrapper' real func'
For more information about the decorator, see vamei's article Python in-depth 05 decorator
6 References
- The main examples and principles of python source code analysis are as follows:
- Python quick tutorial
- Song Jinsong's Linux C language one-stop Programming
Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.