Python extension-reference count and Memory leakage

Background Python usually contains a large amount of memory allocation and release. It also needs to avoid Memory leakage (forgetting to release the allocated memory) and wild pointers (read and write memory areas not allocated ). The selected method is reference count. The principle is relatively simple: each object contains a counter. The increase or decrease of the counter is directly related to the increase or decrease of the reference. When the reference count is 0, it indicates that the object does not exist, yes. In the pyhton extension, it is very interesting and prone to errors to operate the reference count of python objects in C/C ++. For existing references, call Py_DECREF () to reduce the reference count when you no longer need them. If you forget to reduce the reference count, memory leakage will occur. If you forget to increase the reference count, the interpreter may crash. When do you need to increase or decrease the reference count? A ownership rule cannot be owned by anyone but can be referenced by an object. The reference of an object is a part of the function interface specification whenever a function is passed in or out, regardless of whether the ownership is transferred or not. Most functions that return an object reference transfer ownership by referencing. In particular, functions that create new objects, such as PyInt_FromLong () and Py_BuildValue (), transfer ownership to the recipient. Even if the object is not newly created, you still receive a new reference of the object. For example, PyInt_FromLong () maintains a cache of common values and can return cached objects. Another concept to be noted is the "borrow" object. As the name suggests, the borrowed object does not need to add 1 to its reference count. Therefore, Py_DECREF () cannot be called to reduce the reference count. The advantage of borrow is that you do not need to manage the reference count. The disadvantage is that it may be dizzy by the wild pointer, because you cannot guarantee that the borrowed object has been released during use. PyTuple_GetItem (), PyList_GetItem (), PyDict_GetItem (), and PyDict _-GetItemString () All return references you borrowed from tuples, lists, or dictionaries. If an object reference is passed into another function, it is generally a reference borrowed from you by the function-if the function needs to store the reference, it will use Py_INCREF () to form an independent owner. There are two important exceptions to this rule: PyTuple_SetItem () and PyList_SetItem (). These two functions take over the ownership of the sub-items passed to them-even if these sub-items are invalid! (PyDict_SetItem () class functions do not take over ownership.) When a C function is called in Python, It borrows reference of its own parameter from the caller. The caller has a reference to an object, so the life cycle of "borrow reference" is guaranteed before the return value exits. Only when such a "borrow reference" is to be stored or transferred, you must call Py_INCREF () to convert it to "own reference ". The object reference returned by the C function called by Python must be "with reference" -- the ownership is transferred by the function to its caller. Common Errors are obvious reference errors: PyObject * _ list = PyList_New (0 ); pyObject * _ list1 = PyList_New (0); return_list; Since _ list1 is created, it is neither released (Py_DECREF () called nor returned (return transfer ownership ), therefore, memory leakage occurs. PyList_Append .) In some cases, the extension function needs to return a list to python, so this method is used: PyObject * _ list = PyList_New (0); Intvar = ...; PyList_Append (_ list, PyInt_FromLong (var );... Return_list; it seems that there is nothing wrong with the Return_list; however, if a problem is found on PyList_Append, it will add 1 to the reference count of the second parameter. In this example, PyInt_FromLong (var) returns an object. Its reference is 1. After PyList_Append adds this object to the list, the reference count is 2. Then, the _ list ownership is transferred to the python caller. As you can imagine later, when the caller releases this List, it will only reduce the count by one, that is, the object count will never reach 0 (that is, the memory is leaked ). There are two solutions: Method 1PyObject * _ list = PyList_New (0); Int var = ...; PyObject * tmp = PyInt_FromLong (var); PyList_Append (_ list, tmp); Py_DECREF (jpo );... Return_list; Method 2PyObject * _ list = PyList_New (CNT); Int var = ...; PyList_SetItem (_ list, 0, PyInt_FromLong (var ));... Return_list; CNT indicates the length of the specified list, and then assign values to each subscript element in sequence, instead of using the APPEND method. As to why PyList_SetItem can solve this problem, you can refer to "Introduction to ownership rules"