Python Storage Management Mechanism __python

Python uses the GC module to handle Python objects and the work of the Python garbage collector, as the complete GC module documentation references here. A few examples of GC modules look at:

Gc.enable ()--automatic garbage collection; gc.disable ()--not automatically garbage collection; Gc.set_threshold ()--sets the threshold for Python garbage collection; Gc.set_debug ()-- Sets the debug tag for garbage collection, and the debug information is written to Std.err gc.get_objects ()--Returns a list of all objects tracked by the collector, excluding the returned list. ......

Python's memory management mechanism mainly includes three aspects: reference counting mechanism, garbage collection mechanism, memory pool mechanism

first, the reference counting mechanism

Python internally uses reference counting to keep track of objects in memory, and all objects have reference counts.

Increase in reference count :

(1) The object is created: x=1;

(2) By the object has been referenced to assign value: y=x;

(3) is passed as a parameter to the function: f (x);

(4) put it in a personal container, such as a list, a tuple, or a dictionary.

Decrease in reference count :

(1) A local reference leaves its scope. For example, when the f (X) function above ends, the object reference that X points to will be reduced by 1;

(2) Explicit destruction of object aliases using the DEL statement: Del X;

(3) The reference is assigned a value: x=2;

(4) The object is removed from a Window object: Mylist.remove (x);

(5) The Window object itself is destroyed: Del myList, or the Window object itself left the scope: F (myList)

Use the sys.getrefcount () function to get the current reference count for an object. In most cases, the reference count is much larger than you would guess.

For immutable data , such as numbers and strings, the interpreter shares memory in different parts of the program in order to conserve memory.

second, garbage collection

(1) Reference count

Reference counting is also a garbage collection mechanism and is one of the most intuitive and simple garbage collection techniques. Party the reference count of an object drops to 0 o'clock, indicating that no reference is made to the object, and the object becomes the garbage to be recycled and is recycled. One exception, however, is that circular references are references to each other, making the reference count of a group of objects not 0, and then the objects are not actually referenced by any external objects, which take up memory and are never released.

So Python has introduced other garbage collection mechanisms to make up for the pitfalls of reference counting: "Mark-clean," "generational recycling."

(2) Mark cleanup

For circular references, when two objects are referenced to each other, the DEL statement reduces the reference count for A and B and destroys the name used to reference the underlying object. However, because each object contains a reference to another object, the reference count is not zeroed and the object is not destroyed, resulting in a memory leak. To solve this problem, the interpreter periodically executes a loop detector that searches for the loops of inaccessible objects and deletes them.

Instead of changing the actual reference count in practice, you copy the reference count of the object in the collection and change the copy of the object reference. (Any changes made to the replica do not affect the maintenance of the object lifecycle.) The only effect of this count copy is to look for the root object collection (objects in the collection cannot be reclaimed). When the root object collection is successfully found, the current list of memory chains is split into two, a list of which maintains the root object collection, becomes the root list, and the rest of the list maintains the remaining objects, becoming the unreachable list. The reason to split into two lists is based on the consideration that the current unreachable may have objects that are referenced directly or indirectly by objects in the root list, which cannot be recycled, and once in the process of tagging, such objects are found, Move it from the unreachable list to the root list; When the tag is finished, all the objects remaining in the unreachable list are truly garbage objects, and the next garbage collection is limited to the unreachable list.

(3) Generational recycling

From the garbage collection mechanism in front of "mark-clear", the extra operations of this garbage collection mechanism are actually related to the total number of blocks of memory in the system, and the more memory blocks that need to be recycled, the more additional operations the garbage detection brings, and the less the extra work that garbage collection brings; The less memory blocks that need to be recycled, the less the extra action that garbage detection brings to the garbage collection.
Give an example to illustrate:
When some memory block M is still alive after 3 garbage collection cleaning, we row the memory block m into a set a, and the newly allocated memory is divided into set B. When garbage collection begins to work, most cases are garbage collected only for collection B, and collection A is garbage collected for a considerable amount of time, which makes the garbage collection mechanism less memory to process and improves efficiency. In this process, some of the memory blocks in set B are moved to set a for a long time to live, of course, there is actually some rubbish in set a, which is deferred because of the mechanism of this generational.

three, the memory pool mechanism

Python provides a garbage collection mechanism for memory, but it puts unused memory in the memory pool instead of returning it to the system.

The memory mechanism in Python presents a pyramid shape, where 1,-2 layers are operated primarily by the operating system. The No. 0 layer is the memory allocation and release functions, such as Malloc,free in C, to operate.

(1) The 1th and 2nd layers are memory pools , with Python interface functions Pymem_malloc function implementations. All objects less than 256 bytes in Python use the allocator implemented by Pymalloc, while large objects use the system's malloc. However, by modifying the Python source code, we can change this default value to change Python's default memory management behavior.

Python introduced a memory pool mechanism to speed up Python execution to manage application and release of small chunks of memory.

(3) For Python objects, such as integers, floating-point numbers, and lists, each has its own private pool of memory, and objects do not share their memory pools. That is, if you allocate and release a large number of integers, the memory used to cache these integers can no longer be assigned to floating-point numbers. (Value semantics)

Image source: http://blog.csdn.net/alertbear/article/details/50808178