Python in-depth memory management of Python

Memory management of language is an important aspect of language design. It is an important factor in determining language performance. Whether it is the manual management of C language or the garbage collection of Java, it becomes the most important feature of language. In this example, the Python language illustrates a dynamic type, object-oriented language memory management method.

memory usage of the object

Assignment statements are the most common language feature. But even the simplest assignment statement can have a lot of meaning. Python's assignment statement is worth studying.

A = 1

The integer 1 is an object. And A is a reference. Using an assignment statement, reference a points to object 1. Python is a dynamic type of language (referencing dynamic types), and objects are separated from references. Python, like using chopsticks, touches and flips real food-objects by reference.

References and objects

To explore the storage of objects in memory, we can resort to Python's built-in function ID (). It is used to return the identity of an object. In fact, the so-called identity is the memory address of the object.

A = 1print(ID (a))print(Hex (ID (a)))

On my computer, they return the following:

11246696
' 0xab9c68 '

Decimal and hexadecimal representations of memory addresses, respectively.

In Python, both integers and short characters, Python caches these objects for re-use. When we create multiple references that are equal to 1, we actually make all of these references point to the same object.

A = 1= 1print(ID (a))print(ID (b))

The above program returns

11246696

11246696

Visible A and B are actually two references to the same object.

To verify that two references point to the same object, we can use the IS keyword. IS is used to determine whether the object referred to by two references is the same.

#TrueA = 1b= 1Print(A isb)#TrueA ="Good"b="Good"Print(A isb)#FalseA ="very Good Morning"b="very Good Morning"Print(A isb)#FalseA =[]b= []Print(A isb

The above comment is the corresponding run result. As you can see, because Python caches integers and short strings, each object has only one copy. For example, all integer 1 references point to the same object. Even if an assignment statement is used, only new references are created, not the object itself. Long strings and other objects can have multiple identical objects, and you can use an assignment statement to create a new object.

In Python, each object has a total number of references to that object, that is, the reference count (reference count).

We can use the getrefcount ()in the sys package to see the reference count of an object. It is important to note that when a reference is passed to Getrefcount () as a parameter, the parameter actually creates a temporary reference. As a result, Getrefcount () will get 1 more results than expected.

 from Import  = [1, 2, 3]print(Getrefcount (a))

b = A
Print (Getrefcount (b))

For these reasons, two Getrefcount will return 2 and 3 instead of the expected 1 and 2.

Object Reference Object

A Python container object (container), such as a table, dictionary, and so on, can contain multiple objects. In fact, the container object contains not the element object itself, which is a reference to the individual element objects.

We can also customize an object and reference other objects:

class From_obj (object):     def __init__ (self, to_obj):         == [from_obj= (b)print(ID (a.to_obj))print( ID (b))

As you can see, a References object B.

Object refers to the object, which is the most basic form of Python. Even a = 1 is an assignment that actually allows an element of the dictionary's key value "a" to refer to an Integer object 1. The Dictionary object is used to record all global references. The dictionary references the integer object 1. We can use the built-in function globals () to view the dictionary.

When an object A is referenced by another object B, the reference count of a increases by 1.

 from Import  = [1, 2, 3]print= [A, a]print(Getrefcount (a))

Because object B references two times, the reference count of A,a increases by 2.

A reference to a container object can constitute a complex topology. We can use objgraph packages to draw their reference relationships, such as

x = [1, 2, 3= [x, Dict (key1== [Y, (x, y)]import  objgraphobjgraph.show_refs ([z], F Ilename='ref_topo.png')

Objgraph is a third-party package for Python. You need to install Xdot before installing.

sudo apt-get install xdotsudo pip install Objgraph

Objgraph official website

Two objects may be referenced to each other, thus constituting a so-called reference ring (reference cycle).

A == [A]a.append (b)

Even an object, you just have to refer to yourself, you can form a reference ring.

A = []a.append (a)print(Getrefcount (a))

The reference ring can cause a lot of trouble with the garbage collection mechanism, and I'll elaborate on that later.

Reference Reduction

The reference count for an object may be reduced. For example, you can use the DEL keyword to delete a reference:

 from Import  = [1, 2, 3]= aprint(Getrefcount (b))del  a Print (Getrefcount (b))

Del can also be used to delete elements in a container element, such as:

A = [a[0]del ]print(a)

If a reference points to object A, the reference count of object A is reduced when the reference is redirected to a different object B:

 from Import Getrefcount

A = [1, 2, 3= aprint= 1print(Getrefcount (b))

Garbage Collection

Eating too much is always getting fatter, so does python. As the objects in Python become more and more, they will occupy more and more memory. But you don't have to worry too much about Python's figure, it would be nice to "lose weight" at the right time, start garbage collection (garbage collection), and erase useless objects. There are garbage collection mechanisms in many languages, such as Java and Ruby. Although the ultimate goal is to create a slim reminder, there is a big difference in weight loss schemes in different languages (this can be compared to this article and Java Memory Management and garbage collection

)。

On the basic principle, when the reference count of an object in Python falls to 0 o'clock, it means that there is no reference to the object, and the object becomes garbage to be reclaimed. For example, if a new object is assigned to a reference, the reference count of the object becomes 1. If the reference is deleted and the reference count of the object is 0, then the object can be garbage collected. For example, the following table:

A = [1, 2, 3]del A

After del A, no references have been made to this table previously established [1, 2, 3]. It is impossible for a user to touch or use this object in any way. If the object stays in memory, it becomes unhealthy fat. When garbage collection starts, Python scans the object that has a reference count of 0, emptying the memory it occupies.

However, losing weight is an expensive and laborious task. When garbage collection occurs, Python cannot perform other tasks. Frequent garbage collection will greatly reduce python productivity. If there are not many objects in memory, it is not necessary to always start garbage collection. Therefore, Python automatically starts garbage collection only under certain conditions. When Python runs, it records the number of times it allocates objects (object allocation) and Unassigned Objects (object deallocation). Garbage collection starts when the difference between the two values is higher than a certain threshold .

We can view this threshold by using the Get_threshold () method of the GC module:

Import GC Print (Gc.get_threshold ())

Return (700, 10, 10), followed by two 10 is the threshold associated with generational collection, which can be seen later. 700 is the threshold at which garbage collection starts. Can be reset by the set_threshold () method in the GC.

We can also start the garbage collection manually, that is, using gc.collect ().

Generational Recycling

Python also uses the strategy of generational (generation) recycling. The basic assumption of this strategy is that the longer the surviving object, the less likely it is to become garbage in later programs. Our programs tend to produce a large number of objects, many objects quickly produce and disappear, but there are also some objects that are used for a long time. For the sake of trust and efficiency, we believe in the usefulness of these "longevity" objects, so we reduce the frequency of scanning them in garbage collection.

The little guy needs more tests.

Python divides all the objects into 0,1,2 three generations. All new objects are 0-generation objects. When a generation of objects has gone through garbage collection and is still alive, it is grouped into next-generation objects. When garbage collection starts, all 0-generation objects are scanned. If 0 generations have been garbage collected for a certain number of times , then scan cleanup for 0 generation and 1 generation is initiated. When 1 generations have experienced a certain amount of garbage collection, the 0,1,2 is started, that is, all objects are scanned.

These two times are returned by the above Get_threshold () (700, 10, 10) to return two 10. That is, every 10 times 0 generations of garbage collection, will be combined with 1 1 generations of garbage collection, and every 10 times 1 generations of garbage collection, there will be 1 times of the 2 generation garbage collection.

You can also use Set_threshold () to make adjustments, such as more frequent scanning of 2 generations of objects.

Import Gcgc.set_threshold (700, 10, 5)

Isolated Reference Ring

The existence of a reference ring can be very difficult for the garbage collection mechanism above. These reference rings may constitute some objects that cannot be used, but the reference count is not 0.

A == [A]a.append (b)del  adel b

We first created two table objects and referenced each other to form a reference ring. After you delete a A, b reference, it is not possible for these two objects to be called again from the program. However, because of the existence of the reference ring, the reference count of both objects has not dropped to 0 and will not be garbage collected.

Isolated reference ring

To reclaim such a reference ring, Python copies the reference count for each object, which can be remembered as gc_ref. Suppose, for each object I, the count is gc_ref_i. Python iterates through all of the objects I. For object J referenced by each object I, subtract the corresponding Gc_ref_j by 1.

The result after the traversal

After the end of the traversal, GC_REF objects that are not 0, and objects referenced by those objects, as well as the objects that continue to be referenced further downstream, need to be preserved. Other objects are garbage collected.

Summarize

Python is a dynamic type of language whose objects and references are detached. This is very different from the previous process-oriented language. To effectively free up memory, Python has built-in garbage collection support. Python takes a relatively simple garbage collection mechanism, the reference count, and therefore needs to address the problem of isolating the reference ring. Python is both common and special in other languages. Understanding this memory management mechanism is an important step in improving Python performance.

Python in-depth memory management of Python