Python garbage collection mechanism

Garbage collection 1. Small integer Object pool

Integers are used extensively in programs, and in order to optimize speed, Python uses a small integer pool of objects to avoid frequent application and destruction of memory space for integers.

Python defines small integers as [-5, 257] These integer objects are set up well in advance and are not garbage collected. In a Python program, all integers that are in this range use the same object.

Similarly, a single letter is the same.

However, when you define 2 identical strings, the reference count is 0, which triggers a garbage collection

2. Large integer Object pool

A new object is created for each large integer.

3. Intern mechanism

a1 = "HelloWorld"
a2 = "HelloWorld"
a3 = "HelloWorld"
a4 = "HelloWorld"
a5 = "HelloWorld"
a6 = "HelloWorld"
a7 = "HelloWorld"
a8 = "HelloWorld"
a9 = "HelloWorld"

Will python create 9 objects? Will it open up 9 "HelloWorld" of memory space in memory? Think about it, if this is the case, we write 10,000 objects, such as a1= "HelloWorld" ... a1000= "HelloWorld", then he did not open up 1000 "HelloWorld" occupied the memory space? If so, does the memory explode? So there's a mechanism in Python thatintern机制lets him take up only a "HelloWorld" of memory space. By reference counting to maintain when to release.

Summarize

• Small integer [-5,257) Common object, Resident memory
• Single character shared object, resident memory
• Single word, non-modifiable, default on intern mechanism, common object, reference count of 0, destroy

• string (containing spaces), non-modifiable, no intern mechanism, no shared object, reference count 0, destroy

• Large integers do not share memory, reference count is 0, destroy

• numeric types and string types are immutable in Python, which means that you cannot modify the value of the object, and each time you modify the variable, you are actually creating a new object

Garbage collection (ii) 1. Garbage collection (GC garbage Collection)

Today's high-level languages such as java,c# have adopted a garbage collection mechanism, rather than a way for users to manage and maintain their own memory in c,c++. Self-management memory is extremely free, can arbitrarily apply for memory, but like a double-edged sword, for a large number of memory leaks, dangling pointers and other bugs buried hidden trouble. For a string, list, class, or even numeric value is an object, and positioning a simple and easy-to-use language, nature will not let the user to deal with how to allocate memory collection. Python also uses the same garbage collection mechanism as Java, but not the same: Python is based on the reference counting mechanism, marking-clearing and generational collection of two mechanisms supplemented by the strategy

Reference counting mechanism:

Every thing in Python is an object, the core of which is a struct:PyObject

 
typedef struct_object {
    int ob_refcnt;
    struct_typeobject *ob_type;
} PyObject;

Pyobject are the necessary content for each object, where ob_refcnt is counted as a reference. When an object has a new reference, its ob_refcnt is incremented, and when the object referencing it is deleted, its ob_refcnt will be reduced

#define Py_INCREF (op) ((op)-> ob_refcnt ++) // Increment the count
#define Py_DECREF (op) \ // Decrement the count
     if (-(op)-> ob_refcnt! = 0); else __Py_Dealloc ((PyObject *) (op))

When the reference count is 0 o'clock, the object's life is over.

Advantages of the reference counting mechanism:

• Simple
• Real-time: Once there is no reference, the memory is released directly. Don't wait for a specific moment like any other mechanism. Real-time also brings a benefit: the time it takes to process the recovered memory is allocated to the usual.

Disadvantages of the reference counting mechanism:

• Maintain reference count consumption resources
• Circular references

list1 = []
list2 = []
list1.append(list2)
list2.append(list1)

List1 and List2 refer to each other, and if there are no other objects referencing them, the reference count of List1 and List2 is still 1, and the memory consumed will never be recycled, which would be fatal. For today's powerful hardware, the disadvantage of 1 is acceptable, but circular references lead to memory leaks, destined Python will also introduce a new recycling mechanism. (Mark Purge and generational collection)

2. Painting says Ruby and Python garbage collection

English Original: Visualizing garbage collection in Ruby and Python

2.1 Application that heart that's beating

GC Systems work far more than "garbage collection". In fact, they are responsible for three important tasks. They

• Newly generated object allocates memory
  Identify those junk objects, and
  Reclaim memory from garbage objects.
  

If the application is likened to a human body: all the elegant code, business logic, and algorithms that you write should be the brain. So, the garbage collection mechanism should be the body organ? (I heard a lot of interesting answers from Rupy listeners: kidney, white blood cells:) ）

I think garbage collection is the heart of the app. Like the heart provides blood and nutrients to other organs of the body, the garbage collector provides memory and objects for your application. If the heart stops beating, it won't take a few seconds for the person to finish. If the garbage collector stops working or slows down, like an arterial blockage, your application will become less efficient until it eventually dies.

2.2 A simple example

The use of examples has always contributed to the understanding of theory. Here's a simple class, written in Python and Ruby, for example today:

In other words, the code in both languages can be so similar: Ruby and Python are really just slightly different in expressing the same thing. But is it so similar in the internal implementation of the two languages?

2.3 Object Assignment for Ruby

What did Ruby do when we executed the above node.new (1)? How does Ruby create new objects for us? Surprisingly, it did very little. In fact, before the code began to execute, Ruby created hundreds of objects ahead of time, and strung them up on the list, called: the available lists. A conceptual diagram of the available list is shown:

Imagine that each white square is marked with a "unused pre-created object". When we call Node.new, Ruby simply takes a pre-created object for us to use:

The left Gray in the middle represents the current object used in our code, while the other white lattices are unused objects. (Note: No doubt mine is a simplification of the actual.) In fact, Ruby uses another object to load the string "ABC", another object to load the node class definition, and an object to load the abstract syntax tree parsed in the code, and so on)

If we call Node.new,ruby again, we will pass another object to us:

This simple algorithm for pre-allocating objects with a linked list has been invented for more than 50 years, and the inventor, John McCarthy, a famous computer scientist, was first implemented with Lisp. Lisp is not only the earliest functional programming language, but also has many innovations in the field of computer science. One is to use the garbage collection mechanism to automate the concept of program memory management.

The standard version of Ruby, known as the "Matz's Ruby Interpreter" (MRI), uses a GC algorithm similar to the way McCarthy was implemented in the 1960. For better or worse, Ruby's garbage collection mechanism is 53 years old. Like Lisp, Ruby Pre-creates some objects and then you use them when you assign new objects or variables.

2.4 Object Assignment for Python

We've learned that Ruby pre-creates objects and stores them in the available list. And what about Python?

Python and Ruby are different in terms of allocating memory for new objects and variables, although Python also uses available lists for many reasons (to reclaim certain objects such as list).

For example, we use Pyhon to create a node object:

Unlike Ruby, Python immediately requests memory from the operating system when an object is created. (Python actually implements a set of its own memory allocation system, which provides an abstraction layer on top of the operating system heap.) But I'm not going to start talking today. )

When we create a second object, again like the OS request memory:

It seems simple enough that Python will take some time to find and allocate memory for us when we create the object.

2.5 Ruby developer living in a messy room

Ruby leaves useless objects in memory until the next GC executes

Back to see Ruby. As we create more and more objects, Ruby will continue to seek out the available list of pre-created objects for us. As a result, the list of available becomes shorter:

... And then shorter:

Notice that I've been assigning new values to the variable n1, and Ruby left the old values in place. "ABC", "JKL" and "MNO" Three node instances are also stuck in memory. Ruby does not immediately erase old objects that are no longer in use in your code! Ruby developers are like living in a messy room with clothes piled on the floor or dirty dishes in the sink. As a ruby programmer, useless garbage objects will always surround you.

2.6 Python developer lives in a family of health

The Trash object will be cleaned by Python immediately.

Python is quite different from Ruby's garbage collection mechanism. Let's go back to the three Python node objects we mentioned earlier:

Internally, when an object is created, Python always holds an integer in the object's C struct, called引用数. At the beginning of the period, Python sets this value to 1:

A value of 1 indicates that there is one pointer pointing to or referencing the three objects. If we now create a new node instance, JKL:

As before, Python sets the JKL reference number to 1. However, please note that since we changed the N1 point to JKL, we no longer point to Abc,python to set the reference number of ABC to 0. At the moment, the Python garbage collector immediately stands up! Whenever the object's reference count is reduced to 0,python, it is released immediately, and the memory is returned to the operating system:

Above Python recycles the memory used by the ABC node instance. Remember that Ruby discards old objects and does not release their memory.

This garbage collection algorithm for Python is called reference counting. Invented by George-collins in 1960, the available list algorithm that happened to be invented with John McCarthy appeared in the same year. As Mike-bernstein in the June Gotham Ruby Conference's outstanding garbage Collection mechanism speech, "1960 is the Golden age of garbage collectors ..."

Python developers work in the home of health, and you can imagine a roommate with mild OCD (an obsessive compulsive disorder) constantly cleaning up behind you, and as you lay down the dirty dishes or cups, a guy is ready to put it in the dishwasher!

Let's look at the second example. Join us to let N2 quote N1:

The number of references to the DEF on the left has been reduced by python, and the garbage collector reclaims the Def instance immediately. At the same time, the number of references to JKL has changed to 2 because both N1 and N2 point to it.

2.7 Mark-Clear

Eventually the messy room was filled with rubbish and could not be quiet again. After the Ruby program has run for a while, the available list is eventually exhausted:

All Ruby pre-created objects are now used by the program (they are all dimmed), and the list is empty (no white squares).

At the moment Ruby sacrifices another McCarthy invented algorithm, called: Mark-Erase. First, Ruby stops the program, and Ruby uses the "Earth to stop recycling Dafa". After Ruby polls all pointers, variables and code produce other reference objects and other values. Ruby also facilitates internal pointers through its own virtual machines. Mark each object referenced by these pointers. I use M in the diagram.

The three objects marked M are still in use by the program. Internally, Ruby actually uses a string of bits, known as: Available bitmaps: Do you remember the "Programming Zhu Ji Nanxiong" for the burst sort, which has a strong compression effect on the finite integer set with a low degree of dispersion, to conserve the resources of the machine. ) to keep track of whether the object is marked.

If the tagged object is alive, the remaining unlabeled object can only be garbage, which means that our code will no longer use it. I'm going to be using a white grid to represent garbage objects:

Next, Ruby clears these useless garbage objects and sends them back to the available list:

Inside it all happens to be lightning fast, because Ruby does not actually object from this copy to that. Instead, the garbage object is returned to the available list by adjusting the internal pointer to a new linked table.

Now, when the next time you create the object, Ruby will be able to divide the garbage objects into our use. In Ruby, objects are reincarnation, reincarnated, and enjoying many lives.

2.8 Tags-delete vs. reference count

At first glance, Python's GC algorithm seems far better than Ruby's: Ning House Jie Yu and living in the room? Why does Ruby prefer to periodically force programs to stop running and not use Python's algorithms?

However, the reference count is not as simple as it looks at first glance. There are many reasons why many languages do not use a reference counting GC algorithm like Python:

First, it is not good to achieve. Python has to leave some space inside each object to handle the number of references. This has paid a little bit of space for the price. What's worse, every simple operation, like modifying a variable or reference, becomes a more complex operation because Python needs to add a count, reduce the other, and possibly release the object.

2nd, it is relatively slow. Although Python is robust as the program executes the GC (a dirty dish starts to wash in the dishwasher), it's not necessarily faster. Python constantly updates numerous reference values. Especially when you no longer use a large data structure, such as a list of many elements, Python may have to release a large number of objects at once. Reducing the number of references is a complex recursive process.

In the end, it doesn't always work. A reference count cannot handle a circular data structure-that is, a struct containing a circular reference.

3. Circular data structure in Python and reference count 3.1 circular Reference

In this article, we know that in Python, each object holds an integer value called the reference count, to track how many references point to the object. Python will increase this count value whenever a variable or other object in our program references the target object, and Python will reduce the count if the program stops using the object. Once the count is reduced to zero, Python frees the object and reclaims the associated memory space.

Since the 60 's, the computer science community has faced a serious theoretical problem, that is, for reference counting this algorithm, if a data structure refers to itself, that is, if the data structure is a circular data structure, then some of the value of the reference meter will definitely not become zero. To better understand the problem, let's give an example. The following code shows some of the node classes we used last week:

We have a "constructor" (called __init__ in Python) that stores a separate property in an instance variable. After the class definition we create two nodes, ABC and DEF, in the diagram as the left rectangle. The reference count for two nodes is initialized to 1 because each of the two references points to each node (N1 and N2).

Now, let's define two additional attributes in the node, Next and prev:

Unlike Ruby, Python allows you to dynamically define instance variables or object properties while the code is running. This may seem a bit like ruby missing some interesting magic. (I am not a Python programmer, so there may be some naming errors). We set the N1.next to point to N2, and set N2.prev back N1. Now, our two nodes form one using circular referencing双向链表. Also note that the reference count values for ABC and DEF have increased to 2. Here are two pointers to each node: first, N1 and N2, followed by Next and Prev.

Now, assuming our program no longer uses these two nodes, we set both N1 and N2 to null (none in Python).

Well, Python will reduce the reference count of each node to 1 as usual.

3.2 in Python 0 generation (Generation Zero)

Note that in the example above, we end up with a less common scenario: we have an "island" or a set of unused, pointed objects, but no one has an external reference. In other words, our programs no longer use these node objects, so we want Python's garbage collection mechanism to be smart enough to release these objects and reclaim the memory space they occupy. However, this is not possible because all reference counts are 1 instead of 0. The Python reference counting algorithm cannot handle objects that point to one another.

That's why Python is introducingGenerational GCalgorithms! Just as Ruby uses a list of links to keep track of unused, free objects, Python uses a different list to keep track of active objects. Instead of calling it an "active list," Python's internal C code calls it a 0-generation (Generation Zero). Each time you create an object or other value, Python adds it to the 0-generation list:

From the top you can see that when we created the ABC node, Python added it to the 0-generation list. Please note that this is not a real list and cannot be accessed directly from your code, in fact the linked list is a fully internal Python runtime. Similarly, when we create a def node, Python adds it to the same list:

The 0 generation now contains two node objects. (he will also include each of the other values that Python creates, with some of the internal values that Python uses for itself.) )

3.3 Detecting circular references

Python then loops through each object on the 0-generation list, checking each mutually referenced object in the list, and losing its reference count based on the rule. In this process, Python will count the number of internal references one after the other to prevent the object from being released prematurely.

For ease of understanding, consider an example:

From the above you can see that the ABC and DEF nodes contain 1 references. There are three other objects that exist in the 0-generation list at the same time, and the blue arrows indicate that some objects are being referenced by objects other than the 0-generation linked list. (Next we'll see that there are two other linked lists in Python that are called generation and second generation, respectively). These objects have a higher reference count because they are being pointed toward by other pointers.

Next you'll see how the python GC handles the 0-generation list.

By identifying internal references, Python can reduce the number of reference counts for many 0-generation linked list objects. In the first line you can see that the reference count for ABC and DEF has become zero, which means the collector can free them and reclaim the memory space. The remaining active objects are moved to a new linked list: A list of the generations.

In a sense, the Python GC algorithm is similar to the one used by Ruby for the token recycling algorithm. Periodically tracing references from one object to another to determine whether the object is still active is being used by the program, which is similar to Ruby's markup process.

GC Thresholds in Python

When will python carry out this tagging process? As your program runs, the Python interpreter keeps track of the newly created objects and objects that are freed because the reference count is zero. Theoretically, these two values should be consistent, since each object created by the program should eventually be freed.

Of course, that is not the case. Because of the reason for the circular reference, and because your program uses objects that are longer than other objects, the difference between the count value of the allocated object and the count value of the freed object is growing gradually. Once the variance accumulates above a certain threshold, the Python collection mechanism starts and triggers the 0-generation algorithm mentioned above, releasing "floating garbage" and moving the remaining objects to a generation list.

Over time, the objects used by the program are gradually moved from the zero-generation list to the generation list. Python's handling of objects in a generation list follows the same approach, and Python moves the remaining active objects to the second-generation list once the assigned count value and the released count value accumulate to a certain threshold.

In this way, the objects that your code uses for the long term, the active objects that your code continues to access, are transferred from the zero-generation list to the generation and then to the second generation. With different threshold settings, Python can process these objects at different intervals. Python handles the most frequent 0 generations, followed by a generation and then the second generation.

The weak generation hypothesis

Take a look at the core behavior of the garbage collection algorithm: The garbage collector handles new objects more frequently. A new object is what your program just created, and one comes with objects that are still there after several periods of time. Python promotes (promote) This object when an object is moved from zero to a generation, or from a generation to a second generation.

Why do you do this? The source of this algorithm is derived from the weakly-generational hypothesis (weak generational hypothesis). The hypothesis is made up of two points: first of all, the object of the year's pro usually dies fast, while the old object is likely to survive for longer periods of time.

Suppose I now create a new object in Python or ruby:

According to the hypothesis, my code will probably only use ABC for a very short time. This object may be just an intermediate result of a method, and the object will become garbage as the method returns. Most of the new objects are so quickly turned into rubbish. Occasionally, however, the program creates important, long-lived objects-such as session variables or configuration items in a Web application.

By frequently processing new objects in the 0-generation list, Python's garbage collector will spend time in more meaningful places: it handles new objects that can quickly become garbage. At the same time, only at very few times, when the threshold is met, the collector goes back to dealing with those old variables.

Garbage collection (iii)-GC module I. Garbage collection mechanism

Garbage collection in Python is based on reference counts and is supplemented by generational collections.

1, the case that causes the reference count +1

• 对象被创建，例如a=23
• 对象被引用，例如b=a
• 对象被作为参数，传入到一个函数中，例如func(a)
• 对象作为一个元素，存储在容器中，例如list1=[a,a]

2, resulting in reference count-1 case

• 对象的别名被显式销毁，例如del a
• 对象的别名被赋予新的对象，例如a=24
• 一个对象离开它的作用域，例如f函数执行完毕时，func函数中的局部变量（全局变量不会）
• 对象所在的容器被销毁，或从容器中删除对象

3. View a reference count for an object

import sys
a = "hello world"
sys.getrefcount(a)

You can view the reference count of a object, but it is 1 larger than the normal count, because a is passed in when the function is called, which gives a reference count of +1

Two. Circular references cause memory leaks

The flaw in reference counting is the problem of circular referencing

import gc

class ClassA ():
     def __init __ (self):
         print (‘object born, id:% s‘% str (hex (id (self))))

def f2 ():
     while True:
         c1 = ClassA ()
         c2 = ClassA ()
         c1.t = c2
         c2.t = c1
         del c1
         del c2

#Close python's gc
gc.disable ()

f2 ()

Executes F2 (), and the memory that the process consumes grows.

• After the C1,C2 has been created, the reference count for both blocks of memory is 1,c1.t=c2andc2.t=c1after that, the reference count for both blocks of memory becomes 2.
• After Del C1, the reference count of the memory 1 object becomes 1, because it is not 0, so the memory 1 object is not destroyed, so the number of references to memory 2 is still 2, after Del C2, in the same vein, memory 1 object, memory 2 object reference count is 1.
• Although all two of their objects can be destroyed, the garbage collector does not recycle them because of the circular references, which results in a memory leak.

Three. Garbage collection

# coding = utf-8
import gc

class ClassA ():
     def __init __ (self):
         print (‘object born, id:% s‘% str (hex (id (self))))
     # def __del __ (self):
     # print (‘object del, id:% s‘% str (hex (id (self))))

def f3 ():
     print ("----- 0 ------")
     # print (gc.collect ())
     c1 = ClassA ()
     c2 = ClassA ()
     c1.t = c2
     c2.t = c1
     print ("----- 1 ------")
     del c1
     del c2
     print ("----- 2 ------")
     print (gc.garbage)
     print ("----- 3 ------")
     print (gc.collect ()) #Explicitly perform garbage collection
     print ("----- 4 ------")
     print (gc.garbage)
     print ("----- 5 ------")

if __name__ == ‘__main__’:
     gc.set_debug (gc.DEBUG_LEAK) #Set the log of the gc module
     f3 ()

Python2 Running results:

-----0------
object born,id:0x724b20
object born,id:0x724b48
-----1------
-----2------
[]
-----3------
gc: collectable <ClassA instance at 0x724b20>
gc: collectable <ClassA instance at 0x724b48>
gc: collectable <dict 0x723300>
gc: collectable <dict 0x71bf60>
4
-----4------
[<__main__.ClassA instance at 0x724b20>, <__main__.ClassA instance at 0x724b48>, {‘t‘: <__main__.ClassA instance at 0x724b48>}, {‘t‘: <__main__.ClassA instance at 0x724b20>}]
-----5------

Description

• Garbage collected objects are placed in the Gc.garbage list
• Gc.collect () returns the number of Unreachable objects, 4 equals two objects, and their corresponding dict

There are three scenarios that trigger garbage collection:

1. Call Gc.collect (),
2. When the counter of the GC module reaches the threshold.
3. When the program exits

Four. GC Module common function analysis

gc模块提供一个接口给开发者设置垃圾回收的选项。 As mentioned above, one flaw in managing memory using reference counting is circular referencing, and one of the main functions of GC modules is to solve the problem of circular references.

Common functions:

1. Gc.set_debug (flags) sets the debug log of the GC, which is generally set to GC. Debug_leak

2, Gc.collect ([generation]) explicit garbage collection, you can enter parameters, 0 is to check only the first generation of objects, 1 for checking one, two generations of objects, 2 for checking one, two, three generations of objects, if not pass parameters, execute a full collection, That is equal to pass 2. Returns the number of unreachable (Unreachable objects) objects

3, Gc.get_threshold () Gets the frequency of garbage collection automatically in the GC module.

4, Gc.set_threshold (threshold0[, threshold1[, Threshold2]) sets the frequency at which garbage collection is performed automatically.

5, Gc.get_count () Gets the counter that is currently automatically performing garbage collection, and returns a list of length 3

Automatic garbage collection mechanism of GC module

You must import the GC module, and theis_enable()=Trueautomatic garbage collection will not start.

The main role of of this mechanism is to find and process unreachable garbage objects.





Garbage collection = garbage inspection + garbage collection

In Python, the method of generational collection is used. The object is divided into three generations, the beginning, the object at the time of creation, placed in a generation, if in a generation of garbage inspection, the change to survive, will be placed in the second generation, the same time in a second generation of garbage inspection, the object survived, will be placed in three generations.

The GC module will have a counter with a 3-length list, which can be obtained by Gc.get_count ().

For example (488,3,0), where 488 is the distance from the last generation of garbage checks, the number of Python allocated memory minus the number of freed memory, note that the memory allocation, rather than the increase in the reference count. For example:

print gc.get_count() # (590, 8, 0)
a = ClassA()
print gc.get_count() # (591, 8, 0)
del a
print gc.get_count() # (590, 8, 0)

3 refers to the last second generation of garbage inspection, a generation of garbage inspection, the same, 0 is the distance from the last three generations of garbage inspection, the second generation of garbage check the number of times.

GC modulo fast has an automatic garbage collection阀值, that is, through the Gc.get_threshold function to obtain a length of 3 tuples, such as (700,10,10) each time the increase of the counter, the GC module will check whether the increase in the count of the number of thresholds, if so, The corresponding algebraic garbage check is performed, and then the counter is reset

For example, suppose the threshold is (700,10,10):

When the counter is increased from (699,3,0) to (700,3,0), the gc module will execute gc.collect (0), which checks the garbage of the generation object and resets the counter to (0,4,0 )
When the counter is increased from (699,9,0) to (700,9,0), the gc module will execute gc.collect (1), which checks the garbage of the first and second generation objects and resets the counter to (0, 0,1)
When the counter is increased from (699,9,9) to (700,9,9), the gc module will execute gc.collect (2), which checks the garbage of the first, second and third generation objects and resets the counter to (0 , 0,0)

Watch out.

The only thing that the GC module cannot handle is that the __del__ method is used in the class of the circular reference, so it is necessary to avoid defining the __del__ method in the project

import gc

class ClassA():
    pass
    # def __del__(self):
    #     print(‘object born,id:%s‘%str(hex(id(self))))

gc.set_debug(gc.DEBUG_LEAK)
a = ClassA()
b = ClassA()

a.next = b
b.prev = a

print "--1--"
print gc.collect()
print "--2--"
del a
print "--3--"
del b
print "--3-1--"
print gc.collect()
print "--4--"

Operation Result:

--1--
0
--2--
--3--
--3-1--
gc: collectable <ClassA instance at 0x21248c8>
gc: collectable <ClassA instance at 0x21248f0>
gc: collectable <dict 0x2123030>
gc: collectable <dict 0x2123150>
4
--4--

If you open del, the result is:

--1--
0
--2--
--3--
--3-1--
gc: uncollectable <ClassA instance at 0x6269b8>
gc: uncollectable <ClassA instance at 0x6269e0>
gc: uncollectable <dict 0x61bed0>
gc: uncollectable <dict 0x6230c0>
4
--4--

Python garbage collection mechanism