[python]--garbage collection mechanism

Transfer from http://www.cnblogs.com/kaituorensheng/p/4449457.html

In Python, in order to solve the problem of memory leaks, object reference counts are used, and automatic garbage collection is implemented based on reference counting.

　　memory leaks: also known as "storage leaks." A dynamically opened space with dynamic storage allocation functions is not released after use, resulting in the memory unit being occupied until the end of the program.

Memory leaks The image metaphor is that "the storage space that the operating system can provide to all processes is being squeezed by a process", and the end result is that the longer the program runs, the more storage space is consumed, the more storage space is exhausted, and the entire system crashes. So "memory leaks" is from the operating system's point of view. Here the storage space is not the physical memory, but the virtual memory size, the virtual memory size depends on the size of the disk swap settings, a piece of memory requested by the program, if there is no one pointer to it, then this block of memory leaks.

　　Memory Leak Classification:

Common Hair Sex:

　　　　　　The code that occurs in memory leaks is executed multiple times, causing a memory leak each time it is executed .

　　　　Incidental:

　　　　　　The code that occurs in memory leaks occurs only under certain circumstances or operations. The occurrence and the occurrence are relative. For a given environment, the occasional may become regular. So the test environment and test methods are critical to detecting memory leaks.

　　　　Disposable:

The code for a memory leak will only be executed once, or due to an algorithmic flaw, there will always be a block and only one piece of memory leaks. For example, allocating memory in a class's constructor does not release the memory in a destructor, so a memory leak occurs only once.

　　　　Implicit:

　　　　　　The program allocates memory while it is running, but it does not release the memory until the end. Strictly speaking, there is no memory leak, because the final program frees up all the requested memory. But for a server program that needs to run for days, weeks or months, Not releasing memory in a timely manner can also result in the eventual exhaustion of all of the system's memory. So we call this kind of memory leak an implicit memory leak.

　　　　Performance:

　　　　　　Memory leaks or, what does the system show when the resources are exhausted?

CPU Resource exhaustion: It is estimated that the machine is unresponsive, keyboard, mouse, network and so on. It is very common in computer viruses.

Process ID Exhausted: Unable to create a new process, serial or telnet can not be created.

Hard drive exhaustion: The machine is dying, swapping memory is useless, and the logs are useless.

Memory leaks or memory exhaustion: The new connection cannot be created, free memory is less, there are a lot of memory leaks, but to have some consequences, it is necessary that the process is an infinite loop, is a service process. Of course, the kernel is also infinite loop, so if the kernel has a memory leak, The situation is even worse. Memory leaks are difficult to locate and track errors, and there are no useful tools to see (of course, user space has some tools, static analysis, but also dynamic analysis, but to find the kernel of memory leaks, there is no good open source tools).

Because Python has automatic garbage collection, many beginners mistakenly think that there is no need to be bothered by memory leaks. But if you look at the Python document's description of the __del__ () function, you know that there is a cloud in this good day.

A circular reference between objects with the __del__ () function is the main culprit that causes a memory leak. But a circular reference between objects without the __del__ () function can be reclaimed by the garbage collector.

The Python extension GC can view the details of objects that cannot be reclaimed.

Example: A memory leak is not present　　　

ImportGCImportSYSclassCgcleak (object):def __init__(self): Self._text='#'* 10def __del__(self):Passdefmake_circle_ref (): _gcleak=Cgcleak ()Print "_gcleak ref count0:%d"%(Sys.getrefcount (_gcleak))del_gcleakTry:        Print "_gcleak ref Count1:%d"%(Sys.getrefcount (_gcleak))exceptUnboundlocalerror:#local variable XXX reference not defined before        Print "_gcleak is invalid!"deftest_gcleak (): gc.enable ()#Set the garbage collector debug FlagGc.set_debug (GC. debug_collectable | Gc. debug_uncollectable | Gc. debug_instances |GC. debug_objects)Print "begin leak Test ..."make_circle_ref ()Print "\nbegin Collect ..."_unreachable=Gc.collect ()Print "Unreachable Object num:%d"%(_unreachable)Print "Garbage Object num:%d"% (len (gc.garbage))#Gc.garbage is a list object, which is an object that the garbage collector finds unreachable (that is, garbage), but cannot be freed (not recyclable), and typically objects in Gc.garbage are objects that reference objects. Because Python does not know in what order to invoke the object's __del__ function, causing the object to always survive in gc.garbage, resulting in a memory leak if __name__ = = "__main__": Test_gcleak (). If you know a security order, then you can break the citation refresh and then execute del gc.garbage[:] To empty the list of junk objectsif __name__=="__main__": Test_gcleak ()

2         # Object _gcleak has a reference count of 2 is invalid!           # because the Del function was executed, the _gcleak became an unreachable object begin collect               ... # Start garbage collection Unreachable object num:0      # The number of unreachable objects found for this garbage collection is 0garbage an-object num:0          # The number of garbage objects in the interpreter is 0

Results

Example 2: A memory leak to your own circular reference

ImportGCImportSYSclassCgcleak (object):def __init__(self): Self._text='#'* 10def __del__(self):Passdefmake_circle_ref (): _gcleak=cgcleak () _gcleak._self= _gcleak#self-cycle quote yourself    Print "_gcleak ref count0:%d"%(Sys.getrefcount (_gcleak))del_gcleakTry:        Print "_gcleak ref Count1:%d"%(Sys.getrefcount (_gcleak))exceptUnboundlocalerror:Print "_gcleak is invalid!"deftest_gcleak (): Gc.enable () Gc.set_debug (GC. Debug_collectable| Gc. debug_uncollectable | Gc. debug_instances |GC. debug_objects)Print "begin leak Test ..."make_circle_ref ()Print "\nbegin Collect ..."_unreachable=Gc.collect ()Print "Unreachable Object num:%d"%(_unreachable)Print "Garbage Object num:%d"%(Len (gc.garbage))if __name__=="__main__": Test_gcleak ()

View Code

<cgcleak 00000000026366a0>3is <dict 0000000002667bd8>begin Collect...unreachable Object num:2       # The number of objects that cannot be reached by this collection is 2garbage object num:1           # The total number of garbage in the interpreter is 1

Results

Example 3: A circular reference between multiple objects causes a memory leak

ImportGCImportSYSclassCgcleaka (object):def __init__(self): Self._text='$'* 10def __del__(self):Passclasscgcleakb (object):def __init__(self): Self._text='$'* 10def __del__(self):Passdefmake_circle_ref (): _a=Cgcleaka () _b=cgcleakb () _a.s=_b _B.D=_aPrint "ref count0:a=%d b=%d"%(Sys.getrefcount (_a), Sys.getrefcount (_b))del_adel_bTry:        Print "ref count1:a%d"%(Sys.getrefcount (_a))exceptUnboundlocalerror:Print "_a is invalid!"deftest_gcleak (): Gc.enable () Gc.set_debug (GC. Debug_collectable| Gc. debug_uncollectable | Gc. debug_instances |GC. debug_objects)Print "begin leak Test ..."make_circle_ref ()Print "\nbegin Collect ..."_unreachable=Gc.collect ()Print "Unreachable Object num:%d"%(_unreachable)Print "Garbage Object num:%d"%(Len (gc.garbage))if __name__=="__main__": Test_gcleak ()

View Code

begin leak Test...ref count0:a=3 b=3 is invalid! begin collect...unreachable Object num:< /c3>4Garbage Object num:2<cgcleaka 00000000022766d8><cgcleakb 0000000002276710 ><dict 00000000022a7e18><dict 00000000022df3c8>

Results

Conclusion:

Python's GC has strong features, such as setting Gc.set_debug (GC). Debug_leak) can be checked for memory leaks caused by circular references. If a memory leak check is made at development time, and when it is released to ensure that no memory leaks are available, you can increase the operational efficiency by extending the garbage collection interval for Python, or even proactively shutting down the garbage collection mechanism.

[python]--garbage collection mechanism