Python Memory leakage

1. Python has an automatic garbage collection mechanism (when the object reference count is zero, the interpreter will automatically release the memory ), memory leakage usually occurs when the extended library memory leaks or circular references (another is that the objects in the global container are not deleted)

The former does not need to be discussed. The latter example (OBJ ('B') and OBJ ('C') memory are not recycled)

[dongsong@localhost python_study]$ cat leak_test2.py #encoding=utf-8class Obj:    def __init__(self,name='A'):        self.name = name        print '%s inited' % self.name    def __del__(self):        print '%s deleted' % self.nameif __name__ == '__main__':    a = Obj('A')    b = Obj('B')    c = Obj('c')    c.attrObj = b    b.attrObj = c[dongsong@localhost python_study]$ vpython leak_test2.py A initedB initedc initedA deleted

Ii. objgraph Module

This module can find the fastest growing objects and the most practical objects. It can draw the reference relationship diagram of all elements in an object and all reference relationships behind an object. It can obtain Objects Based on the address.

However, using it to find out memory leaks is still a bit of a haystack feeling: You need to determine the suspicious objects (usually common objects such as list/dict/tuple, this is difficult to troubleshoot; it is better to determine the cause if the most rapid is a custom unconventional object)

1. show_refs () show_backrefs () show_most_common_types () show_growth ()

[Dongsong @ localhost python_study] $! Catcat objgraph1.py # encoding = utf-8import objgraplif _ name _ = '_ main _': x = [] y = [x, [x], dict (x = x)] objgraph. show_refs ([y], filename = '/tmp/sample-graph.png') # Draw the reference of all objects in [y] objgraph. show_backrefs ([x], filename = '/tmp/sample-backref-graph.png') # draws all references to x objects # objgraph. show_most_common_types () # statistics on all common types of objects. The data volume is too large and it is of little significance to objgraph. show_growth (limit = 4) # print the added objects from the beginning of the program or the last show_growth (sort by the size of the increase) [Dongsong @ localhost python_study] $! Vpythonvpython objgraph1.py Graph written to/tmp/tmpuSFr9A. dot (5 nodes) Image generated as/tmp/sample-graph.pngGraph written to/tmp/tmpAn6niV. dot (7 nodes) Image generated as/tmp/sample-backref-graph.pngtuple 3393 + 3393wrapper_descriptor 945 + 945 function 830 + 830builtin_function_or_method 622 + 622

Sample-graph.png

Sample-backref-graph.png

2. show_chain ()

[dongsong@localhost python_study]$ cat objgraph2.py #encoding=utf-8import objgraph, inspect, randomclass MyBigFatObject(object):        passdef computate_something(_cache = {}):        _cache[42] = dict(foo=MyBigFatObject(),bar=MyBigFatObject())        x = MyBigFatObject()if __name__ == '__main__':        objgraph.show_growth(limit=3)        computate_something()        objgraph.show_growth(limit=3)        objgraph.show_chain(                objgraph.find_backref_chain(random.choice(objgraph.by_type('MyBigFatObject')),                        inspect.ismodule),                filename = '/tmp/chain.png')        #roots = objgraph.get_leaking_objects()        #print 'len(roots)=%d' % len(roots)        #objgraph.show_most_common_types(objects = roots)        #objgraph.show_refs(roots[:3], refcounts=True, filename='/tmp/roots.png')[dongsong@localhost python_study]$ !vpythonvpython objgraph2.py tuple                  3400     +3400wrapper_descriptor      945      +945function                831      +831wrapper_descriptor      956       +11tuple                  3406        +6member_descriptor       165        +4Graph written to /tmp/tmpklkHqC.dot (7 nodes)Image generated as /tmp/chain.png

Chain.png

Iii. gc Module

This module determines that the garbage collection cycle cannot reference (unreachable) and cannot release (uncollectable) objects, which is unique compared with objgraph.

Gc. collect () Forcibly recycles garbage and returns the number of unreachable objects

Gc. garbage returns the list of uncollectable objects in the unreachable object (all objects with _ del _ () destructor and trapped in reference loops) IfDebug_saveall
Is set, then all unreachable objects will be added to this list rather than freed.

Warning: If gc. disable () is used to disable automatic garbage collection, and gc. collect () is not active, you will see the consumption of memory refreshing ....

[dongsong@bogon python_study]$ cat gc_test.py #encoding=utf-8import gcclass MyObj:        def __init__(self, name):                self.name = name                print "%s inited" % self.name        def __del__(self):                print "%s deleted" % self.nameif __name__ == '__main__':        gc.disable()        gc.set_debug(gc.DEBUG_COLLECTABLE | gc.DEBUG_UNCOLLECTABLE | gc.DEBUG_INSTANCES | gc.DEBUG_OBJECTS | gc.DEBUG_SAVEALL)        a = MyObj('a')        b = MyObj('b')        c = MyObj('c')        a.attr = b        b.attr = a        a = None        b = None        c = None        if gc.isenabled():                print 'automatic collection is enabled'        else:                print 'automatic collection is disabled'        rt = gc.collect()        print "%d unreachable" % rt        garbages = gc.garbage        print "\n%d garbages:" % len(garbages)        for garbage in garbages:                if isinstance(garbage, MyObj):                        print "obj-->%s name-->%s attrrMyObj-->%s" % (garbage, garbage.name, garbage.attr)                else:                        print str(garbage)[dongsong@bogon python_study]$ vpython gc_test.py a initedb initedc initedc deletedautomatic collection is disabledgc: uncollectable <MyObj instance at 0x7f3ebd455b48>gc: uncollectable <MyObj instance at 0x7f3ebd455b90>gc: uncollectable <dict 0x261c4b0>gc: uncollectable <dict 0x261bdf0>4 unreachable4 garbages:obj--><__main__.MyObj instance at 0x7f3ebd455b48> name-->a attrrMyObj--><__main__.MyObj instance at 0x7f3ebd455b90>obj--><__main__.MyObj instance at 0x7f3ebd455b90> name-->b attrrMyObj--><__main__.MyObj instance at 0x7f3ebd455b48>{'name': 'a', 'attr': <__main__.MyObj instance at 0x7f3ebd455b90>}{'name': 'b', 'attr': <__main__.MyObj instance at 0x7f3ebd455b48>}

Iv. pdb Module

Manual: http://www.ibm.com/developerworks/cn/linux/l-cn-pythondebugger/

Command and gdb are good (only p is not required when printing data, and the debugging interface and operations are similar to the python interaction mode)

H (elp) Help

C (ontinue) continue

N (ext) next statement

S (tep) Next Step (inside the function)

B (reak) sets the breakpoint

L (ist) display code

Bt call stack

Press enter to repeat the previous command.

....

The birdman prefers to add PDB. set_trace () to the debugging and then enter the status... (there are many other options)

5. Django Memory leakage

Why is Django leaking memory?

Django isn't known to leak memory. If you find your Django processes areallocating more and more memory, with no sign of releasing it, check to makesure yourDebug
Setting is setFalse. IfDebugIsTrue,
Then Django saves a copy of every SQL statement it has executed.

(The queries are saved inDjango. DB. Connection. Queries. Seehow
Can I see the raw SQL queries Django is running ?.)

To fix the problem, set
DebugToFalse.

If you need to clear the query list manually at any point in your functions, just callReset_queries (), Like this:

from django import dbdb.reset_queries()