Python Code Refinement and optimization

Python is simple, easy to use, highly developed, well-ported, and rich in code resources, and is widely used. But Python code is made up of dynamic cubby larger, Python library is full, the disadvantage is cubby larger.

In memory consumption method, with the introduction of the Py Library, memory also multiplied, here to discuss how to thin the python, and how to optimize the use of memory.

First, how to thin the Python dynamic library.

Python code is still very concise, so it is difficult to reduce the size of small code, but there are still some ideas to reduce the size of the Python library.

1, strip Python dynamic library.

Dynamic libraries are generally included in the symbol table, these are useful when called, but for release version, it is possible to tune the symbol table, the method is to use the Strip command, so the size can be reduced from eight or nine trillion to less than 3 trillion.

2, using the Code optimization option:-O 3, this parameter will optimize the code, including the size of the generated binary code, the disadvantage is that the optimization will be difficult to debug.

3. Remove the doc String from the code.

Python's code is useful for PYDOC_STRVAR macro-defined modules, which can be removed by specifying--without-doc-strings at configure time, There will be no definition of the following in the generated pyconfig.h:

#define WITH_DOC_STRINGS 1

This reduces the size of the generated dynamic libraries and, of course, reduces the memory footprint of the modules at run time because they no longer contain help information.

4. Get rid of Unicode support.

Unicode support in Python is not required, of course Python 3 is another matter. Python supports Unicode in a way that utf-8 encoding can be used. The support for removing Unicode can use the following parameters when configure:

--enable-unicode=no

In this way, the following definitions are removed from the pyconfig.h:

#define PY_USING_UNICODE 1

Two: How to reduce the size of the Python extension library?

Python's extension library is placed in the Lib directory, you can execute the following command in the Lib directory to compile the Python code:

Python-oo-m Compileall.

This generates a library file with the PYO extension, and the-oo parameter removes the doc string so that the size of the compilation target file can be significantly reduced when the comments are more in the py file.

Be careful not to use absolute paths:

such as python-oo-m compileall/path/to/python/lib use absolute path commands, because when generating pyo files, each function and the method of the class will generate a code object object, each code Object saves the path to the module it is in, and if the absolute path is used, the size of the PYO file is significantly increased when the path is longer and the function is relatively large.

Of course, when the code is running, it can also reduce the amount of memory consumed.

Third, how to reduce the expansion of the library.

There is a py2exe tool that can package Python code and dependent dynamic libraries, package Python's necessary extensions into a zip file, but in fact the zip package is often not the most streamlined. In fact, the biggest difficulty of the cut is to find all the dependent modules, you can use the following method to find out the other modules that a module depends on:

Import importlibdef Module_diff (mod):    import sys    keys = [] for    key in Sys.modules.keys ():        keys.append ( Key)    Importlib.import_module (mod)    for key in Sys.modules.keys ():        if isn't key in keys:            print key, Sys.modules[key]

To view the modules that the Multiprocessing module relies on, you can use the following command:

Module_diff (' multiprocessing ')

You will get the following output:

Multiprocessing.atexit nonemultiprocessing.weakref nonemultiprocessing.signal Nonethreading <module ' threading ' From ' C:\Python27\lib\threading.pyc ' >cpickle <module ' cpickle ' (built-in) >_multiprocessing <module ' _ Multiprocessing ' from ' C:\Python27\DLLs\_multiprocessing.pyd ' >multiprocessing.os nonemultiprocessing.itertools nonemultiprocessing.threading nonemultiprocessing.util <module ' multiprocessing.util ' from ' C:\Python27\lib\ Multiprocessing\util.pyc ' >multiprocessing.sys nonecstringio <module ' Cstringio ' (built-in) > multiprocessing._multiprocessing nonemultiprocessing.multiprocessing nonethread <module ' thread ' (built-in) > Atexit <module ' atexit ' from ' C:\Python27\lib\atexit.pyc ' >multiprocessing <module ' multiprocessing ' from ' \ C ' Python27\lib\multiprocessing\__init__.pyc ' >weakref <module ' weakref ' from ' C:\Python27\lib\weakref.pyc ' > Itertools <module ' itertools ' (built-in) >time <module ' time ' (built-in) &GT;MULTIPROCESSING.PROcess <module ' multiprocessing.process ' from ' C:\PYTHON27\LIB\MULTIPROCESSING\PROCESS.PYC ' 

This allows you to know which modules you depend on.

To see all of the modules, it's easier:

Def print_all_module ():    import sys    keys = [] for    key in Sys.modules.keys ():        print Key,sys.modules[key]

After the code is initialized, execute the above function to know the module that the program needs to run.

Python Code Refinement and optimization