Python Performance diagnostics and code optimization techniques

    program code optimization usually includes: Reduce the size of the code, improve the efficiency of the Code. This allows the program to run faster. Here's a concrete discussion of Python code optimization techniques.


Improved algorithm, select appropriate data structure

A good algorithm can play a key role in performance, so the first point of performance improvement is to improve the algorithm. The time complexity of the algorithm in order is:

O (1)-> O (lg N)-> o (n lg N)-> O (n^2)-> O (n^3)-> O (n^k)-> O (k^n)-> O (n !)

Therefore, if the algorithm can be improved in time complexity, the improvement of performance is self-evident. However, the improvement of the specific algorithm is not the scope of this article, the reader can refer to this information. The following sections will focus on the selection of data structures. A hash table is used in the

    Dictionary (dictionary) and list

Python dictionaries, so the lookup operation has a complexity of O (1), and the list is actually a number of arrays, in the L IST, the lookup needs to traverse the entire list, and its complexity is O (n), so the action dictionary for the member's lookup access is faster than the list.

Listing 1. Code dict.py

from time import time   t = time ()    list = [' A ', ' B ', ' is ', ' python ', ' Jason ', ' Hello ', ' hill ', ' with ', ' phone ', ' Test ',  ' dfdf ', ' Apple ', ' PDDF ', ' ind ', ' basic ', ' none ', ' baecr ', ' var ', ' Bana ', ' dd ', ' WRD ']    #list  =  Dict.fromkeys (list,true)    print list  filter = []   for i in  range  (1000000):       for find in [' is ', ' hat ', ' new ', ' List ', ' old ', '. ']:           if find not in  list:               filter.append ( Find)    print  "Total run time:"  print time ()-T 

The above code will probably take 16.09seconds to run. If you remove the comment for the row #list = Dict.fromkeys (list,true), and then run the list after converting it to a dictionary, the time is approximately 8.375 seconds, and the efficiency is probably increased by half. Therefore, it is a good choice to use dict instead of list when multiple data members are frequently searched or accessed. The Union of the

    Collection (set) and list

Set, which is intersection,difference faster than the list iteration. So if it involves finding the list intersection, the problem of the set or the difference can be converted to set to operate.

Listing 2. Find the intersection of list:

From time import time
t = time ()
lista=[1,2,3,4,5,6,7,8,9,13,34,53,42,44]
listb=[2,4,6,9,23]
Inter Section=[] for
i in range (1000000): For
A in Lista: for B in
Listb:
If a = = B:
intersection.append (a)
print "Total run time:"
print Time ()-T

The running time of the above program is probably:

Total run Time:
38.4070000648

Listing 3. Using set to find intersection

from time import time   t = time ()    lista=[1,2,3,4,5,6,7,8,9,13,34,53,42,44]   listb=[2,4,6,9,23]   intersection=[]    for i in range  (1000000):       list (Set (Lista) &set (LISTB))    print  "Total run time:"  print time ()-T 

When the

is changed to set, the program runs at a reduced time of 8.75, up to 4 times times more, and the running time is greatly shortened. The reader can use the table 1 other actions to test.

Set common usage

Syntax                       Operations            Instructions
Set (LIST1) | set ( LIST2)   Union            new collection containing all data for List1 and List2
Set (List1) & Set (LIST2)   intersection  contains a new collection of List1 and List2 communist elements
Set (List1)? Set (List2)   difference & nbsp;  the collection of elements appearing in List1 but not appearing in List2


optimization of loops

The optimization of loops follows the principle of minimizing the amount of computation in the loop. There are multiple loops that try to refer to the upper layer of the calculation of the inner layer. The following examples are used to compare the performance improvement of the cyclic optimization. In Listing 4 of the program, the approximate elapsed time is about 132.375 if the loop optimization is not done.

Listing 4. Pre-
    

for loop optimization

From time to import time
t = time ()
lista = [1,2,3,4,5,6,7,8,9,10]
Listb =[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8, 0.9,0.01] for
i in range (1000000): For
A in range (Len (lista)): for
B in range (Len (LISTB)):
X=lista[a]+listb[b]
print ' Total run time: '
print time ()-T

The following optimizations are now made to refer to the length calculation as a loop, and range is replaced with xrange, while the third layer of the computation lista[a] refers to the second layer of the loop.

Listing 5. After loop optimization

From time to import time
t = time ()
lista = [1,2,3,4,5,6,7,8,9,10]
Listb =[0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8, 0.9,0.01]
len1=len (lista)
Len2=len (LISTB) for I-xrange (1000000)
: For A in
xrange (len1): 
   Temp=lista[a] for
B-Xrange (LEN2):
x=temp+listb[b]
print "total run t IME:
print time ()-T

The optimized program has a shortened running time of 102.171999931. In Listing 4 Lista[a] The number of times calculated is 1000000*10*10, and the number of times calculated in the optimized code is 1000000*10, the number of computations is shortened significantly, so performance improves.

take full advantage of the attributes of the Lazy if-evaluation

The conditional expression in Python is Lazy evaluation, that is, if there is a conditional expression if x and Y, the If x is false, the value of the Y expression is no longer evaluated. Therefore, this feature can be used to improve program efficiency to some extent.

Listing 6. Using the Lazy if-evaluation feature

From time import time
t = time ()
abbreviations = [' cf. ', ' e.g ', ' ex. ', ' etc. ', ' fig. ', ' i.e. ', ' Mr. ', ' vs. ']         For-
I in Range (1000000): For
W in (' Mr. ', ' Hat ', ' is ', ' chasing ', ' the ', ' black ', ' cat ', '. '):
If w in abbreviations:
#if w[-1] = = '. ' and W abbreviations:
Pass
Print Total Run time: '
print time ()-T

The program runs about 8.84 times before it is optimized, and if you use a comment line instead of the first if, you run about 6.17.

　　Optimization of strings

String objects in Python are immutable, so manipulating any string, such as stitching, modifying, and so on, will produce a new string object instead of the original string, so this continuous copy can affect Python's performance to some extent. Optimization of strings is also an important aspect of improving performance, especially in cases where more text is processed. The optimization of strings is mainly focused on the following aspects:

Use Join () instead of + in string concatenation: Using + for string concatenation in Listing 7 probably requires 0.125 s, and the join is shortened to 0.016s. Therefore, the join is faster than the + in the operation of the character, so try to use join instead of +.

Listing 7. Use join instead of + connection string

From time to import time
t = time ()
s = ""
list = [' A ', ' B ', ' B ', ' d ', ' e ', ' f ', ' g ', ' h ', ' I ', ' j ', ' K ', ' l ', ' m ' , ' n '] for
i in range (10000): for
substr in list:
s+= substr
print "Total run time : "
print Time ()-T

At the same time to avoid:

s = ""
For x in list:
S + + func (x)

Instead, you use:

Slist = [Func (ELT) for ELT in Somelist]
s = "". Join (Slist)

2. When you can use regular expressions or built-in functions to handle strings, select the built-in functions. such as Str.isalpha (), Str.isdigit (), Str.startswith (' x ', ' yz '), Str.endswith ((' x ', ' YZ '))

3. Formatting characters is faster than direct inline reading, so use the

out = "%s%s%s%s"% (head, prologue, query, tail)

and avoid

out = "" + head + Prologue + query + tail + ""

Working with List resolution (list comprehension) and builder expressions (generator expression)

List resolution is more efficient than rebuilding a new list in a loop, so we can use this feature to improve the efficiency of the operation.

From time to import time
t = time ()
list = [' A ', ' B ', ' are ', ' python ', ' Jason ', ' Hello ', ' hill ', ' with ', ' phone ', ' test ', c2/> ' dfdf ', ' apple ', ' PDDF ', ' ind ', ' basic ', ' none ', ' baecr ', ' var ', ' Bana ', ' dd ', ' WRD ']
total=[] for
i in range ( 1000000): For
W in list:
total.append (w)
print ' Total run time: '
print time ()-T

Use list resolution:

For I in Range (1000000):
A = [w for w in list]

The above code directly runs about 17s, and instead of using list resolution, the elapsed time is shortened to 9.29s. Nearly half the increase. The builder expression is the new content introduced in 2.4, syntax is similar to list resolution, but the advantage of the generator expression in large data processing is obvious, it does not create a list, it simply returns a generator, so it is more efficient. In the example above, the Code a = [w for w in list] is modified to a = (W for w in list), the running time is further reduced, shortened by about 2.98s.

　　Other optimization techniques

1, if you need to exchange the value of two variables using a,b=b,a instead of the use of intermediate variable t=a;a=b;b=t;

>>> from Timeit import timer
>>> timer ("T=a;a=b;b=t", "a=1;b=2"). Timeit ()
0.25154118749729365
>>> Timer ("A,b=b,a", "a=1;b=2"). Timeit ()
0.171566777

34181258
>>>

2. Use xrange instead of range when looping, and using xrange can save a lot of system memory because Xrange () produces only one integer element per call in the sequence. and range () ⒅ benzene 臃 prayer pseudo tonyuquq, Qi oblique angry, the fat was sheng the plaque? The xrange no longer exists in T Python3, which provides a iterator that can traverse a range of arbitrary lengths.

3, the use of local variables to avoid the "global" keyword. Python accesses local variables much faster than global variables, so you can use this feature to improve performance.

4, if do is not none of the statement if done!= None faster, the reader can verify;

5, in a more time-consuming cycle, you can change the function of the call to inline mode;

6, using the Cascade comparison "x < Y < Z" instead of "X < Y and y < Z";

7, while 1 is faster than while True (the latter is, of course, more readable);

8, build in function is usually faster, add (A,B) is superior to a+b.

　Positioning Program Performance Bottlenecks

The premise of code optimization is the need to understand where the performance bottlenecks, the main time the program runs is consumed, for more complex code can use some tools to locate, Python built a wealth of performance analysis tools, such as Profile,cprofile and hotshot. The Profiler is a python-brought set of programs that describe the performance of the program at run time and provide a variety of statistics to help users locate the program's performance bottlenecks. The Python standard module provides three kinds of profilers:cprofile,profile and hotshot.

Profile is very simple to use, just import it before you use it. Specific examples are as follows:

Listing 8. Profiling by using profile

Import Profile
def profiletest (): Total
=1;
For I in range:
total=total* (i+1) Print Total return total
If __name__ = "__ main__ ":
profile.run (" Profiletest () ")

The results of the program's operation are as follows:

Figure 1. Performance Analysis Results

The specific explanations for each column of output are as follows:





Ncalls: Represents the number of function calls;


Tottime: Represents the total elapsed time of the specified function, removing the run-time of the calling child function in the function;


Percall: (first percall) equals tottime/ncalls;


Cumtime: Represents the time that the call of the function and all its child functions runs, that is, when the function starts calling to the time of return;


Percall: (second Percall) is the average time when a function is run, equal to Cumtime/ncalls;


Filename:lineno (function): The specific information of each function call;





If you need to save the output as a log, just add another argument when you call. such as Profile.run ("Profiletest ()", "Testprof").





For profile profiling data, if the binary file to save the results of the time, you can through the Pstats module for text report analysis, it supports a variety of forms of report output, text interface is a more practical tool. Very simple to use:





Import pstats


p = pstats. Stats (' testprof ')


P.sort_stats ("name"). Print_stats ()





where the Sort_stats () method can sort the split data and accept multiple sort fields, such as Sort_stats (' name ', ' file ') will first be sorted by the function name, and then sorted by the file name. The common sort fields are calls (the number of calls), time (function internal runtime), cumulative (total elapsed time), and so on. In addition, Pstats also provides command-line interaction tools to execute Python? M pstats can help you learn more about how to use it.





For large applications, if the results of performance analysis can be presented graphically, will be very practical and intuitive, common visual tools such as Gprof2dot,visualpytune,kcachegrind, readers can access the relevant official website, this article does not do a detailed discussion.

　Python Performance Tuning Tool

Python Performance Optimization In addition to improved algorithms, the selection of appropriate data structures, there are several key technologies, such as the key Python code part to rewrite the C extension module, or to choose more performance-optimized interpreter, etc., these are called optimization tools in this article. Python has many of its own optimization tools, such as Psyco,pypy,cython,pyrex, which have different advantages, and this section chooses several to introduce.

Psyco

Psyco is a just-in-time compiler that can improve performance without altering the source code, Psyco compiles the operation into a somewhat optimized machine code, The operation is divided into three different levels, with the run-time, compile-time, and virtual-time variables. and to increase and decrease the level of variables as needed. Run-time variables are simply raw bytecode and object structures that are processed by the regular Python interpreter. Once the Psyco is compiled into machine code, the compile-time variables are represented in machine registers and directly accessible memory locations. At the same time Python can cache the compiled machine code for future reuse, which will save a little time. But Psyco also has its drawbacks, and its own running memory is larger. Currently, Psyco is no longer supported in python2.7, and no maintenance and updates are available, and you can refer to the http://psyco.sourceforge.net/

for which you are interested.

pypy

PyPy represents "python implemented in Python", but it is actually implemented using a python subset called Rpython, which can turn Python code into C,. NET, Java and other languages and platform code. PyPy integrates a just-in-time (JIT) compiler. Unlike many compilers, interpreters, it doesn't care about the lexical parsing and syntax tree of Python code. Because it's written in Python, it uses the Python language's Code Object directly. The code Object is the representation of the Python bytecode, which means that pypy directly analyzes the bytecode of the Python code, which is not stored in characters or in a binary format in a file, but in the Python runtime environment. The current version is 1.8. Support for different platform installations, install PyPy on Windows need to download first https://bitbucket.org/pypy/pypy/downloads/ Pypy-1.8-win32.zip, then unzip to the relevant directory and add the extracted path to the environment variable path. To run PyPy on the command line, if the following error occurs: "MSVCR100.dll is not found, so the application could not be started, reinstalling the application may fix the problem", you will also need to download VS. Runtime libraries on Microsoft's official web site to resolve the issue. The specific address is http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=5555

Run pypy on the command line after successful installation, The output results are as follows:

C:\Documents and Settings\administrator>pypy
Python 2.7.2 (0e28b379d8b3, Feb, 18:31:47)
[PyPy 1.8.0 With MSC v.1500-bit] on Win32
Type ' help ', ' copyright ', ' credits ' or ' license ' for the more information.
Something completely different: ' PyPy is vast, and contains
multitudes '
>>>>

In the example of listing 5, using Python and pypy to run separately, the results are as follows:

C:\Documents and settings\administrator\ Desktop \doc\python>pypy loop.py
Total run Time:
8.42199993134

C:\Documents and settings\administrator\ Desktop \doc\python>python loop.py
Total run Time:
106.391000032

Visible use PyPy to compile and run the program, its efficiency is greatly improved.

Cython





Cython is a language implemented in Python that can be used to write Python extensions, and the libraries that are written with it can be loaded through import, and performance is faster than Python. A python extension (such as import math) can be loaded into the Cython, or it can be loaded into the header file of C's library (for example, cdef extern from "math.h"), and it can also be used to write Python code. Rewrite key sections as C extension modules





　　 installation of Linux CPython:





The first step: Download





[root@v5254085f259 cpython]# wget-n Http://cython.org/release/Cython-0.15.1.zip


--2012-04-16 22:08:35--Http://cython.org/release/Cython-0.15.1.zip


Resolving cython.org ... 128.208.160.197


Connecting to cython.org|128.208.160.197|:80 ... Connected.


HTTP request sent, awaiting response ... OK


length:2200299 (2.1M) [Application/zip]


Saving to: ' Cython-0.15.1.zip '





100%[======================================&GT] 2,200,299 1.96m/s in 1.1s





2012-04-16 22:08:37 (1.96 mb/s)-' cython-0.15.1.zip ' saved [2200299/2200299]





The second step: decompression





[root@v5254085f259 cpython]# Unzip-o Cython-0.15.1.zip





Step Three: Install





pythonsetup.py Install





When the installation is complete, enter Cython directly, which indicates that the installation is successful if the following occurs.

[root@v5254085f259 cython-0.15.1]# cython   Cython  (http://cython.org)  is a  compiler for code written in the   Cython language.   cython is based on pyrex by greg ewing.     Usage: cython  [options] sourcefile. {pyx,py} ...     Options:    -V, --version                   display version number  of cython compiler    -l, --create-listing            write error messages to a listing file    -I, --include-dir   Search for include files in named  directory                                     (multiple include directories are allowed) .    -o, -- Output-file    specify name of generated c file   -t,  --timestamps                only compile newer source files    -f, --force                     Compile  all source files  (overrides implied -t)     -q, --quiet                      Don ' t print module names in recursive mode    -v, --verbose   &nbsP;               be verbose,  print file names on multiple compil ation    -p, -- embed-positions          if specified, the  positions in cython files of each    function definition is  embedded in its docstring.    --cleanup     release  interned objects on python exit, for memory debugging.       level indicates aggressiveness, default 0 releases nothing.     -w, --working     sets the working directory for  Cython  (the directory modules are searched from)     --GDB  output debug information for cygdb    -D, --no-docstrings                 Strip docstrings from the  compiled module.    -a, --annotate                 Produce a colorized HTML version of  the source.    --line-directives                 Produce  #line  directives pointing to the  pyx source    --cplus                 Output a C++ rather than C file.    -- embed[=]                generate  a main ()  function that embeds the Python interpreter.    -2           compile based on python-2 syntax and code  seman tics.    -3          compile  based on Python-3 syntax and code seman tics.    -- Fast-fail     abort the compilation on the first error     --warning-error, -Werror       Make all  warnings into errors    --warning-extra, -Wextra        Enable extra warnings    -X, --directive =    [, <name=value,...]  overrides a compiler directive

installation on other platforms can refer to the documentation: http://docs.cython.org/src/quickstart/install.html

Cython code is different from Python and must be compiled first, Compiling typically takes two phases, compiling the Pyx file into a. c file, and then compiling the. c file as a. so file. There are several ways to compile:

    compile from the command line: Suppose you have the following test code, which is compiled using the command line as a. c file.

def sum (int a,int b):
print a+b
[root@v5254085f259 test]# cython sum.pyx
[root@v5254085f259 test]# ls Total
4 drwxr-xr-x 2 root 4096 Apr 02:45.
4 drwxr-xr-x 4 root root 4096 APR 16 2 2:20..
4-rw-r--r--1 root 02:45 1
60-rw-r--r--1 root 55169 Apr 02:45 sum.c 4-rw-r-
-r--1 root Apr 02:45 Sum.pyx

Compile with GCC on Linux as. so file:

[root@v5254085f259 test]# gcc -shared -pthread  -fPIC -fwrapv -O2   -wall -fno-strict-aliasing -i/usr/include/python2.4
 -o sum.so sum.c   [root@v5254085f259 test]# ls   total 96
 4 drwxr-xr-x 2 root root  4096 Apr 17 02:47 .   4 drwxr-xr-x 4 root root  4096 apr 16 22:20     4 -RW-R--R-- 1 ROOT ROOT    35 APR 17 02:45  1  60 -RW-R--R-- 1 ROOT ROOT 55169 APR 17 02:45 SUM.C    4 -RW-R--R-- 1 ROOT ROOT    35 APR 17 02:45  sum.pyx   20 -rwxr-xr-x 1 root root 20307 apr 17 02:47  sum.so 

to build a setup.py script using Distutils compilation

:

from distutils.core import setup   from distutils.extension import  extension   from Cython.Distutils import build_ext     ext_modules =  [extension ("Sum",  ["Sum.pyx"])]     setup (      name =
  ' Sum app ',      cmdclass = {' Build_ext ': build_ext},      ext_modules = ext_modules   )        [ root@v5254085f259 test]#  python setup.py build_ext --inplace   running  build_ext   cythoning sum.pyx to sum.c   building  ' Sum '   extension   gcc -pthread -fno-strict-aliasing -fpic -g -o2 -dndebug  -g -fwrapv -O3   -wall -wstrict-prototypes -fpic -i/opt/ activepython-2.7/include/python2.7 &Nbsp; -c sum.c -o build/temp.linux-x86_64-2.7/sum.o   gcc -pthread - shared build/temp.linux-x86_64-2.7/sum.o   -o /root/cpython/test/sum.so

After the compilation is complete, you can import it into Python using:

[root@v5254085f259 test]# python
activepython 2.7.2.5 (ActiveState Software Inc.) based on
python 2.7.2 (defau LT, June, 11:24:26)
[GCC 4.0.2 20051125 (Red Hat 4.0.2-8)]
on linux2 Type ' help ', ' copyright ', ' credits "or" license for the more information.
>>> import Pyximport; Pyximport.install ()
>>> import sum
>>> sum.sum (1,3)

Here's a simple performance comparison:

Listing 9. Cython Test Code

From time import time
def test (int n):
cdef int a =0
cdef int i to
i in Xrang E (n):
a+= I return
a
t = time ()
test (10000000)
print Total run Time: '
print time ()-T

Test results:

[GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2
Type ' help ', ' copyright ', ' credits ' or ' license ' for the more Informati On.
>>> Import Pyximport pyximport.install ()
>>> import ctest Total
run time:
0.007 14015960693

Listing 10. Python Test Code

From time to import time
def test (n):
a =0;
For I in Xrange (n):
a+= I return
a
t = time ()
test (10000000)
pri NT "Total run time:"
print Time ()-T
[root@v5254085f259 test]# python test.py total
run time: 
   0.971596002579

From the above comparison you can see that the speed of using Cython is increased by nearly 100 times.

Summarize

This paper focuses on the common techniques of performance optimization in Python. How to use tools to diagnose performance bottlenecks in programs, and provide tools or languages that can be optimized for performance, may help Python learning and development friends.