20 recommendations for Python performance optimization (reprint)

• Optimization of the algorithm time complexity

  The time complexity of the algorithm has the greatest impact on the execution efficiency of the program, and in Python it is possible to optimize the time complexity by selecting the appropriate data structure, such as the time complexity of the list and set to find an element, respectively O (n) and O (1). Different scenarios have different optimization methods, in general, there are divided, branch boundaries, greed, dynamic planning and other ideas.

• Reduce redundant data

  To save a large symmetric matrix in a triangular or triangular way. The sparse matrix representation is used in the matrix of the 0 element majority.

• Reasonable use of copy and Deepcopy

  For objects such as data structures such as dict and lists, the direct assignment uses a reference method. In some cases, you need to copy the entire object, you can use copy and deepcopy in the copy package, the difference between the two functions is recursive replication. The efficiency is not the same: (The following program runs in Ipython)

  Import Copy
  A = range (100000)
  %timeit-n Copy.copy (a) # run 10 times copy.copy (a)
  %timeit-n Copy.deepcopy (a)
  Ten loops, Best of 3:1.55 ms Per loop
  Ten loops, Best of 3:151 ms Per loop

  The Timeit followed by-n indicates the number of runs, and the next two lines correspond to the output of two Timeit. This shows that the latter is one order of magnitude slower.

• To find an element using Dict or set

  Python dict and set are implemented using a hash table (similar to unordered_map in the C++11 Standard library), and the time complexity of finding elements is O (1)

  A = range (1000)
  s = Set (a)
  D = dict ((i,1) for I in a)
  %timeit-n 10000 in D
  %timeit-n 10000-in S
  10000 loops, Best of 3:43.5 ns per loop
  10000 loops, Best of 3:49.6 ns per loop

  The efficiency of the dict is slightly higher (more space occupied).

• Rational use of generators (generator) and yield

  %timeit-n A = (I for I in range (100000))
  %timeit-n B = [i-I in range (100000)]
  Loops, Best of 3:1.54 ms Per loop
  Loops, Best of 3:4.56 ms Per loop

  using () is a generator object that requires no more memory space than the size of the list, so the efficiency is higher. For specific applications, such as set (I for I in range (100000)) is faster than set ([I for I in range (100000)]).

  But for situations where loop traversal is required:

  %timeit-n x in (I-I in range (100000)): Pass
  %timeit-n x in [i-I in range (100000)]: Pass
  Ten loops, Best of 3:6.51 ms Per loop
  Ten loops, Best of 3:5.54 ms Per loop

  The latter is more efficient, but if there is a break in the loop, the benefits of using generator are obvious. Yield is also used to create generator:

  def yield_func (LS):
  For i in LS:
  Yield i+1
  
  def not_yield_func (LS):
  return [i+1 for i in LS]
  
  ls = range (1000000)
  %timeit-n for I in Yield_func (LS):p
  %timeit-n for I in Not_yield_func (LS):p
  Ten loops, Best of 3:63.8 ms Per loop
  Ten loops, Best of 3:62.9 ms Per loop

  For a list that is not very large in memory, you can return a list directly, but the readability yield is better (a person's preference).

  Python2.x has xrange function, Itertools package, etc. built-in generator function.

• Optimize loops

  What you can do outside of the loop is not in the loop, for example, the following optimizations can be faster:

  A = range (10000)
  Size_a = Len (a)
  %timeit-n-I in a:k = Len (a)
  %timeit-n-I in a:k = Size_a
  Loops, Best of 3:569µs per loop
  Loops, Best of 3:256µs per loop

• Optimize the order of multiple judgment expressions

  For and, the satisfying conditions should be placed in the front, for or, the satisfying conditions are placed in front. Such as:

  A = Range (2000)
  %timeit-n [I for I in a If < I < + or + < I < 2000]
  %timeit-n [I for I in a If < I < + + < I < 20]
  %timeit-n [I for I in a if I% 2 = = 0 and i > 1900]
  %timeit-n [I for I in a If i > 1900 and i% 2 = = 0]
  Loops, Best of 3:287µs per loop
  Loops, Best of 3:214µs per loop
  Loops, Best of 3:128µs per loop
  Loops, Best of 3:56.1µs per loop

• Use the join to merge the string in the iterator in

  [1]:%%timeit
  ...: s = ""
  ...: For i in a:
  ...: s + = i
  ...:
  10000 loops, Best of 3:59.8µs per loop
  
  In [2]:%%timeit
  s = "". Join (a)
  ...:
  100000 loops, Best of 3:11.8µs per loop

  Joins have a 5 times-fold increase in the way they accumulate.

• Select the appropriate format character mode

  s1, s2 = "ax", "BX"
  %timeit-n 100000 "abc%s%s"% (S1, S2)
  %timeit-n 100000 "Abc{0}{1}". Format (s1, S2)
  %timeit-n 100000 "abc" + S1 + s2
  100000 loops, Best of 3:183 ns per loop
  100000 loops, Best of 3:169 ns per loop
  100000 loops, Best of 3:103 ns per loop

  Of the three cases, the% is the slowest, but the difference between the three is not large (both very fast). (Individuals feel the best readability of%)

• Exchange values of two variables without using intermediate variables in

  [3]:%%timeit-n 10000
  a,b=1,2
  ....: c=a;a=b;b=c;
  ....:
  10000 loops, Best of 3:172 ns per loop
  
  In [4]:%%timeit-n 10000
  a,b=1,2
  A,b=b,a
  ....:
  10000 loops, Best of 3:86 ns per loop

  Use a,b=b,a instead of c=a;a=b;b=c; to exchange a A, a, a, 1 time times faster value.

• Use if is

  a = range (10000)
  %timeit-n [i-I in a if i = = True]
  %timeit-n [i-I in a if I is True]
  Loops, Best of 3:531µs per loop
  Loops, Best of 3:362µs per loop

  Use if is true nearly one times faster than if = true.

• use a cascading comparison of x < Y < Z

  x, y, z =%timeit-n
  1000000 if x < y < Z:pass
  %timeit- n 1000000 if x < y and y < Z:pass
  1000000 loops, Best of 3:101 ns per loop
  1000000 loops, Best of 3:12 1 ns per loop

  x < Y < Z are slightly more efficient and more readable.

• while 1 is faster than while True

  def while_1 ():
      n = 100000
      while 1: br>        n-= 1
          if n <= 0:break
  def While_true ():
      n = 100000
      While true:
          n -= 1
          if n <= 0:break    
  
  m, n = 1000000, 1000 000 
  %timeit-n while_1 ()
  %timeit-n while_true ()
  Loops, Best of 3:3.69 ms Per loop
  Loops, Best of 3:5.61 ms Per loop

  while 1 is much faster than while true because in python2.x, true is a global variable, not a keyword.

• Use * * Instead of POW

  %timeit-n 10000 c = POW (2,20)
  %timeit-n 10000 C = 2**20
  10000 loops, Best of 3:284 ns per loop
  10000 loops, Best of 3:16.9 ns per loop

  * * is faster than 10 times times!

• Use CProfile, Cstringio and cpickle to achieve the same function (respectively, the profile, Stringio, pickle) of the package

  import Cpickle
  Import Pickle
  A = range (10000)
  %timeit-n x = Cpickle.dumps (a)
  %timeit-n x = Pickle.dumps (a)
  Loops, Best of 3:1.58 ms Per loop
  Loops, Best of 3:17 ms Per loop

  C Implementation of the package, speed faster than 10 times times!

• The
• uses the best deserialization method

  import JSON
  Import cPi Ckle
  A = range (10000)
  S1 = str (a)
  S2 = Cpickle.dumps (a)
  S3 = Json.dumps (a)
  %timeit-n x = eval (S1)
  %timeit-n x = cpickle.loads (s2)
  %timeit-n x = json.loads (S3)
  Loops, Best of 3: 16.8 ms Per loop
  Loops, best of 3:2.02 ms Per loop
  from loops, best of 3:798µs per loop

• The
• use C extension (Extension)

  cpython native API: By introducing the Python.h header file, Python's data structure can be used directly in the corresponding C program. The implementation process is relatively cumbersome, but has a relatively large scope of application.

  ctypes: typically used for encapsulating (wrap) C programs, allowing pure Python programs to invoke functions in a dynamic-link library (DLL in Windows or so files in Unix). If you want to use a C class library in Python already, using cTYPES is a good choice, and with some benchmarks, python2+ctypes is the best way to perform. The

  cython:cython is a superset of CPython and is used to simplify the process of writing C extensions. The advantage of Cython is that the syntax is concise and can be well compatible with NumPy and other libraries that contain a large number of C extensions. The Cython scenario is typically optimized for an algorithm or process in the project. In some tests, you can have hundreds of times times the performance boost.

  cffi:cffi is the implementation of cTYPES in PyPy (see below), also compatible with CPython. Cffi provides a way to use Class C libraries in Python, writing C code directly in Python code, and supporting links to existing C-class libraries.

  using these optimizations is generally optimized for existing project performance bottleneck modules, which can greatly improve the efficiency of the entire program in the case of minor changes to the original project.

• Parallel programming

  Because of the Gil's presence, Python is difficult to take advantage of multicore CPUs. However, there are several parallel modes that can be implemented through the built-in module multiprocessing:

  Multi-process: for CPU-intensive programs, you can use multiprocessing Process,pool and other packaged classes to implement parallel computing in a multi-process manner. However, because the communication cost in the process is relatively large, the efficiency of the program that requires a lot of data interaction between processes may not be greatly improved.

  Multithreading: For IO-intensive programs, the Multiprocessing.dummy module uses multiprocessing's interface to encapsulate threading, making multithreaded programming very easy (such as the ability to use the pool's map interface for simplicity and efficiency).

  Distributed: The managers class in multiprocessing provides a way to share data in different processes, on which a distributed program can be developed.

  Different business scenarios can choose one or several of these combinations to achieve program performance optimization.

• End-stage big kill device: PyPy

  PyPy is a python implemented using Rpython (a subset of CPython), which is 6 times times faster than the CPython implementation of Python based on the benchmark data of the website. The reason for this is that the Just-in-time (JIT) compiler, a dynamic compiler, is different from a static compiler (such as GCC,JAVAC, etc.) and is optimized using data from the process that the program is running. The Gil is still in pypy for historical reasons, but the ongoing STM project attempts to turn PyPy into Python without Gil.

  If the Python program contains a c extension (non-cffi), the JIT optimization effect will be greatly reduced, even slower than CPython (than NumPy). Therefore, it is best to use pure python or cffi extension in PyPy.

  With the improvement of stm,numpy and other projects, I believe PyPy will replace CPython.

• Using the Performance analysis tool

  In addition to the Timeit modules used above in Ipython, there are cprofile. CProfile is also very simple to use: Python-m cProfile filename.py,filename.py is the file name to run the program, you can see in the standard output the number of times each function is called and the elapsed time, so as to find the program's performance bottleneck, It can then be optimized in a targeted manner.

Citation: http://saebbs.com/forum.php?mod=viewthread&tid=37696&page=1&extra=

http://www.maxburstein.com/blog/speeding-up-your-python-code/

http://www.ibm.com/developerworks/cn/linux/l-cn-python-optim/

20 recommendations for Python performance optimization (reprint)