Python performance chicken soup, very effective to improve the performance of tips

Part 1

Read Zen of Python and input import this in the python parser. A novice Python may notice the word "resolution" and think that python is just another scripting language. "It must be slow! "

There is no doubt that the python program is not compiled in a language that is efficient and fast. even Python advocates will tell you that python is not suitable for these fields. however, YouTube has used python to provide services for tens of millions of video requests per hour. all you need to do is write efficient code and use external implementation (C/C ++) code when necessary. here are some suggestions to help you become a better Python developer:

1. Use built-in functions:

You can use python to write efficient code, but it is difficult to beat built-in functions. After verification, they are very fast.

 

2. Use join () to connect a string.

You can use "+" to connect strings. however, because string is unchangeable in Python, each "+" operation creates a new string and copies the old content. A common usage is to use a single character modified by the python array module. When the modification is complete, use the join () function to create the final string.
>>># This is good to glue a large number of strings
>>> For chunk in input ():
>>> My_string.join (chunk)

3. Use python to assign multiple values and exchange variables.
This is elegant and fast in Python:
>>> X, Y = Y, X
This is slow:
>>> Temp = x
>>> X = y
>>> Y = temp

4. Use local variables whenever possible
Python retrieves local variables faster than global variables. This means that the "Global" keyword is avoided.

5. Try to use "in"
Use the "in" keyword. Concise and fast.
>>> For key in sequence:
>>> Print "found"

6. Use delayed loading Acceleration
Import the "import" declaration to the function only when necessary. in other words, if some modules do not need to be used immediately, import them later. for example, you do not have to import a large number of modules as soon as you enable the Acceleration Program. this technology cannot improve overall performance. however, it helps you allocate the module loading time more evenly.

7. Use "while 1" for an infinite loop"
Sometimes you need an infinite loop in the program. (for example, an instance that listens to sockets) Although "While true" can accomplish the same thing, "While 1" is a single-step operation. this method can improve your python performance.
>>> While 1:
>>># Do stuff, faster with while 1
>>> While true:

>>># Do stuff, slower with wile true

8. Use List Comprehension
In Python 2.0, you can use list comprehension to replace a large number of "for" and "while" blocks. using list comprehension is usually faster, and the python parser can find in a loop that it is a predictable mode and optimized. the additional benefit is that list comprehension is more readable (functional programming) and can save an additional counting variable in most cases. For example, let's calculate the number of even numbers between 1 and 10:
>>> # The good way to iterate a range
>>> Evens = [I for I in range (10) If I % 2 = 0]
>>> [0, 2, 4, 6, 8]
>>> # The following is not so pythonic
>>> I = 0
>>> Evens = []
>>> While I <10:
>>> If I % 2 = 0: evens. append (I)
>>> I + = 1
>>> [0, 2, 4, 6, 8]

9. Use xrange () to process long sequences:
This saves you a lot of system memory, because xrange () generates only one integer element for each call in the sequence. In contrast, range () directly gives you a complete list of elements, which is unnecessary for loop.

10. Use Python generator:
This can also save memory and improve performance. For example, a video stream can be sent in one byte block instead of the whole stream. For example,
>>> Chunk = (1000 * I for I in xrange (1000 ))
>>> Chunk
<Generator object <genexpr> at 0x7f65d90dcaa0>
>>> Chunk. Next ()
0
>>> Chunk. Next ()
1000
>>> Chunk. Next ()
2000

11. Understand the itertools module:
This module is very effective for iteration and combination. Let's generate a list of all the permutation and combinations of [1, 2, 3]. Only three lines of Python code are required:
>>> Import itertools
>>> Iter = itertools. permutations ([1, 2, 3])
>>> List (ITER)
[(1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1)]

12. Sort the learning bisect module to keep the list:
This is a free tool for implementing binary search and fast insertion of ordered sequences. In other words, you can use:
>>> Import bisect
>>> Bisect. insort (list, element)
You have inserted an element into the list, and you do not need to call sort () again to maintain the sorting of containers, because this will be very expensive in long sequences.

13. Understanding the python list is actually an array:
The List Implementation in python is not based on common single-chain tables in computer science that people usually talk about. The list in python is an array. That is to say, you can use constant time O (1) to retrieve an element in the list, instead of searching from the beginning. What is the significance of this? When using the list object insert (), Python developers need to think twice. For example: >>> list. insert (0, item)
Inserting an element before the list is inefficient because all subsequent subscript in the list has to be changed. however, you can use list. append () effectively adds elements to the end of the list. select deque first, if you want to quickly insert or at two times. It is fast, because deque in python is implemented using double-stranded tables. I will not talk about it more. :)

14. Test members using dict and set:

Check whether an element exists in dicitonary or set, which is very fast in Python. This is because dict and set use hash tables. The search efficiency can reach O (1 ). Therefore, if you need to check the members frequently, use set or dict as your container.
>>> Mylist = ['A', 'B', 'C'] # slower, check membership with list:
>>> 'C' in mylist
>>> True
>>> Myset = set (['A', 'B', 'C']) # Faster, check membership with set:
>>> 'C' in myset:
>>> True

15. Use the sort () of the schwartzian transform ():

The native list. Sort () function is very fast. Python sorts the list in the natural order. Sometimes, you need to sort in a non-natural order. For example, the IP address you want to sort by server location. Python supports custom comparison. You can use list. Sort (CMP (), which is slower than list. Sort () because it increases the overhead of function calls. If you have any questions about the performance, you can apply for Gutman-rosler Transform Based on schwartzian transform. it is only interested in the algorithms actually used. Its brief working principle is that you can change the list and call the python built-in list. sort ()-> faster without using list. sort (CMP ()-> slow.

16. cache results of the python decorator:
The "@" symbol is the decoration Syntax of Python. It is not only used for tracing, locking, or logging. You can describe a python function and remember the call results for later use. This technology is called memoization. The following is an example:

>>> From functools import wraps
>>> Def memo (f ):
>>> Cache = {}
>>> @ Wraps (f)
>>> Def wrap (* Arg ):
>>> If Arg not in cache: cache ['arg '] = f (* Arg)
>>> Return cache ['arg ']
>>> Return wrap
We can also use the decorator for the Fibonacci function:
>>> @ Memo
>>> Def fib (I ):
>>> If I <2: return 1
>>> Return fib (I-1) + fib (I-2)

The key idea here is: Enhance the function (decoration) function, and remember each calculated Fibonacci value. If they are in the cache, you do not need to calculate them.

17. Understand Python Gil (Global interpreter lock ):
Gil is necessary because the memory management of cpython is non-thread-safe. You cannot simply create multiple threads and want python to run faster on multiple core machines. This is because Gil will prevent multiple native threads from simultaneously executing Python bytecode. In other words, Gil will serialize all your threads. However, you can use threads to manage multiple derivative process acceleration programs, which run independently outside your Python code.

18. Familiar with Python source code like familiar with documents:
Some Python modules use C for performance. When performance is critical and official documentation is insufficient, You can freely explore the source code. You can find the underlying data structure and algorithms. The Python source code library is a great place: http://svn.python.org/view/python/trunk/Modules

Conclusion:
These cannot replace brain thinking. open the hood to fully understand the responsibilities of developers, so that they will not quickly piece together a spam design. the Python suggestions in this article can help you achieve good performance. if the speed is not fast enough, python will need external forces to analyze and run external code. in the second part of this article.

Part 2

It is helpful to note that static compilation code is still important. just a few examples, chrome, Firefox, MySQL, MS office and Photoshop are highly optimized software that we use every day. as a parsing language, python is obviously not suitable. python alone is not enough to satisfy those areas where performance is the primary indicator. this is why Python allows you to access the underlying bare metal infrastructure and delegate heavier work to faster languages such as C. this is a key feature in high-performance computing and embedded programming. the first part discusses how to use Python efficiently. in the second part, we will involve monitoring and scaling python.

1. First, reject the temptation to tune

Optimization adds complexity to your code. before integrating other languages, check the list below. If your algorithm is "good enough", optimization is not so urgent.
1. Have you made a performance test report?
2. Can you reduce hard disk I/O access?
3. Can you reduce network I/O access?
4. Can you upgrade the hardware?
5. Are you compiling libraries for other developers?
6. Is your third-party library the latest version?

2. Use tools to monitor code, rather than intuition
Speed issues may be subtle, so don't rely on intuition. Thanks to the "cprofiles" module, You can monitor Python code by simply running it.
"Python-M cprofile myprogram. py"
 

We wrote a test program. black box monitoring. the bottleneck here is "very_slow ()" function call. we can also see that both "Fast ()" and "slow ()" Are called 200 times. this means that if we can improve the "Fast ()" and "slow ()" functions, we can achieve comprehensive performance improvement. the cprofiles module can also be imported at runtime. this is very useful for checking long-running processes.

3. Review time complexity
After control, a basic algorithm performance analysis is provided. The constant time is the ideal value, and the logarithm time complexity is stable. The factorial complexity is difficult to expand.

O (1)-> O (lg n)-> O (N ^ 2)-> O (N ^ 3) -> O (N ^ K)-> O (K ^ N)-> O (N !)

4. Use a third-party package

There are many high-performance third-party libraries and tools designed for python. Below is a brief list of useful acceleration packages.

1. numpy: an open source package equivalent to MATLAB
2. scipy: Another numeric processing database
3. gpulib: Use GPUs to accelerate code
4. pypy: Use the just-in-time compiler to optimize Python code
5. cython: Convert Python code to C

6. shedskin: Convert Python code to C ++

5. Use the multiprocessing module to implement true concurrency
Because Gil will serialize threads, multithreading in Python cannot be accelerated on multiple-core machines and clusters. therefore, Python provides the multiprocessing module, which can derive additional processes to replace threads and jump out of Gil restrictions. in addition, you can also combine this suggestion in external C code to make the program faster.

Note that the overhead of a process is usually higher than that of a thread because the thread automatically shares the memory address space and file descriptor. this means that creating a process consumes more or more memory than creating a thread. this should be kept in mind when you use multi-processor computing.

6. Local Code

Now you decide to use local code for performance. in the standard ctypes module, you can directly load the programmed binary Library (. DLL or. so file) in Python, do not worry about writing C/C ++ code or building dependencies. for example, we can write a program to load libc to generate a random number.

However, the cost of binding ctypes is non-lightweight. you can think of ctypes as a bonding operation library function or hardware device-driven glue. several Python-embedded libraries such as swig, cython, and boost have lower calling costs than ctypes. python supports object-oriented features such as class and inheritance. as we can see, we can retain the regular C ++ code and import it later. the main task here is to compile a package (Line 10 ~ 18 ).

Summary:

I hope these Python suggestions can make you a better developer. finally, I need to point out that the pursuit of performance limit is an interesting game, and over-optimization will become a ridicule. although Python allows you to seamlessly integrate with the C interface, you must ask yourself if you have bought the account after several hours of hard work optimization. on the other hand, it is worthwhile to sacrifice the maintainability of the Code in exchange for several milliseconds of improvement. the team members often Thank you for writing concise code. try to be close to the python method, because life is too short.

Http://blog.monitis.com/index.php/2012/02/13/python-performance-tips-part-1/.

Http://blog.monitis.com/index.php/2012/03/21/python-performance-tips-part-2/.

Oschina Original translation. For more information, see Source: oschina.