Python Performance Chicken Soup

The first part

Read Zen of Python and enter import this in the Python parser. A sharp Python novice might notice the word "parsing", thinking that Python is just another scripting language. "It must be slow!"

No doubt: Python programs are fast and efficient without a compiled language. Even Python advocates will tell you that Python is not suitable for these areas. However, YouTube has used python to serve 40 million video requests per hour. All you have to do is write efficient code and use the external implementation (c + +) code when needed. Here are some tips to help you become a better Python developer:

1. Use the built-in function:

You can write efficient code in Python, but it's hard to beat the built-in functions. Verified. They are very fast.

2. Use the Join () connection string.

You can use "+" to concatenate strings. But because string is immutable in Python, each "+" operation creates a new string and copies the old content. A common usage is to use the Python array module to modify the characters individually, and when finished, use the join () function to create the final string.
>>> #This is good to glue a large number of strings
>>> for chunk in input ():
>>> My_string.join (Chunk)

3. Use Python multiple assignments to swap variables
This is elegant and fast in Python:
>>> x, y = y, X
This is slow:
>>> temp = x
>>> x = y
>>> y = Temp

4. Use local variables as much as possible
Python retrieves local variables faster than retrieving global variables. This means avoiding the "global" keyword.

5. Use "in" as much as possible
Use the "in" keyword. Simple and fast.
>>> for key in sequence:
>>> print "Found"

6. Using Lazy load acceleration
Move the "import" declaration into the function and import it only when needed. In other words, if some modules do not need to be used immediately, import them later. For example, you do not have to import a large number of modules in a single drive to speed up the program. This technology does not improve overall performance. But it can help you more evenly distribute the load time of the module.

7. Use "while 1" for infinite loops
Sometimes you need an infinite loop in the program. (such as an instance of a listening socket) although "while true" can do the same thing, "while 1" is a single-step operation. This will improve your Python performance.
>>> while 1:
>>> #do stuff, faster with while 1
>>> while True:

>>> # do stuff, slower with Wile True

8. Using the list comprehension
Starting with Python 2.0, you can use the list comprehension instead of a large number of "for" and "while" blocks. Using the list comprehension is usually faster, and the Python parser is optimized to see it as a predictable pattern in the loop. The additional benefit is that list comprehension is more readable (functional programming) and, in most cases, it saves an extra count variable. For example, let's calculate an even number of numbers between 1 and 10:
>>> # The good to iterate a range
>>> evens = [i-I in range] if i%2 = = 0]
>>> [0, 2, 4, 6, 8]
>>> # The following is not so Pythonic
>>> i = 0
>>> evens = []
>>> While I < 10:
>>> if I%2 = = 0:evens.append (i)
>>> i + = 1
>>> [0, 2, 4, 6, 8]

9. Use Xrange () to process long sequences:
This can save you a lot of system memory because Xrange () produces only one integer element per call in the sequence. Instead of the range (), it will give you a complete list of elements that can be used for loops with unnecessary overhead.

10. Use Python Generator:
This can also save memory and improve performance. For example a video stream, you can send a byte block instead of the entire stream. For example
>>> Chunk = (+ * I for I in xrange (1000))
>>> Chunk
At 0x7f65d90dcaa0>
>>> Chunk.next ()
0
>>> Chunk.next ()
1000
>>> Chunk.next ()
2000

11. Learn about the Itertools module:
This module is very effective for iterations and combinations. Let's generate a list of all permutations of [all-in-one], with just three lines of Python code:
>>> Import Itertools
>>> iter = itertools.permutations ([+])
>>> List (ITER)
[(1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1)]

12. Learn the Bisect module to keep the list sorted:
This is a free binary search tool that implements and quickly inserts an ordered sequence. In other words, you can use:
>>> Import bisect
>>> bisect.insort (list, Element)
You have inserted an element into the list, and you do not need to call sort () again to keep the container sorted, because this can be very expensive in a long sequence.

13. Understand the Python list, which is actually an array:
The list implementations in Python are not implemented in a common single-linked list of computer science that people usually talk about. The list in Python is an array. That is, you can retrieve an element of the list in constant time O (1) without having to search from scratch. What's the point? Python developers need to think twice when using the list object Insert (). For example:>>> List.insert (0,item)
Inserting an element in front of the list is inefficient because all subsequent subscripts in the list have to be changed. However, you can use List.append () to effectively add elements at the end of the list. Pick First deque If you want to quickly insert or when in two. It is fast because the deque in Python is implemented with a doubly linked list. Say no more. :)

14. Test members with Dict and set:

Checks whether an element is present in the Dicitonary or set, which is very fast in python. This is because Dict and set are implemented using a hash table. The search efficiency can reach O (1). Therefore, if you need to check members frequently, use set or dict as your container.
>>> mylist = [' A ', ' B ', ' C '] #Slower, check membership with list:
>>> ' C ' in MyList
>>> True
>>> MySet = set ([' A ', ' B ', ' C ']) # Faster, check membership with set:
>>> ' C ' in MySet:
>>> True

15. Use Schwartzian Transform's sort ():

The native List.sort () function is very fast. Python sorts the list in natural order. Sometimes, you need to sort in unnatural order. For example, you want to sort the IP address according to the server location. Python supports custom comparisons, you can use List.sort (CMP ()), which is slower than list.sort (), because it increases the overhead of function calls. If there is a problem with performance, you can apply Guttman-rosler Transform, based on Schwartzian Transform. It is only interested in the actual algorithm to be used, and it briefly works, you can transform the list and call Python built-in List.sort ()-> faster, without using List.sort (CMP ())-Slow.


The Python adorner caches the results:
The "@" symbol is the decorative syntax for Python. It is not only used for tracing, locking or logging. You can decorate a Python function and remember to invoke the result for later use. This technique is called memoization. Here is an example:

>>> from Functools Import wraps
>>> def memo (f):
>>> cache = {}
>>> @wraps (f)
>>> def Wrap (*arg):
>>> if arg not in cache:cache[' arg '] = f (*arg)
>>> return cache[' arg ']
>>> return Wrap
We can also use adorners with the Fibonacci function:
>>> @memo
>>> def fib (i):
>>> If I < 2:return 1
>>> return fib (i-1) + fib (i-2)

The key idea here is to enhance the function (decoration) function, remembering each Fibonacci value that has been computed, and if they are in the cache, no more calculations are needed.

17. Understanding Python's Gil (Global Interpreter Lock):
Gil is necessary because CPython's memory management is non-thread safe. You can't simply create multiple threads and expect Python to run faster on multi-core machines. This is because the Gil will prevent multiple native threads from executing python bytecode at the same time. In other words, the Gil will serialize all your threads. However, you can use threads to manage multiple derived process accelerators, which run independently of your Python code.

18. Familiar with Python source code as familiar with the document:
Some of the Python modules are implemented for performance using C. When performance is critical and official documentation is insufficient, you can freely explore the source code. You can find the underlying data structures and algorithms. Python's source library is a great place to be: http://svn.python.org/view/python/trunk/Modules

Conclusion:
These are no substitute for brain thinking. Opening the Hood it's a developer's job to get a good idea, so they don't quickly piece together a garbage design. The Python recommendations in this article can help you get good performance. If the speed is not fast enough, Python will need an external force: analyze and run outside code. We'll cover this in part two of this article.

Part II

Useful reminders, statically compiled code is still important. For just a few examples, Chrome,firefox,mysql,ms Office and Photoshop are highly optimized software that we use every day. Python is obviously not suitable as a parsing language.  It is not possible to rely on python alone to meet those areas where performance is the primary indication. That's why Python supports your exposure to the underlying bare metal infrastructure, and the more onerous work agent for faster languages like C. This is a key feature in high-performance computing and embedded programming. Python Performance Chicken Soup The first part discusses how to use Python efficiently. In the second part, we will involve monitoring and extending python.

1. First, refuse to tune the temptation

Tuning adds complexity to your code. Before you integrate other languages, check the list below. If your algorithm is "good enough", optimization is less urgent.
1. Did you make a performance test report?
2. Can you reduce the I/O access to the hard drive?
3. Can you reduce network I/o access?
4. Can you upgrade your hardware?
5. Are you compiling libraries for other developers?
6. Is your third-party library software The latest version?

2. Use tools to monitor code, not intuition
The problem of speed can be subtle, so don't rely on intuition. Thanks to the "Cprofiles" module, you can monitor Python code by simply running it
"Python-m cProfile myprogram.py"

We've written a test program. Based on black box monitoring. The bottleneck here is the "very_slow ()" function call. We can also see that both "fast ()" and "slow ()" are called 200 times. This means that if we can improve the "fast ()" and "Slow ()" functions, we can get a full performance boost. The Cprofiles module can also be imported at run time. This is useful for checking long-running processes.

3. Review of Time complexity
After control, provides a basic algorithm for performance analysis. The constant time is the ideal value. The logarithmic time complexity is stable. Factorial complexity is difficult to scale.

O (n^k)-O (k^n)-O (n!), O (n^2), O (n^3) o (n lg N) (1)

4. Use of third-party packages

There are many high-performance third-party libraries and tools designed for Python. Here is a short list of some useful accelerator packs.

1. NumPy: An open source equivalent of MATLAB package
2. SciPy: Another numerical processing library
3. Gpulib: Use GPUs acceleration Code
4. PyPy: Using the Just-in-time compiler to optimize Python code
5. Cython: Turn python code into C

6. Shedskin: Turn python code into C + +

5. Use the multiprocessing module for real concurrency
Because the Gil serializes threads, multithreading in Python cannot be accelerated in multi-core machines and clusters. So Python provides a multiprocessing module that can derive additional processes instead of threads, jumping out of the Gil limit. In addition, you can combine the recommendation in the external C code to make the program faster.

Note that the cost of the process is usually more expensive than the thread because the thread automatically shares the memory address space and the file descriptor. This means that creating a process can be more expensive or more memory than creating a thread. This is something you should keep in mind when you calculate a multiprocessor use.

6. Local Code

Well, now you decide to use native code for performance. In the standard cTYPES module, you can directly load a programmed binary library (. dll or. so file) into Python without having to worry about writing C + + code or building dependencies. For example, we can write a program to load libc to generate random numbers.

However, the overhead of binding ctypes is non-lightweight. You can think of ctypes as a glue operation library function or a hardware-driven adhesive. There are several SWIG, Cython, and boost such Python direct-implanted libraries that are less expensive to call than cTYPES. Python supports object-oriented features such as classes and inheritance. As we see in the example, we can keep the regular C + + code and import it later. The main task here is to write a wrapper (line 10~18).

Summarize:

I hope these python suggestions will make you a better developer. Finally, I need to point out that the pursuit of performance limits is an interesting game, and that over-optimization becomes a mockery. While Python grants you the ability to seamlessly integrate with the C interface, you must ask yourself if you spend hours of hard work optimizing whether the user buys the account. On the other hand, sacrificing the maintainability of the code is worth a few milliseconds of promotion. Members of the team often thank you for writing concise code. As close to Python as possible, because life is short. :)

English Original: http://blog.monitis.com/index.php/2012/02/13/python-performance-tips-part-1/

English Original: http://blog.monitis.com/index.php/2012/03/21/python-performance-tips-part-2/

Oschina Original Translation

Python Performance Chicken Soup