Python performance optimization
Performance optimization – making a program run faster – is closely related to refactoring. Refactoring makes existing source code more readable and less complex on the inside without changing its behavior on the outside. Refactoring is a costly operation. Typically, an 80/20 rule applies. A 20% effort is needed to implement the bulk of the program. Then another 80% effort (and budget) is needed to optimize, refactor, extend and/or document the program.
Do not worry about performance during development. You first need the program to produce correct results (with correct input) before you know what is making it slow.
Not all parts of the source code will be optimizable. In Python, many operations such as dictionary lookups and regular expressions are as efficient as they can be. Some operations in your code simply need to happen to make the program work. In this case, micro-optimizations (e.g., making a function 1% faster) are a waste of time.
Profiling
Typically, the top 20% slow parts of the program may benefit from optimization. To find the slow parts you need to profile the source code. The profile() function below uses Python's cProfile module to return a string of performance statistics. To use it you can simply copy it int0 a test script:
| def profile (function, *args, **kwargs): """ Returns performance statistics (as a string) for the given function. """ def _run (): function(*args, **kwargs) import cProfile as profile import pstats import os import sys; sys.modules[ '__main__' ].__profile_run__ = _run id = function.__name__ + '()' |