tutorial on using the optimizer to improve the execution efficiency of a Python program _python

It would be unwise to begin the process of optimization without first thinking about this Knuth famous saying. However, you quickly write code to add some features, it can be ugly, you need to pay attention to. This article is prepared for this time.

Then there are some very useful tools and patterns to quickly optimize python. Its main purpose is simple: find bottlenecks as quickly as possible, fix them, and make sure you fix them.
Write a test

Before you start tuning, write an advanced test to prove that the code is slow. You may need to use some minimal data sets to reproduce it slowly enough. Usually one or two programs that show runtime seconds are enough to handle some of the improvements.

There are some basic tests to ensure that your optimization does not change the behavior of the original code is also necessary. You can also modify the benchmarks of these tests a little while running tests many times to optimize the code.

So now, let's take a look at the optimization tool put.
a simple timer

The timer is simple, and this is one of the most flexible ways to record execution time. You can put it anywhere and the side effects are small. Running your own timer is very simple, and you can customize it so that it works the way you want it to. For example, you have a simple timer as follows:

Import Time
 
def timefunc (f):
 def f_timer (*args, **kwargs):
  start = time.time () result
  = f (*args, * * Kwargs) End
  = Time.time ()
  print f.__name__, ' took ', End-start, ' time ' return to result return
 F_timer
 
def get_number (): For
 x in Xrange (5000000):
  yield x
 
@timefunc
def expensive_function (): For
 x in Get_number ():
  i = x ^ x ^ x return
 ' some result! '
 
# prints "Expensive_function took 0.72583088875 seconds" result
= Expensive_function ()

Of course, you can use context management to make it more powerful, add checkpoints or some other features:

Import time Class Timewith (): Def __init__ (self, name= '): self.name = name Self.start = Time.time () @property  def elapsed (self): return Time.time ()-Self.start def checkpoint (self, name= '): print ' {timer} {checkpoint} took {Elapsed} seconds '. Format (Timer=self.name, Checkpoint=name, elapsed=self.elapsed,). Strip () def __enter__ ( Self): return self def __exit__ (self, type, value, Traceback): Self.checkpoint (' finished ') Pass Def Get_number ( ): For x in Xrange (5000000): Yield x def expensive_function (): For x in Get_number (): i = x ^ x ^ x return ' some
 
Result! ' # prints something like: # Fancy thing do with something took 0.582462072372 seconds # fancy thing do with something E LSE took 1.75355315208 seconds # fancy thing finished took 1.7535982132 seconds with timewith (' fancy thing ') as Timer:ex
 Pensive_function () timer.checkpoint (' Done with something ') expensive_function () expensive_function () Timer.checkpoint (' Done and soMething Else ') # or directly timer = Timewith (' Fancy Thing ') expensive_function () Timer.checkpoint ("Done with something"
 )

The timer also requires you to do some digging. Wrap some of the more advanced functions and determine where the bottleneck is, and then go deep into the function to reproduce it continuously. When you find some inappropriate code, fix it, and then test it again to make sure it's fixed.

Some tips: Don't forget to use the Timeit module! It is more useful for small chunks of code to be benchmarked rather than for actual investigation.

• Timer Benefits: It's easy to understand and implement. It is also very easy to compare after a modification. Applies to many languages.
• Timer drawback: Sometimes it's a bit too simple for very complex code, and you might spend more time placing or moving the reference code instead of fixing the problem!

Built-in Optimizer

Enabling the built-in optimizer is like using a cannon. It's very powerful, but it's a little bit less useful, and it's more complicated to use and explain.

You can learn more about the profile module, but the basics are very simple: You can enable and disable the optimizer, and it can print all function calls and execution times. It gives you the ability to compile and print out output. A simple adorner is as follows:

Import CProfile
 
def do_cprofile (func):
 def profiled_func (*args, **kwargs): Profile
  = Cprofile.profile ()
  Try:
   profile.enable () result
   = Func (*args, **kwargs)
   profile.disable ()
  Finally:
   profile.print_stats () return
 profiled_func
 
def get_number (): For
 x in Xrange (5000000):
  yield x
 
@do_cprofile
def expensive_function (): For
 x in Get_number ():
  i = x ^ x ^ x
 return ' Some result! '
 
# perform profiling result
= Expensive_function ()

In the case of the above code, you should see something printed on the terminal, and the printed content is as follows:

5000003 function calls in 1.626 seconds
 
 Ordered by:standard name
 
 ncalls tottime percall cumtime percall E:lineno (function)
 5000001 0.571 0.000 0.571 0.000 timers.py:92 (get_number)
  1 1.055 1.055 1.626 1.626 timers.py:96 (expensive_function)
  1 0.000 0.000 0.000 0.000 {method ' disable ' of ' _lsprof '. Profiler ' Objects}

As you can see, it gives the number of calls to different functions, but it misses out some key information: which function makes running so slow?

However, this is a good start for basic optimization. Sometimes even less energy is available to find solutions. I often use it to drill down to find out which function is slow or too many calls to debug the program.

• Built-in benefits: No extra dependencies and very fast. is useful for fast, high level checks.
• Built-in disadvantage: The information is relatively limited, the need for further debugging; The report is a bit less straightforward, especially for complex code.

Line Profiler

If the built-in optimizer is a cannon, line Profiler can be regarded as an ion cannon. It's very heavyweight and powerful.

In this case, we'll use a very good line_profiler library. In order to be easy to use, we will again use the adorner packaging, this simple method can also prevent it in the production code.

Try: From
 line_profiler import Lineprofiler
 
 def do_profile (follow=[):
  def Inner (func):
   def profiled _func (*args, **kwargs):
    try:
     profiler = Lineprofiler ()
     profiler.add_function (func) for
     F in follow:
      profiler.add_function (f)
     Profiler.enable_by_count () return
     func (*args, **kwargs)
    finally:
     profiler.print_stats () return
   profiled_func return
  inner
 
except importerror:
 def do_ Profile (follow=[]):
  "Helpful if your accidentally leave in production!"
  def inner (func):
   def nothing (*args, **kwargs): Return
    func (*args, **kwargs)
  return Nothing return inner
 
def get_number (): For
 x in Xrange (5000000):
  yield x
 
@do_profile (follow=[get_number)
def expensive_function (): For
 x in Get_number ():
  i = x ^ x ^ x return
 ' some result! '
 
result = Expensive_function ()

If you run the above code, you can see the report:

Timer unit:1e-06 s
 
File:test.py
function:get_number at line all
time:4.44195 S Line
 
#  Hits
   time per Hit% time line Contents
==============================================================           Def get_number ():
 5000001  2223313  0.4 50.1 for x in  xrange (5000000):
 45 5000000  2218638  0.4  49.9   yield x
 
File:test.py
function:expensive_function
at line Total time:16.828 S Line
 
#  Hits time/Hit% time line   Contents
================================= =============================           def expensive_function ():
 5000001  14090530  2.8  83.7 for  x in Get_number ():
 5000000  2737480  0.5   i = x ^ x ^ x   1   0  0.0  0.0 return  ' some result! '

As you can see, there is a very detailed report that gives you a complete insight into the operation of the code. The Cprofiler, which does not want to be built, calculates the time of the language's core characteristics, such as loops and imports, and gives the time to spend on different lines.

These details make it easier to understand the interior of the function. If you're studying a third-party library, you can simply import it and add an adorner to analyze it.

Some tips: Just decorate your test function and use the problem function as the next argument.

• Line Profiler Advantages: There are very direct and detailed reports. Ability to track functions in third party libraries.
• Line Profiler drawback: because it makes the code a lot slower than it really is, so don't use it for benchmark testing. This is an additional requirement.

Summary and Best practices

You should use a simpler tool to do a basic check on the test cases and drill down into the inside of the function with a slower line_profiler that shows more detail.

In the case of nine, you may find that a circular call in a function or an incorrect data structure consumes 90% of the time. Some of the adjustment tools are perfect for you.

If you still feel that this is too slow, instead use some of your own secret weapons, such as comparing attribute access techniques or adjusting balance checking techniques. You can also use the following methods:

1. Endure slow or cache them

2. Rethinking the whole realization

3. More use of optimized data structures

4. Write a C extension

Note that optimizing code is a kind of sinful pleasure! It's interesting to speed up your Python code in the right way, but be careful not to break the logic. Readable code is more important than running speed. It's better to cache it first and then optimize it.