Python code checking for line-level code optimization

Sometimes a Python program runs and it takes a long time to run. You may want to improve the efficiency of the program. Then what should I do?

First you need to find out where the program bottlenecks are-for example, which function takes longer to run? Which function consumes more memory, does it need to optimize the use of memory? Which CPU time takes longer?   etc... These need to be considered, Python has a few libraries can help you solve these problems ~ nonsense to say, cut into the topic.

First write a Python script that reads a file:

Touch c9.py#!/usr/bin/env python#date:2015/07/21def Read_file (fpath): block_size=1024 with open (Fpath, ' RB ') as FD: block = Fd.read (block_size) if Block:yield BLOCK else:returndef main (): F Or I in Read_file (' ~/access.log ') print iif __name__ = = "__main__": Main ()

Then, test the code.

First test the code run time:

It is an external python measurement.

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6F/EE/wKiom1Wtq0WjGaA4AAEVu-Nf1Sc199.jpg "title=" 1.png " alt= "Wkiom1wtq0wjgaa4aaevu-nf1sc199.jpg"/>

Real shows the total time spent executing the script.

User indicates the time spent executing the script on the CPU.

SYS indicates the time spent executing the script in kernel functions.

Therefore, the difference between Real time and User+sys may indicate that it is spent waiting for I/O or the system is busy performing other tasks.

Using the Cprofile module

If you want to know the time spent on each function and method, and how many times they are called, you can use the Cprofile module.

$ python-m cprofile-s Cumulative + Python script to execute (-s cumulative it will be sorted by accumulating the time spent on each function)

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/6F/EF/wKiom1Wtr_7SVO0dAAQLcFGOoa8346.jpg "title=" 3.png " alt= "Wkiom1wtr_7svo0daaqlcfgooa8346.jpg"/>

You will see that the total time spent in running your script is higher than before, and this is the loss we take to measure the execution time of each function.

Using the Line_profile module

Line_profile gives you the CPU time spent on your code in the US line.

First you need to install Line_profiler:

Pip Install Line_profiler

Next, you need to define which function you want to evaluate using the adorner @profile (you don't need to import it into your file) 650) this.width=650; "Src=" http://s3.51cto.com/wyfs02/M01/6F/EF/wKiom1WtuhSCJLWTAAIfyDyspWo213.jpg " Title= "4.png" alt= "Wkiom1wtuhscjlwtaaifydyspwo213.jpg"/>

Next Test the code:

$ kernprof-l-V + code to execute

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6F/EF/wKiom1WtunGjlaE-AARCxdynI2o340.jpg "title=" 5.png " alt= "Wkiom1wtungjlae-aarcxdyni2o340.jpg"/>

The-L ID indicates that the row-by-line and-v identities indicate verbose output.

Using the Memory_profile module

The Memory_profile module is used to measure the memory utilization of your code on a progressive basis. Nonetheless, it may make your code run more slowly.

Install Memory_profiler First

$pip Install Memory_profiler

It is also recommended to install the Psutil package, making the Memory_profile module run faster.

$ pip Install Psutil

Similar to Line_profile, use the adorner @profile to mark which function is tracked.

$python-M Memory_profiler + code file to execute

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6F/EC/wKioL1WtvmGiFjWaAAQImIsWaaQ189.jpg "title=" 6.png " alt= "Wkiol1wtvmgifjwaaaqimiswaaq189.jpg"/>

Looking at the output above, note that the memory usage is in the MIB, which represents megabytes (1MiB = 1.05MB).

Using the Guppy module

With the Guppy module you can track how many objects each type has in your code at each stage (characters, tuples, dictionaries, and so on).

Install Guppy:

$ pip Install Guppy

Then change your code to the following:

#!/usr/bin/env pythonfrom guppy import hpydef read_file (Fpath):     hp = hpy ()     print  "heap at the beginning of  the function\n ",  hp.heap ()     BLOCK_SIZE = 1024     with open (fpath,  ' RB ')  as fd:         while true:            block =  Fd.read (block_size)             if block:                 yield  block            else:                 print  "Heap at the  end of the function\n ", &Nbsp;hp.heap ()                  Returndef main ():     hp = hpy ()     print  Heap  at the beginning of the function\n ",  hp.heap ()     for  i in read_file ('/users/david/applications/py/dt '):          #print  i        pass    print  " Heap at the end of the function\n ",  hp.heap () if __name__ ==  "__main__":     main ()

Execute the code:

$ python c9.py

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/6F/EC/wKioL1WtwlWC7D7qAAgVRIuUZSc843.jpg "title=" 8.png " alt= "Wkiol1wtwlwc7d7qaagvriuuzsc843.jpg"/>

Through the data results, we can see that each str, dict, function and other objects are created.

Through the above modules, you can more clearly understand the implementation of Python code and Resource usage. A lot of Help for code optimization ~ ~ ~

This article is from the "David" blog, so be sure to keep this source http://davidbj.blog.51cto.com/4159484/1676671

Python code checking for line-level code optimization