Profile Analysis of python program and pythonprofile Analysis

Profile Analysis of python program and pythonprofile Analysis

Operating System: CentOS7.3.1611 _ x64

Python version: 2.7.5

Problem description

1. The program developed by Python is very slow to use. I want to determine which code segment is relatively slow;

2. The program developed by Python occupies a large amount of memory during use and wants to determine which code segment is causing it;

Solution

Use profile to analyze cpu usage

Profile Introduction: https://docs.python.org/2/library/profile.html

You can use profile and cProfile to analyze the python program. Here we mainly record the use of cProfile. For details about profile, refer to cProfile.

Assume that the following code needs to be analyzed (cProfileTest1.py ):

#! /usr/bin/env python#-*- coding:utf-8 -*-def foo():    sum = 0    for i in range(100):        sum += i    return sumif __name__ == "__main__" :    foo()

You can use the following two methods for analysis:

1. Do not modify the program

Analysis Program:

python -m cProfile -o test1.out cProfileTest1.py

View the running result:

python -c "import pstats; p=pstats.Stats('test1.out'); p.print_stats()"

View the running result after sorting:

python -c "import pstats; p=pstats.Stats('test1.out'); p.sort_stats('time').print_stats()"

2. modify the program

Add the following code:

import cProfilecProfile.run("foo()")

Complete code: https://github.com/mike-zhang/pyExamples/blob/master/profileOpt/cpuProfile1/cProfileTest2.py

The running effect is as follows:

Ordered by: standard namencalls  tottime  percall  cumtime  percall filename:lineno(function)     1    0.000    0.000    0.000    0.000 <string>:1(<module>)     1    0.000    0.000    0.000    0.000 cProfileTest2.py:4(foo)     1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}     1    0.000    0.000    0.000    0.000 {range}

Result description:

Ncils: Number of function calls tottime: total function running time, except for the function running time called in the function percall: average time of the function running once, equal to tottime/ncallscumtime: total function running time, including the called function running time percall: the average time of a function running time, which is equal to the name of the file where the cumtime/ncallsfilename: lineno (function) function is located, the row number of the function, and the function name

Use memory_profiler to Analyze memory usage

Https://pypi.python.org/pypi/memory_profiler

Install memory_profiler:

pip install psutilpip install memory_profiler

Assume that the following code needs to be analyzed:

def my_func():    a = [1] * (10*6)    b = [2] * (10*7)    del b    return a

To use memory_profiler, You need to modify the code. Here we record the following two usage methods:

1. Do not import modules

@profiledef my_func():    a = [1] * (10*6)    b = [2] * (10*7)    del b    return a

Complete code: https://github.com/mike-zhang/pyExamples/blob/master/profileOpt/memoryProfile1/test1.py

Profile Analysis:

python -m memory_profiler test1.py

2. Use the import module

from memory_profiler import profile@profiledef my_func():    a = [1] * (10*6)    b = [2] * (10*7)    del b    return a

The complete code is as follows:

Directly run the program for analysis.

The running effect is as follows:

(py27env) [mike@local test]$ python test1.pyFilename: test1.pyLine #    Mem usage    Increment   Line Contents================================================     6     29.5 MiB      0.0 MiB   @profile     7                             def my_func():     8     29.5 MiB      0.0 MiB       a = [1] * (10*6)     9     29.5 MiB      0.0 MiB       b = [2] * (10*7)    10     29.5 MiB      0.0 MiB       del b    11     29.5 MiB      0.0 MiB       return a

Profile Analysis Complete Code address: https://github.com/mike-zhang/pyExamples/tree/master/profileOpt

Okay, that's all. I hope it will help you.

Github address:

Bytes

Please add