10 methods for detecting Python program running time, CPU usage, and memory usage

This article mainly introduces 10 methods for detecting Python program running time, CPU usage, and memory usage, including using the Python decorator or external UnixShell commands, for more information about how to run complex Python programs, it may take a long time. in this case, you may want to improve the program execution efficiency. But what should we do?

First, you must have a tool that can detect the bottlenecks in the code. for example, you must find a part of the tool that has been executed for a long time. Next, we will optimize this part.

At the same time, you also need to control the memory and CPU usage, so that you can optimize the code on the other hand.

Therefore, in this article, I will introduce seven different Python tools to check the execution time of functions in the code and the usage of memory and CPU.
1. use the decorator to measure the function execution time

A simple method is to define a decorator to measure the function execution time and output the result:

import timefrom functools import wraps def fn_timer(function):  @wraps(function)  def function_timer(*args, **kwargs):    t0 = time.time()    result = function(*args, **kwargs)    t1 = time.time()    print ("Total time running %s: %s seconds" %        (function.func_name, str(t1-t0))        )    return result  return function_timer

Next, add the decorator to the function to be measured, as shown below:

@fn_timerdef myfunction(...):...

For example, the time required to sort an array containing 2 million random numbers by function:

@fn_timerdef random_sort(n):  return sorted([random.random() for i in range(n)]) if __name__ == "__main__":  random_sort(2000000)

The following result is displayed when the script is executed:

Total time running random_sort: 1.41124916077 seconds

2. use the timeit module

Another method is to use the timeit module to calculate the average time consumption.

Run the following script to run this module.

python -m timeit -n 4 -r 5 -s "import timing_functions" "timing_functions.random_sort(2000000)"

Here timing_functions is the name of the Python script file.

At the end of the output, you can see the following results:

4 loops, best of 5: 2.08 sec per loop

This indicates that four tests were performed, and an average of five tests were repeated each time. The best test result is 2.08 seconds.

If no number of tests or repeated times is specified, the default value is 10 tests, and 5 times are repeated each time.
3. use the time command in Unix

However, decorator and timeit are both Python-based. When testing Python in an external environment, the unix time utility is very useful.

Run time utility:

$ time -p python timing_functions.py

Output result:

Total time running random_sort: 1.3931210041 secondsreal 1.49user 1.40sys 0.08

The first line comes from the predefined Decorator. The other three actions:

• Real indicates the total time of script execution.
• User indicates the CPU time consumed by executing the script.
• Sys indicates the time consumed to execute kernel functions.

Note: according to Wikipedia, the kernel is a computer program used to manage the input and output of software and translate it into data processing commands that can be executed by CPU and other computer electronic devices.

Therefore, the difference between Real execution time and User + Sys execution time is the time consumed when input/output and other tasks are executed by the system.
4. use the cProfile module

If you want to know how much time each function and method has consumed and how many times these functions have been called, you can use the cProfile module.

$ python -m cProfile -s cumulative timing_functions.py

Now we can see the detailed description of the function in the code, which contains the number of calls to each function. because the-s option is used (accumulative), the final result is sorted according to the cumulative execution time of each function.

The reader will find that the total time required to execute the script is more than before. This is because it takes time to measure the execution time of each function.
5. use the line_profiler module

The line_profiler module provides the CPU time required to execute each line of code.

First, install the module:

$ pip install line_profiler

Next, you need to specify the function to be checked with @ profile (you do not need to use the import module in the code ):

@profiledef random_sort2(n):  l = [random.random() for i in range(n)]  l.sort()  return l if __name__ == "__main__":  random_sort2(2000000)

It is best to use the following command to obtain a line-by-line description of the random_sort2 function.

$ kernprof -l -v timing_functions.py

-L indicates line-by-line explanation, and-v indicates output detailed results. Using this method, we can see that building an array consumes 44% of the computing time, while the sort () method consumes 56% of the remaining time.

Similarly, because the execution time needs to be checked, the script execution time is longer.
6. use the memory_profiler module

The memory_profiler module is used for memory usage based on the row-by-row measurement code. Using this module slows down code execution.

The installation method is as follows:

 pip install memory_profiler

In addition, we recommend that you install the psutil package so that memory_profile runs faster:

$ pip install psutil

Similar to line_profiler, the @ profile modifier is used to identify the function to be traced. Then, enter:

$ python -m memory_profiler timing_functions.py

The script execution time is 1 or 2 seconds longer than before. If the psutil package is not installed, it may be longer.

The results show that the memory usage is measured in MiB, indicating mebibyte (1 MiB = 1.05 MB ).
7. use the guppy package

Finally, we can use this package to know how many objects are created for each type (str, tuple, dict, etc.) in each stage of code execution.

The installation method is as follows:

$ pip install guppy

Then, add it to the code:

from guppy import hpy def random_sort3(n):  hp = hpy()  print "Heap at the beginning of the functionn", hp.heap()  l = [random.random() for i in range(n)]  l.sort()  print "Heap at the end of the functionn", hp.heap()  return l if __name__ == "__main__":  random_sort3(2000000)

Run the code:

$ python timing_functions.py

The output result is as follows:

By placing heap () in different locations in the code, you can understand the process of creating and deleting objects in the script.

If you want to learn