Common Python debugging tools, required for Python development

Common Python debugging tools, which must read logs for Python development

Yes, it is the log. It is not important to keep enough logs in your application. You should log important content. If your logs are good enough, you can find the problem by reading the logs. This saves you a lot of time.

If you have been using print statements in your code for a long time, stop immediately. Use logging. debug. In the future, you can continue to reuse, or stop all of them.

Tracking

Sometimes it is better to see which statements are executed. You can use several IDE debuggers for single-step execution, but you need to be clear about the statements you are looking for. Otherwise, the process will be very slow.

The trace module in the standard library can print all statements executed in the module. (Just like preparing a project report)

python -mtrace –trace script.py

This will generate a large amount of output (each row will be printed, and you may want to use grep to filter those modules you are interested in ).

For example:

python -mtrace –trace script.py | egrep '^(mod1.py|mod2.py)'

Debugger

The following is a well-known basic introduction:

Import pdbpdb. set_trace () # prompt for enabling pdb

Or

Try :( a piece of code that throws an exception) export T: import pdb. pm () # or pdb. post_mortem () or (Enter c to start executing the script)

　　

Python-mpdb script. py

In the input-compute-output loop (note: REPL, short for READ-EVAL-PRINT-LOOP) environment, you can perform the following operations:

C or continue

Q or quit

L or list: displays the source code of the current step frame.

W or where: trace the call process

D or down. move back to the frame (note: rollback is equivalent)

U or up, one frame forward

(Press enter), repeat the previous command

Almost all other commands (except a few other commands) are parsed as python code on the current frame.

If you don't think it is challenging enough, try smiley-it will show you the variables and you can use it to remotely track programs.

Better debugger

Direct replacement of pdb:

Ipdb (easy_install ipdb)-similar to ipython (automatic completion, display color, etc)

Pudb (easy_install pudb)-based on curses (similar to the GUI interface), especially suitable for browsing source code

Remote debugger

Installation method:

Sudo apt-get install winpdb

Replace the previous pdb. set_trace () with the following method ():

import rpdb2rpdb2.start_embedded_debugger("secretpassword")

Run winpdb now, file-join

Do not like Winpdb? You can also directly package PDB to run on TCP!

In this way:

import logggingclass Rdb(pdb.Pdb):    """    This will run pdb as a ephemeral telnet service. Once you connect no one    else can connect. On construction this object will block execution till a    client has connected.      Based on https://github.com/tamentis/rpdb I think ...      To use this::          Rdb(4444).set_trace()      Then run: telnet 127.0.0.1 4444    """    def __init__(self, port=0):        self.old_stdout = sys.stdout        self.old_stdin = sys.stdin        self.listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)        self.listen_socket.bind(('0.0.0.0', port))        if not port:            logging.critical("PDB remote session open on: %s", self.listen_socket.getsockname())            print >> sys.__stderr__, "PDB remote session open on:", self.listen_socket.getsockname()            sys.stderr.flush()        self.listen_socket.listen(1)        self.connected_socket, address = self.listen_socket.accept()        self.handle = self.connected_socket.makefile('rw')        pdb.Pdb.__init__(self, completekey='tab', stdin=self.handle, stdout=self.handle)        sys.stdout = sys.stdin = self.handle      def do_continue(self, arg):        sys.stdout = self.old_stdout        sys.stdin = self.old_stdin        self.handle.close()        self.connected_socket.close()        self.listen_socket.close()        self.set_continue()        return 1      do_c = do_cont = do_continue  def set_trace():    """    Opens a remote PDB on first available port.    """    rdb = Rdb()    rdb.set_trace()

Only one REPL environment is required? How about IPython?

If you do not need a complete debugger, you only need to start IPython in the following way:

import IPythonIPython.embed()

Standard linux tools

I am often surprised that they are far from being fully utilized. You can use these tools to solve a wide range of problems: from performance problems (too many system calls, memory allocation, etc.) to deadlocks, network problems, disk problems, and so on.

The most useful is the most direct strace. you only need to run the sudo strace-p 12345 or strace-f command (-f is the sub-process that tracks fork at the same time ), that's all. The output is usually very large, so you may want to redirect it to a file for more analysis (just add the &> file name ).

The ltrace is similar to strace. The difference is that it outputs library function calls. The parameters are roughly the same.

Also, lsof is used to point out the meaning of the handle value you see in ltrace/strace. For example:

Lsof-p 12345

Better tracking

It's easy to use and can do a lot of things-everyone should install htop!

Sudo apt-get install htop

Sudo htop

Find the processes you want, and then enter:

S-represents the system call process (similar to strace)

L-represents the library call process (similar to ltrace)

L-represents lsof

Monitoring

There is no good continuous server monitoring, but if you have encountered some strange situations, such as why everything is running so slowly, what are the system resources ,... If you want to understand these problems, you don't have to use iotop, iftop, htop, iostat, and vmstat. just use dstat! It can do most of the work we mentioned before, and maybe it can do better!

It uses a compact, code highlighting method (different from iostat and vmstat) to continuously display data to you. you can also often see the past data (different from iftop, iostop, htop ).

Just run:

Dstat -- cpu -- io -- mem -- net -- load -- fs -- vm -- disk-util -- disk-tps -- freespace -- swap -- top-io -- top-bio-adv

There may be a simpler way to write the above command,

This is a very complex and powerful tool, but here I only mention some basic content (Installation and basic commands)

Sudo apt-get install gdb python-dbg

Zcat/usr/share/doc/python2.7/gdbinit.gz> ~ /. Gdbinit

Run the program with python2.7-dbg:

Sudo gdb-p 12345

Use now:

Bt-stack trace (Level C)

Pystack-python stack tracing, unfortunately you need ~ /. Gdbinit and use python-dbg

C-continue

Segment error? Use faulthandler!

Python 3.3 and later versions provide a great feature that can be migrated to python 2.x. You only need to run the following statement to find out the cause of the error.

Import faulthandler

Faulthandler. enable ()

Memory leakage

Well, there are a lot of tools to use in this case, some of which are specifically for WSGI programs such as Dozer, but of course my favorite is objgraph. The ease of use is surprising!

It does not integrate WSGI or others, so you need to discover the method for running the code, as shown below:

Import objgraph

Objs = objgraph. by_type ("Request") [: 15]

Objgraph. show_backrefs (objs, max_depth = 20, highlight = lambda v: v in objs,

Filename = "/tmp/graph.png ")

Graph written to/tmp/objgraph-zbdM4z.dot (107 nodes)

Image generated as/tmp/graph.png

You will get an image like this (note: it is very large ). You can also get a vertex output.

Memory usage

Sometimes you want to use less memory. Less memory allocation can often make the program run faster and better, and the user wants the memory to work properly)

There are many available tools, but in my opinion it is best to use pytracemalloc. Compared with other tools, it has very low overhead (does not need to rely on sys. settrace, which seriously affects the speed) and the output is very detailed. However, it is difficult to install python. you need to re-compile python, but with apt, it is also very easy to do.

You just need to run these commands and then have a lunch or something else:

Apt-get source python2.7

Cd python2.7 -*

Wget? Https://github.com/wyplay/pytracemalloc/raw/master/python2.7_track_free_list.patch

Patch-p1 <python2.7 _ track_free_list.patch

Debuild-us-uc

Cd ..

Sudo dpkg-I python2.7-minimal_2.7 *. deb python2.7-dev _ *. deb

Install pytracemalloc (note that if you operate in a virtualenv virtual environment, you need to re-install python and re-create it-just run virtualenv myenv)

Pip install pytracemalloc

Now package your application in the code like below

import tracemalloc, timetracemalloc.enable()top = tracemalloc.DisplayTop(    5000, # log the top 5000 locations    file=open('/tmp/memory-profile-%s' % time.time(), "w"))top.show_lineno = Truetry:    # code that needs to be tracedfinally:    top.display()

The output will look like this:

18: 05: 07: Top 5000 allocations per file and line

#1:.../site-packages/billiard/_ connection. py: 198: size = 1288 KiB, count = 70 (+ 0 ),

Average = 18 KiB

#2:.../site-packages/billiard/_ connection. py: 199: size = 1288 KiB, count = 70 (+ 0 ),

Average = 18 KiB

#3:.../python2.7/importlib/_ init _. py: 37: size = 459 KiB, count = 5958 (+ 0 ),

Average = 78 B

#4:.../site-packages/amqp/transport. py: 232: size = 217 KiB, count = 6960 (+ 0 ),

Average = 32 B

#5:.../site-packages/amqp/transport. py: 231: size = 206 KiB, count = 8798 (+ 0 ),

Average = 24 B

#6:.../site-packages/amqp/serialization. py: 210: size = 199 KiB, count = 822 (+ 0 ),

Average = 248 B

#7:.../lib/python2.7/socket. py: 224: size = 179 KiB, count = 5947 (+ 0), average = 30

B

#8:.../celery/utils/term. py: 89: size = 172 KiB, count = 1953 (+ 0), average = 90 B

#9:.../site-packages/kombu/connection. py: 281: size = 153 KiB, count = 2400 (+ 0 ),

Average = 65 B

#10:.../site-packages/amqp/serialization. py: 462: size = 147 KiB, count = 4704

(+ 0), average = 32 B

...