Common python Debugging tools, Python development must-read

Log

Yes, that's the log. It is not too important to emphasize the importance of keeping enough logs in your application. You should make a log of important content. If your log is good enough, just look at the log and you will find the problem. That will save you a lot of time.

If you've been messing with the print statement in your code all along, stop right now. Swap with Logging.debug. Later you can continue to reuse, or all to deactivate and so on.

Tracking

Sometimes it's better to look at what statements are executed. You can use some IDE's debugger for stepping, but you need to know exactly what you are looking for, otherwise the whole process will be very slow.

The trace module inside the standard library allows you to print all the statements that are executed in the module that is contained in the runtime. (Just like making a project report)

Python-mtrace–trace script.py

This produces a lot of output (every line that executes will be printed, and you may want to use grep to filter the modules you are interested in).

Like what:

Python-mtrace–trace script.py | Egrep ' ^ (mod1.py|mod2.py) '

Debugger

Here is a basic introduction that should now be known to everyone:

Import Pdbpdb.set_trace () # Turn on PDB hints

Or

Try: (a code that throws an exception) except:    import pdb    pdb.pm () # or Pdb.post_mortem () or (enter C to start executing the script)

　　

Python-mpdb script.py

In the input-calculation-Output loop (note: repl,read-eval-print-loop abbreviation) environment, you can do the following:

C or Continue

Q or quit

L or list, showing the source of the current step frame

W or where, backtracking call procedure

D or down, step back frame (Note: equivalent rollback)

U or up, front further frame

(enter), repeat the previous instruction

Almost all of the remaining instructions (with the exception of a few other commands) are parsed as Python code on the current step frame.

If you find it challenging enough, try smiley,-it can show you the variables and you can use it to track the program remotely.

A better debugger

Direct substitution of PDB:

IPDB (Easy_install ipdb) – Similar to Ipython (with auto-complete, display color, etc.)

Pudb (Easy_install pudb) – based on curses (similar GUI interface), ideal for browsing source code

Remote Debugger

Installation method:

sudo apt-get install winpdb

Replace the previous pdb.set_trace () in the following way:

Import Rpdb2rpdb2.start_embedded_debugger ("Secretpassword")

Now run winpdb, file-Association

Don't like winpdb? You can also directly wrap the PDB on top of TCP to run!

Do this:

Import Logggingclass Rdb (PDB. PDB): "" This would run PDB as a ephemeral Telnet service. Once you connect no one else can connect.      On construction This object would block execution till a client has connected.      Based on https://github.com/tamentis/rpdb I think ...        To use the This:: Rdb (4444). Set_trace () then run:telnet 127.0.0.1 4444 "" Def __init__ (Self, port=0): Self.old_stdout = sys.stdout Self.old_stdin = Sys.stdin Self.listen_socket = Socket.socket (Socket.AF_IN ET, Socket.  Sock_stream) Self.listen_socket.bind ((' 0.0.0.0 ', port)) if not port:logging.critical ("PDB remote Session Open on:%s ", Self.listen_socket.getsockname ()) Print >> sys.__stderr__," PDB Remote session Ope N on: ", Self.listen_socket.getsockname () Sys.stderr.flush () Self.listen_socket.listen (1) Self.con Nected_socket, address = self.listen_socket.accept () Self.handle = Self.connected_socket. Makefile (' RW ') pdb.pdb.__init__ (self, completekey= ' tab ', Stdin=self.handle, Stdout=self.handle) sys.stdout = Sys.stdin = Self.handle def do_continue (self, arg): Sys.stdout = self.old_stdout Sys.stdin = Self.old _stdin self.handle.close () self.connected_socket.close () self.listen_socket.close () Self.set_c Ontinue () return 1 Do_c = Do_cont = Do_continue def set_trace (): "" "Opens a remote PDB on first Availa    BLE port. "" "RDB = Rdb () rdb.set_trace ()

Just want a REPL environment? How about trying Ipython?

If you don't need a full-fledged debugger, you just need to start a ipython in a single way:

Import ipythonipython.embed ()

Standard Linux Tools

I am often amazed that they are far from being fully exploited. You can use these tools to solve a wide range of problems: from performance issues (too many system calls, memory allocations, etc.) to deadlocks, network problems, disk problems, and so on.

The most useful of these is the most direct strace, only need to run the sudo strace-p 12345 or strace-f instruction (-f that is the same time to track the fork out of the sub-process), this is OK. The output is usually very large, so you might want to redirect it to a file for more analysis (just add the &> filename).

And then Ltrace, a bit like strace, the difference is that it outputs a library function call. Parameters are roughly the same.

There is also the lsof used to indicate the meaning of the handle values you see in the Ltrace/strace. Like what:

Lsof-p 12345

Better tracking

Easy to use and can do a lot of things-everyone should put on htop!

sudo apt-get install Htop

sudo htop

Now find the process you want, and then enter:

S-Represents the system call procedure (similar to strace)

L-Represents the library call procedure (similar to ltrace)

L-Representative Lsof

Monitoring

There is no good continuous server monitoring, but if you have encountered some very strange situation, such as why everything is running so slow, those system resources to do, ... Wait for these questions, want to understand but there is no place to start the occasion, do not need to use iotop,iftop,htop,iostat,vmstat these tools, use Dstat Bar! It can do most of the work we've mentioned before, and maybe we can do better!

It will continue to show you data in a compact, code-highlighting way (unlike Iostat,vmstat), and you can often see past data (unlike Iftop,iostop,htop).

Just run:

Dstat--cpu--io--mem--net--load--fs--vm--disk-util--disk-tps--freespace--swap--top-io--top-bio-adv

There is probably a shorter way to write the above command,

This is a very complex and powerful tool, but here I only mention some basic content (installation and basic commands)

sudo apt-get install gdb python-dbg

zcat/usr/share/doc/python2.7/gdbinit.gz > ~/.gdbinit

To run the program with python2.7-dbg:

sudo gdb-p 12345

Now use:

BT-Stack trace (C level)

Pystack-python stack Trace, unfortunately you need to have ~/.gdbinit and use python-dbg

C-Continue

A segment error occurred? With Faulthandler!

A great addition to the Python 3.3 version that can be ported back to the python2.x version. Just run the following statement, and you'll probably know what's causing the segment error.

Import Faulthandler

Faulthandler.enable ()

Memory leaks

Well, in this case there are a lot of tools to use, some of which are specifically for WSGI programs such as Dozer, but my favorite of course is objgraph. Easy to use, amazing!

It has no integration wsgi or other, so you need to find a way to run the code yourself, like this:

Import Objgraph

OBJS = Objgraph.by_type ("Request") [: 15]

Objgraph.show_backrefs (Objs, max_depth=20, Highlight=lambda v:v in Objs,

Filename= "/tmp/graph.png")

Graph written To/tmp/objgraph-zbdm4z.dot (107 nodes)

Image generated As/tmp/graph.png

You will get a picture like this (note: It is very large). You can also get a single point output.

Memory usage

Sometimes you want to use less memory. Less memory allocation can often make programs execute faster, better, and users want memory to fit

There are many tools available, but in my opinion it is best to use Pytracemalloc. Compared to other tools, it costs very little (no need to rely on the sys.settrace of severe impact speed) and the output is very verbose. But it's painful to install, you need to recompile python, but with APT, it's easy to do.

Just run these commands and go to lunch or something else:

Apt-get Source python2.7

CD python2.7-*

Wget? Https://github.com/wyplay/pytracemalloc/raw/master/python2.7_track_free_list.patch

Patch-p1 < Python2.7_track_free_list.patch

Debuild-us-uc

Cd..

sudo dpkg-i python2.7-minimal_2.7*.deb python2.7-dev_*.deb

Then install Pytracemalloc (Note that if you are operating in a virtualenv virtual environment, you need to re-rebuild the python once again – just run Virtualenv myenv)

Pip Install Pytracemalloc

Now wrap your app in code like this

Import Tracemalloc, timetracemalloc.enable () top = Tracemalloc. Displaytop (    *, # Log the top locations    file=open ('/tmp/memory-profile-%s '% time.time (), "w")) Top.show_ Lineno = truetry:    # code, needs to be tracedfinally:    top.display ()

The output will look like this:

2013-05-31 18:05:07:top allocations per file and line

#1:/site-packages/billiard/_connection.py:198:size=1288 KiB, Count=70 (+0),

Average=18 KiB

#2:/site-packages/billiard/_connection.py:199:size=1288 KiB, Count=70 (+0),

Average=18 KiB

#3:/python2.7/importlib/__init__.py:37:size=459 KiB, count=5958 (+0),

average=78 B

#4:/site-packages/amqp/transport.py:232:size=217 KiB, count=6960 (+0),

Average=32 B

#5:/site-packages/amqp/transport.py:231:size=206 KiB, count=8798 (+0),

Average=24 B

#6:/site-packages/amqp/serialization.py:210:size=199 KiB, count=822 (+0),

average=248 B

#7:/lib/python2.7/socket.py:224:size=179 KiB, count=5947 (+0), average=30

B

#8:/celery/utils/term.py:89:size=172 KiB, count=1953 (+0), average=90 B

#9:/site-packages/kombu/connection.py:281:size=153 KiB, count=2400 (+0),

Average=65 B

#10:/site-packages/amqp/serialization.py:462:size=147 KiB, count=4704

(+0), average=32 B

...

