Debugging a multi-process program using GDB

GDB is a commonly used debugging tool on Linux/C + +, which is very powerful. For more complex systems, such as multi-process systems, how to use GDB debugging? Consider the following three-process system:

Process

PROC2 is the child process of Proc1, PROC3 is also the child process of PROC2. How do I debug proc2 or proc3 with GDB?

In fact, GDB does not provide direct support for debugging multi-process programs. For example, using GDB to debug a process, if the process fork a child process, GDB will continue to debug the process, and the child process will run undisturbed. If you set a breakpoint in the child process code beforehand, the child process receives the SIGTRAP signal and terminates. So how do you debug a child process? In fact, we can use the features of GDB or some other auxiliary means to achieve the purpose. In addition, GDB also joins some multi-process debugging support on newer kernels.

Next we introduce several methods in detail, namely Follow-fork-mode method, attach Subprocess method and GDB wrapper method.

Follow-fork-mode

In the 2.5.60 Linux kernel and later, GDB provides the Follow-fork-mode option to support multi-process debugging for programs that use Fork/vfork to create child processes.

The usage of Follow-fork-mode is:

set follow-fork-mode [parent|child]

• Parent:fork continues to debug the parent process after the child process is not affected.
• Child:fork after the child process is debugged, the parent process is unaffected.

So if you need to debug a child process, after you start gdb:

1	(gdb) set follow-fork-mode child

and set breakpoints in the child Process code.

There is also the Detach-on-fork parameter, which instructs GDB whether to disconnect (detach) The debug of a process after fork, or to be controlled by GDB:

set detach-on-fork [on|off]

• On: Disconnects the specified process from debug Follow-fork-mode.
• Off:gdb will control the parent and child processes. Follow-fork-mode The specified process is debugged and the other process is placed in a paused (suspended) state.

Note that it is best to use GDB 6.6 or later, if you are using GDB6.4, there is only Follow-fork-mode mode.

The use of Follow-fork-mode/detach-on-fork is still relatively simple, but due to the/GDB version limitation of the system kernel, we can only use it on a system that meets the requirements. Moreover, since Follow-fork-mode's debugging must start from the parent process, it is not convenient to debug a system that has multiple fork so that the grandchild process or great-grandchild process, such as the 3 process system, is debugged.

Attach Child process

As is known to all, GDB has the ability to attach (attach) to a running process, namely attach <pid> commands. So we can use this command to attach the child process and then debug it.

For example, to debug a process rim_oracle_agent.9i, we first get the PID of the process

123	[[email protected] tianq]# ps -ef|grep RIM_Oracle_Agent.9inobody    6722  6721  0 05:57 ?        00:00:00 RIM_Oracle_Agent.9iroot      7541 27816  0 06:10 pts/3    00:00:00 grep -i rim_oracle_agent.9i

As can be seen through Pstree, this is a three-process system, Oserv is the parent process of Rim_oracle_prog, Rim_oracle_prog is also the parent process of rim_oracle_agent.9i.

1	[[email protected] root]# pstree -H 6722

Through the Pstree view process

Start Gdb,attach to the process

Connecting processes with GDB

Now it's time to debug. A new problem is that the sub-process has been running, attach to go up and do not know where to run. Is there any way to solve it?

One way is to debug the child process in the initial code, such as the beginning of the main function, add a special code, so that the child process when a condition is set up to loop the sleep wait, attach to the process after the code snippet after the breakpoint, and then set the conditions to cancel, so that the code can continue to execute.

As for the criteria used in this code, look at your preferences. For example, we can check the value of a specified environment variable, or check that a particular file does not exist. Take the file as an example, in the form of the following:

12345678910

void debug_wait(char *tag_file){    while(1)    {        if (tag_file存在)            睡眠一段时间;        else            break;    }}

After the attach to the process, set a breakpoint after the code, and then delete the file is OK. Of course you can also use other conditions or forms, as long as this condition can be set/detected.

The attach process approach is also convenient, and it can handle a variety of complex process systems, such as the grandson/grandchild process, such as the daemon (daemon process), and the only thing needed is to add a small piece of code.

GDB Wrapper

Many times, the parent process fork out the child process, and the child process immediately calls the Exec family function to execute the new code. For this scenario, we can also use the GdB wrapper method. It has the advantage of not having to add extra code.

The basic principle is to execute the EXEC function with GDB calling execution code as a new whole, so that the pending code is always in the control of GDB, so that we can naturally debug the child Process code.

Or the above example, the Rim_oracle_prog fork will execute the rim_oracle_agent.9i binary code file immediately after the child process. We renamed the file to Rim_oracle_agent.9i.binary and created a new shell script file named rim_oracle_agent.9i with the following contents:

1234	[[email protected] bin]# mv RIM_Oracle_Agent.9i RIM_Oracle_Agent.9i.binary[[email protected] bin]# cat RIM_Oracle_Agent.9i#!/bin/shgdb RIM_Oracle_Agent.binary

When a sub-process of fork executes a file named Rim_oracle_agent.9i, GDB is started first so that the code to be debugged is under GDB control.

The new question has come. The sub-process is under GDB's control, but still cannot be debugged: how do I interact with GDB? We have to start gdb in some way so that we can interact with GDB in a window/terminal. Specifically, you can use xterm to generate this window.

Xterm is an analog terminal program under the X Window System. For example, we typed the xterm command in the Linux desktop environment gnome:

Xterm

You will jump out of a terminal window:

Terminal

If you are debugging on a remote Linux server, you can use the VNC (Virtual Network Computing) Viewer to connect to the server using xterm from your local machine. Prior to this, you need to install the VNC viewer on your local machine, install and start VNC server on the servers. Most Linux distributions have vnc-server packages preinstalled, so we can run the Vncserver command directly. Note that the first time you run Vncserver, you will be prompted for a password to use as the VNC Viewer password when connecting from the client. The password can be modified using the VNCPASSWD command on the VNC server machine.

123456789101112131415

[[email protected] root]# vncserver New ‘tivf09:1 (root)‘ desktop is tivf09:1Starting applications specified in /root/.vnc/xstartupLog file is /root/.vnc/tivf09:1.log[[email protected] root]#[[email protected] root]# ps -ef|grep -i vncroot     19609     1  0 Jun05 ?        00:08:46 Xvnc :1 -desktop tivf09:1 (root)   -httpd /usr/share/vnc/classes -auth /root/.Xauthority -geometry 1024x768   -depth 16 -rfbwait 30000 -rfbauth /root/.vnc/passwd -rfbport 5901 -pnroot     19627     1  0 Jun05 ?        00:00:00 vncconfig -iconicroot     12714 10599  0 01:23 pts/0    00:00:00 grep -i vnc[[email protected] root]#

Vncserver is a Perl script used to start Xvnc (X VNC server). X client applications, such as the Xterm,vnc Viewer, are all communicating with it. As shown above, the display value we can use is tivf09:1. You can now connect to the past using the VNC Viewer from your local machine:

VNC Viewer: Input server

Enter Password:

VNC Viewer: Enter your password

Log on successfully, the interface is the same as on the server's local desktop:

VNC Viewer

Let's modify the rim_oracle_agent.9i script so that it looks like this:

12	#!/bin/shexport DISPLAY=tivf09:1.0; xterm -e gdb RIM_Oracle_Agent.binary

If your program has passed the parameters in exec, you can change it to:

12	#!/bin/shexport DISPLAY=tivf09:1.0; xterm -e gdb --args RIM_Oracle_Agent.binary [email protected]

Finally add Execute permission

1	[[email protected] bin]# chmod 755 RIM_Oracle_Agent.9i

Now it's time to debug. Run the program that started the subprocess:

123456789101112

[[email  Protected] root]# wrimtest-l 9i_linux resource type : RIM resource label:9i_linux host name      : tivf09 user name     : Mdstatus vendor        : Oracle Code class= "Htmlscript Plain" >database      : Rim database home :/data/oracle9i/920 server id     : Rim instance home : instance name : opening Regular Session ...

The program stopped. As you can see from the VNC Viewer, a new GDB xterm window opens on the server side

GDB xterm window

1234	[[email protected] root]# ps -ef|grep gdbnobody   24312 24311  0 04:30 ?        00:00:00 xterm -e gdb RIM_Oracle_Agent.binarynobody   24314 24312  0 04:30 pts/2    00:00:00 gdb RIM_Oracle_Agent.binaryroot     24326 10599  0 04:30 pts/0    00:00:00 grep gdb

Run the program that you want to debug. Set breakpoints and start debugging!

Note that the following error is generally a permission issue, using the Xhost command to modify permissions:

Xterm Error

123	[[email protected] bin]# export DISPLAY=tivf09:1.0[[email protected] bin]# xhost +access control disabled, clients can connect from any host

Xhost + disables access control and can be connected from any machine. For security reasons, you can also use Xhost + < your machine name >.

Summary

The above three methods have their own characteristics and advantages and disadvantages, so they adapt to different occasions and environments:

• Follow-fork-mode method: Easy to use, limited to the system kernel and GDB version, suitable for simpler multi-process systems
• Attach Subprocess method: Flexible and powerful, but need to add additional code, suitable for a variety of complex situations, especially daemons
• GDB Wrapper method: Dedicated to fork+exec mode, without adding extra code, but requires X environment support (XTERM/VNC).

Related Topics

• GDB Official reference: http://sourceware.org/gdb/documentation/
• More VNC information: http://www.realvnc.com/

Debugging a multi-process program using GDB