Fix bugs with core dump

A perplexity developers usually meet is they release the product to qa team and get a feedback of occasional crash. And the testers don't have a solid reproduction steps. In this case, it's a time-consuming task to find out the cause and fix it. What we need is an efficient postmortem debugging method.

Core dump is a mechanism provided by operating system to automatically capture the address apace of a crashed process into a dump file that can be used to help us debugging.

How to enable it
In most linux distributions, core dump is disabled by default. This can be validated by running "ulimit -a" command from shell. We are very likely to see the results below:
core file size (blocks, -c) 0

It indicates core dump is diabled. To enble it, the simpliest way is invoking "ulimit -c unlimited" command. But it only takes effect for current session.
If we want to enable it permanently for a user, we can add the command to the ~/.bashrc file.
If we want to enable it globally, we can edit the /etc/security/limits.conf file by setting
* soft core unlimited

Customize core dump name
By default, the core dump file will be created in the same directory that the application is running in with fixed file name "core". The name conforms to the definition in /proc/sys/kernal/core_pattern file.
This pattern can be changed via "sysctl -w kernel.core_pattern=DesiredValue" command. Again, this setting is't persisted. To make it permanent, we can edit the /etc/sysctl.conf file and add the line "kernel.core_pattern=DesiredValue" to it.
The list below is some place holder can be used in the naming pattern to make it more flexible.

%% A single % character
%p PID of dumped process
%u real UID of dumped process
%g real GID of dumped process
%s number of signal causing dump
%t time of dump (seconds since 0:00h, 1 Jan 1970)
%h hostname (same as ’nodename’ returned by uname(2))
%e executable filename

Example
And here is an exmple.

#Makefile
EXE = test
CC = g++
#CFLAGS += -Os
# add debug information
CFLAGS += -g
CFLAGS += -Wall

all:
$(CC) $(CFLAGS) test.cpp -o $(EXE)

.PHONY: clean

clean:
rm $(EXE) -rf

// source code
#include

using namespace std;

void foo()
{
char *buf = "aa";
cout
buf[0] = 'b';// invalid code
cout
}

int main()
{
foo();
return 0;
}


Every time we run the application, it will crash and generate a core dump file. We can analysis the dump file with gdb (gdb test core_dump_file).

Capture live core dump
It's also desirable to capture core dump of a process when it's still running. Such dump is useful for trouble shooting contention and dead lock issues. gcore is the right tool for this purpose.

Reference
http://linuxfocus.berlios.de/English/July2004/article343.shtml