TECH: getting a stack trace from a core file on Unix (File ID 1812.1)

Modification time:Type:Bulletin

Introduction~~~~~~~~~~~~ This short article aims to explain how to get a stack trace from a core dump produced by any of the Oracle products on Unix platforms. By following the steps below you can provide Oracle Support with vital information to help identify the cause of a problem. Please note that it is important to include information about the tool being used, any code involved, the operation being performed, environment etc.. in addition to the details below.What is a ‘core dump‘ ?~~~~~~~~~~~~~~~~~~~~~~~ A core dump is an image copy of a processes state at the instant it ‘aborted‘. It is produced in the form of a file called ‘core‘ usually located in the current directory. What causes a core dump ?~~~~~~~~~~~~~~~~~~~~~~~~~ There are many situations which can cause a core dump to be produced, but it is usually because the process has attempted to do something which the operating system does not like. The most common causes of this are: The program tried to access memory outside its allowed range. The program tried to obtain a resource which was either exhausted or unavailable. An attempt was made to execute illegal instructions.An attempt was made to read unaligned data In Unix systems the offending process is sent one of a number of signals which force a core dump to be produced. It is also possible for a user to produce a core dump by sending one of these signals to a process manually. What should I do if I get a core dump ?~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As with any problem you should first note down the FULL version numbers of the product, the RDBMS, PL/SQL (if used) and any related products. You should also note the EXACT command you were running when this occurred. Eg: If it was a SQL*Forms problem and you were using ‘mrunform30‘, write this down. This command will be referred to as ‘program‘ below. Now follow the instructions below in order: 1) Check if you have a ‘core‘ file, it should be in the directory where the command was issued, or in CORE_DUMP_DEST/core_NNNNNN if it is the ‘oracle‘ executable. "oracle" can also produce core files in $ORACLE_HOME/dbs/core_NNNNN or $ORACLE_HOME/dbs . 2) Log in as ORACLE and cd to the directory containing the core file. Then issue: file core This should identify the "program" name to use in the next step, e.g.: oracle 3) Log in as ORACLE and change in to the $ORACLE_HOME/bin directory. Enter the command: file program and write the result down letter for letter. If the word ‘dynamic‘ or ‘dynamically linked‘ appears in the output of this command then please make a note of this as there are a few platforms on which Oracle does NOT support dynamic linking and this may be the cause of your problem. 4) Now enter: chmod +r program to add read permission to the program. 5) Log out , then log in as the user who encountered the error. The next step will vary slightly depending on which version of Unix you are using. One of the following commands should exist on your machine - try each in order until you find one that exists. In some cases you may be asked for stacks from all threads so use the "thread" version of the command if it exists. If the core file contains multiple threads see Note:118252.1. An alternative to the commands below is to use the stackx.sh script from Note:362791.1 . That script will try to find a suitable debugger and extract the stack tracee for you . Common Debuggers and commands to show a symbolic stack trace: Command NB Exit command Stack Trace command ------- -- ------------ ------------------- dbx quit where xdb (HPUX 10) quit t gdb (HPUX 11) q bt dde (HPUX 11) q bt sdb q t adb $q (or Ctrl-D) $cdebug(PTX only) quit stack gdb (Linux) quit bt or thread apply all where pstack (HPUX, Linux, Solaris) Change to the directory where the core dump is located and enter the commands as in the relevant example below. If you are not sure which program produced the ‘core‘ file then on some Unix platforms the command ‘file core‘ will tell you the executable name that the core file is from (this does not work on ALL Unix platforms, see note below.). Example commands: DBX: $ script /tmp/mystack $ dbx $ORACLE_HOME/bin/<program> core (dbx) where ... << Stack should appear here (dbx) quit $ exit XDB: $ script /tmp/mystack $ xdb $ORACLE_HOME/bin/<program> core (xdb) t ... << Stack should appear here (xdb) quit $ exit SDB: $ script /tmp/mystack $ sdb $ORACLE_HOME/bin/<program> core (sdb) t ... << Stack should appear here (sdb) q $ exit(NOTE: In the ‘adb‘ commands below literally type the $c & $q) ADB: $ script /tmp/mystack $ adb $ORACLE_HOME/bin/<program> core $c << NB: adb has no prompt so just enter $c ... $q $ exit DEBUG: $ script /tmp/mystack $ debug -c core $ORACLE_HOME/bin/<program> debug> stack ... << Stack should appear here debug> quit $ exit GDB: $ script /tmp/mystack $ gdb $ORACLE_HOME/bin/<program> core (gdb) bt ... << Stack should appear here (gdb) thread apply all where ... << Stacks for all threads here (gdb) quit $ exit DDE: $ script /tmp/mystack $ dde -ui line core $ORACLE_HOME/bin/<program> dde> bt ... dde> q $exit PSTACK: $ script /tmp/mystack $ pstack core $ exit Assuming this worked then the stack trace should be shown in the file ‘/tmp/mystack‘. Upload this to Oracle Support. 6) If the debug command failed to give a stack trace then try using a different debugger from the list above (if available). If all debuggers fail then there is probably a problem with either the permissions or the file type - see the section below and then contact Oracle Support with all the details you have so far.Common reasons for not getting a sensible stack~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Filesize Limits: Note that on some machines there may be a kernel parameter or user limit which controls the maximum size of core file that can be produced - you can usually check this by typing: limit in the C shell OR ulimit -a in the Bourne / Korn shells. If this limit is too small the core file will be useless - raise the limit and reproduce the problem. Stripped Executable Some program executables are stripped of symbol information. This makes the stack trace useless. If ‘file program‘ shows the word ‘stripped‘ or ‘nm program‘ shows no output then it is likely that the executable is stripped of symbolic information. In this case the problem tool must be relinked without being stripped - on most Unix platforms this involves ensuring there is no ‘-s‘ option on the link line. Contact Oracle Support with details of the link line used to link the tool. HP Unix Some platforms like HP Unix need a special object file linking in at link time to ensure symbols in shared objects can be reported by the debug tool. Typically this involves relinking the tool including /usr/lib/end.o on the link line. The location of this special file may be different depending on your HPUX version. ‘xdb‘ generally tells you the location of this file if it was not linked into the executable.If ‘file core‘ does not return the executable name:~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Try using the ‘strings‘ command: csh> setenv LANG C or ksh> LANG=C;export LANG ...to get rid of non-ASCII characters return by ‘strings‘ > strings -a core | more The first part of the output may reveal the executable name. Referencesnote: 118252.1-how to process an express Core File Using DBX, dbg, DDE, GDB or ladebug Note: 362791.1-stackx User Guide