Analyze whether the Oracle process dead or fails with Oradebug Short_stack and strace-p

1, you can use Oradebug or strace-p to track whether the background or the foreground process dead or hang
2, if the process fails, will be in the corresponding TRC file to write the latest information, based on this can obtain very important information for further analysis and diagnosis
Log files in Background_dump_dest
3. trc file with Ll-lhrt *lgwr*|tail-10f for the latest process
4, and in the event of a failure, most likely in the alert log records related information, this is the troubleshooting important and primary methods and ideas
5,oradebug Setospid Ospid
Oradebug Short_stack
The stack information for the process is displayed, note that it can be run multiple times, and if the stack information is displayed consistently, it is certain that the process is dead or malfunctioning.

6, can be used STRACE-P ospid tracking analysis,

Similar information---hang or failure is as follows
Semtimedop (9273344, 0x7fffe66199d0, 1, {1, 0}) = 1 Eagain (Resource temporarily unavailable)

A similar message---normal is as follows
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440015944
Semtimedop (9273344, 0x7fffe661b1f0, 1, {1, 800000000}) = 1 Eagain (Resource temporarily unavailable)
Getrusage (rusage_self, {ru_utime={0, 123981}, Ru_stime={0, 132979}, ...}) = 0
Getrusage (rusage_self, {ru_utime={0, 123981}, Ru_stime={0, 132979}, ...}) = 0
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016124
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016124
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016124
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016124
Semtimedop (9273344, 0x7fffe661b1f0, 1, {3, 0}) = 1 Eagain (Resource temporarily unavailable)
Getrusage (rusage_self, {ru_utime={0, 123981}, Ru_stime={0, 132979}, ...}) = 0
Getrusage (rusage_self, {ru_utime={0, 123981}, Ru_stime={0, 132979}, ...}) = 0
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016424
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016424
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016424
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016424
Semtimedop (9273344, 0x7fffe661b1f0, 1, {3, 0}) = 1 Eagain (Resource temporarily unavailable)
Getrusage (rusage_self, {ru_utime={0, 123981}, Ru_stime={0, 132979}, ...}) = 0
Getrusage (rusage_self, {ru_utime={0, 123981}, Ru_stime={0, 132979}, ...}) = 0
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016725
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016725
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016725
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440016725
Semtimedop (9273344, 0x7fffe661b1f0, 1, {3, 0}) = 1 Eagain (Resource temporarily unavailable)
Getrusage (rusage_self, {ru_utime={0, 123981}, Ru_stime={0, 132979}, ...}) = 0
Getrusage (rusage_self, {ru_utime={0, 123981}, Ru_stime={0, 132979}, ...}) = 0
Times ({tms_utime=12, tms_stime=13, tms_cutime=0, tms_cstime=0}) = 440017025
Open ("/proc/4385/stat", o_rdonly) = 35
Read ("4385 (Oracle) S 1 4385 4385 0-1" ..., 999) = 225

Plainly, is to see whether the information changes, there is a change in the process is normal, otherwise it is not normal

Test

Sql> SELECT * from V$version where rownum=1;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0-64bit Production

View background processes
Sql> Select Pid,spid,pname,username from V$process order by 1;

PID SPID PNAME USERNAME
---------- ---------- ---------- ------------------------------
1
2 4385 Pmon Oracle
3 4387 Vktm Oracle
4 4391 GEN0 Oracle
5 4393 DIAG Oracle
6 4395 DBRM Oracle
7 4397 PSP0 Oracle
8 4399 DIA0 Oracle
9 4401 Mman Oracle
4403 DBW0 Oracle
4405 LGWR Oracle


PID SPID PNAME USERNAME
---------- ---------- ---------- ------------------------------
4407 CKPT Oracle
4409 Smon Oracle
4411 RECO Oracle
4413 Mmon Oracle
4415 MMNL Oracle
4417 D000 Oracle
4419 S000 Oracle
4652 SMCO Oracle
5266 W000 Oracle
4936 Oracle
4468 ARC0 Oracle
PID SPID PNAME USERNAME
---------- ---------- ---------- ------------------------------
4481 ARC1 Oracle
4486 ARC2 Oracle
4489 ARC3 Oracle
4496 QMNC Oracle
4549 Q000 Oracle
4551 Q001 Oracle
4568 Oracle

Rows selected.

Sql>
---view TRC file directories
[Email protected] trace]$ Ll-lhrt *lgwr*|tail-10f
-RW-R-----1 Oracle oinstall 213 DEC 19:05 GUOWANG_LGWR_5297.TRM
-RW-R-----1 Oracle oinstall 2.4K DEC 19:05 guowang_lgwr_5297.trc
-RW-R-----1 Oracle oinstall 2.3K DEC 01:05 GUOWANG_LGWR_22295.TRM
-RW-R-----1 Oracle oinstall 27K DEC 01:05 guowang_lgwr_22295.trc
-RW-R-----1 Oracle Oinstall 02:18 GUOWANG_LGWR_31280.TRM
-RW-R-----1 Oracle Oinstall 903 DEC 02:18 GUOWANG_LGWR_31280.TRC
-RW-R-----1 Oracle Oinstall 02:44 GUOWANG_LGWR_32077.TRM
-RW-R-----1 Oracle Oinstall 906 DEC 02:44 guowang_lgwr_32077.trc
-RW-R-----1 Oracle Oinstall 03:27 GUOWANG_LGWR_1032.TRM
-RW-R-----1 Oracle Oinstall 887 DEC 03:27 guowang_lgwr_1032.trc


---hang LGWR
sql> Oradebug setospid 4405
Oracle pid:11, Unix process pid:4405, Image: [email protected] (LGWR)
Sql> Oradebug Suspend
Statement processed.

--alert record the above information synchronously
Tue Dec 15 04:46:15 2015
Unix process pid:4405, Image: [email protected] (LGWR) flash frozen [command #1]

---the TRC directory to record the above information synchronously
[Email protected] trace]$ Ll-lhrt *lgwr*|tail-10f
-RW-R-----1 Oracle oinstall 2.3K DEC 01:05 GUOWANG_LGWR_22295.TRM
-RW-R-----1 Oracle oinstall 27K DEC 01:05 guowang_lgwr_22295.trc
-RW-R-----1 Oracle Oinstall 02:18 GUOWANG_LGWR_31280.TRM
-RW-R-----1 Oracle Oinstall 903 DEC 02:18 GUOWANG_LGWR_31280.TRC
-RW-R-----1 Oracle Oinstall 02:44 GUOWANG_LGWR_32077.TRM
-RW-R-----1 Oracle Oinstall 906 DEC 02:44 guowang_lgwr_32077.trc
-RW-R-----1 Oracle Oinstall 03:27 GUOWANG_LGWR_1032.TRM
-RW-R-----1 Oracle Oinstall 887 DEC 03:27 guowang_lgwr_1032.trc
-RW-R-----1 Oracle Oinstall 04:46 GUOWANG_LGWR_4405.TRM
-RW-R-----1 Oracle Oinstall 896 DEC 04:46 GUOWANG_LGWR_4405.TRC
[Email protected] trace]$

Analyze whether the Oracle process dead or fails with Oradebug Short_stack and strace-p