Oracle BUG導致執行個體宕機：ORA-07445

Oracle BUG導致執行個體宕機：ORA-07445

現象：
客戶的資料庫（RAC環境：11.1.0.6）發生了執行個體異常宕機現象，伴隨有ORA-07445錯誤：
Sun Jun 23 01:00:06 2013
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0xF] [PC:0x755773D, kcbw_get_bh()+67]
Errors in file /Oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_mman_2015.trc (incident=298938):
ORA-07445: exception encountered: core dump [kcbw_get_bh()+67] [SIGSEGV] [ADDR:0xF] [PC:0x755773D] [Address not mapped to object] []
Incident details in: /oracle/app/11gR1/diag/rdbms/xij/xij1/incident/incdir_298938/xij1_mman_2015_i298938.trc
Sun Jun 23 01:00:07 2013
Trace dumping is performing id=[cdmp_20130623010007]
Sun Jun 23 01:00:09 2013
Sweep Incident[298938]: completed
Sun Jun 23 01:00:09 2013
Errors in file /oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_pmon_1981.trc:
ORA-00822: MMAN process terminated with error
PMON (ospid: 1981): terminating the instance due to error 822
Sun Jun 23 01:00:09 2013
Errors in file /oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_j000_22268.trc:
ORA-00822: MMAN process terminated with error
Sun Jun 23 01:00:09 2013
Errors in file /oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_m000_22430.trc:
ORA-00822: MMAN process terminated with error
System state dump is made for local instance
System State dumped to trace file /oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_diag_1987.trc
Sun Jun 23 01:00:09 2013
ORA-1092 : opiodr aborting process unknown ospid (11096_47524616916112)
Sun Jun 23 01:00:09 2013
ORA-1092 : opitsk aborting process
Sun Jun 23 01:00:09 2013
ORA-1092 : opiodr aborting process unknown ospid (6317_47353365785744)
Sun Jun 23 01:00:09 2013
ORA-1092 : opitsk aborting process
Sun Jun 23 01:00:09 2013
ORA-1092 : opiodr aborting process unknown ospid (28698_47056912551056)
Sun Jun 23 01:00:09 2013
ORA-1092 : opitsk aborting process
Sun Jun 23 01:00:09 2013
ORA-1092 : opiodr aborting process unknown ospid (18927_47567504653456)
Sun Jun 23 01:00:10 2013
ORA-1092 : opitsk aborting process
Sun Jun 23 01:00:10 2013
Errors in file /oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_q001_3487.trc:
ORA-00822: MMAN process terminated with error
ORA-1092 : opidrv aborting process Q001 ospid (3487_47252506410128)
Sun Jun 23 01:00:11 2013
ORA-1092 : opitsk aborting process
Sun Jun 23 01:00:11 2013
License high water mark = 510
Errors in file /oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_m000_22430.trc:
ORA-00822: MMAN process terminated with error
ORA-00822: MMAN process terminated with error
Errors in file /oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_j000_22268.trc:
ORA-00449: background process 'LGWR' unexpectedly terminated with error 822
ORA-00822: MMAN process terminated with error
Errors in file /oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_j000_22268.trc:
ORA-00449: background process 'LGWR' unexpectedly terminated with error 822
ORA-00822: MMAN process terminated with error
Errors in file /oracle/app/11gR1/diag/rdbms/xij/xij1/trace/xij1_j000_22268.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00822: MMAN process terminated with error
ORA-06512: at "WKSYS.WK_JOB", line 442
ORA-00449: background process 'MMON' unexpectedly terminated with error 822
ORA-00822: MMAN process terminated with error
ORA-06512: at line 1
ORA-1092 : opidrv aborting process J000 ospid (22268_47357930925200)
Sun Jun 23 01:00:20 2013
Instance terminated by PMON, pid = 1981
Sun Jun 23 01:00:21 2013
USER (ospid: 22527): terminating the instance
Instance terminated by USER, pid = 22527
Sun Jun 23 01:00:26 2013
Starting ORACLE instance (normal)

分析：
Ora-07445通常是Oracle自身的BUG導致的，
首先使用IPS收集了alert中的錯誤資訊（IPS使用方法見我的另一篇文章《IPS簡單使用方法》）：
搜尋了一下metalink，發現客戶的問題跟以下三篇Note中描述的BUG類似：
ORA-7445 (kcbw_get_bh) [ID 1341402.1]
Bug 9728912 [https://bug.oraclecorp.com/pls/bug/webbug_edit.edit_info_top?rptno=9728912] - PMON terminates instance due to ORA-7445 [kcbw_numperchunk] / ORA-7445 [kcbw_get_bh]] [ID 9728912.8]
Instance Crashed On ORA-7445 kcbw_numperchunk [ID 1364264.1]
但根據Note可以看到，相關的BUG已經在11.1.0.6中fix掉了。
看看客戶資料庫中的其餘嚴重錯誤資訊：
Node1:
adrci> show problem

ADR Home = /oracle/app/11gR1/diag/rdbms/xij/xij1:
*************************************************************************
PROBLEM_ID PROBLEM_KEY LAST_INCIDENT LASTINC_TIME
-------------------- ----------------------------------------------------------- -------------------- ----------------------------------------
5 ORA 7445 [kcbw_get_bh()+67] 298938 2013-06-23 01:00:06.373716 +08:00
11 ORA 600 276161 2013-06-04 18:12:12.709933 +08:00
10 ORA 600 [729] 276160 2013-06-04 18:09:27.857128 +08:00
7 ORA 7445 [kgghash()+367] 253234 2013-06-03 15:27:04.349337 +08:00
9 ORA 7445 [kksMapCursor()+323] 256538 2013-05-27 09:54:58.684956 +08:00
8 ORA 7445 [qkabxo()+22] 251194 2013-05-01 22:03:37.715416 +08:00
2 ORA 600 [kghfrh:ds] 238818 2013-01-28 11:35:23.755034 +08:00
6 ORA 7445 [eoa_pm_push()+31] 239218 2013-01-28 11:24:42.835685 +08:00
3 ORA 7445 [ioei_get_method_counts()+39] 71129 2012-10-17 11:17:39.735719 +08:00
4 ORA 7445 [jol_calculate_transitive_interface_set()+1165] 74233 2012-10-17 11:05:51.570021 +08:00
1 ORA 600 [kghfru:ds] 6369 2012-09-07 17:35:55.001585 +08:00
11 rows fetched
Node2:
[oracle@XIJ02 ~]$ adrci

ADRCI: Release 11.1.0.6.0 - Beta on Mon Jun 24 14:59:37 2013

Copyright (c) 1982, 2007, Oracle. All rights reserved.
ADR base = "/oracle/app/11gR1"
adrci>
adrci>
adrci> set homepath diag/rdbms/xij/xij2
adrci>
adrci> show problem
ADR Home = /oracle/app/11gR1/diag/rdbms/xij/xij2:
*************************************************************************
PROBLEM_ID PROBLEM_KEY LAST_INCIDENT LASTINC_TIME
-------------------- ----------------------------------------------------------- -------------------- ----------------------------------------
1 ORA 7445 [kgghash()+367] 209965 2013-06-16 23:34:39.333982 +08:00
2 ORA 7445 [kksMapCursor()+323] 190129 2013-05-27 09:54:56.121652 +08:00
2 rows fetched
adrci>
解決方案：
在客戶的2個節點中一共發現了13個疑似BUG引起的資料庫故障，總體而言，Oracle 11.1.0.6不算太穩定的版本，存在著各種BUG，
Oracle在11.1.0.7中Fix掉了11.1.0.6中發現的大部分BUG，所以相對而言要穩定得多，因此建議客戶升級資料庫至11.1.0.7或者11.2.0.3。

附：
(Triage Tool 3.01, routed by file analysis):
Failing Function: kcbw_get_bh
Route To: BUFFER CACHE:MANAGEABILITY
Error Argument: [kcbw_get_bh]
Type of Error: ORA-07445
File Name: xij1_mman_2015_i298938.trc
Comment: Routed by Error Argument, Conventional routing
DB Version: 11.1.0.6.0
Platform: Linux CPU: x86_64
OS Version: 2.6.18-194.el5
Stack Trace: kcbw_get_bh kcbw_get_first_buffer kcbw_next_free kmgs_extract_mem_from_granule kmgs_process_request_immediate kmgs_process_request kmgsdrv ksbabs ksbrdp opirip

在CentOS 6.4下安裝Oracle 11gR2(x64)

Oracle 11gR2 在VMWare虛擬機器中安裝步驟

Debian 下 安裝 Oracle 11g XE R2