AIX 下Oracle Rac dbca建庫報錯 ora-7445 [PC:0x103E2AFA0]，dbca0x103e2afa0

AIX 下Oracle Rac dbca建庫報錯 ora-7445 [PC:0x103E2AFA0]，dbca0x103e2afa0
在AIX 7100-02-03-1334 上安裝Oracle Rac，grid和oracle都已安裝完成。但是dbca建庫的時候探索資料庫crash，以下是建庫時的alert.log，資料庫報ora-07445報錯，dbca的日誌中可以發現在Create database時出錯。在mos上沒有找到匹配的文檔，嘗試使用其他方法。/oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace/alert_rmbtodb1.logMMNL started with pid=26, OS id=7733452 Exception [type: SIGILL, Illegal opcode] [ADDR:0x103E2AFA0] [PC:0x103E2AFA0, {empty}] [flags: 0x0, count: 1]Errors in file /oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace/rmbtodb1_asmb_6357148.trc  (incident=105793):ORA-07445: exception encountered: core dump [PC:0x103E2AFA0] [SIGILL] [ADDR:0x103E2AFA0] [PC:0x103E2AFA0] [Illegal opcode] []Incident details in: /oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/incident/incdir_105793/rmbtodb1_asmb_6357148_i105793.trcUse ADRCI or Support Workbench to package the incident.See Note 411.1 at My Oracle Support for error and packaging details.lmon registered with NM - instance number 1 (internal mem no 0)Reconfiguration started (old inc 0, new inc 2)List of instances: 1 (myinst: 1)  Global Resource Directory frozen* allocate domain 0, invalid = TRUE  Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info  Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resourcesReconfiguration completeThu Dec 11 11:19:18 2014LCK0 started with pid=27, OS id=10420304 Starting background process RSMNThu Dec 11 11:19:18 2014RSMN started with pid=28, OS id=9306256 ORACLE_BASE from environment = /oraapp/oracleException [type: SIGSEGV, Address not mapped to object] [ADDR:0x496568BB8] [PC:0x10029B4D0, {empty}] [flags: 0x8, count: 3]Errors in file /oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace/rmbtodb1_asmb_6357148.trc  (incident=105794):ORA-07445: exception encountered: core dump [PC:0x10029B4D0] [SIGSEGV] [ADDR:0x496568BB8] [PC:0x10029B4D0] [Address not mapped to object] []ORA-07445: exception encountered: core dump [PC:0x103E2AFA0] [SIGILL] [ADDR:0x103E2AFA0] [PC:0x103E2AFA0] [Illegal opcode] []Incident details in: /oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/incident/incdir_105794/rmbtodb1_asmb_6357148_i105794.trcUse ADRCI or Support Workbench to package the incident.See Note 411.1 at My Oracle Support for error and packaging details.Thu Dec 11 11:19:21 2014Sweep [inc][105794]: completedUse ADRCI or Support Workbench to package the incident.See Note 411.1 at My Oracle Support for error and packaging details.Sweep [inc][105793]: completedSweep [inc2][93794]: completedSweep [inc2][105794]: completedPMON (ospid: 16318602): terminating the instance due to error 486System state dump requested by (instance=1, osid=16318602 (PMON)), summary=[abnormal instance termination].System State dumped to trace file /oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace/rmbtodb1_diag_14352568.trcDumping diagnostic data in directory=[cdmp_20141211111922], requested by (instance=1, osid=16318602 (PMON)), summary=[abnormal instance termination].Instance terminated by PMON, pid = 16318602oracle@urmbtodb1:/oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace>1/incident/incdir_105794/rmbtodb1_asmb_6357148_i105794.trc                              <"/oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/incident/incdir_105794/rmbtodb1_asmb_6357148_i105794.trc" 2832 lines, 161159 characters Dump file /oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/incident/incdir_105794/rmbtodb1_asmb_6357148_i105794.trc
首先懷疑是oracle對ASM磁碟沒有寫入權限，嘗試用oracle在ASM上建立spfile，成功建立。檢查CRS_HOME和ORACLE_HOME的執行檔案oracle，並未發現許可權問題。1、首先嘗試在1號節點上手動建庫，編寫一份pfile，嘗試將資料庫nomount，探索資料庫nomount起來後立即crash。2、嘗試在2號節點上dbca建庫，其中報錯資訊如下：/oraapp/oracle/cfgtoollogs/dbca/rmbtodb/trace.log[Thread-178] [ 2014-12-11 12:47:49.813 CST ] [PostDBCreationStep.executeImpl:889]  Starting Database HA Resource[Thread-178] [ 2014-12-11 12:48:16.318 CST ] [CRSNative.internalStartResource:389]  Failed to start resource: Name: ora.rmbtodb.db, node: null, filter: null, msg CRS-5017: The resource action "ora.rmbtodb.db start" encountered the following error:ORA-03113: end-of-file on communication channelProcess ID: 14287060Session ID: 126 Serial number: 1. For details refer to "(:CLSN00107:)" in"/oraapp/grid/gridhome/log/urmbtodb1/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.rmbtodb.db' on 'urmbtodb1' failedCRS-2632: There are no more servers to try to place resource 'ora.rmbtodb.db' on that would satisfy its placement policy[Thread-178] [ 2014-12-11 12:48:16.319 CST ] [PostDBCreationStep.executeImpl:897]  Exception while Starting with HA Database Resource PRCR-1079 : Failed to start resource ora.rmbtodb.dbCRS-5017: The resource action "ora.rmbtodb.db start" encountered the following error:ORA-03113: end-of-file on communication channelProcess ID: 14287060Session ID: 126 Serial number: 1. For details refer to "(:CLSN00107:)" in "/oraapp/grid/gridhome/log/urmbtodb1/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.rmbtodb.db' on 'urmbtodb1' failedCRS-2632: There are no more servers to try to place resource 'ora.rmbtodb.db' on that would satisfy its placement policy
ora.rmbtodb.db在rmbtodb1上啟動失敗，但是資料庫可以成功建立在2號節點上。
具體查看oraagent_oracle.log日誌：/oraapp/grid/gridhome/log/urmbtodb1/agent/crsd/oraagent_oracle/oraagent_oracle.log
2014-12-10 22:48:11.505: [ USRTHRD][1800] {2:52141:473} Value of LOCAL_LISTENER is2014-12-10 22:48:11.549: [ USRTHRD][1800] {2:52141:473} ORA-01405: fetched column value is NULL
2014-12-10 22:48:11.549: [ USRTHRD][1800] {2:52141:473} Value of LISTENER_NETWORKS is2014-12-10 22:48:11.549: [ USRTHRD][1800] {2:52141:473} sqlStmt = ALTER SYSTEM SET LOCAL_LISTENER=' (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=200.31.155.225)(PORT=1521))))' SCOPE=MEMORY SID='rmbtodb1' /* db agent *//* {2:52141:473} */2014-12-10 22:48:13.011: [ USRTHRD][1800] {2:52141:473} ORA-03113: end-of-file on communication channelProcess ID: 14287060Session ID: 126 Serial number: 1發現在設定LOCAL_LISTENER時，資料庫crash。此時問題已經非常明顯，肯定是網路方面的問題。AIX管理員表示之前在1號節點上做過更改網卡綁定的模式。grid@urmbtodb1:/home/grid>oifcfg getif -globalen10  192.168.4.0  global  cluster_interconnecten9  200.31.155.0  global  public查看public IP和priv IP並無異常。嘗試將Public IP重新設定一下：刪除en9資訊：grid@urmbtodb1:/home/grid>oifcfg -delif -global en9grid@urmbtodb1:/home/grid>oifcfg getif -globalen10  192.168.4.0  global  cluster_interconnect重設public IP：grid@urmbtodb1:/home/grid>oifcfg -setif -global en9/200.31.155.0:publicgrid@urmbtodb1:/home/grid>oifcfg getif -globalen10  192.168.4.0  global  cluster_interconnecten9  200.31.155.0  global  public
之後將crs重新啟動。並再次在1號節點dbca建庫，沒有出現此前類似的問題。