Ora-7445 [PC: 0x103E2AFA0], dbca0x103e2afa0

Ora-7445 [PC: 0x103E2AFA0], dbca0x103e2afa0
Install Oracle Rac on AIX 7100-02-03-1334. Both grid and oracle have been installed. However, dbca database found crash, the following is the database creation of the alert. log, the database reported ora-07445 error, dbca logs can be found in the Create database error. No matching document found on mos. Try other methods. /Oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace/alert_rmbtodb1.logMMNL started with pid = 26, OS id = 7733452 Exception [type: SIGILL, Illegal opcode] [ADDR: 0x103E2AFA0] [PC: 0x103E2AFA0, {empty}] [flags: 0x0, count: 1] Errors in file/oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace/rmbtodb1_asmb_6357148.trc (incident = 105793): ORA-07445: exception encountered: core dump [PC: 0x103E2AFA0] [SIGILL] [ADDR: 0x103E2AFA0] [PC: 0x103E2AFA0] [Illegal opcode] [] Incident details in: /oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/incident/incdir_105793/rmbtodb1_asmb_6357148_i105793.trcUse ADRCI or Support Workbench to package the incident. see Note 411.1 at My Oracle Support for error and packaging details. lmon registered with NM-instance number 1 (internal mem no 0) Reconfiguration started (old inc 0, new inc 2) List of instances: 1 (myinst: 1) global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 1: 0 GCS shadows canceled, 0 closed, 0 Xw primary ved LMS 0: 0 GCS shadows canceled, 0 closed, 0 Xw primary ved Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Post SMON to start 1st pass IR Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resourcesReconfiguration completeThu Dec 11 11:19:18 2014LCK0 started with pid = 27, OS id = 10420304 Starting background process RSMNThu Dec 11 11:19:18 2014 RSMN started with pid = 28, OS id = 9306256 ORACLE_BASE from environment =/oraapp/oracleException [type: SIGSEGV, address not mapped to object] [ADDR: 0x496568BB8] [PC: 0x10029B4D0, {empty}] [flags: 0x8, count: 3] Errors in file/oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace/rmbtodb1_asmb_6357148.trc (incident = 105794): ORA-07445: exception encountered: core dump [PC: 0x10029B4D0] [SIGSEGV] [ADDR: 0x496568BB8] [PC: 0x10029B4D0] [Address not mapped to object] [] ORA-07445: exception encountered: core dump [PC: 0x103E2AFA0] [SIGILL] [ADDR: 0x103E2AFA0] [PC: 0x103E2AFA0] [Illegal opcode] [] Incident details in: /oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/incident/incdir_105794/rmbtodb1_asmb_6357148_i105794.trcUse ADRCI or Support Workbench to package the incident. see Note 411.1 at My Oracle Support for error and packaging details. thu Dec 11 11:19:21 2014 Sweep [inc] [105794]: completedUse ADRCI or Support Workbench to package the incident. see Note 411.1 at My Oracle Support for error and packaging details. sweep [inc] [105793]: completedSweep [inc2] [93794]: completedSweep [inc2] [105794]: completedPMON (ospid: 16318602 ): terminating the instance due to error 486 System state dump requested by (instance = 1, osid = 16318602 (PMON), summary = [abnormal instance termination]. system State dumped to trace file/oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace/diagnostic data in directory = [cdmp_201412111922], requested by (instance = 1, osid = 16318602 (PMON), summary = [abnormal instance termination]. instance terminated by PMON, pid = 16318602oracle @ urmbtodb1: /oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/trace> 1/incident/messages <"/oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/incident/messages /rmbtodb1_asmb_6357148_i105794.trc "2832 lines, 161159 characters Dump file/oraapp/oracle/diag/rdbms/rmbtodb/rmbtodb1/incident/incdir_105794/rmbtodb1_asmb_6357148_i105794.trc
First, we suspect that oracle does not have the write permission on the ASM disk. We tried to use oracle to create a spfile on the ASM. Check the execution files of CRS_HOME and ORACLE_HOME in oracle. No permission error is found. 1. First, try to manually create a database on node 1, write a pfile, and try to mount the database nomount. Then crash is immediately triggered when the database nomount is found. 2. Try to create a database in dbca on node 2. The error message is as follows:/oraapp/oracle/export toollogs/dbca/rmbtodb/trace. log [Thread-178] [12:47:49. 813 CST] modify postdbcreationstep.exe cuteImpl: 889] Starting Database HA Resource [Thread-178] [12:48:16. 318 CST] [CRSNative. internalStartResource: 389] Failed to start resource: Name: ora. rmbtodb. db, node: null, filter: null, msg CRS-5017: The resource action "ora. rmbtodb. db start "encountered the following error: ORA-03113: end-of-file on communication channelProcess ID: 14287060 Session ID: 126 Serial number: 1. for details refer to "(: CLSN00107 :)" in "/oraapp/grid/gridhome/log/urmbtodb1/agent/crsd/oraagent_oracle/oraagent_oracle.log ".
CRS-2674: Start of 'ora. rmbtodb. db 'on 'urmbtodb1' failedCRS-2632: There are no more servers to try to place resource' ora. rmbtodb. db 'on that wocould satisfy its placement policy [Thread-178] [12:48:16. 319 CST] parse postdbcreationstep.exe cuteImpl: 897] Exception while Starting with HA Database Resource PRCR-1079: Failed to start resource ora. rmbtodb. dbCRS-5017: The resource action "ora. rmbtodb. db start "encountered the following error: ORA-03113: end-of-file on communication channelProcess ID: 14287060 Session ID: 126 Serial number: 1. for details refer to "(: CLSN00107 :)" in "/oraapp/grid/gridhome/log/urmbtodb1/agent/crsd/oraagent_oracle/oraagent_oracle.log ".
CRS-2674: Start of 'ora. rmbtodb. db 'on 'urmbtodb1' failedCRS-2632: There are no more servers to try to place resource' ora. rmbtodb. db 'on that wowould satisfy its placement policy
Ora. rmbtodb. db failed to start on rmbtodb1, but the database can be successfully created on node 2.
For details, see oraagent_oracle.log:/oraapp/grid/gridhome/log/urmbtodb1/agent/crsd/oraagent_oracle/oraagent_oracle.log
22:48:11. 505: [USRTHRD] [1800] {2: 52141: 473} Value of LOCAL_LISTENER is2014-12-10 22:48:11. 549: [USRTHRD] [1800] {2: 52141: 473} ORA-01405: fetched column value is NULL
22:48:11. 549: [USRTHRD] [1800] {2: 52141: 473} Value of LISTENER_NETWORKS is2014-12-10 22:48:11. 549: [USRTHRD] [1800] {2: 52141: 473} sqlStmt = alter system set LOCAL_LISTENER = '(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = 200.31.155.225) (PORT = 1521) 'SCOPE = memory sid = 'rmbtodb1'/* db agent * // * {2: 52141: 473} */22:48:13. 011: [USRTHRD] [1800] {2: 52141: 473} ORA-03113: e Nd-of-file on communication channelProcess ID: 14287060 Session ID: 126 Serial number: 1 found that when LOCAL_LISTENER is set, the database crash. The problem is already very obvious, and it must be a network problem. The AIX administrator indicates that the NIC binding mode has been changed on node 1. Grid @ urmbtodb1:/home/grid> oifcfg getif-globalen10 192.168.4.0 global cluster_interconnecten9 200.31.155.0 global public check public IP and priv IP is normal. Try to reset the Public IP: Delete the en9 information: grid @ urmbtodb1:/home/grid> oifcfg-delif-global en9grid @ urmbtodb1: /home/grid> oifcfg getif-globalen10 192.168.4.0 global cluster_interconnect reset public IP: grid @ urmbtodb1:/home/grid> oifcfg-setif-global en9/200.31.155.0: publicgrid @ urmbtodb1: /home/grid> oifcfg getif-globalen10 192.168.4.0 global cluster_interconnecten9 200.31.155.0 global public
Then restart crs. The database was created again in dbca on node 1 without similar problems.