Oracle11g Control File Corruption Problem Analysis

For databases earlier than oracle11g, when the control file is corrupted, when we mount the database, there will be a significant ora-600 error, so it is easy to know that the control file is corrupt

When the control file is corrupted for databases earlier than oracle 11g, we mount the database with a significant ora-600 error, which makes it easy to know that the control file is corrupt.

When the control file is corrupted for databases earlier than Oracle 11g, when we mount the database, there will be a significant ora-600 error, so it is easy to know the control file corruption error, however, oracle 11g R2,

At that time, it was an ORACLE 11g RAC system. When a problem occurs, the database instance can be opened by nomount, but the following alarm occurs when the mount control file is mounted:

ORA-3113 "end of file on communication channel"

Then the whole sqlplus connection is terminated and needs to be reconnected. Of course, we know that the mount phase is usually unavailable, and the problem lies in the damage to the control file, but for professional personnel, it is obviously impossible to satisfy such a mentality, so we need to analyze it further:

However, we can see the following information in the ASM log:

Tue Mar 27 13:35:11 2012
NOTE: client PROD1: PROD registered, osid 6726, mbr 0x1
Tue Mar 27 13:35:24 2012
NOTE: ASM client PROD1: PROD disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file/u01/app/oracle/diag/asm/+ ASM1/trace/+ ASM1_ora_6726.trc
Tue Mar 27 13:40:35 2012
NOTE: client PROD1: PROD registered, osid 7477, mbr 0x1
Tue Mar 27 13:41:45 2012
NOTE: ASM client PROD1: PROD disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file/u01/app/oracle/diag/asm/+ ASM1/trace/+ ASM1_ora_7477.trc
Tue Mar 27 13:41:47 2012
NOTE: client PROD1: PROD registered, osid 7736, mbr 0x1
Tue Mar 27 13:42:01 2012
NOTE: ASM client PROD1: PROD disconnected unexpectedly.
NOTE: check client alert log.
NOTE: Trace records dumped in trace file/u01/app/oracle/diag/asm/+ ASM1/trace/+ ASM1_ora_7736.tr

We can only see the following information for the generated trace file:

13:41:08. 022438: 802EEFE8: KFNS: kfn. c @ 702: kfnDispatch (): calling server stub for KFNOP = 5
13:41:13. 027006: 802EF0F4: KFNU: kfns. c @ 1924: kfnsBackground (): kfnsBackground completed in 5 seconds (KFNPM = 0)
13:41:13. 027012: 802EF0F5: KFNS: kfn. c @ 729: kfnDispatch (): completed KFNOP = 5
13:41:13. 027122: 802EF0F6: KFNS: kfn. c @ 702: kfnDispatch (): calling server stub for KFNOP = 5

It is obviously useless for this problem, and the problem should still be in the database.

Therefore, the alert alarm check for database instances. The logs when the alter database mount status is executed are as follows:

Tue Mar 27 11:42:01 2012
Alter database mount
This instance was first to mount
Tue Mar 27 11:42:01 2012
NOTE: Loaded library:/opt/oracle/extapi/64/asm/orcl/1/libasm. so
NOTE: Loaded library: System
Tue Mar 27 11:42:01 2012
SUCCESS: diskgroup PRODDATA was mounted
Tue Mar 27 11:42:01 2012
NOTE: dependency between database PROD and diskgroup resource ora. PRODDATA. dg is established
USER (ospid: 26774): terminating the instance
Tue Mar 27 11:42:07 2012
System state dump requested by (instance = 1, osid = 26774), summary = [abnormal instance termination].
System State dumped to trace file/d01/oracle/11.2.0/admin/prodmongodb01/diag/rdbms/prod/PROD1/trace/PROD1_diag_26656.trc
Dumping diagnostic data in directory = [cdmp_20120327114207], requested by (instance = 1, osid = 26774), summary = [abnormal instance termination].
Instance terminated by USER, pid = 26774

Check the alarm trace file:/d01/oracle/11.2.0/admin/PROD1_db01/diag/rdbms/prod/PROD1/trace/PROD1_diag_26656.trc. There is no detailed information.

Later, we used the 10046 event to track the mount process and then saw a detailed prompt,

Alter session set events = '10046 trace name context forever, level 12 ';
Trace file/d01/oracle/11.2.0/admin/PROD1_db01/diag/rdbms/prod/PROD1/trace/PROD1_ora_7764.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0-64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME =/d01/oracle/11.2.0
System name: Linux
Node name: db01.clc.com
Release: 2.6.18-238. el5
Version: #1 SMP Sun Dec 19 14:22:44 EST 2010
Machine: x86_64
Instance name: PROD1
Redo thread mounted by this instance: 0
Oracle process number: 31
Unix process pid: 7764, image: oracle@db01.clc.com (TNS V1-V3)


* ** 2012-03-27 13:41:55. 101
* ** Session id: (1751.3) 13:41:55. 101
* ** Client id :() 13:41:55. 101
* ** Service name :() 13:41:55. 101
* Module name :( oraagent.bin@db01.clc.com (TNS V1-V3) 2012-03-27 13:41:55. 101
* ** Action name :() 13:41:55. 101

Error: kccpb_sanity_check_2
Control file sequence number mismatch!
Fhcsq: 312916 bhcsq: 313137 cfn 0

* ** 2012-03-27 13:41:55. 101
Submitting synchronized dump request [268435460]. summary = [Controlfile header dump (kccpbsc)].

* ** 2012-03-27 13:41:57. 102
Kjzduptcctx: policying DIAG for crash event
----- Abridged Call Stack Trace -----
Ksedsts () + 461 <-kjzdssdmp () + 267 <-kjzduptcctx () + 232 <-kjzdicrshnfy () + 53 <-ksuitm () + 1325 <-kccpb_sanity_check () + 341 <-kccbmp _get () + 309 <-kccsed_rbl () + 111 <-kccocx () + 1154 <-kccocf () + 136 <-kcfcmb () + 1025 <-kcfmdb () + 54 <-adbdrv () + 63122 <-opiexe () + 18173 <-opiosq0 () + 3993 <-kpooprx () + 274
<-Kpoal8 () + 800 <-opiodr () + 910 <-ttcpip () + 2289 <-opitsk () + 1670
----- End of Abridged Call Stack Trace -----

* ** 2012-03-27 13:41:57. 141
USER (ospid: 7764): terminating the instance
Ksuitm: waiting up to [5] seconds before killing DIAG (7652)

As shown in the red field above, the Consistency Verification of the control file is damaged because the serial number does not match in the control file, and the database cannot be mounted normally.

This makes it clear that you can modify or recreate the control file to open the database.

For more information about Oracle, see the Oracle topic page? Tid = 12