停電遭遇ORA-600

1：周末遭遇停電，新配置的內網測試資料庫active dataguard 環境自動關閉，今早啟動主庫後發現報錯如下

[oracle@db1 ~]$ sqlplus /nologSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 17 17:00:57 2013Copyright (c) 1982, 2009, Oracle.  All rights reserved.SQL> conn /as sysdbaConnected to an idle instance.SQL> startupORACLE instance started.Total System Global Area 1.3362E+10 bytesFixed Size                  2217952 bytesVariable Size            6777997344 bytesDatabase Buffers         6576668672 bytesRedo Buffers                4960256 bytesDatabase mounted.ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1],[6], [301353], [301354], [], [], [], [], [], [], []

2：第一反應先recover下database，再嘗試拉起資料庫，問題依舊

SQL> recover database;Media recovery complete.SQL> alter database open;alter database open*ERROR at line 1:ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1],[6], [301353], [301354], [], [], [], [], [], [], []

3：於是看alert日誌和相關的trace檔案，綜合判斷可能控制檔案出現問題

[oracle@db1 ~]$ tail -f alert_db.logErrors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc:ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []Errors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc:ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []ORA-600 signalled during: ALTER DATABASE OPEN...Trace dumping is performing id=[cdmp_20130617170117]Mon Jun 17 17:02:13 2013Sweep [inc][24153]: completedSweep [inc2][24153]: completedMon Jun 17 17:02:38 2013ALTER DATABASE RECOVER  databaseMedia Recovery Start started logmerger processParallel Media Recovery started with 4 slavesMon Jun 17 17:02:38 2013Recovery of Online Redo Log: Thread 1 Group 3 Seq 6 Reading mem 0  Mem# 0: /u01/app/oracle/oradata/DB/onlinelog/o1_mf_3_8vpmjgoq_.log  Mem# 1: /u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_3_8vpmkm9x_.logMedia Recovery Complete (db)Completed: ALTER DATABASE RECOVER  databaseMon Jun 17 17:02:54 2013alter database openBeginning crash recovery of 1 threads parallel recovery started with 3 processesStarted redo scanCompleted redo scan read 152 KB redo, 0 data blocks need recoveryErrors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc  (incident=24154):ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []Incident details in: /u01/app/oracle/diag/rdbms/db1/db/incident/incdir_24154/db_ora_32462_i24154.trcMon Jun 17 17:02:55 2013Trace dumping is performing id=[cdmp_20130617170255]Aborting crash recovery due to error 600Errors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc:ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []Errors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_ora_32462.trc:ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr], [1], [6], [301353], [301354], [], [], [], [], [], [], []ORA-600 signalled during: alter database open...Mon Jun 17 17:03:13 2013Sweep [inc][24154]: completedSweep [inc2][24154]: completed

4：於是產生控制檔案trace控制指令碼對控制檔案執行恢複

SQL> select open_mode from v$database;OPEN_MODE--------------------MOUNTEDSQL> alter database backup controlfile to trace as '/tmp/1.ctl';Database altered.[oracle@db1 ~]$ cat /tmp/1.ctlCREATE CONTROLFILE REUSE DATABASE "DB" NORESETLOGS FORCE LOGGING ARCHIVELOG    MAXLOGFILES 16    MAXLOGMEMBERS 3    MAXDATAFILES 100    MAXINSTANCES 8    MAXLOGHISTORY 292LOGFILE  GROUP 1 (    '/u01/app/oracle/oradata/DB/onlinelog/o1_mf_1_8vpmdkl9_.log',    '/u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_1_8vpmdryt_.log'  ) SIZE 512M BLOCKSIZE 512,  GROUP 2 (    '/u01/app/oracle/oradata/DB/onlinelog/o1_mf_2_8vpmfqmw_.log',    '/u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_2_8vpmhk2z_.log'  ) SIZE 512M BLOCKSIZE 512,  GROUP 3 (    '/u01/app/oracle/oradata/DB/onlinelog/o1_mf_3_8vpmjgoq_.log',    '/u01/app/oracle/flash_recovery_area/DB/onlinelog/o1_mf_3_8vpmkm9x_.log'  ) SIZE 512M BLOCKSIZE 512--STANDBY LOGFILE--   GROUP 4 (--     '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_4_8vpq6nvy_.log',--     '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_4_8vpq7wk8_.log'--   ) SIZE 512M BLOCKSIZE 512,--   GROUP 5 (--     '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_5_8vpqbh6s_.log',--     '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_5_8vpqcmbj_.log'--   ) SIZE 512M BLOCKSIZE 512,--   GROUP 6 (--     '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_6_8vpqf3rz_.log',--     '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_6_8vpqfv5w_.log'--   ) SIZE 512M BLOCKSIZE 512,--   GROUP 7 (--     '/u01/app/oracle/oradata/DB1/onlinelog/o1_mf_7_8vpqgw0j_.log',--     '/u01/app/oracle/flash_recovery_area/DB1/onlinelog/o1_mf_7_8vpqhcql_.log'--   ) SIZE 512M BLOCKSIZE 512DATAFILE  '/u01/app/oracle/oradata/DB/datafile/o1_mf_system_8vpm8hf3_.dbf',  '/u01/app/oracle/oradata/DB/datafile/o1_mf_sysaux_8vpm8hjq_.dbf',  '/u01/app/oracle/oradata/DB/datafile/o1_mf_undotbs1_8vpm8hl4_.dbf',  '/u01/app/oracle/oradata/DB/datafile/o1_mf_users_8vpm8hn1_.dbf'CHARACTER SET ZHS16GBK;SQL> shutdown immediateORA-01109: database not openDatabase dismounted.ORACLE instance shut down.SQL> startup nomountORACLE instance started.Total System Global Area 1.3362E+10 bytesFixed Size                  2217952 bytesVariable Size            6777997344 bytesDatabase Buffers         6576668672 bytesRedo Buffers                4960256 bytesSQL> @/tmp/1.ctl;Control file created.

5：重建控制檔案後，對資料庫進行recover，成功開啟資料庫

SQL> alter database open;alter database open*ERROR at line 1:ORA-01113: file 1 needs media recoveryORA-01110: data file 1:'/u01/app/oracle/oradata/DB/datafile/o1_mf_system_8vpm8hf3_.dbf'SQL> recover database;Media recovery complete.SQL> alter database open;Database altered.SQL> select open_mode,database_role from v$database;  OPEN_MODE            DATABASE_ROLE-------------------- ----------------READ WRITE           PRIMARY

6：通過觀察alert日誌，添加暫存資料表空間資料檔案

[oracle@db1 ~]$ tail -f alert_db.logErrors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_m001_381.trc:ORA-25153: Temporary Tablespace is EmptyMon Jun 17 17:17:25 2013Errors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_j007_400.trc:ORA-25153: Temporary Tablespace is EmptyErrors in file /u01/app/oracle/diag/rdbms/db1/db/trace/db_j007_400.trc:ORA-12012: error on auto execute of job 12696ORA-25153: Temporary Tablespace is EmptyORA-06512: at "DBSNMP.BSLN_INTERNAL", line 2073ORA-06512: at line 1Mon Jun 17 17:18:20 2013alter tablespace temp add tempfileCompleted: alter tablespace temp add tempfileSQL> alter tablespace temp add tempfile;Tablespace altered.

7：觀察physical database，發現出現如下問題

[root@db2 ~]# su - oracle[oracle@db2 ~]$ sqlplus /nologSQL*Plus: Release 11.2.0.1.0 Production on Mon Jun 17 17:18:50 2013Copyright (c) 1982, 2009, Oracle.  All rights reserved.SQL> conn /as sysdbaConnected to an idle instance.SQL> startupORACLE instance started.Total System Global Area 1.3362E+10 bytesFixed Size                  2217952 bytesVariable Size            6777997344 bytesDatabase Buffers         6576668672 bytesRedo Buffers                4960256 bytesDatabase mounted.ORA-10458: standby database requires recoveryORA-01196: file 1 is inconsistent due to a failed media recovery sessionORA-01110: data file 1:'/u01/app/oracle/oradata/DB2/datafile/o1_mf_system_08oc7c38_.dbf'SQL> alter database open read only;alter database open read only*ERROR at line 1:ORA-10458: standby database requires recoveryORA-01196: file 1 is inconsistent due to a failed media recovery sessionORA-01110: data file 1:'/u01/app/oracle/oradata/DB2/datafile/o1_mf_system_08oc7c38_.dbf'

8：於是關閉physical standby,在主庫上啟動system資料表空間的熱備份，拷貝相應的檔案至從庫指定位置，重新同步正常

SQL> shutdown immediateORA-01109: database not openDatabase dismounted.ORACLE instance shut down.SQL> alter tablespace system begin backup;Tablespace altered.[oracle@db1 datafile]$ pwd/u01/app/oracle/oradata/DB/datafile[oracle@db1 datafile]$ scp o1_mf_system_8vpm8hf3_.dbf root@db2:/tmpSQL> alter tablespace system end backup;Tablespace altered.

本文出自 “斬月” 部落格，謝絕轉載！