One rhel5.8 file system read-only caused by ORA-00354: corruptredologblockheader

One rhel5.8 file system read-only caused by ORA-00354: corruptredologblockheader

An rhel5.8 file system read-only caused by ORA-00354: upt redo log block header

Wed Jun 03 03:31:24 CST 2015 Thread 1 advanced to log sequence 396 (LGWR switch) Current log #3 seq #396 mem #0: /u01/app/oracle/oradata/orcl/redo03.logWed Jun 03 19:05:48 CST 2015 Thread 1 advanced to log sequence 397 (LGWR switch) -----------------> you can see from this point: sequence 397 is the current redo log Current log #1 seq #397 mem #0: /u01/app/oracle/oradata/orcl/redo01.logWed Jun 03 20:44:19 CST 2015 Errors in file/u01/app/oracle/admin/orcl/bdump/orcl_j000_30071.trc: ORA-07445: exception encountered: core dump [kglats () + 106] [SIGSEGV] [Address not mapped to object] [0x000000000] [] [] ORA-01403: no data foundWed Jun 03 22:00:26 CST 2015 Errors in file/u01/app/oracle/admin/orcl/bdump/orcl_j001_30285.trc: ORA-07445: exception encountered: core dump [kglhdiv_callback () + 79] [SIGSEGV] [Address not mapped to object] [0x0000000D4] [] [] Thu Jun 04 20:28:17 CST 2015 Starting ORACLE instance (normal)

Then the database instance is restarted (manually restart ):

Completed: alter database MOUNTThu Jun 04 20:28:24 CST 2015 alter database OPENThu Jun 04 20:28:24 CST 2015 Beginning crash recovery of 1 threads parallel recovery started with 3 processesThu Jun 04 20:28:24 CST 2015 Started redo scanThu Jun 04 20:28:24 CST 2015 Completed redo scan 1210 redo blocks read, 287 data blocks need recoveryThu Jun 04 20:28:25 CST 2015 Started redo application at Thread 1: logs Eq 397, block 660827Thu Jun 04 20:28:25 CST 2015 Recovery of Online Redo Log: Thread 1 Group 1 Seq 397 Reading mem 0 Mem #0: /u01/app/oracle/oradata/orcl/redo01.logThu Jun 04 20:28:25 CST 2015 Completed redo applicationThu Jun 04 20:28:29 CST 2015 Completed crash recovery at Thread 1: logseq 397, block 662037, scn 1359645 287 data blocks read, 287 data blocks written, 1210 redo blocks readThu Jun 04 20:28:30 CST 2015 LGWR: starting arch PROCESSESARC0 started with pid = 19, OS id = 3609Thu Jun 04 20:28:30 CST 2015ARC0: Archival startedARC1: Archival restart: STARTING ARCH PROCESSES COMPLETEARC1 started with pid = 20, OS id = 3611Thu Jun 04 20:28:30 CST 2015 Thread 1 advanced to log sequence 398 (thread open) ---> you have used sequence 398, sequence 398 is current redo logThread 1 opened at log sequence 398 Cur Rent log #2 seq #398 mem #0: /u01/app/oracle/oradata/orcl/empty open of redo thread 1Thu Jun 04 20:28:30 CST 2015 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not setThu Jun 04 20:28:30 CST 2015ARC0: becoming the 'no fal' ARCHARC0: Becoming the 'no srl' ARCHThu Jun 04 20:28:30 CST 2015 SMON: enabling cache recoveryThu Jun 04 20:28:30 CST 2015ARC1: Becoming the heartbeat ARCHThu Jun 04 20:28:35 CST 2015ARC0: Log upload uption near block 385521 change 1314765 time? ---> Found that the redo log is damaged Thu Jun 04 20:28:35 CST 2015 Errors in file/u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc: ORA-00354: upload upt redo log block header ---> found that the redo log header is damaged ORA-00353: log upload uption near block 385521 change 1314765 time 06/03/2015 19: 05: 47ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' ARC0: All Archive destinations made inactive due to error 354Thu Jun 04 20:28:35 CST 2015ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/home/oracle/1_397_880950378.dbf' (error 354) (orcl) committing creation of archivelog '/home/oracle/zookeeper _880950378.dbf' (error 354) Thu Jun 04 20:28:43 CST 2015 Successfully onlined Undo Tablespace 1.Thu Jun 04 20:28:43 CST 2015 SMON: enabling tx recoveryThu Jun 04 20:28:45 CST 2015 ARCH: Archival stopped, error occ Urred. will continue retryingThu Jun 04 20:28:45 CST 2015 ORACLE Instance orcl-Archival ErrorThu Jun 04 20:28:45 CST 2015 Database Characterset is ZHS16GBKThu Jun 04 20:28:45 CST 2015ORA-16038: log 1 sequence #397 cannot be archivedORA-00354: upt redo log block headerORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' Thu Jun 04 20:28:45 CST 2015 Errors in file/u01/app/o Racle/admin/orcl/bdump/orcl_arc0_3609.trc: ORA-16038: log 1 sequence #397 cannot be archivedORA-00354: upt redo log block headerORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' Opening with internal Resource Manager plan replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNCQMNC started with pid = 21, OS id = 363 7Thu Jun 04 20:28:49 CST 2015 Completed: alter database open -------> db open. If open is open and cannot be archived, the business will be hang sooner or later. ---> At this time,/u01/app/oracle/oradata/orcl/redo02.log is sequence 398, which is the current redo log.

Thread 1 advanced to log sequence 399 (LGWR switch) Current log #3 seq #399 mem #0: /u01/app/oracle/oradata/orcl/redo03.log ---> at this time,/u01/app/oracle/oradata/orcl/redo03.log is sequence 399, which is the current redo log.

---> Then, sqlplus sends the log switch statement: alter system switch logfile; Thread 1 cannot allocate new log, sequence 400 ----> no log switch is allowed, because sequence 397 is not archived and cannot be overwritten, db hang is in place. All online logs needed archiving Current log #3 seq #399 mem #0:/u01/app/oracle/oradata/orcl/redo03.log Thu Jun 04 21:28:31 CST 2015ARC0: log uption near block 385521 change 1314765 time? Thu Jun 04 21:28:31 CST 2015 Errors in file/u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc: ORA-00354: Drawing upt redo log block headerORA-00353: log partition uption near block 385521 change 1314765 time 06/03/2015 19: 05: 47ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log 'arc0: all Archive destinations made inactive due to error 354Thu Jun 04 21:28:31 CST 2015ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/home/oracle/hangzhou' (error 354) (orcl) ARCH: archival stopped, error occurred. will continue retryingThu Jun 04 21:28:34 CST 2015 ORACLE Instance orcl-Archival ErrorThu Jun 04 21:28:34 CST 2015ORA-16038: log 1 sequence #397 cannot be archivedORA-00354: Could upt redo log block headerORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' Thu Jun 04 21:28:34 CST 2015 Errors in file/u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc: ORA-16038: log 1 sequence #397 cannot be archivedORA-00354: upt redo log block headerORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' ARCH: Archival stopped, error occurred. will continue retryingThu Jun 04 21:28:34 CST 2015 ORACLE Instance orcl-Archival ErrorThu Jun 04 21:28:34 CST 2015ORA-16014: log 1 sequence #397 not archived, no available destinationsORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' Thu Jun 04 21:28:34 CST 2015 Errors in file/u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc: ORA-16014: log 1 sequence #397 not archived, no available destinationsORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'

Start the database to the mount status for processing.

[oracle@lei1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 10.2.0.5.0 - Production on Fri Jun 12 23:14:24 2015Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.Connected to an idle instance.SQL> startup mount;ORACLE instance started.Total System Global Area 2147483648 bytesFixed Size                  2097696 bytesVariable Size             536874464 bytesDatabase Buffers         1593835520 bytesRedo Buffers               14675968 bytesDatabase mounted.SQL> set line 200SQL> select * from v$log;    GROUP#    THREAD#  SEQUENCE#      BYTES    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------         1          1        397  536870912          1 NO  INACTIVE               1294462 03-JUN-15         3          1        399  536870912          1 NO  CURRENT                1361682 04-JUN-15         2          1        398  536870912          1 YES INACTIVE               1359647 04-JUN-15SQL> alter database clear logfile group 1;alter database clear logfile group 1*ERROR at line 1:ORA-00350: log 1 of instance orcl (thread 1) needs to be archivedORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'SQL> alter database clear unarchived logfile group 1;Database altered.SQL> select * from v$log;    GROUP#    THREAD#  SEQUENCE#      BYTES    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------         1          1          0  536870912          1 YES UNUSED                 1294462 03-JUN-15         3          1        399  536870912          1 NO  CURRENT                1361682 04-JUN-15         2          1        398  536870912          1 YES INACTIVE               1359647 04-JUN-15SQL> archive log list;Database log mode              Archive ModeAutomatic archival             EnabledArchive destination            /home/oracleOldest online log sequence     0Next log sequence to archive   399Current log sequence           399SQL>

The output of the alert Log is as follows:

Fri Jun 12 23:20:27 CST 2015alter database clear logfile group 1Fri Jun 12 23:20:27 CST 2015ORA-350 signalled during: alter database clear logfile group 1...Fri Jun 12 23:22:30 CST 2015alter database clear unarchived logfile group 1Fri Jun 12 23:22:31 CST 2015WARNING! CLEARING REDO LOG WHICH HAS NOT BEEN ARCHIVED. BACKUPS TAKEN    BEFORE 06/04/2015 20:28:30 (CHANGE 1359647) CANNOT BE USED FOR RECOVERY.Clearing online log 1 of thread 1 sequence number 397Fri Jun 12 23:22:43 CST 2015Completed: alter database clear unarchived logfile group 1