Once rhel5.8 causes read-only file system ora-00354:corrupt Redo log block header

Once rhel5.8 causes read-only file system ora-00354:corrupt Redo log block header

Wed June 03:31:24 CST 2015Thread 1 advanced to log sequence 396 (LGWR switch) current  log# 3 seq# 396 mem# 0:/u01/a pp/oracle/oradata/orcl/redo03.logwed June 19:05:48 CST 2015Thread 1 advanced to log sequence 397 (LGWR switch)--------- --------> can be seen from this place: sequence 397 for current redo log current  log# 1 seq# 397 mem# 0:/u01/app/oracle/oradata/orcl/ redo01.logwed June 20:44:19 CST 2015Errors in file/u01/app/oracle/admin/orcl/bdump/orcl_j000_30071.trc:ora-07445: Exception Encountered:core dump [Kglats () +106] [SIGSEGV] [Address not mapped to object] [0x000000000] [] []ora-01403:no Data foundwed June 22:00:26 CST 2015Errors in file/u01/app/oracle/admin/orcl/bdump/orcl_j001_30285.trc:ora-07445: Exception Encountered:core dump [Kglhdiv_callback () +79] [SIGSEGV] [Address not mapped to object] [0X0000000D4] [] []thu J Un 20:28:17 CST 2015Starting ORACLE instance (normal)

Subsequent database instance restarts (man-made reboot):

Completed:alter database Mountthu June 20:28:24 CST 2015ALTER DATABASE openthu June 20:28:24 CST 2015Beginning CRAs h Recovery of 1 threads parallel recovery started with 3 Processesthu June 20:28:24 CST 2015Started redo Scanthu June 04 20:28:24 CST 2015Completed Redo Scan 1210 Redo blocks read 287 data blocks need Recoverythu June 20:28:25 CST 2015Start  Ed Redo application at Thread 1:logseq 397, block 660827Thu June 20:28:25 CST 2015Recovery of Online redo Log:thread 1 Group 1 Seq 397 Reading mem 0 mem# 0:/u01/app/oracle/oradata/orcl/redo01.logthu June 20:28:25 CST 2015Completed Redo Applicationthu June 20:28:29 CST 2015Completed crash recovery at Thread 1:logseq 397, Block 662037, SCN 1359645 287 dat  A blocks read, 287 data blocks written, 1210 redo blocks Readthu June 20:28:30 CST 2015lgwr:starting ARCH PROCESSESARC0 Started with pid=19, OS id=3609thu June 20:28:30 CST 2015arc0:archival startedarc1:archival startedlgwr:starting ARC H PROCESSES COMPLETEARC1 Started with pid=20, OS id=3611thu June 20:28:30 CST 2015Thread 1 advanced to log sequence 398 (thread open)---> has made With sequence 398, at this time, sequence 398 is current redo Logthread 1 opened in log sequence 398 current log# 2 seq# 398 mem# 0:/u01/ap P/oracle/oradata/orcl/redo02.logsuccessful Open of Redo thread 1Thu June 20:28:30 CST 2015MTTR advisory is disabled Beca Use Fast_start_mttr_target are not setthu June 20:28:30 CST 2015arc0:becoming the ' no FAL ' archarc0:becoming the ' no SR L ' Archthu June 20:28:30 CST 2015smon:enabling cache Recoverythu June 20:28:30 CST 2015arc1:becoming the heartbeat A   Rchthu June 20:28:35 CST 2015arc0:log corruption near block 385521 change 1314765 time? ---> Found redo log damaged Thu June 20:28:35 CST 2015Errors in File/u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:o Ra-00354:corrupt Redo Log block Header---> found redo log header damaged Ora-00353:log corruption near block 385521 change 1314 765 Time 06/03/2015 19:05:47ora-00312:online log 1 ThreaD 1: '/u01/app/oracle/oradata/orcl/redo01.log ' arc0:all Archive destinations made inactive due to error 354Thu June 04 20:2 8:35 CST 2015arc0:closing Local archive destination log_archive_dest_1: '/home/oracle/1_397_880950378.dbf ' (Error 354) ( ORCL) committing creation of Archivelog '/home/oracle/1_397_880950378.dbf ' (Error 354) Thu June 20:28:43 CST 2015SUCCESSF ully onlined Undo tablespace 1.Thu June 20:28:43 CST 2015smon:enabling TX recoverythu June 20:28:45 CST 2015arch:arc Hival stopped, error occurred. Would continue Retryingthu June 20:28:45 CST 2015ORACLE Instance orcl-archival errorthu June 20:28:45 CST 2015Databas E Characterset is Zhs16gbkthu June 20:28:45 CST 2015ora-16038:log 1 sequence# 397 cannot be archivedora-00354:corrupt Redo Log block Headerora-00312:online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log ' Thu June 20:28:45 CST 20 15Errors in File/u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:ora-16038:log 1 sequence# 397 cannot be archiveDora-00354:corrupt Redo Log block headerora-00312:online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log ' Openi ng with internal Resource Manager plan replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNCQMNC started with pid=21, OS id=3637thu June 20:28:49 CST 2015completed:alter DATABASE Open------->db Open. Open is open and cannot be archived. Business early evening hang. ---> At this time/u01/app/oracle/oradata/orcl/redo02.log, is sequence 398, is current redo log.

Thread 1 advanced to log sequence 399 (LGWR switch) current  log# 3 seq# 399 mem# 0:/u01/app/oracle/oradata/orcl/redo0 3.log---> At this time/u01/app/oracle/oradata/orcl/redo03.log, is sequence 399, is the current redo log.

---> Then, a log switching statement was issued in Sqlplus: alter system switch logfile; Thread 1 cannot allocate new log, sequence----> cannot log switching because sequence 397 is not archived and does not agree to be overwritten when DB is stuck.
All online logs needed archiving log# 3 seq# 399 mem# 0:/u01/app/oracle/oradata/orcl/redo03.log Thu June 04 21:2 8:31 CST 2015arc0:log corruption near block 385521 change 1314765 time? Thu June 21:28:31 CST 2015Errors in File/u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:ora-00354:corrupt redo Lo G Block Headerora-00353:log corruption near block 385521 change 1314765 time 06/03/2015 19:05:47ora-00312:online log 1 t Hread 1: '/u01/app/oracle/oradata/orcl/redo01.log ' arc0:all Archive destinations made inactive due to error 354Thu June 04 21:28:31 CST 2015arc0:closing Local archive destination log_archive_dest_1: '/home/oracle/1_397_880950378.dbf ' (Error 354) (ORCL) Arch:archival stopped, error occurred. Would continue Retryingthu June 21:28:34 CST 2015ORACLE Instance orcl-archival errorthu June 21:28:34 CST 2015ora-160 38:log 1 sequence# 397 cannot be archivedora-00354:corrupt redo log block headerora-00312:online log 1 thread 1: '/u01/ App/oracle/oradata/orcl/reDo01.log ' Thu June 21:28:34 CST 2015Errors in File/u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:ora-16038:log 1 sequence# 397 cannot be archivedora-00354:corrupt redo log block headerora-00312:online log 1 thread 1: '/u01/app/oracle /oradata/orcl/redo01.log ' Arch:archival stopped, error occurred. Would continue Retryingthu June 21:28:34 CST 2015ORACLE Instance orcl-archival errorthu June 21:28:34 CST 2015ora-160 14:log 1 sequence# 397 not archived, no available destinationsora-00312:online log 1 thread 1: '/u01/app/oracle/oradata/ Orcl/redo01.log ' Thu June 21:28:34 CST 2015Errors in File/u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:o Ra-16014:log 1 sequence# 397 not archived, no available destinationsora-00312:online log 1 thread 1: '/u01/app/oracle/or Adata/orcl/redo01.log '

Start the DB to Mount State. For processing

[[email protected] ~]$ sqlplus/as sysdbasql*plus:release 10.2.0.5.0-production on Fri June 23:14:24 2015Copyri  Ght (c) 1982, Oracle. All rights reserved.connected to an idle instance. Sql> Startup Mount;oracle instance started. Total System Global area 2147483648 bytesfixed size 2097696 bytesvariable size 536874464 byte Sdatabase buffers 1593835520 Bytesredo buffers 14675968 bytesdatabase mounted.    Sql> Set Line 200sql> select * from V$log; group# thread# sequence# BYTES members ARC STATUS first_change# first_tim-------------------------          ------------------------------------------------------------------1 1 397 536870912                1 No INACTIVE 1294462 03-jun-15 3 1 399 536870912 1 No current 1361682 04-jun-15 2 1 398 536870912 1 YES INACTIVE 1359604-jun-15sql> ALTER DATABASE clear logfile Group 1;alter database clear logfile Group 1*error at line 1:ora-00350:l OG 1 of instance ORCL (thread 1) needs to be archivedora-00312:online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo0 1.log ' sql> ALTER DATABASE clear unarchived logfile group 1;database altered.    Sql> select * from V$log; group# thread# sequence# BYTES members ARC STATUS first_change# first_tim-------------------------          ------------------------------------------------------------------1 1 0 536870912                1 YES UNUSED 1294462 03-jun-15 3 1 399 536870912 1 NO current 1361682 04-jun-15 2 1 398 536870912 1 YES INACTIVE 1359647 04-jun -15sql> Archive Log list;database log mode archive modeautomatic archival enabledarchive desti Nation/home/oracleoldesT online log sequence 0Next log sequence to archive 399Current log sequence 399sql> 

At this point the output of the alert log is as follows:

Fri June 23:20:27 CST 2015alter database Clear logfile Group 1Fri June 23:20:27 CST 2015ora-350 signalled During:alte R Database Clear logfile group 1...Fri June 23:22:30 CST 2015alter database Clear unarchived logfile Group 1Fri June 12 2 3:22:31 CST 2015warning! Clearing REDO LOG which have not BEEN archived. BACKUPS taken    before 06/04/2015 20:28:30 (change 1359647) cannot is used for RECOVERY. Clearing online log 1 of thread 1 sequence number 397Fri June 23:22:43 CST 2015completed:alter database Clear unarchive D LogFile Group 1

Once rhel5.8 causes read-only file system ora-00354:corrupt Redo log block header