Database failure recovery due to an Oracle database power down or out of line

XX Hospital store suddenly dropped the line, resulting in database anomalies, on-site engineers toss a day, the problem is still unresolved, helpless to find us, I hope we can help restore the database.

Start report ora-00600[2131] Error
Fri Nov 06 14:50:59 2015
ALTER DATABASE MOUNT
This is instance was the Mount
Fri Nov 06 14:50:59 2015
ALTER SYSTEM SET local_listener= ' (address= (PROTOCOL=TCP) (host=192.168.4.4) (port=1521)) ' Scope=memory sid= ' Xifenfei1 ';
note:loaded Library:system
Success:diskgroup DATA was mounted
Note:dependency between database Xifenfei and DiskGroup resource Ora. DATA.DG is established
Errors in File/home/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_13221.trc (incident=191085):
Ora-00600:internal error code, arguments: [2131], [33], [32], [], [], [], [], [], [], [], [], []
Incident details in:/home/app/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_191085/xifenfei1_ora_13221_ I191085.trc
Fri Nov 06 14:51:10 2015
Use Adrci or Support Workbench to package the incident.
411.1 at my Oracle Support for error and packaging details.
ORA-600 signalled During:alter DATABASE MOUNT ...
The reason for this error is that: We are attempting to write a controlfile checkpoint progress record, but found We do not have the progress Rd generating this exception. As a result of the control file exception, we typically use a single control file once, and if you can't consider rebuilding the control file

Database instance recovery failed due to bad block (logical/physical)

Beginning crash Recovery of 2 threads
Started Redo scan
Kcrfr_rnenq:use Log NAB 393216
Kcrfr_rnenq:use Log NAB 2
Completed Redo Scan
Read 4427 KB redo, data blocks need recovery
Started Redo application at
Thread 1:logseq 5731, Block 391398
Thread 2:logseq 4252, Block 520815
Recovery of Online Redo log:thread 1 Group 2 Seq 5731 Reading mem 0
mem# 0: +data/xifenfei/onlinelog/group_2.266.835331047
Recovery of Online Redo log:thread 2 Group 8 Seq 4252 Reading Mem 0
mem# 0: +data/xifenfei/onlinelog/group_8.331.835330421
Errors in File/home/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_13770.trc (incident=197486):
Ora-00600:internal error code, arguments: [Kdxlin:psno out of Range], [], [], [], [], [], [], [], [], [], [], []
Incident Details in:/home/app/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_197486/xifenfei1_ora_13770_ I197486.trc
Fri Nov 06 15:03:09 2015
Use Adrci or Support Workbench to package the incident.
411.1 at my Oracle Support for error and packaging details.
Errors in File/home/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_13770.trc (incident=197487):
Ora-01578:oracle data Block corrupted (file # 2, Block # 65207)
Ora-01110:data file 2: ' +data/xifenfei/datafile/sysaux.257.835324753 '
Ora-10564:tablespace Sysaux
Ora-01110:data file 2: ' +data/xifenfei/datafile/sysaux.257.835324753 '
Ora-10561:block type ' TRANSACTION MANAGED INDEX block ', Data object# 81045
Ora-00600:internal error code, arguments: [Kdxlin:psno out of Range], [], [], [], [], [], [], [], [], [], [], []
Incident Details in:/home/app/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_197487/xifenfei1_ora_13770_ I197487.trc
Errors in FILE/HOME/APP/ORACLE/DIAG/RDBMS/XIFENFEI/XIFENFEI1/TRACE/XIFENFEI1_ORA_13770.TRC:
Ora-01578:oracle data Block corrupted (file # 2, Block # 65207)
Ora-01110:data file 2: ' +data/xifenfei/datafile/sysaux.257.835324753 '
Ora-10564:tablespace Sysaux
Ora-01110:data file 2: ' +data/xifenfei/datafile/sysaux.257.835324753 '
Ora-10561:block type ' TRANSACTION MANAGED INDEX block ', Data object# 81045
Ora-00600:internal error code, arguments: [Kdxlin:psno out of Range], [], [], [], [], [], [], [], [], [], [], []
Recovery of Online Redo log:thread 2 Group 3 Seq 4253 Reading mem 0
mem# 0: +data/xifenfei/onlinelog/group_3.332.835330505
Hex Dump of (file, block 62536) in trace File/home/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_ 13770.trc
Reading datafile ' +data/xifenfei/datafile/ht01.dbf ' for corruption in rdba:0x0380f448 (file, block 62536)
Reread (file, block 62536) found same corrupt data (logically corrupt)
RECOVERY of THREAD 1 stuck at block 62536 of FILE 14
Fri Nov 06 15:03:13 2015
Abort Recovery for domain 0
Aborting crash recovery due to error 1172
Errors in FILE/HOME/APP/ORACLE/DIAG/RDBMS/XIFENFEI/XIFENFEI1/TRACE/XIFENFEI1_ORA_13770.TRC:
Ora-01172:recovery of Thread 1 stuck at block 62536 of file 14
Ora-01151:use Media Recovery to recover block, restore backup if needed
Abort Recovery for domain 0
Errors in FILE/HOME/APP/ORACLE/DIAG/RDBMS/XIFENFEI/XIFENFEI1/TRACE/XIFENFEI1_ORA_13770.TRC:
Ora-01172:recovery of Thread 1 stuck at block 62536 of file 14
Ora-01151:use Media Recovery to recover block, restore backup if needed
ORA-1172 signalled During:alter DATABASE OPEN ...
View data discovery and bugs 14301592–several errors by corrupt blocks shifted by 2 bytes in buffer cache during recovery from caused by INDEX Redo apply, can be temporarily resolved through allow 1 corruption

Recovery with allow 1 corruption with Ora-07445[kdxlin] Error
Specify log: {<ret>=suggested | AUTO | CANCEL}
+data/xifenfei/onlinelog/group_3.332.835330505
Ora-00279:change 700860458 generated at 11/05/2015 21:20:15 for thread
1
Ora-00289:suggestion: +archive/xifenfei/xifenfei_1_5731_835324843.arc
Ora-00280:change 700860458 for thread 1 was in sequence #5731


Specify log: {<ret>=suggested | AUTO | CANCEL}
+data/xifenfei/onlinelog/group_2.266.835331047
Ora-00283:recovery session canceled due to errors
Ora-10562:error occurred while applying redo to data block (file# 2, block#
70104)
Ora-10564:tablespace Sysaux
Ora-01110:data file 2: ' +data/xifenfei/datafile/sysaux.257.835324753 '
Ora-10561:block type ' TRANSACTION MANAGED INDEX block ', Data object# 82289
Ora-00607:internal error occurred while making a change to a data block
Ora-00602:internal Programming exception
Ora-07445:exception Encountered:core Dump [Kdxlin () +4088] [SIGSEGV]
[ADDR:0XC] [pc:0x95fb572] [address isn't mapped to object] []


Ora-01112:media Recovery not started
Ora-07445[kdxlin () +4088] did not find a similar description, to this point, can not be a simple recovery success, only by setting the implied parameters skip instance recovery, try to Resetlog library

Resume by setting the _allow_resetlogs_corruption parameter
sql> startup pfile= '/tmp/pfile.ora ' mount;
ORACLE instance started.

Total System Global area 7315603456 bytes
Fixed Size 2267384 bytes
Variable Size 2566915848 bytes
Database buffers 4731174912 bytes
Redo buffers 15245312 bytes
Database mounted.
sql> ALTER DATABASE open resetlogs;
ALTER DATABASE open Resetlogs
*
ERROR at line 1:
Ora-01092:oracle instance terminated. Disconnection forced
Ora-00600:internal error code, arguments: [Kclchkblk_4], [0], [700869927],
[0], [700860464], [], [], [], [], [], [], []
Process id:13563
Session id:157 Serial Number:3
Alert Log Error
Fri Nov 06 19:26:39 2015
Smon:enabling Cache Recovery
Instance recovery:looking for dead threads
Instance recovery:lock domain Invalid but no dead threads
Errors in File/home/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_13563.trc (incident=319140):
Ora-00600:internal error code, arguments: [Kclchkblk_4], [0], [700869927], [0], [700860464], [], [], [], [], [], [], []
Incident Details in:/home/app/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_319140/xifenfei1_ora_13563_ I319140.trc
Use Adrci or Support Workbench to package the incident.
411.1 at my Oracle Support for error and packaging details.
Redo thread 2 internally disabled at SEQ 1 (CKPT)
Arc1:archiving disabled thread 2 sequence 1
Archived Log Entry 9956 added for thread 2 sequence 1 ID 0x0 dest 1:
Arc3:archival started
Arc0:starting ARCH Processes COMPLETE
Errors in FILE/HOME/APP/ORACLE/DIAG/RDBMS/XIFENFEI/XIFENFEI1/TRACE/XIFENFEI1_ORA_13563.TRC:
Ora-00600:internal error code, arguments: [Kclchkblk_4], [0], [700869927], [0], [700860464], [], [], [], [], [], [], []
Errors in FILE/HOME/APP/ORACLE/DIAG/RDBMS/XIFENFEI/XIFENFEI1/TRACE/XIFENFEI1_ORA_13563.TRC:
Ora-00600:internal error code, arguments: [Kclchkblk_4], [0], [700869927], [0], [700860464], [], [], [], [], [], [], []
Error happened during DB Open, shutting down database
USER (ospid:13563): Terminating the instance due to error 600
Fri Nov 06 19:26:42 2015
Instance terminated by USER, PID = 13563
ORA-1092 signalled during:alter database open resetlogs ...
OPIODR aborting process unknown ospid (13563) as a result of ORA-1092
Fri Nov 06 19:26:42 2015
Ora-1092:opitsk aborting Process
Here is a more familiar ora-600[kclchkblk_4] error, similar to the ora-600[2662] error, the SCN needs to be tuned, because the database version is 11.2.0.4, the SCN cannot be adjusted using the normal method, in the Modify control file, Oradebug, Bbed method is available for selection

Using the Oradebug method to process
Because it's an ASM environment, other methods are relatively cumbersome to handle.

[Oracle@wisetop1 ~]$ Sqlplus/as SYSDBA

Sql*plus:release 11.2.0.4.0 Production on Fri Nov 6 19:30:59 2015

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to a idle instance.

sql> startup pfile= '/tmp/pfile.ora ' mount;
ORACLE instance started.

Total System Global area 7315603456 bytes
Fixed Size 2267384 bytes
Variable Size 2566915848 bytes
Database buffers 4731174912 bytes
Redo buffers 15245312 bytes
Database mounted.
Sql> Oradebug Setmypid
Statement processed.
Sql> oradebug poke 0x06001ae70 4 0x2faf0800
Before: [06001ae70, 06001ae74) = 00000000
After: [06001ae70, 06001ae74) = 2faf0800
sql> ALTER DATABASE open;

Database altered.

To this database open successfully, the follow-up is to deal with some bad blocks of work, and recommend the client to rebuild the library logically.