Oracle error ORA-00600 [2131] bad block storage offline recovery case

O & m dba reflects production Oracle database storage offline Oracle database faults cannot start error: ORA-00600 [2131] ORA-07445 [kdxlin] Errors
1. Startup reports ORA-00600 [2131] error
Fri Nov 06 14:50:59 2015
ALTER DATABASE MOUNT
This instance was first to mount
Fri Nov 06 14:50:59 2015
Alter system set local_listener = '(ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.4.4) (PORT = 1521)' SCOPE = memory sid = 'oracleplus1 ';
NOTE: Loaded library: System
SUCCESS: diskgroup DATA was mounted
NOTE: dependency between database oracleplus and diskgroup resource ora. DATA. dg is established
Errors in file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13221.trc (incident = 191085 ):
ORA-00600: internal error code, arguments: [2131], [33], [32], [], [], [], [], [], [], [], [], [], []
Incident details in:/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/incident/incdir_191085/oracleplus%ora_1322%i19%5.trc
Fri Nov 06 14:51:10 2015
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORA-600 signalled during: alter database mount...
This error occurs because: We are attempting to write a controlfile checkpoint progress record, but find we do not have the progress record generating this exception. due to an exception in the control file, we usually use a single control file for one attempt. If neither of them can be considered, re-create the control file.
2. Failed database instance recovery due to bad blocks (logical/physical)
Beginning crash recovery of 2 threads
Started redo scan
Kcrfr_rnenq: use logs nab 393216
Kcrfr_rnenq: use log nab 2
Completed redo scan
Read 4427 KB redo, 500 data blocks need recovery
Started redo application
Thread 1: logseq 5731, block 391398
Thread 2: logseq 4252, block 520815
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5731 Reading mem 0
Mem #0: + DATA/oracleplus/onlinelog/group_2.266.835331047
Recovery of Online Redo Log: Thread 2 Group 8 Seq 4252 Reading mem 0
Mem #0: + DATA/oracleplus/onlinelog/group_8.331.838530421
Errors in file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc (incident = 197486 ):
ORA-00600: internal error code, arguments: [kdxlin: psno out of range], [], [], [], [], [], [], [], [], [], [], []
Incident details in:/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/incident/incdir_197133/oracleplus1_ora_13770_i197486.trc
Fri Nov 06 15:03:09 2015
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc (incident = 197487 ):
ORA-01578: ORACLE data block upted (file #2, block #65207)
ORA-01110: data file 2: '+ DATA/oracleplus/datafile/sysaux.257.820.24753'
ORA-10564: tablespace SYSAUX.
ORA-01110: data file 2: '+ DATA/oracleplus/datafile/sysaux.257.820.24753'
ORA-10561: block type 'transactional managed index block', data object #81045
ORA-00600: internal error code, arguments: [kdxlin: psno out of range], [], [], [], [], [], [], [], [], [], [], []
Incident details in:/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/incident/incdir_197487/oracleplus1_ora_13770_i197487.trc
Errors in file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc:
ORA-01578: ORACLE data block upted (file #2, block #65207)
ORA-01110: data file 2: '+ DATA/oracleplus/datafile/sysaux.257.820.24753'
ORA-10564: tablespace SYSAUX.
ORA-01110: data file 2: '+ DATA/oracleplus/datafile/sysaux.257.820.24753'
ORA-10561: block type 'transactional managed index block', data object #81045
ORA-00600: internal error code, arguments: [kdxlin: psno out of range], [], [], [], [], [], [], [], [], [], [], []
Recovery of Online Redo Log: Thread 2 Group 3 Seq 4253 Reading mem 0
Mem #0: + DATA/oracleplus/onlinelog/group_3.332.81_30505
Hex dump of (file 14, block 62536) in trace file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc
Reading datafile '+ DATA/oracleplus/datafile/ht01.dbf' for upload uption at rdba: 0x0380f448 (file 14, block 62536)
Reread (file 14, block 62536) found same Upload UPT data (logically Upload UPT)
Recovery of thread 1 stuck at block 62536 of file 14
Fri Nov 06 15:03:13 2015
Abort recovery for domain 0
Aborting crash recovery due to error 1172
Errors in file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc:
ORA-01172: recovery of thread 1 stuck at block 62536 of file 14
ORA-01151: use media recovery to recover block, restore backup if needed
Abort recovery for domain 0
Errors in file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc:
ORA-01172: recovery of thread 1 stuck at block 62536 of file 14
ORA-01151: use media recovery to recover block, restore backup if needed
ORA-1172 signalled during: alter database open...
View information and Bug 14301592-Several errors by minute UPT blocks shifted by 2 bytes in buffer cache during recovery caused by INDEX redo apply, which can be temporarily solved through ALLOW 1 commit uption
3. Use ALLOW 1 resume uption for recovery with the ORA-07445 [kdxlin] error
Specify log: {= suggested | filename | AUTO | CANCEL}
+ DATA/oracleplus/onlinelog/group_3.332.81_30505
ORA-00279: change 700860458 generated at 21:20:15 needed for thread
1
ORA-00289: suggestion: + ARCHIVE/oracleplus/oracleplus_00005731_8000024843.arc
ORA-00280: change 700860458 for thread 1 is in sequence #5731

Specify log: {= suggested | filename | AUTO | CANCEL}
+ DATA/oracleplus/onlinelog/group_2.266.835331047
ORA-00283: recovery session canceled due to errors
ORA-10562: Error occurred while applying redo to data block (file #2, block #
70104)
ORA-10564: tablespace SYSAUX.
ORA-01110: data file 2: '+ DATA/oracleplus/datafile/sysaux.257.820.24753'
ORA-10561: block type 'transactional managed index block', data object #82289
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [kdxlin () + 4088] [SIGSEGV]
[ADDR: 0xC] [PC: 0x95FB572] [Address not mapped to object] []

ORA-01112: media recovery not started
ORA-07445 [kdxlin () + 4088] does not find similar instructions, at this step, you cannot simply recover successfully, you can only skip instance recovery by setting implicit parameters, try resetlog Library

Resume the restoration by setting the _ allow_resetlogs_resume uption parameter.
SQL> startup pfile = '/tmp/pfile. ora' mount;
ORACLE instance started.

Total System Global Area 7315603456 bytes
Fixed Size 2267384 bytes
Variable Size 2566915848 bytes
Database Buffers 4731174912 bytes
Redo Buffers 15245312 bytes
Database mounted.
SQL> alter database open resetlogs;
Alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [700869927],
[0], [700860464], [], [], [], [], [], [], []
Process ID: 13563
Session ID: 157 Serial number: 3
Alert Log error
Fri Nov 06 19:26:39 2015
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Errors in file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13563.trc (incident = 319140 ):
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [700869927], [0], [700860464], [], [], [], [], [], [], []
Incident details in:/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/incident/incdir_319140/oracleplus1_ora_13563_i319140.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Redo thread 2 internally disabled at seq 1 (CKPT)
ARC1: Archiving disabled thread 2 sequence 1
Archived Log entry 9956 added for thread 2 sequence 1 ID 0x0 dest 1:
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Errors in file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13563.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [700869927], [0], [700860464], [], [], [], [], [], [], []
Errors in file/home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13563.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [700869927], [0], [700860464], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 13563): terminating the instance due to error 600
Fri Nov 06 19:26:42 2015
Instance terminated by USER, pid = 13563
ORA-1092 signalled during: alter database open resetlogs...
Opiodr aborting process unknown ospid (13563) as a result of ORA-1092
Fri Nov 06 19:26:42 2015
ORA-1092: opitsk aborting process
Here is the familiar ora-600 [kclchkblk_4] error, and ora-600 [2662] error similar, need to adjust scn, because the database version is 11.2.0.4, unable to use the general method to adjust scn, in the modify control file, the oradebug and bbed methods are available.
4. Use oradebug method to handle
Because it is an asm environment, other methods are relatively difficult to handle.
[Oracle @ wisetop1 ~] $ Sqlplus/as sysdba

SQL * Plus: Release 11.2.0.4.0 Production on Fri Nov 6 19:30:59 2015

Copyright (c) 1982,201 3, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup pfile = '/tmp/pfile. ora' mount;
ORACLE instance started.

Total System Global Area 7315603456 bytes
Fixed Size 2267384 bytes
Variable Size 2566915848 bytes
Database Buffers 4731174912 bytes
Redo Buffers 15245312 bytes
Database mounted.
SQL> oradebug setmypid
Statement processed.
SQL> oradebug poke 0x06001AE70 4 0x2FAF0800
BEFORE: [06001AE70, 06001AE74) = 00000000
AFTER: [06001AE70, 06001AE74) = 2FAF0800
SQL> alter database open;

Database altered.

At this point, the database is successfully opened, and some bad blocks will be processed in the future. We recommend that you rebuild the database logically.