Oracle Database ORA-600 Error Problem solving method

Friend asked me to help deal with a library that can not open, open alert log a look, dumbfounded, which is a variety of ORA-600 error everything, was tossing enough miserable
Reboot after failure, unable to start main performance in block bad blocks, caused by various ORA-600 errors
Mon Mar 02 16:09:27 2015
ALTER DATABASE OPEN
Beginning crash recovery of 1 threads
Parallel recovery started with processes
Started Redo scan
Completed Redo Scan
Read 962 KB redo, 256 data blocks need recovery
Started Redo application at
Thread 1:logseq 726, block 37343
Recovery of Online Redo log:thread 1 Group 3 Seq 726 Reading mem 0
mem# 0:/u01/app/oracle/oradata/oa/redo03.log
Mon Mar 02 16:09:27 2015
RECOVERY of THREAD 1 stuck at block 1673 of FILE 3
Completed Redo Application of 0.27MB
Mon Mar 02 16:09:27 2015
RECOVERY of THREAD 1 stuck at block 3104 of FILE 3
Mon Mar 02 16:09:27 2015
RECOVERY of THREAD 1 stuck at block 3613 of FILE 3
Mon Mar 02 16:09:28 2015
RECOVERY of THREAD 1 stuck at block 272 of FILE 3
Mon Mar 02 16:09:28 2015
RECOVERY of THREAD 1 stuck at block 2512 of FILE 3
Hex Dump of (file 2, block 92889) in trace file/u01/app/oracle/diag/rdbms/oa/oa/trace/oa_dbw2_4158.trc
Corrupt block relative dba:0x00816ad9 (file 2, block 92889)
Bad header found during preparing blocks for write
Data in bad block:
type:0 format:0 rdba:0x6ad90000
Last Change scn:0x0000.00c6a052 seq:0x1 flg:0x00
spare1:0x6 Spare2:0xa2 spare3:0x5d7e
Consistency value in tail:0xa0520001
Check value in block header:0x0
Block checksum disabled
Mon Mar 02 16:09:28 2015
Errors in File/u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p007_4196.trc (incident=3833):
Ora-00600:internal error code, arguments: [4502], [1], [], [], [], [], [], [], [], [], [], []
Mon Mar 02 16:09:28 2015
Errors in File/u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p013_4208.trc (incident=3881):
Ora-00600:internal error code, arguments: [2037], [4259067], [4244307968], [159], [243], [0], [2162032704], [100728832], [], [], [], []
Slave exiting with ORA-1172 exception
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_P009_4200.TRC:
Ora-01172:recovery of Thread 1 stuck at block 3613 of file 3
Ora-01151:use Media Recovery to recover block, restore backup if needed
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_P001_4184.TRC:
Ora-01172:recovery of Thread 1 stuck at block 2512 of file 3
Ora-01151:use Media Recovery to recover block, restore backup if needed
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_P021_4224.TRC:
Ora-10388:parallel Query server interrupt (failure)
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_P021_4224.TRC:
Errors in File/u01/app/oracle/diag/rdbms/oa/oa/trace/oa_dbw2_4158.trc (incident=3697):
Ora-00600:internal error code, arguments: [Kcbzpbuf_1], [4], [1], [], [], [], [], [], [], [], [], []
Incident details in:/u01/app/oracle/diag/rdbms/oa/oa/incident/incdir_3697/oa_dbw2_4158_i3697.trc
Exception [TYPE:SIGSEGV, Si_kernel (general_protection)] [addr:0x0] [Pc:0xd2ddb7, Kcbs_shrink_pool () +705] [flags:0x0, COUNT:1]
Errors in File/u01/app/oracle/diag/rdbms/oa/oa/trace/oa_mman_4152.trc (incident=3673):
Ora-07445:exception Encountered:core Dump [Kcbs_shrink_pool () +705] [SIGSEGV] [addr:0x0] [PC:0XD2DDB7] [Si_kernel ( General_protection)] []
Incident details in:/u01/app/oracle/diag/rdbms/oa/oa/incident/incdir_3673/oa_mman_4152_i3673.trc
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_DBW2_4158.TRC:
Mon Mar 02 16:09:34 2015
Instance terminated by DBW2, PID = 4158
Add new error after second reboot ora-00600[17182]
Mon Mar 02 16:39:50 2015
Errors in File/u01/app/oracle/diag/rdbms/oa/oa/trace/oa_p002_4321.trc (incident=4993):
Ora-00600:internal error code, arguments: [17182], [0x7f548c2bdba8], [], [], [], [], [], [], [], [], [], []
After some recovery processing, the error in the log
It is mainly reflected in the incomplete recovery, and it should be a series of hints for renaming the redo or redo the lock of the head.
Beginning crash recovery of 1 threads
Started Redo scan
Completed Redo Scan
Read 962 KB redo, 256 data blocks need recovery
Started Redo application at
Thread 1:logseq 726, block 37343
Recovery of Online Redo log:thread 1 Group 3 Seq 726 Reading mem 0
mem# 0:/u01/app/oracle/oradata/oa/redo03.log
RECOVERY of THREAD 1 stuck at block 1673 of FILE 3
Aborting crash recovery due to error 1172
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_6644.TRC:
Ora-01172:recovery of Thread 1 stuck at block 1673 of File 3
Ora-01151:use Media Recovery to recover block, restore backup if needed
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_6644.TRC:
Ora-01172:recovery of Thread 1 stuck at block 1673 of File 3
Ora-01151:use Media Recovery to recover block, restore backup if needed
ORA-1172 signalled during:alter database open ...
Tue Mar 03 11:17:59 2015
Sweep [inc][17178]: Completed
Sweep [inc][17177]: Completed
Sweep [inc2][17178]: Completed
Tue Mar 03 11:18:00 2015
ALTER database RECOVER database until cancel
Media Recovery Start
Started Logmerger process
Parallel Media Recovery started with slaves
ORA-279 signalled during:alter database RECOVER database until cancel ...
ALTER DATABASE RECOVER CONTINUE DEFAULT
Tue Mar 03 11:18:06 2015
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_PR00_6701.TRC:
Ora-00266:name of archived log file needed
ORA-266 signalled During:alter DATABASE RECOVER CONTINUE DEFAULT ...
ALTER DATABASE RECOVER CANCEL
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_PR00_6701.TRC:
Ora-01547:warning:recover succeeded but OPEN Resetlogs would get error below
Ora-01194:file 1 needs more recovery to be consistent
Ora-01110:data file 1: '/U01/APP/ORACLE/ORADATA/OA/SYSTEM01.DBF '
Slave exiting with ORA-1547 exception
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_PR00_6701.TRC:
Ora-01547:warning:recover succeeded but OPEN Resetlogs would get error below
Ora-01194:file 1 needs more recovery to be consistent
Ora-01110:data file 1: '/U01/APP/ORACLE/ORADATA/OA/SYSTEM01.DBF '
ORA-10879 signalled During:alter DATABASE RECOVER CANCEL ...
Tue Mar 03 11:18:06 2015
Checker Run found 4 new persistent data failures
Tue Mar 03 11:18:13 2015
ALTER DATABASE open Resetlogs
Resetlogs is being done without consistancy checks. This could result
In a corrupted database. The database should be recreated.
Resetlogs after incomplete recovery UNTIL change 12986989
Resetting resetlogs activation ID 3278679642 (0xc36cae5a)
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_6644.TRC:
Ora-00367:checksum Error in log file header
Ora-00322:log 1 of Thread 1 is isn't current copy
Ora-00312:online Log 1 Thread 1: '/u01/app/oracle/oradata/oa/redo01.log '
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_6644.TRC:
Another step toss, added _allow_resetlogs_corruption= True after the database reported ora-600[2662]
Tue Mar 03 11:19:26 2015
Smon:enabling Cache Recovery
Errors in File/u01/app/oracle/diag/rdbms/oa/oa/trace/oa_ora_6864.trc (incident=18195):
Ora-00600:internal error code, arguments: [2662], [0], [13007002], [0], [13016626], [4194545], [], [], [], [], [], []
Incident details in:/u01/app/oracle/diag/rdbms/oa/oa/incident/incdir_18195/oa_ora_6864_i18195.trc
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_6864.TRC:
Ora-00704:bootstrap Process Failure
Ora-00704:bootstrap Process Failure
Ora-00600:internal error code, arguments: [2662], [0], [13007002], [0], [13016626], [4194545], [], [], [], [], [], []
Error 704 happened during DB Open, shutting down database
USER (ospid:6864): Terminating the instance due to error 704
Instance terminated by USER, PID = 6864
ORA-1092 signalled during:alter database open ...
OPIODR aborting process unknown ospid (6864) as a result of ORA-1092
Tue Mar 03 11:19:29 2015
Ora-1092:opitsk aborting Process
Further toss, can be seen to undo has been its offline, not normal access, resulting in the system reported ORA-704 and ORA-00376
Wed Mar 04 21:10:58 2015
Smon:enabling Cache Recovery
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_17074.TRC:
Ora-00704:bootstrap Process Failure
Ora-00604:error occurred at recursive SQL level 2
Ora-00376:file 3 cannot is read at this time
Ora-01110:data file 3: '/U01/APP/ORACLE/ORADATA/OA/UNDOTBS01.DBF '
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_17074.TRC:
Ora-00704:bootstrap Process Failure
Ora-00604:error occurred at recursive SQL level 2
Ora-00376:file 3 cannot is read at this time
Ora-01110:data file 3: '/U01/APP/ORACLE/ORADATA/OA/UNDOTBS01.DBF '
Error 704 happened during DB Open, shutting down database
USER (ospid:17074): Terminating the instance due to error 704
Instance terminated by USER, PID = 17074
ORA-1092 signalled during:alter database open ...
OPIODR aborting process unknown ospid (17074) as a result of ORA-1092
Wed Mar 04 21:11:00 2015
Ora-1092:opitsk aborting Process
The Oracle database exception Recovery Check script (Oracle Recovery Check) results are shown in the attachment (xifenfei_db_recover_20150304), where you know the Undo Don't know how to toss the data file SCN larger and also offline,
The database SCN is adjusted by some column methods (bbed, implied parameters, etc.), and the database is forced to start, reporting the following error
Wed Mar 04 22:50:23 2015
Smon:enabling Cache Recovery
ORA-01555 caused by SQL statement below (SQL ID:3NKD3G3JU5PH1, scn:0x0000.4000003e):
Select Obj#,type#,ctime,mtime,stime, status, dataobj#, flags, oid$, Spare1, spare2 from obj$ where Owner#=:1 and name=:2 a nd namespace=:3 and Remoteowner is null and linkname are null and subname is NULL
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_17807.TRC:
Ora-00704:bootstrap Process Failure
Ora-00604:error occurred at recursive SQL level 2
Ora-01555:snapshot too old:rollback segment number with name "_syssmu10_3550978943$" too small
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_17807.TRC:
Ora-00704:bootstrap Process Failure
Ora-00604:error occurred at recursive SQL level 2
Ora-01555:snapshot too old:rollback segment number with name "_syssmu10_3550978943$" too small
Error 704 happened during DB Open, shutting down database
USER (ospid:17807): Terminating the instance due to error 704
Instance terminated by USER, PID = 17807
ORA-1092 signalled during:alter database open resetlogs ...
OPIODR aborting process unknown ospid (17807) as a result of ORA-1092
According to experience, the error is suspected to be that the file head SCN is not large enough, block delay cleanup led to further increase the SCN attempt, the last is still ora-00704/ora-00604/ora-01555 error
Wed Mar 04 22:50:23 2015
Smon:enabling Cache Recovery
ORA-01555 caused by SQL statement below (SQL ID:3NKD3G3JU5PH1, scn:0x0000.4000003e):
Select Obj#,type#,ctime,mtime,stime, status, dataobj#, flags, oid$, Spare1, spare2 from obj$ where Owner#=:1 and name=:2 a nd namespace=:3 and Remoteowner is null and linkname are null and subname is NULL
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_17807.TRC:
Ora-00704:bootstrap Process Failure
Ora-00604:error occurred at recursive SQL level 2
Ora-01555:snapshot too old:rollback segment number with name "_syssmu10_3550978943$" too small
Errors in FILE/U01/APP/ORACLE/DIAG/RDBMS/OA/OA/TRACE/OA_ORA_17807.TRC:
Ora-00704:bootstrap Process Failure
Ora-00604:error occurred at recursive SQL level 2
Ora-01555:snapshot too old:rollback segment number with name "_syssmu10_3550978943$" too small
Error 704 happened during DB Open, shutting down database
USER (ospid:17807): Terminating the instance due to error 704
Instance terminated by USER, PID = 17807
ORA-1092 signalled during:alter database open resetlogs ...
OPIODR aborting process unknown ospid (17807) as a result of ORA-1092
According to experience, it is difficult to solve the problem on the SCN, and do 10046 and Errorstack analysis on its start-up process.
Parsing in CURSOR #3 len=202 dep=2 uid=0 oct=3 lid=0 tim=1425481940448439 hv=3819099649 ad= ' 64ff91af8 ' sqlid= ' 3nkd3g3ju5ph 1 '
Select Obj#,type#,ctime,mtime,stime, status, dataobj#, flags, oid$, Spare1, spare2 from obj$ where Owner#=:1 and name=:2 a nd namespace=:3 and Remoteowner is null and linkname are null and subname is NULL
End of STMT
PARSE #3: c=1000,e=334,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=4,plh=0,tim=1425481940448439
Binds #3:
Bind#0
oacdty=02 mxl=22 mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
KXSBBBFP=7F5B3253A6F0 bln=22 avl=01 flg=05
Value=0
Bind#1
Oacdty=01 mxl=32 (a) mxlc=00 mal=00 scl=00 pre=00
Oacflg=18 fl2=0001 frm=01 csi=852 siz=32 off=0
Kxsbbbfp=7f5b3253a6b8 bln=32 avl=06 flg=05
Value= "props$"
Bind#2
oacdty=02 mxl=22 mxlc=00 mal=00 scl=00 pre=00
oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=7f5b3253a688 bln=24 avl=02 flg=05
Value=1
EXEC #3: c=0,e=640,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=4,plh=2853959010,tim=1425481940449147
Wait #3: nam= ' db file sequential read ' ela= 5 file#=1 block#=345 Blocks=1 obj#=37
Wait #3: nam= ' db file sequential read ' ela= 4 file#=1 block#=44528 Blocks=1 obj#=37
Wait #3: nam= ' db file sequential read ' ela= 3 file#=1 block#=5505 Blocks=1 obj#=37

2015-03-04 23:12:20.450
Dbkeddefdump (): Starting a non-incident diagnostic dump (flags=0x0, level=3, mask=0x0)
-----Error Stack Dump-----
Ora-00604:error occurred at recursive SQL level 2
Ora-01555:snapshot too old:rollback segment number with name "_syssmu10_3550978943$" too small
-----Current SQL Statement for this session (SQL_ID=G64R07V2JN8NQ)-----
SELECT NULL from props$ WHERE name= ' Bootstrap_upgrade_error '
Here you can see that the database needs to execute a select NULL from props$ where name= ' bootstrap_upgrade_error ' statement when it is started, and the statement recursively calls the Select Obj#,type#,ctime , Mtime,stime, status, dataobj#, flags, oid$, Spare1, spare2 from obj$ where Owner#=:1 and Name=:2 and REM Oteowner is null and linkname are null and subname is NULL statement. Since this is done by some means of querying the select NULL from props$ WHERE name= ' bootstrap_upgrade_error ' statement when the database is started, the database started successfully.

Knowledge points Supplement
ORA-600 [4502] [a]
Arg [A] ITL entry with a lock count
Meaning:during ITL cleanout We clear all row locks but the ITL entry
Still thinks there is a uncleared lock. IE:ITL has a locked
Row but there are no locked rows in the block
The general meaning is that the database found that the ITL of undo has been cleared, but the ITL in block is still present, resulting in ora-600[4502], causing the problem in addition to the bug mainly due to the bad block

ORA-600 [2037] [A] [b] [d] [E] [F] [G]
ARG [A] relative Data block Address (RDBA), the redo vector is for
ARG [b] The block format
Arg 1 Rdba in the block itself
ARG [d] The block type
ARG [E] the sequence number
Arg [F] Flags, if set
ARG [G] The return value from the Block Head/tail checker.
DESCRIPTION:
During recovery We are examining a blocks to ensure this it is not
Corrupt prior to applying no change vectors.
The block has failed this check and this exception is raised
The general meaning is that during the recovery process, the block is being checked to make sure it does not damage before applying any change vectors. If the check fails to exclude the exception ora-600[2037], the problem is caused by bad blocks except for bugs

ORA-600 [Kcbzpbuf_1],[a],[b]
ARG [A] corruption reason
Arg [b] Calculate checksum flag

Corruption reason:
#define Kcbh_good 0/* block is valid * *
#define Kcbh_zero 1/* block header is entirely ZERO on disk */
#define KCBH_BROKEN 2/* Corruption could is from a partial disk write * * *
#define KCBH_CHKVAL 3/* The check value for the "Block failed * *
#define KCBH_CORRUPT 4/* is the "wrong block" or "not" a data block * *
#define KCBH_ZERONG 5/* All zero blocks and it is not allowed * *

Calculate Checksum flag:
The possible values are 1 (Generate checksum-db_block_checksum is Enabled-default value)
0 (does not generate Checksum-db_block_checksum=false)
Kcbzpbuf_1 is the source function of the error

ORA-600 [17182] [A] [b] [d] [E]
description:       
  Oracle has dete CTED That's magic number in a memory chunk header has been.
  IS-a heap (in memory) corruption and there is no underlying data corruption.
  The error may occur in the "the" process specific heaps
  (the call heap, PGA heap, or session Heap) or in the shared heap (SGA).
1
ORACLE discovers that important blocks in memory are reset, but there is no underlying data corruption, most of which is related to data block or memory corruption.
<br>
ORA-600 [4552] [A] [b] 1 [d] [E]
1
DESCRIPTION:
  This assertion are raised because we are trying to unlock the rows in a
  blocks, but re Ceive an incorrect block type. The
  The second argument is the block type received.
Oracle attempted to unlock a row but received an incorrect data block type, ARG [b] is the data block type received

ORA-600 [2662] [A] [b] [d] [E]
DESCRIPTION:
 
  A data block SCN are ahead of the current SCN.
&NBSP
  The ORA-600 [2662] occurs when a SCN are compared to the dependent SCN
  stored in a UGA vari Able.
 
  If The SCN is less than dependent SCN then we signal the ORA-600 [2662]
  Internal err Or.
 
ARGUMENTS:
  ARG [a]  current SCN WRAP
  ARG [b]  current SCN BASE
  Arg 1  dependent SCN WRAP
  ARG [d]  dependent SCN BASE
  ARG [e]  Where present this is The DBA where the dependent SCN came from. The main implication of the
is that the Oracle header SCN is smaller than a block dependent SCN and this problem occurs