Customer database exception (ORA-600 kcbz_check_objd_typ_1), let's remotely handle the analysis
ORA-600 kcbz_check_objd_typ_1 exception
Mon Aug 8 12:19:28 2016
Completed: ALTER DATABASE OPEN
Mon Aug 8 12:19:29 2016
Db_recovery_file_dest_size of 20480 MB is 0.00% used. This is
User-specified limit on the amount of space that will be used by this
Database for recovery-related files, and does not reflect the amount
Space available in the underlying filesystem or ASM diskgroup.
Mon Aug 8 12:19:33 2016
Errors in file/home/oracle/admin/RT/bdump/rt_smon_1514.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_1], [0], [0], [1], [], [], [], [], []
Mon Aug 8 12:20:21 2016
Shutting down archive processes
Mon Aug 8 12:20:26 2016
ARCH shutting down
ARC3: Archival stopped
Mon Aug 8 13:12:25 2016
Thread 1 advanced to log sequence 13804
Current log #3 seq #13804 mem #0:/home/oracle/product/10.2.0/oradata/RT/redo03a. log
Mon Aug 8 14:01:37 2016
Thread 1 advanced to log sequence 13805
Current log #2 seq #13805 mem #0:/home/oracle/product/10.2.0/oradata/RT/redo02a. log
Mon Aug 8 14:20:51 2016
Errors in file/home/oracle/admin/RT/bdump/rt_smon_1514.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_1], [0], [0], [1], [], [], [], [], []
Mon Aug 8 15:54:47 2016
Thread 1 advanced to log sequence 13808
Current log #2 seq #13808 mem #0:/home/oracle/product/10.2.0/oradata/RT/redo02a. log
Mon Aug 8 16:21:48 2016
Errors in file/home/oracle/admin/RT/bdump/rt_smon_1514.trc:
ORA-00600: internal error code, arguments: [kcbz_check_objd_typ_1], [0], [0], [1], [], [], [], [], []
Mon Aug 8 16:22:05 2016
Errors in file/home/oracle/admin/RT/bdump/rt_pmon_1500.trc:
ORA-00474: SMON process terminated with error.
Here is more obvious, the database reported a large number of ORA-600 kcbz_check_objd_typ_1 error, then the smon process terminated, the instance crash.
Smon trace file
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0-Production
With the Partitioning, OLAP and Data Mining options
ORACLE_HOME =/home/oracle/product/10.2.0/db_1
System name: SunOS
Node name: st104
Release: 5.10
Version: Generic_141445-09
Machine: i86pc
Instance name: RT
Redo thread mounted by this instance: 1
Oracle process number: 12
Unix process pid: 1514, image: oracle @ st104 (SMON)
* ** 12:19:26. 868
* ** Service name :() 12:19:26. 868
* ** Session id: (383.1) 12:19:26. 868
Dead transaction 0x003d. 002.20.f964 recovered by SMON
Dead transaction 0x0041. 017.4154d55 recovered by SMON
Dead transaction 0x0047. 002.20.180c recovered by SMON
Dead transaction 0x004c. 01c. 20171b57 recovered by SMON
* ** Session id: (383.1) 12:19:27. 470
DATA seg. obj = 0, on-disk obj = 925949, dsflg = 0, dsobj = 923715, cls = 4
Formatted dump of block:
Buffer tsn: 4 rdba: 0x0100336b (4/13163)
Scn: 0x09c6. b2c7f7a2 seq: 0x02 flg: 0x04 tail: 0xf7a20602
Frmt: 0x02 chkval: 0x649b type: 0x06 = trans data
Hex dump of block: st = 0, typ_found = 1
* ** Session id: (383.1) 12:19:34. 244
DATA seg. obj = 0, on-disk obj = 925950, dsflg = 0, dsobj = 923671, cls = 4
Formatted dump of block:
Buffer tsn: 4 rdba: 0x01003343 (4/13123)
Scn: 0x09c6. b2c7f7dc seq: 0x02 flg: 0x04 tail: 0xf7dc0602
Frmt: 0x02 chkval: 0x8013 type: 0x06 = trans data
Hex dump of block: st = 0, typ_found = 1
* ** Session id: (383.1) 12:19:35. 197
DATA seg. obj = 0, on-disk obj = 925941, dsflg = 0, dsobj = 923657, cls = 4
Formatted dump of block:
Buffer tsn: 7 rdba: 0x01c03d53 (7/15699)
Scn: 0x09c6. b2c7f570 seq: 0x02 flg: 0x04 tail: 0xf5700602
Frmt: 0x02 chkval: 0xe5c5 type: 0x06 = trans data
Hex dump of block: st = 0, typ_found = 1
* ** Session id: (383.1) 12:19:38. 965
DATA seg. obj = 0, on-disk obj = 925948, dsflg = 0, dsobj = 923656, cls = 4
Formatted dump of block:
Buffer tsn: 7 rdba: 0x01c03a6b (7/14955)
Scn: 0x09c6. b2c7f745 seq: 0x02 flg: 0x04 tail: 0xf7450602
Frmt: 0x02 chkval: 0x58c5 type: 0x06 = trans data
Hex dump of block: st = 0, typ_found = 1
The obj and dataobj in the block do not match.
Query seg $. type = 3
Type = 3 is a temporary object. Due to exceptions, smon cannot be properly cleared during temp object cleanup, resulting in termination of smon and crash of the instance.
SQL> select file #, block #, ts # from seg $ where type # = 3;
FILE # BLOCK # TS #
------------------------------
4 13163 4
4 13123 4
7 15699 7
7 14955 7
Processing Method of ORA-600 kcbz_check_objd_typ_1
1) Check tablespace bitmap
SQL> oradebug setmypid
SQL> exec dbms_space_admin.tablespace_verify ('& TBSP_NAME ')
SQL> oradebug tracefile_name
Or if the tablespace involved is an ASSM tablespace:
SQL> oradebug setmypid
SQL> exec dbms_space_admin.assm_tablespace_verify ('& TBSP_NAME', dbms_space_admin.TS_VERIFY_BITMAPS)
SQL> oradebug tracefile_name
I am expecting to fail
2) upload UPT these temp segments
SQL> exec dbms_space_admin.segment_corrupt ('& TBSP_NAME', & FILE #, & BLOCK #)
3) Drop them
SQL> exec dbms_space_admin.segment_drop_corrupt ('& TBSP_NAME', & FILE #, & BLOCK #)
4) Rebuild tablespace bitmap
Exec DBMS_SPACE_ADMIN.TABLESPACE_REBUILD_BITMAPS ('& TBSP_NAME ')
5) Verify the tablespace again
SQL> oradebug setmypid
SQL> exec dbms_space_admin.tablespace_verify ('& TBSP_NAME ')
SQL> oradebug tracefile_name
Or if the tablespace involved is an ASSM tablespace:
SQL> oradebug setmypid
SQL> exec dbms_space_admin.assm_tablespace_verify ('& TBSP_NAME', dbms_space_admin.TS_VERIFY_BITMAPS)
SQL> oradebug tracefile_name