Oracle errors ORA-00604 ORA-00376 database redo undo loss recovery example

O & M DBAs reflect database storage faults, resulting in the loss of data files in two tablespaces of redo undo, and the database cannot be started open.
A group's ebs system stores redo and undo on raid 0 due to insufficient disk space, and the database does not have any backup. In the end, the tragedy occurred. A raid 0 exception causes all redo undo tasks to be lost, and the database cannot be started properly. (when I took over the attack, the database had already resetlogs, but failed)

1. Oracle error ORA-00604 ORA-00376

Sun Jul 27 11:31:27 2014
SMON: enabling cache recovery
SMON: enabling tx recovery
Sun Jul 27 11:31:27 2014
Database Characterset is ZHS16GBK
Sun Jul 27 11:31:27 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/bdump/prod_smon_454754.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00376: file 42 cannot be read at this time
ORA-01110: data file 42: '/prod/oracle/PROD/logdata/undo/undo1.dbf'
Sun Jul 27 11:31:27 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/bdump/prod_smon_454754.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00376: file 42 cannot be read at this time
ORA-01110: data file 42: '/prod/oracle/PROD/logdata/undo/undo1.dbf'
Sun Jul 27 11:31:27 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/bdump/prod_smon_454754.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00376: file 42 cannot be read at this time
ORA-01110: data file 42: '/prod/oracle/PROD/logdata/undo/undo1.dbf'
Sun Jul 27 11:31:27 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_661_0.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00376: file 41 cannot be read at this time
ORA-01110: data file 41: '/prod/oracle/PROD/logdata/undo/undo2.dbf'
Error 604 happened during db open, shutting down database
USER: terminating instance due to error 604.
Instance terminated by USER, pid = 663670
ORA-1092 signalled during: alter database open...

The undo tablespace file is lost and is processed by offline.
In the past, the alert log was cleared. Here we probably guess that it is the undo file lost by offline, and then resetlogs the database. Now the processing method is
Use _ upted_rollback_segments to block Rollback segments, and then try to start the database

2. Oracle error ORA-00604 ORA-01555

Tue Jul 29 11:40:39 2014
SMON: enabling cache recovery
SMON: enabling tx recovery
Tue Jul 29 11:40:39 2014
Database Characterset is ZHS16GBK
Tue Jul 29 11:40:39 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/bdump/prod_smon_1_378.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number with name "" too small
Tue Jul 29 11:40:39 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/bdump/prod_smon_1_378.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number with name "" too small
Tue Jul 29 11:40:39 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/bdump/prod_smon_1_378.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number with name "" too small
Tue Jul 29 11:40:39 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_585786.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number with name "" too small
Error 604 happened during db open, shutting down database
USER: terminating instance due to error 604.
Instance terminated by USER, pid = 585786
ORA-1092 signalled during: alter database open...

This error occurs because the corresponding rollback segment needs to be found during database startup, but the rollback segment cannot be found due to an undo exception. The solution is to modify the data scn, so that it does not retrieve the rollback segment to block this error. after the database is started, delete undo and recreate the new undo

3. Oracle error ORA-01122 ORA-01110 ORA-01565 ORA-27037

Tue Jul 29 15:59:22 2014
Drop tablespace undo2 including contents and datafiles
Tue Jul 29 15:59:23 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_782490.trc:
ORA-01122: database file 41 failed verification check
ORA-01110: data file 41: '/prod/oracle/PROD/logdata/undo/undo2.dbf'
ORA-01565: error in identifying file '/prod/oracle/PROD/logdata/undo/undo2.dbf'
ORA-27037: unable to obtain file status
Ibm aix risc System/6000 Error: 2: No such file or directory
Additional information: 3
Tue Jul 29 15:59:23 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_782490.trc:
ORA-01259: unable to delete datafile/prod/oracle/PROD/logdata/undo/undo2.dbf
Tue Jul 29 15:59:23 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_782490.trc:
ORA-01122: database file 42 failed verification check
ORA-01110: data file 42: '/prod/oracle/PROD/logdata/undo/undo1.dbf'
ORA-01565: error in identifying file '/prod/oracle/PROD/logdata/undo/undo1.dbf'
ORA-27037: unable to obtain file status
Ibm aix risc System/6000 Error: 2: No such file or directory
Additional information: 3
ORA-01259: unable to delete datafile/prod/oracle/PROD/logdata/undo/undo2.dbf
Tue Jul 29 15:59:23 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_782490.trc:
ORA-01259: unable to delete datafile/prod/oracle/PROD/logdata/undo/undo1.dbf
Tue Jul 29 15:59:23 2014
Completed: drop tablespace undo2 including contents and datafiles
Tue Jul 29 15:59:56 2014
Create undo tablespace undotbs1 datafile '/prod/oracle/PROD/logdata/undo_new01.dbf' size 100 M autoextend on next 128 M maxsize 30G
Tue Jul 29 15:59:57 2014
Completed: create undo tablespace undotbs1 datafile '/prod/oracle/PROD/logdata/undo_new01.dbf' size 100 M autoextend on next 128 M maxsize 30G
Tue Jul 29 16:00:03 2014
Alter tablespace undotbs1 add datafile '/prod/oracle/PROD/logdata/undo_new02.dbf' size 100 M autoextend on next 128 M maxsize 30G
Completed: alter tablespace undotbs1 add datafile '/prod/oracle/PROD/logdata/undo_new02.dbf' size 100 M autoextend on next 128 M maxsize 30G
5. During business operation, the database reports a large number of ORA-600 4097, ORA-600 kdsgrp1, ORA-600 kcfrbd_3 errors
Tue Jul 29 16:07:03 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_950484.trc:
ORA-00600: internal error code, arguments: [4097], [], [], [], [], [], [], []
Tue Jul 29 16:07:06 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_950484.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], []
Tue Jul 29 16:10:06 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_917702.trc:
ORA-00600: internal error code, arguments: [4097], [], [], [], [], [], [], []
Tue Jul 29 16:10:07 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_917702.trc:
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], []
Tue Jul 29 16:12:45 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/bdump/prod_m000_880692.trc:
ORA-00600: internal error code, arguments: [4097], [], [], [], [], [], [], []
Tue Jul 29 16:21:23 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_1040638.trc:
ORA-00600: Internal error code, parameter: [kcfrbd_3], [41], [231381], [1], [12800], [12800], [], []
Tue Jul 29 16:21:37 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_1040638.trc:
ORA-00600: Internal error code, parameter: [kcfrbd_3], [41], [231381], [1], [12800], [12800], [], []
Tue Jul 29 16:21:56 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_1040638.trc:
ORA-00600: Internal error code, parameter: [kcfrbd_3], [41], [231381], [1], [12800], [12800], [], []
Tue Jul 29 16:22:18 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_1040638.trc:
ORA-00600: Internal error code, parameter: [kcfrbd_3], [41], [231381], [1], [12800], [12800], [], []
Tue Jul 29 16:22:28 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_1105950.trc:
ORA-00600: Internal error code, parameter: [4097], [], [], [], [], [], [], []
Tue Jul 29 16:22:33 2014
Errors in file/prod/oracle/PROD/db/tech_st/10.2.0/admin/PROD_erpserver/udump/prod_ora_1159232.trc:
ORA-00600: Internal error code, parameter: [kcfrbd_3], [42], [61235], [1], [12800], [12800], [], []
There are several causes and solutions for this error:
ORA-600 kdsgrp1 is caused by related bad blocks (tab, index, memory, cr block, etc.), combined with the log analysis object exception cause, determine the object based on the actual situation and select the appropriate processing solution (for details, refer to NOTE: 1332252.1)
ORA-600 4097 due to database exception close and then open, creating Rollback segments may trigger bugs that cause the problem (although said fixed in the current version, but actually handled I did follow NOTE: 1030620.6)
ORA-600 kcfrbd_3 after the transaction block is accessed, according to the rollback slot information to locate the relevant rollback segment, and exactly the new rollback segment information and the previous name number is consistent, this error occurs when the data file size is insufficient (for details, refer to NOTE: 601798.1)
In the end, although the database is restored and a large amount of data is saved, the loss of redo and undo data is huge for the ebs system. tip again: database redo and undo are also important, and database backup is even more important.