DBA work Memo-Array corruption resolution

Solve
Problem Description:
In the afternoon, the site said that the product library was terminated unexpectedly, followed by the possibility that the array was damaged because the lights were not lit.
So the emergency dial to the remote, check the database, 5 minutes later, the database restarted successfully.
The problem is actually very simple, but later look at Alertlog, found that the failure from the array until you call me, the middle has nearly 1 hours of time, this time, because the field personnel do not know the problem, so repeatedly switch cluster, trying to make the database can start success, Khan!

Question Explanation:
A total of three copies of the control files in the product library were kept in/global/oracle,/global/backup1,/global/backup2
Under Redo logs and archived logs two copies, respectively, in/global/backup1,/global/backup2
Under
/global/oracle,/global/backup1,/global/backup2 are three separate arrays.
Since the control file must be consistent with Oracle, any one of the three control files cannot be read
, the database will be closed. and redo log and archive logs, as long as a copy can be read and write, then the data
The library can also continue to function correctly.
So, if any array fails, the database will abort unexpectedly because of the problem with the control file.

Solution process:
1. Database aborted unexpectedly

2. Check the $oracle_home/admin/dpshdb/bdump/alert_dpshdb.log file to confirm the unexpected
The reasons for this, such as today's content are:
Tue Oct 26 16:36:40 2004
Errors in FILE/EXPORT/ORACLE/PRODUCT/817/ADMIN/DPSHDB/BDUMP/DPSHDB_CKPT_16541.TRC:
Ora-00206:error in writing (Block 3, # blocks 1) of Controlfile
Ora-00202:controlfile: '/global/backup1/oradata/dpshdb/control02.ctl '
Ora-27063:skgfospo:number of bytes Read/written is incorrect
SVR4 error:6: No such device or address
Additional Information:-1
Additional information:8192
Tue Oct 26 16:36:40 2004
Errors in FILE/EXPORT/ORACLE/PRODUCT/817/ADMIN/DPSHDB/BDUMP/DPSHDB_LGWR_16539.TRC:
Ora-00345:redo Log write error block 38713 count 2
Ora-00312:online Log 1 Thread 1: '/global/backup1/oradata/dpshdb/redo1b.log '
Ora-27063:skgfospo:number of bytes Read/written is incorrect
SVR4 error:5: I/O Error
Additional Information:-1
Additional information:1024
The first part points out that the Control02.ctl control file cannot be read, and the second part points out that the Redo1b.log
Cannot read as log.
The database is then automatically closed, as shown below, which occurs 16:36:43, the first discovery in 16:36:40
After 3 seconds of array failure (we know Ckpt's timeout value is 3 seconds):
Tue Oct 26 16:36:43 2004
Errors in FILE/EXPORT/ORACLE/PRODUCT/817/ADMIN/DPSHDB/UDUMP/DPSHDB_ORA_10206.TRC:
ORA-00221: Error writing control file
Instance terminated by CKPT, PID = 16541
From these logs we can see that the/global/backup1 array is not valid.

3. Modify initialization parameter file $oracle_home/dbs/initdpshdb.ora, ignore invalid control file
And the failed archive path
Original content:
Control_files = ("/global/oracle/oradata/dpshdb/control01.ctl",
"/global/backup1/oradata/dpshdb/control02.ctl",
"/global/backup2/oradata/dpshdb/control03.ctl")
log_archive_dest_1 = "Location=/global/backup1/oradata/dpshdb/arch"
Modified content:
Control_files = ("/global/oracle/oradata/dpshdb/control01.ctl",
"/global/backup2/oradata/dpshdb/control03.ctl")
#log_archive_dest_1 = "Location=/global/backup1/oradata/dpshdb/arch"

4. Start the database, at which point the database is ready to use.

5. To ensure that the redo log files are always 2 copies, after the database is started, we need to remove
The original defunct file, add a new redo log file. This part of the operation can be done after the database is started
, but it is best not to have a large number of update operations in order to prevent the redo log switching too fast, causing the deletion to redo
The log operation failed.
ALTER DATABASE drop LOGFILE member
'/global/backup1/oradata/dpshdb/redo1b.log ';
ALTER DATABASE add LogFile Member
'/global/oracle/oradata/dpshdb/redo1c.log ' to group 1;
ALTER DATABASE drop LOGFILE member
'/global/backup1/oradata/dpshdb/redo2b.log ';
ALTER DATABASE add LogFile Member
'/global/oracle/oradata/dpshdb/redo2c.log ' to group 2;
ALTER DATABASE drop LOGFILE member
'/global/backup1/oradata/dpshdb/redo3b.log ';
ALTER DATABASE add LogFile Member
'/global/oracle/oradata/dpshdb/redo3c.log ' to group 3;
ALTER DATABASE drop LOGFILE member
'/global/backup1/oradata/dpshdb/redo4b.log ';
ALTER DATABASE add LogFile Member
'/global/oracle/oradata/dpshdb/redo4c.log ' to group 4;
Check the V$log view before executing ALTER DATABASE drop LOGFILE member to confirm that the file
The Status field for the Redo log group that belongs to is not current, not active, but inactive. Otherwise
Delete will error, if delete error, then run two times the following statement:
alter system Swtich logfile;
Then rerun the delete old log and add new log operation.