Oracle Database TB-level Exadata database recovery

A customer's exadata environment causes database downtime due to a large number of hardware faults. After analysis, it is confirmed that a disk of a cell node has an exception, leading to some

After the disk of the cell node is offline, the asm reblance cannot be performed. In the end, the diskgroup is forcibly dismounted. Alert log:

Fri Oct 09 15:44:26 2015
NOTE: process _ user4655 _ + asm1 (4655) initiating offline of disk 46.3915916191 (DATA_KLYX_CD_10_KLYXCEL02) with mask 0x7e in group 1
NOTE: checking PST: grp = 1
ERROR: Disk 18 cannot be offlined, since all the disks [18, 49] with mirrored data wocould be offline.
ERROR: too offline disks in PST (grp 2)
GMON checking disk modes for group 1 at 55 for pid 41 and osid 4655
Fri Oct 09 15:44:26 2015
NOTE: cache dismounting (not clean) group 2/0x4AD8D253 (DBFS_DG)
ERROR: too offline disks in PST (grp 1)
NOTE: checking PST for grp 1 done.
WARNING: Disk 46 (DATA_KLYX_CD_10_KLYXCEL02) in group 1 in mode 0x7f is now being taken offline on ASM inst 1
NOTE: initiating PST update: grp = 1, dsk = 46/0 xe968239f, mode = 0x6a, op = 4
GMON updating disk modes for group 1 at 56 for pid 41 and osid 4655
ERROR: Disk 46 cannot be offlined, since all the disks [46, 72] with mirrored data wocould be offline.
ERROR: too offline disks in PST (grp 1)
WARNING: Offline of disk 46 (DATA_KLYX_CD_10_KLYXCEL02) in group 1 and mode 0x7f failed on ASM inst 1
WARNING: Disk 18 (DBFS_DG_CD_10_KLYXCEL02) in group 2 mode 0x7f offline is being aborted
Fri Oct 09 15:44:26 2015
NOTE: process _ user4663 _ + asm1 (4663) initiating offline of disk 46.3915916191 (DATA_KLYX_CD_10_KLYXCEL02) with mask 0x7e in group 1
NOTE: checking PST: grp = 1
WARNING: Offline of disk 18 (DBFS_DG_CD_10_KLYXCEL02) in group 2 and mode 0x7f failed on ASM inst 1
Fri Oct 09 15:44:26 2015
NOTE: halting all I/OS to diskgroup 2 (DBFS_DG)
NOTE: unable to offline disks after getting write error for diskgroup DBFS_DG
NOTE: write to disk 32 succeeded
NOTE: disk 18 had IO error
NOTE: write to disk 6 succeeded
NOTE: halting all I/OS to diskgroup 1 (DATA_KLYX)
NOTE: LGWR doing non-clean dismount of group 1 (DATA_KLYX)
NOTE: LGWR sync ABA = 21.5465 last written ABA 21.5465
GMON checking disk modes for group 1 at 57 for pid 44, osid 4663
ERROR: too offline disks in PST (grp 1)
NOTE: checking PST for grp 1 done.

We can see that disk 10th of cell 2 has an exception. You can use the cellcli command line to further confirm:

CellCLI> list physicaldisk;
20:0 E07Q5N normal
20:1 E05SMD normal
20:2 E07GV6 normal
20:3 E060YD normal
20:4 E05ZQE normal
20:5 E07L9M normal
20:6 E07FED normal
20:7 E05N88 normal
20:8 E08DY5 normal
20:9 E06ADQ normal
20:10 E08E5C critical
20:11 E08DYF normal
By performing a repair check on the disk, a bad track is also found, as shown below:

SEAGATE ST360057SSUN600G --> SEAGATE ST360057SSUN600G
Sector 0 --> Sector 0
1,172,123,568 Sectors
Sectors that cocould not be read:
102,578,137
102,578,139
102,578,140
102,578,141
102,578,142
102,578,149
102,578,152
......
1,172,123,561 sector (s) successfully copied.
7 bad source sectors encountered.
Corresponding destination sectors filled with: UNREADABLESECTOR

After the repair is completed, the diskgroup is successfully mounted. However, check that some bad data blocks exist in the database as follows:

SQL> select * from V $ DATABASE_BLOCK_CORRUPTION;
 
FILE # BLOCK # BLOCKS corrcorruption_change # CORRUPTION_TYPE
-----------------------------------------------------------------
1 5186816 3 0 slave UPT
1 5186813 1 0 slave UPT
1 3343734 2 0 UPT
25 2391372 1 1.4561E + 13 bytes UPT
35 538916 1 0 bytes UPT
36 543804 3 0 slave UPT

The repair of bad data blocks is relatively simple, but it is difficult to process the asm environment. In addition, the customer does not have a backup, so blockcover cannot be implemented. For several bad data blocks of the system, it is found to be an audit object after Check. Therefore, you can format the data by truncate and then insert it;
In addition, the bad data block of file 25th is a logical bad data block. You can directly rebuild the index by checking that the data block is an index through the query.
As for the final part of file 35 and 36, since all are data files and the objects involved are partition table partiiton, the processing is relatively troublesome.

Here, I create a test table, clear the data, construct an empty data block, and copy the data block to the asm diskgroup to replace these bad blocks, you need to modify the rdba address and obj id of the bad data block. The following describes how to replace the data block copy process:

SQL> @ copy_block_to_asm. SQL
Enter value for file_with_patched_block:/u01/.../blk. dbf
Old 15: v_FsFileName: = '& file_with_patched_block ';
New 15: v_FsFileName: = '/u01/.../blk. Dbf ';
Enter value for file_to_patch_in_asm: + DATA_KLYX/xxxx1/datafile/epmrange107.dbf
Old 16: v_AsmFileName: = '& file_to_patch_in_ASM ';
New 16: v_AsmFileName: = '+ DATA_KLYX/xxxx1/datafile/epmrange107.dbf ';
Enter value for block_to_patch: 543804
Old 17: v_offstart: = '& block_to_patch ';
New 17: v_offstart: = '20140901 ';
PL/SQL procedure successfully completed.

Finally, run the rman check to confirm that everything is normal. The following is the result of rman detection:

.......
RMAN> backup validate datafile 36;
 
Starting backup at 16-OCT-15
Using target database control file instead of recovery catalog
Allocated channel: ORA_DISK_1
Channel ORA_DISK_1: SID = 1337 instance = xxx1 device type = DISK
Channel ORA_DISK_1: starting full datafile backup set
Channel ORA_DISK_1: specifying datafile (s) in backup set
Input datafile file number = 00036 name = + DATA_KLYX/xxx1/datafile/epmrange107.dbf
Channel ORA_DISK_1: backup set complete, elapsed time: 00:00:15
List of Datafiles
========================
File Status Marked UPT Empty Blocks Examined High SCN
-------------------------------------------------------------
36 OK 0 315258 3276800 14561048845049
File Name: + DATA_KLYX/xxx1/datafile/epmrange107.dbf
Block Type Blocks Failing Blocks Processed
----------------------------------------
Data 0 2926354
Index 0 20
Other 0 35168
......