Case study of one TB-level ERP (asm rac) database restoration for oracle databases

First, when the customer adds disk to one of the rac nodes, it finds that the node is not successfully added to the other node, and then repeatedly adds or even adds the dd disk header.
One of the most fatal actions is to force add disk. In fact, before this step, these disks have been added once and reblance has been completed, but drop disk
But it was not successful. Finally, the customer tried to add it forcibly, as shown below:

SQL> ALTER DISKGROUP xxxx ADD DISK 'ORCL: VOL1_xxx' SIZE 2097152 M FORCE,
'Orcl: VOL2_xxx 'SIZE 2097152 m force,
'Orcl: VOL3_xxx 'SIZE 2097152 M FORCE
........
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Feb 18 06:09:32 2014
SQL> alter diskgroup xxx MOUNT
NOTE: cache registered group xxx number = 1 incarn = 0x6c42d680
.......
Tue Feb 18 06:09:32 2014
NOTE: when at: instance not first (grp 1)
Tue Feb 18 06:09:32 2014
NOTE: cache dismounting group 1/0x6C42D680 (xxx)
NOTE: dbwr not being msg 'd to dismount
Tue Feb 18 06:09:32 2014
NOTE: PST enabling heartbeating (grp 1)
Tue Feb 18 06:09:32 2014
ERROR: diskgroup xxx was not mounted
Tue Feb 18 06:10:22 2014
ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]
Tue Feb 18 06:10:22 2014
.........

Eventually, the disk groups cannot be mounted. Of course, the database cannot be opened successfully, and the following error is reported;

Tue Feb 18 05:53:57 2014
Errors in file/opt/oracle/admin/xxx/bdump/xxx_lmon_17095.trc:
ORA-00202: control file: '+ xxx/controlfile/current.256.741_6671'
ORA-15078: ASM diskgroup was forcibly dismounted.
Tue Feb 18 05:53:58 2014
Errors in file/opt/oracle/admin/xxx/bdump/hxxx_lmon_17095.trc:
ORA-00204: error in reading (block 35, # blocks 1) of control file
ORA-00202: control file: '+ xxx/controlfile/current.256.741_6671'
ORA-15078: ASM diskgroup was forcibly dismounted.
Tue Feb 18 05:53:58 2014
LMON: terminating instance due to error 204
Tue Feb 18 05:53:58 2014

First, use kfed to read the related disk and find that the related metadata of asm basically does not exist, because the previous add disk force is actually equivalent.
Diskgroup is rebuilt once, and the most critical file directory metadata is missing. This is a very troublesome task.
During the restoration process, we even tried Oracle DUL and found that the file directory was completely lost, and dul could not work properly. Of course,
We successfully restored the database using the old Bear's ODU. ODU currently has such a powerful function that allows you to scan the asm disk and then upload all the data files.
The AU allocation information is scanned and recorded in a file asm_fileext_meta.odu. Then, the AU can be automatically spliced to form a complete
The syntax is as follows:

Extract asm file 1 to/xxxxx/system01.dbf
Extract asm file 2 to/xxxxx/undotbs1_01.dbf
Extract asm file 3 to/xxxxx/sysaux01.dbf

Note:

1. The first time I used the file extraction method for recovery, the data dictionary was damaged seriously, resulting in continuous problems after the database opened (which also fixed
A large number of blocks, because some blocks are empty, and some empty blocks are because some disks have completed reblance and some are not, resulting in incomplete data dictionaries ),
Therefore, we re-extracted the data and re-imported the database, which cost a lot of manpower. Of course, this is the first time I used ODU
Restore the ERP database and solve the problem perfectly. Here you must like one!