Disk loss in hp platform rdisk in oracle causes asm startup to report ORA-15042 recovery

An old friend found me and said that a customer's database was abnormal. The problem was that asm could not mount normally, and the error was that two disks were missing. ask me if I can recover. because it is an intranet environment, the scattered information sent over it is roughly analyzed as follows:
Asm alert log error
ERROR: diskgroup DGROUP1 was not mounted
Fri Aug 12 16:03:12 EAT 2016
SQL> alter diskgroup DGROUP1 mount
Fri Aug 12 16:03:12 EAT 2016
NOTE: cache registered group DGROUP1 number = 1 incarn = 0xf6781b5c
Fri Aug 12 16:03:12 EAT 2016
NOTE: when at: instance first (grp 1)
Fri Aug 12 16:03:16 EAT 2016
NOTE: start heartbeating (grp 1)
Fri Aug 12 16:03:16 EAT 2016
NOTE: cache dismounting group 1/0xF6781B5C (DGROUP1)
NOTE: dbwr not being msg 'd to dismount
ERROR: diskgroup DGROUP1 was not mounted
The front-end tries to mount the asm disk group and reports an error ORA-15042

It can be seen from this that the asm disk group cannot be properly mounted due to the lack of asm disk 15, 16. if you want to restore the asm, the best way is to find the two disks. kfed analyzes the current disk and finds that the disk for asm disk 14 is disk160, and the disk for asm disk 17 is disk163, according to the first feeling, it is likely that the disk161 and disk161 disks are abnormal, so that no warning is triggered for the hardware inspection in the IDC.

OS-level analysis

Omit records irrelevant to this conclusion
Ls-l/dev/rdisk
Crw-rw ---- 1 oracle dba 13 0x000070 Jan 1 2016 disk160
Crw-rw ---- 1 oracle dba 13 0x000073 Jan 1 2016 disk163
 
Ls-l/dev/disk
Brw-r ----- 1 bin sys 1 0x000070 Jan 13 2015 disk160
Brw-r ----- 1 bin sys 1 0x000071 Jan 13 2015 disk161
Brw-r ----- 1 bin sys 1 0x000072 Jan 13 2015 disk162
Brw-r ----- 1 bin sys 1 0x000073 Jan 13 2015 disk163

Here, we found that the disks under/dev/disk exist in hp unix, but are lost under/dev/rdisk. We can continue the analysis using ioscan commands.

Ioscan-fNnkC disk
Disk 160 64000/0 xfa00/0x70 esdisk claimed device hp OPEN-V
/Dev/disk/disk160/dev/rdisk/disk160
Disk 161 64000/0 xfa00/0x71 esdisk claimed device hp OPEN-V
/Dev/disk/disk161
Disk 162 64000/0 xfa00/0x72 esdisk claimed device hp OPEN-V
/Dev/disk/disk162
Disk 163 64000/0 xfa00/0x73 esdisk claimed device hp OPEN-V
/Dev/disk/disk163/dev/rdisk/disk163

Here we can basically determine whether the disk under/dev/rdisk is lost. Further analysis, because rdisk is an aggregated drive letter, then we analyze whether the drive letter before aggregation is normal.

Ioscan-m dsf
/Dev/rdisk/disk160/dev/rdsk/c29t12d4
/Dev/rdsk/c28t12d4
/Dev/rdisk/disk163/dev/rdsk/c29t12d7
/Dev/rdsk/c28t12d7
 
Ls-l/dev/rdsk
Crw-r ----- 1 bin sys 188 0x1dc000 Apr 22 2014 c29t12d0
Crw-r ----- 1 bin sys 188 0x1dc100 Apr 22 2014 c29t12d1
Crw-r ----- 1 bin sys 188 0x1dc300 Jan 13 2015 c29t12d3
Crw-r ----- 1 bin sys 188 0x1dc400 Jan 13 2015 c29t12d4
Crw-r ----- 1 bin sys 188 0x1dc500 Jan 13 2015 c29t12d5
Crw-r ----- 1 bin sys 188 0x1dc600 Jan 13 2015 c29t12d6
Crw-r ----- 1 bin sys 188 0x1dc700 Jan 13 2015 c29t12d7
 
Crw-r ----- 1 bin sys 188 0x1cc100 Apr 22 2014 c28t12d1
Crw-r ----- 1 bin sys 188 0x1cc300 Jan 13 2015 c28t12d3
Crw-r ----- 1 bin sys 188 0x1cc400 Jan 13 2015 c28t12d4
Crw-r ----- 1 bin sys 188 0x1cc500 Jan 13 2015 c28t12d5
Crw-r ----- 1 bin sys 188 0x1cc600 Jan 13 2015 c28t12d6
Crw-r ----- 1 bin sys 188 0x1cc700 Jan 13 2015 c28t12d7

Here we can basically judge/dev/rdsk/c28t12d5,/dev/rdsk/c28t12d6,/dev/rdsk/c29t12d5, /dev/rdsk/c29t12d6 is the drive letter before the aggregation of/dev/rdisk/disk161 and disk162. that is to say, we now determine that only the character devices under/dev/rdisk are faulty, and the other devices are normal.

Use system commands to fix exceptions
Insf-e-H 64000/0 xfa00/0x71
Insf-e-H 64000/0 xfa00/0x72
Hp-asm-disk
Now, we can see the/dev/rdisk/disk161 and/dev/rdisk/disk162 drive letters. We can make a preliminary judgment that the OS-level drive letter has been restored to normal. Modify the disk permission and group
Chmod 660/dev/rdisk/disk161
Chmod 660/dev/rdisk/disk162
Chown oracle: dba/dev/rdisk/disk161
Chown oracle: dba/dev/rdisk/disk162
Start asm, mount disk, and open database normally

Asm-mount
This recovery mainly solves the problem from the operating system level, so as to achieve perfect database recovery and zero data loss. There are similar recovery cases.