Oracle RAC extended disk mining pit record

Oracle RAC disk expansion Mining pit record:

System environment: VMware ESXi Oracle RAC 11.2.0.4

Yesterday, I expanded the ASM Disk group, the FRA disk group, and the data disk group, and I added a piece of hard disk 320g,200g to the data group, and 120g extended to the FRA group.
Because Oracle uses raw bare devices, the first disk is extended normally, without shutting down.

Pit 1:
However, when you expand the FRA group, you need to shut down to find a second device. Extended normally after reboot, it is recommended to extend the disk to one group at a time.

Pit Two:
Today, I see the system disk space is not enough, expand the system disk space, add a 100g disk, add to the dynamic coupons, everything is normal.
But after I restarted Oracle, I found that the database was not up, ORA.DATA.DG,ORA.FRA.DG, ora.bol.db.
When I look at ASM disk groups, I find:
Select Path,header_status,state from V$asm_disk;
Select Name,path from V$asm_disk_stat;
My original 7 bare devices, now just identified 6, I checked the SD device ID:LL/DEV/SD. Found the ID of the SD device was changed and I
Configuration changes in the/etc/udev/rules.d/60-raw.rules configuration, which is why the database data and FRA do not come.

Analysis Reason:
I added a shared disk, which is on bus SCSI bus 1
I added a non-shared disk on the bus scsi0, but the system defaults to identifying the SCSI bus before, causing my raw device code to be garbled.
Workaround:
Fortunately I remember how to allocate space. Fdisk-l view the corresponding space size, according to the control/dev/sd* device ID, change to/etc/udev/rules.d/60-raw.rules inside. Be sure to correspond to the previous device
Code, or Oracle will not get up.
It's really a pit-dead.
Keep a record, lest you go into the pit again.

Oracle RAC extended disk mining pit record