Oracle 11.2 RAC automatic start error diagnosis

A customer's 11.2 RAC for Linux x86-64 environment fails after a power outage and the RAC environment does not start automatically.

After you log on to the database server, you discover that the cluster process exists and then try to start the database manually:

$ sqlplus/as SYSDBA

Sql*plus:release 11.2.0.2.0 Production on Sun May 29 20:17:16 2011

Copyright (c) 1982, Oracle. All rights reserved.

Connected to a idle instance.

Sql> Startup

Ora-01078:failure in processing system parameters

Ora-01565:error in identifying file ' +dg_data/spsp/spfilespsp.ora '

Ora-17503:ksfdopn:2 Failed to open file +dg_data/spsp/spfilespsp.ora

Ora-15056:additional error message

Ora-17503:ksfdopn:2 Failed to open file +dg_data/spsp/spfilespsp.ora

Ora-15001:diskgroup "Dg_data" does not exist or isn't not mounted

Ora-06512:at Line 4

Looks like the problem is on the ASM Disk group, check the alert log for the ASM Disk Group:

Note:client SPSP2:SPSP Registered, Osid 14319, MBR 0x1

Sun May 29 16:58:31 2011

Warning:write Failed. Group:1 disk:0 Au:12 offset:856064 size:4096

Note:unable to write no mirror side for DiskGroup Dg_arch

Note:cache initiating offline of disk 0 group Dg_arch

Note:process 13597 initiating offline of disk 0.3915929597 (ARCH1) with mask 0x7e in Group 1

Sun May 29 16:58:31 2011

Warning:disk ARCH1 in mode 0x7f are now being offlined

Warning:disk ARCH1 in mode 0x7f be now being taken offline

note:initiating PST update:grp = 1, DSK = 0/0xe96857fd, mode = 0x15

Gmon Updating disk modes for group 1 at the for PID Osid 13597

Error:too many offline disks in PST (GRP 1)

Sun May 29 16:58:31 2011

Note:cache dismounting (not clean) group 1/0x9c08a705 (Dg_arch)

Warning:disk ARCH1 in mode 0x7f offline aborted

Warning:offline for disk ARCH1 in mode 0x7f failed.

Sun May 29 16:58:31 2011

Note:halting all I/Os to DiskGroup 1 (dg_arch)

Note:unable to offline disks after getting write error for DiskGroup Dg_arch

Note:disk 0 had IO error

NOTE:LGWR doing Non-clean dismount of Group 1 (dg_arch)

NOTE:LGWR sync aba=237.2255 last written ABA 237.2256

Sun May 29 16:58:32 2011

error:ora-15130 in COD Recovery for DiskGroup 1/0x9c08a705 (Dg_arch)

error:ora-15130 thrown in Rbal for group number 1

Errors in FILE/ORACLE/GRIDBASE/DIAG/ASM/+ASM/+ASM2/TRACE/+ASM2_RBAL_13603.TRC:

Ora-15130:diskgroup "Dg_arch" is being dismounted

Warning:read Failed. Group:1 disk:0 Au:1 offset:4096 size:4096

Warning:read Failed. Group:1 disk:0 Au:1 offset:0 size:4096

Error:no PST Quorum in group:required 1, found 0

Error:could not heartbeat PST for GRP 1. Force dismounting the disk group.

Warning:write Failed. Group:2 disk:2 au:1 offset:1044480 size:4096

Warning:write Failed. Group:2 disk:1 au:1 offset:1044480 size:4096

Warning:write Failed. Group:2 disk:0 Au:1 offset:1044480 size:4096

Warning:disk 0.3915929598 (VD1) Not responding to heart beat

Warning:disk 1.3915929599 (VD2) Not responding to heart beat

Error:too many offline disks in PST (GRP 2)

Warning:disk 2.3915929600 (VD3) Not responding to heart beat

Error:too many offline disks in PST (GRP 2)

Sun May 29 16:58:32 2011

Errors in FILE/ORACLE/GRIDBASE/DIAG/ASM/+ASM/+ASM2/TRACE/+ASM2_B001_14488.TRC: