CRS cannot be started. Running crsctl start crs has no response

CRS cannot be started. Running crsctl start crs has no response

View messages:
[Root @ UNID02 ~] # Tail-f/var/log/messages
Dec 9 08:11:13 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7259.
Dec 9 08:12:13 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7468.
Dec 9 08:12:13 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7356.
Dec 9 08:12:13 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7259.
Dec 9 08:13:13 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7468.
Dec 9 08:13:13 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7356.
Dec 9 08:13:14 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7259.
Dec 9 08:14:13 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7468.
Dec 9 08:14:13 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7356.
Dec 9 08:14:14 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7259.
The reason why CRS cannot be started is: Cluster Ready Services waiting on dependencies
It seems that dependent components do not exist.
View More detailed information:
[Root @ UNID02 ~] # Less/tmp/crsctl.7259
Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]

[Root @ UNID02 ~] # Service rawdevices status
/Dev/raw/raw5: bound to major 120, minor 97
/Dev/raw/raw6: bound to major 120, minor 113
/Dev/raw/raw7: bound to major 120, minor 129

It is found that only three RAW files are available, so the remaining RAW files are started.
[Root @ UNID02 ~] # Service rawdevices start
Assigning devices:
/Dev/raw/raw1 -->/dev/emcpowera1
/Dev/raw/raw1: bound to major 120, minor 1
/Dev/raw/raw2 -->/dev/emcpowerb1
/Dev/raw/raw2: bound to major 120, minor 17
/Dev/raw/raw3 -->/dev/emcpowerc1
/Dev/raw/raw3: bound to major 120, minor 33
/Dev/raw/raw4 -->/dev/emcpowerd1
/Dev/raw/raw4: bound to major 120, minor 49
/Dev/raw/raw5 -->/dev/emcpowerg1
/Dev/raw/raw5: bound to major 120, minor 97
/Dev/raw/raw6 -->/dev/emcpowerh1
/Dev/raw/raw6: bound to major 120, minor 113
/Dev/raw/raw7 -->/dev/emcpoweri1
/Dev/raw/raw7: bound to major 120, minor 129
Done
[Root @ UNID02 ~] #
Why are these RAW items not available? The reason is that the storage is restarted due to a fault. After the disk is restarted, it is mounted to the host, but it is not identified as RAW.
Then start rawservices again.
[Root @ UNID02 ~] # Service rawdevices start
Assigning devices:
/Dev/raw/raw1 -->/dev/emcpowera1
/Dev/raw/raw1: bound to major 120, minor 1
/Dev/raw/raw2 -->/dev/emcpowerb1
/Dev/raw/raw2: bound to major 120, minor 17
/Dev/raw/raw3 -->/dev/emcpowerc1
/Dev/raw/raw3: bound to major 120, minor 33
/Dev/raw/raw4 -->/dev/emcpowerd1
/Dev/raw/raw4: bound to major 120, minor 49
/Dev/raw/raw5 -->/dev/emcpowerg1
/Dev/raw/raw5: bound to major 120, minor 97
/Dev/raw/raw6 -->/dev/emcpowerh1
/Dev/raw/raw6: bound to major 120, minor 113
/Dev/raw/raw7 -->/dev/emcpoweri1
/Dev/raw/raw7: bound to major 120, minor 129
Done
[Root @ UNID02 ~] #
Continue to view Messages and find the previous error:

Dec 9 08:15:14 UNID02 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.7259.
It seems that there is still a problem with CRS during the startup process, so check the crsctl. * file again:
[Root @ UNID02 ~] # Cat/tmp/crsctl.7259
Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage Operating System error [Permission denied] [13]
[Root @ UNID02 ~] #
The same error is reported, but this time [No such file or directory] is changed to [Permission denied].
There must be a problem with RAW permissions. View RAW permissions:
[Root @ UNID02 ~] # Ll/dev/raw
Total 0
Crw ------- 1 root 162, 1 Dec 9 raw1
Crw ------- 1 root 162, 2 Dec 9 raw2
Crw ------- 1 root 162, 3 Dec 9 raw3
Crw ------- 1 root 162, 4 Dec 9 raw4
Crw ------- 1 root 162, 5 Dec 9 raw5
Crw ------- 1 root 162, 6 Dec 9 raw6
Crw ------- 1 root 162, 7 Dec 9 raw7
Indeed, the owner of RAW is oracle:
[Root @ UNID02 ~] # Chown oracle: dba/dev/raw /*
Information about CRS startup is found in messages:
Dec 9 08:16:14 UNID02 logger: Cluster Ready Services completed waiting on dependencies.
Dec 9 08:16:14 UNID02 last message repeated 2 times
Dec 9 08:16:14 UNID02 logger: Running CRSD with TZ =
Dec 9 08:16:14 UNID02 logger: Oracle CSS Family monitor starting.
Dec 9 08:16:15 UNID02 logger: Oracle CSS restart. 0, 1

I thought the problem ended here, but after waiting for a while, I found that asm and instance have never been up:
[Oracle @ UNID02 ~] $ Crs_stat-t
Name Type Target State Host
------------------------------------------------------------
Ora... SM1.asm application ONLINE OFFLINE
Ora... 01. lsnr application ONLINE OFFLINE
Ora... t01.gsd application ONLINE OFFLINE
Ora... t01.ons application ONLINE OFFLINE
Ora... t01.vip application ONLINE OFFLINE
Ora... SM2.asm application ONLINE OFFLINE
Ora... 02. lsnr application ONLINE UNID02
Ora... t02.gsd application ONLINE UNID02
Ora... t02.ons application ONLINE UNID02
Ora... t02.vip application ONLINE UNID02
Ora... RTAL. cs application ONLINE OFFLINE
Ora... ac1.srv application ONLINE OFFLINE
Ora... ac2.srv application ONLINE OFFLINE
Ora. rac. db application ONLINE OFFLINE
Ora... c1.inst application ONLINE OFFLINE
Ora... c2.inst application ONLINE OFFLINE
Ora... rwss. cs application ONLINE OFFLINE
Ora... ac1.srv application ONLINE OFFLINE
Ora... ac2.srv application ONLINE OFFLINE
Ora... _ taf. cs application ONLINE OFFLINE
Ora... ac1.srv application ONLINE OFFLINE
Ora... ac2.srv application ONLINE OFFLINE
Ora... test. cs application ONLINE OFFLINE
Ora... ac1.srv application ONLINE OFFLINE
Ora... rac1.cs application ONLINE OFFLINE
Ora... ac1.srv application ONLINE OFFLINE
Ora... ac2.srv application ONLINE OFFLINE
Ora... rac2.cs application ONLINE OFFLINE
Ora... ac1.srv application ONLINE OFFLINE
Ora... ac2.srv application ONLINE OFFLINE
[Oracle @ UNID02 ~] $
Start the ASM instance manually:
[Oracle @ UNID02 ~] $ Dba

SQL * Plus: Release 11.1.0.7.0-Production on Mon Dec 9 08:25:06 2013

Copyright (c) 1982,200 8, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation: check if cable failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpcini1
ORA-27303: additional information: requested interface eth1 interface not running set _ disable_interface_checking = TRUE to disable this check for single instance cluster. Check output from ifcon
SQL>
SQL>
SQL> quit
Disconnected
This error is reported because the ASM and INSTANCE in RAC check the private network information of other nodes through the private network at startup. At this time, the other node is shut down.
The solution is to set the following two implicit parameters in the asm parameter file:
_ Disable_instance_params_check = TRUE
_ Disable_interface_checking = TRUE
_ Disable_instance_params_check = TRUE indicates that the value of instance_type is ignored when the instance is started, while the disable_interface_checking parameter only uses the db parameter file, which reports an error ORA-15021 when used for the asm instance: parameter "_ disable_interface_checking" is not valid in asm instance, SO _ disable_instance_params_check = TRUE is used to skip the instance_type check.
Set the following implicit parameters in the instance parameter file:
_ Disable_interface_checking = TRUE

Continue to start asm and instance.