Crs-0184:cannot communicate with the CRS Daemon. Node separation from cluster problem solving

1. System Environment :

OEL6.5 Enterprise Edition

Oracle11.2.0.2 RAC + ASM

2. Error message:

$ crs_stat-t
Crs-0184:cannot communicate with the CRS daemon.
$ crsctl Check has
Crs-4639:could not contacted Oracle high availability Services

3, error reasons to find:

view IP address on Node 1 node, VIP no longer exists
$ ifconfig-a
En0:flags=1e080863,480<up,broadcast,notrailers,running,simplex,multicast,grouprt,64bit,checksum_offload ( ACTIVE),chain>
inet 192.168.0.117 netmask 0xffffff00 broadcast 192.168.0.255
Tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
En1:flags=1e080863,480<up,broadcast,notrailers,running,simplex,multicast,grouprt,64bit,checksum_offload ( ACTIVE),chain>
inet 10.10.10.1 netmask 0xff000000 broadcast 10.255.255.255
Tcp_sendspace 65536 tcp_recvspace 65536 rfc1323 1
Running "Crs_stat-t" on Node 2 found that Node 1 seems to have exited the cluster environment
# Su-grid
$ crs_stat-t
Name Type Target State Host
------------------------------------------------------------
Ora. DATA.DG ora....up.type Online Online node2
Ora .... Er.lsnr ora....er.type Online Online node2
Ora .... N1.lsnr Ora....er.type ONLINE OFFLINE
Ora. OCR.DG ora....up.type Online Online node2
Ora.asm ora.asm.type Online Online node2
ORA.CVU Ora.cvu.type ONLINE OFFLINE
ORA.GSD Ora.gsd.type OFFLINE OFFLINE
Ora....network ora....rk.type Online Online node2
ORA.NODE1.VIP Ora....t1.type ONLINE OFFLINE
Ora .... Sm2.asm Application Online Online Node2
Ora .... E2.lsnr Application Online Online Node2
ORA.NODE2.GSD Application OFFLINE OFFLINE
Ora.node2.ons Application Online Online Node2
Ora.node2.vip ora....t1.type Online Online node2
ora.oc4j Ora.oc4j.type ONLINE OFFLINE
Ora.ons ora.ons.type Online Online node2
Ora.prod.db ora....se.type Online Online node2
Ora....ry.acfs ora....fs.type Online Online node2
ORA.SCAN1.VIP Ora....ip.type ONLINE OFFLINE
From the output see Node 1 VIP has been flowing to Node 2, so as to determine that node 1 is out of the cluster.
View the IP information on Node 2 through "Ifconfig".

4, Problem solving:
Cleans up the configuration information for Node 1 and intends to rerun root.sh
#/u01/app/11.2.0/grid/crs/install/rootcrs.pl-verbose-deconfig-force
Using configuration parameter file:/u01/app/11.2.0/grid/crs/install/crsconfig_params
Prcr-1119:failed to look up CRS resources of Ora.cluster_vip_net1.type type
prcr-1068:failed to query resources
Cannot communicate with CRSD
prcr-1070:failed to check if resource ORA.GSD is registered
Cannot communicate with CRSD
prcr-1070:failed to check if resource ora.ons is registered
Cannot communicate with CRSD


Crs-4535:cannot communicate with Cluster ready Services
Crs-4000:command Stop failed, or completed with errors.
Crs-4544:unable to connect to Ohas
Crs-4000:command Stop failed, or completed with errors.
Acfs-9313:no ADVM/ACFS installation detected.
Successfully deconfigured Oracle clusterware stack on this node
#
#
#/u01/app/11.2.0/grid/crs/install/roothas.pl-verbose-deconfig-force
Using configuration parameter file:/u01/app/11.2.0/grid/crs/install/crsconfig_params
Crs-4047:no Oracle Clusterware components configured.
Crs-4000:command Stop failed, or completed with errors.
Crs-4047:no Oracle Clusterware components configured.
Crs-4000:command Delete failed, or completed with errors.
Crs-4047:no Oracle Clusterware components configured.
Crs-4000:command Stop failed, or completed with errors.
You must kill OHASD processes or reboot the system to properly
Cleanup the processes started by Oracle Clusterware
Acfs-9313:no ADVM/ACFS installation detected.
Either/etc/oracle/olr.loc does not exist or was not readable
Make sure the file exists and it have read and execute access
Failure in Execution (rc=-1, MB, A file or directory in the path name does not exist.) for COMMAND/ETC/OHASD Deinstall
Successfully deconfigured Oracle Restart stack


After the environment is cleaned, run the root.sh
#./u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g


The following environment variables is set as:
oracle_owner= Grid
Oracle_home=/u01/app/11.2.0/grid


Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "Dbhome" has not changed. No need to overwrite.
The contents of "oraenv" has not changed. No need to overwrite.
The contents of "coraenv" has not changed. No need to overwrite.


Entries'll be added to The/etc/oratab file as needed by
Database Configuration Assistant When a database is created
Finished running generic part of root script.
Now product-specific root actions would be performed.
Using configuration parameter file:/u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored prerequisites during installation
User Grid has the required capabilities to run CSSD in realtime mode
OLR initialization-successful
Adding Clusterware entries to Inittab
Crs-4402:the CSS Daemon was started in exclusive mode but found a active CSS daemon on node node2, number 2, and are term Inating
An active cluster is found during exclusive startup, restarting to join the cluster


PRKO-2190:VIP exists for node Node1, VIP name Node1-vip
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
View results:
$ crs_stat-t
Name Type Target State Host
------------------------------------------------------------
Ora. DATA.DG ora....up.type Online Online node1
Ora .... Er.lsnr ora....er.type Online Online node1
Ora .... N1.lsnr ora....er.type Online Online node1
Ora. OCR.DG ora....up.type Online Online node1
Ora.asm ora.asm.type Online Online node1
ORA.CVU ora.cvu.type Online Online node2
ORA.GSD Ora.gsd.type OFFLINE OFFLINE
Ora....network ora....rk.type Online Online node1
Ora .... Sm1.asm Application Online Online Node1
Ora .... E1.lsnr Application Online Online Node1
ORA.NODE1.GSD Application OFFLINE OFFLINE
Ora.node1.ons Application Online Online Node1
Ora.node1.vip ora....t1.type Online Online node1
Ora .... Sm2.asm Application Online Online Node2
Ora .... E2.lsnr Application Online Online Node2
ORA.NODE2.GSD Application OFFLINE OFFLINE
Ora.node2.ons Application Online Online Node2
Ora.node2.vip ora....t1.type Online Online node2
ora.oc4j Ora.oc4j.type ONLINE OFFLINE
Ora.ons ora.ons.type Online Online node1
Ora.prod.db ora....se.type Online Online node1
Ora....ry.acfs ora....fs.type Online Online node1
Ora.scan1.vip ora....ip.type Online Online node1

Crs-0184:cannot communicate with the CRS Daemon. Node separation from cluster problem solving