CRS-0184: Cannot communicate with the CRS daemon.之節點脫離叢集問題解決，crs-0184crs

CRS-0184: Cannot communicate with the CRS daemon.之節點脫離叢集問題解決，crs-0184crs

1、系統內容：

                        OEL6.5企業版

                        Oracle11.2.0.2 RAC + ASM

2、報錯資訊：

$ crs_stat -t                
CRS-0184: Cannot communicate with the CRS daemon.                
$ crsctl check has                
CRS-4639: Could not contact Oracle High Availability Services

3、報錯原因尋找：

在節點1節點查看IP地址，VIP已經不存在
$ ifconfig -a               
en0: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>                
        inet 192.168.0.117 netmask 0xffffff00 broadcast 192.168.0.255                
         tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1                
en1: flags=1e080863,480<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),CHAIN>                
        inet 10.10.10.1 netmask 0xff000000 broadcast 10.255.255.255                
         tcp_sendspace 65536 tcp_recvspace 65536 rfc1323 1                               
在節點2上運行“crs_stat -t”發現，節點1好像已經退出叢集環境
# su - grid
$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg     ora....up.type         ONLINE    ONLINE    node2       
ora....ER.lsnr   ora....er.type         ONLINE    ONLINE    node2       
ora....N1.lsnr   ora....er.type         ONLINE    OFFLINE               
ora.OCR.dg     ora....up.type         ONLINE    ONLINE    node2       
ora.asm       ora.asm.type          ONLINE    ONLINE    node2       
ora.cvu       ora.cvu.type          ONLINE    OFFLINE               
ora.gsd       ora.gsd.type          OFFLINE   OFFLINE               
ora....network   ora....rk.type         ONLINE    ONLINE    node2       
ora.node1.vip   ora....t1.type         ONLINE    OFFLINE               
ora....SM2.asm   application          ONLINE    ONLINE    node2       
ora....E2.lsnr   application          ONLINE    ONLINE    node2       
ora.node2.gsd   application          OFFLINE   OFFLINE               
ora.node2.ons   application          ONLINE    ONLINE    node2       
ora.node2.vip   ora....t1.type         ONLINE    ONLINE    node2       
ora.oc4j      ora.oc4j.type         ONLINE    OFFLINE               
ora.ons       ora.ons.type          ONLINE    ONLINE    node2       
ora.prod.db    ora....se.type         ONLINE    ONLINE    node2       
ora....ry.acfs   ora....fs.type         ONLINE    ONLINE    node2       
ora.scan1.vip   ora....ip.type         ONLINE    OFFLINE               
從輸出中看到節點1的VIP已經飄逸到節點2，從而判斷節點1已經脫離叢集。
通過"ifconfig"查看節點2上的ip資訊。

4、問題解決：
清理節點1的配置資訊，打算重新運行root.sh
# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -verbose -deconfig -force 
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd


CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
ACFS-9313: No ADVM/ACFS installation detected.
Successfully deconfigured Oracle clusterware stack on this node
#
#
# /u01/app/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig -force
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Delete failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
You must kill ohasd processes or reboot the system to properly 
cleanup the processes started by Oracle clusterware
ACFS-9313: No ADVM/ACFS installation detected.
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Failure in execution (rc=-1, 512, A file or directory in the path name does not exist.) for command /etc/ohasd deinstall
Successfully deconfigured Oracle Restart stack


環境清理後，運行root.sh
# ./u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g 


The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0/grid


Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.


Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node2, number 2, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster


PRKO-2190 : VIP exists for node node1, VIP name node1-vip
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
查看結果：
$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type         ONLINE    ONLINE    node1       
ora....ER.lsnr   ora....er.type         ONLINE    ONLINE    node1       
ora....N1.lsnr   ora....er.type         ONLINE    ONLINE    node1       
ora.OCR.dg     ora....up.type         ONLINE    ONLINE    node1       
ora.asm       ora.asm.type          ONLINE    ONLINE    node1       
ora.cvu       ora.cvu.type          ONLINE    ONLINE    node2       
ora.gsd       ora.gsd.type          OFFLINE   OFFLINE               
ora....network   ora....rk.type         ONLINE    ONLINE    node1       
ora....SM1.asm   application          ONLINE    ONLINE    node1       
ora....E1.lsnr   application          ONLINE    ONLINE    node1       
ora.node1.gsd   application          OFFLINE   OFFLINE               
ora.node1.ons   application          ONLINE    ONLINE    node1       
ora.node1.vip   ora....t1.type         ONLINE    ONLINE    node1       
ora....SM2.asm   application          ONLINE    ONLINE    node2       
ora....E2.lsnr   application          ONLINE    ONLINE    node2       
ora.node2.gsd   application          OFFLINE   OFFLINE               
ora.node2.ons   application          ONLINE    ONLINE    node2       
ora.node2.vip   ora....t1.type         ONLINE    ONLINE    node2       
ora.oc4j       ora.oc4j.type         ONLINE    OFFLINE               
ora.ons       ora.ons.type          ONLINE    ONLINE    node1       
ora.prod.db    ora....se.type         ONLINE    ONLINE    node1       
ora....ry.acfs   ora....fs.type         ONLINE    ONLINE    node1       
ora.scan1.vip   ora....ip.type         ONLINE    ONLINE    node1