Problem analysis (assuming you don't know where the problem is, analyze it first):
1, check the Cluster service, found that CRS and CSS services did not start properly
CRSCTL Check CRS
2. Check CRS and CSS logs to find OCR disk anomalies
3, the restoration of OCR (in fact, the use of root.sh to reconstruct the process of OCR, after the reconstruction of the need to re-register related resources such as listener/database, etc.)
Clears the cluster configuration information for all nodes: root user execution $GRID _home/crs/install/rootcrs.pl
Node 1
[Email protected] install]#./rootcrs.pl
Using configuration parameter file:./crsconfig_params
User ignored prerequisites during installation
Installing Trace File Analyzer
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Node 2
[Email protected] install]#./rootcrs.pl
Using configuration parameter file:./crsconfig_params
User ignored prerequisites during installation
Installing Trace File Analyzer
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Clears cluster information for all nodes
Node 1
[Email protected] install]#./rootcrs.pl-deconfig-force
Using configuration parameter file:./crsconfig_params
Prcr-1119:failed to look up CRS resources of Ora.cluster_vip_net1.type type
prcr-1068:failed to query resources
Cannot communicate with CRSD
prcr-1070:failed to check if resource ORA.GSD is registered
Cannot communicate with CRSD
prcr-1070:failed to check if resource ora.ons is registered
Cannot communicate with CRSD
Crs-4535:cannot communicate with Cluster ready Services
Crs-4000:command Stop failed, or completed with errors.
crs-2791:starting shutdown of Oracle high availability services-managed resources on ' Node1 '
Crs-2673:attempting to stop ' ora.mdnsd ' on ' node1 '
Crs-2673:attempting to stop ' ORA.CRF ' on ' node1 '
Crs-2673:attempting to stop ' ora.ctssd ' on ' node1 '
Crs-2673:attempting to stop ' ora.evmd ' on ' node1 '
Crs-2673:attempting to stop ' ora.asm ' on ' node1 '
Crs-2673:attempting to stop ' ora.drivers.acfs ' on ' node1 '
Crs-2677:stop of ' ora.evmd ' on ' Node1 ' succeeded
Crs-2677:stop of ' ORA.CRF ' on ' Node1 ' succeeded
Crs-2677:stop of ' ora.mdnsd ' on ' Node1 ' succeeded
Crs-2677:stop of ' ora.ctssd ' on ' Node1 ' succeeded
Crs-2677:stop of ' ora.drivers.acfs ' on ' Node1 ' succeeded
Crs-2677:stop of ' ora.asm ' on ' Node1 ' succeeded
Crs-2673:attempting to stop ' ora.cluster_interconnect.haip ' on ' node1 '
Crs-2677:stop of ' ora.cluster_interconnect.haip ' on ' Node1 ' succeeded
Crs-2673:attempting to stop ' ora.cssd ' on ' node1 '
Crs-2677:stop of ' ora.cssd ' on ' Node1 ' succeeded
Crs-2673:attempting to stop ' ora.gipcd ' on ' node1 '
Crs-2677:stop of ' ora.gipcd ' on ' Node1 ' succeeded
Crs-2673:attempting to stop ' ora.gpnpd ' on ' node1 '
Crs-2677:stop of ' ora.gpnpd ' on ' Node1 ' succeeded
Crs-2793:shutdown of Oracle High availability services-managed resources on ' Node1 ' have completed
Crs-4133:oracle High Availability Services have been stopped.
removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
Node 2
[[email protected] install]#./rootcrs.pl-deconfig-force-lastnode
Using Configuration parameter File:./crsconfig_params
Crs-5702:resource ' ORA.CSSD ' is already running on ' Node2 '
Crs-4000:command Start failed , or completed with errors.
CSS startup failed with return code 1
prcr-1068:failed-to-query resources
cannot communicate with CRSD
PRCR -1068:failed to query resources
cannot communicate with CRSD
prcr-1068:failed to query resources
cannot Commu Nicate with CRSD
prcr-1068:failed to query resources
cannot communicate with CRSD
Prcr-1119:failed CRS Resources of Ora.cluster_vip_net1.type type
prcr-1068:failed to query resources
cannot communicate with crsd< br>prcr-1070:failed to check if resource ORA.GSD are registered
cannot communicate with CRSD
Prcr-1070:failed to Check if resource ora.ons is registered
cannot communicate with CRSD
Crs-4535:cannot communicate with Cluster ready Services
Crs-4000:command Stop failed, or completed with errors.
Crs-4535:cannot communicate with Cluster ready Services
Crs-4000:command Delete failed, or completed with errors.
Crs-2673:attempting to stop ' ora.ctssd ' on ' node2 '
Crs-2673:attempting to stop ' ora.evmd ' on ' node2 '
Crs-2673:attempting to stop ' ora.asm ' on ' node2 '
Crs-2677:stop of ' ora.evmd ' on ' Node2 ' succeeded
Crs-2677:stop of ' ora.ctssd ' on ' Node2 ' succeeded
Crs-2677:stop of ' ora.asm ' on ' Node2 ' succeeded
Crs-2673:attempting to stop ' ora.cluster_interconnect.haip ' on ' node2 '
Crs-2677:stop of ' ora.cluster_interconnect.haip ' on ' Node2 ' succeeded
Crs-2673:attempting to stop ' ora.cssd ' on ' node2 '
Crs-2677:stop of ' ora.cssd ' on ' Node2 ' succeeded
Crs-2672:attempting to start ' ora.cssdmonitor ' on ' node2 '
Crs-2676:start of ' ora.cssdmonitor ' on ' Node2 ' succeeded
Crs-2672:attempting to start ' ora.cssd ' on ' node2 '
Crs-2672:attempting to start ' Ora.diskmon ' on ' node2 '
Crs-2676:start of ' Ora.diskmon ' on ' Node2 ' succeeded
Crs-2676:start of ' ora.cssd ' on ' Node2 ' succeeded
Crs-4611:successful deletion of voting disk +data.
ASM de-configuration trace File location:/tmp/asmcadc_clean2016-10-31_02-02-22-pm.log
ASM Clean Configuration START
ASM Clean Configuration END
ASM with SID +asm1 deleted successfully. Check/tmp/asmcadc_clean2016-10-31_02-02-22-pm.log for details.
crs-2791:starting shutdown of Oracle high availability services-managed resources on ' Node2 '
Crs-2673:attempting to stop ' ora.ctssd ' on ' node2 '
Crs-2673:attempting to stop ' ora.asm ' on ' node2 '
Crs-2673:attempting to stop ' ora.mdnsd ' on ' node2 '
Crs-2677:stop of ' ora.mdnsd ' on ' Node2 ' succeeded
Crs-2677:stop of ' ora.ctssd ' on ' Node2 ' succeeded
Crs-2677:stop of ' ora.asm ' on ' Node2 ' succeeded
Crs-2673:attempting to stop ' ora.cluster_interconnect.haip ' on ' node2 '
Crs-2677:stop of ' ora.cluster_interconnect.haip ' on ' Node2 ' succeeded
Crs-2673:attempting to stop ' ora.cssd ' on ' node2 '
Crs-2677:stop of ' ora.cssd ' on ' Node2 ' succeeded
Crs-2673:attempting to stop ' ORA.CRF ' on ' node2 '
Crs-2677:stop of ' ORA.CRF ' on ' Node2 ' succeeded
Crs-2673:attempting to stop ' ora.gipcd ' on ' node2 '
Crs-2677:stop of ' ora.gipcd ' on ' Node2 ' succeeded
Crs-2673:attempting to stop ' ora.gpnpd ' on ' node2 '
Crs-2677:stop of ' ora.gpnpd ' on ' Node2 ' succeeded
Crs-2793:shutdown of Oracle High availability services-managed resources on ' Node2 ' have completed
Crs-4133:oracle High Availability Services have been stopped.
removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
Rebuilding OCR and OLR, using the root.sh script to complete the rebuild, is actually installing the script executed in the RAC, which is the default location of $grid_home
Node 1
[Email protected] grid]#./root.sh
Performing root user operation for Oracle 11g
The following environment variables is set as:
oracle_owner= Grid
Oracle_home=/u01/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "Dbhome" has not changed. No need to overwrite.
The contents of "oraenv" has not changed. No need to overwrite.
The contents of "coraenv" has not changed. No need to overwrite.
Entries'll be added to The/etc/oratab file as needed by
Database Configuration Assistant When a database is created
Finished running generic part of root script.
Now product-specific root actions would be performed.
Using configuration parameter file:/u01/11.2.0/grid/crs/install/crsconfig_params
User ignored prerequisites during installation
Installing Trace File Analyzer
OLR initialization-successful
Adding Clusterware entries to upstart
Crs-2672:attempting to start ' ora.mdnsd ' on ' node1 '
Crs-2676:start of ' ora.mdnsd ' on ' Node1 ' succeeded
Crs-2672:attempting to start ' ora.gpnpd ' on ' node1 '
Crs-2676:start of ' ora.gpnpd ' on ' Node1 ' succeeded
Crs-2672:attempting to start ' ora.cssdmonitor ' on ' node1 '
Crs-2672:attempting to start ' ora.gipcd ' on ' node1 '
Crs-2676:start of ' ora.cssdmonitor ' on ' Node1 ' succeeded
Crs-2676:start of ' ora.gipcd ' on ' Node1 ' succeeded
Crs-2672:attempting to start ' ora.cssd ' on ' node1 '
Crs-2672:attempting to start ' Ora.diskmon ' on ' node1 '
Crs-2676:start of ' Ora.diskmon ' on ' Node1 ' succeeded
Crs-2676:start of ' ora.cssd ' on ' Node1 ' succeeded
ASM created and started successfully.
Disk Group DATA created successfully.
CLSCFG:-install Mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ' root ', privgrp ' root '.
Operation successful.
Successful addition of voting disk 4331DAD495C14F71BFDB6D4F1A82D2F9.
Successfully replaced voting disk group with +data.
crs-4266:voting file (s) successfully replaced
Node 2
[Email protected] grid]#./root.sh
Performing root user operation for Oracle 11g
The following environment variables is set as:
oracle_owner= Grid
Oracle_home=/u01/11.2.0/grid
Enter The full pathname of the local bin directory: [/usr/local/bin]:
The contents of "Dbhome" has not changed. No need to overwrite.
The contents of "oraenv" has not changed. No need to overwrite.
The contents of "coraenv" has not changed. No need to overwrite.
Entries'll be added to The/etc/oratab file as needed by
Database Configuration Assistant When a database is created
Finished running generic part of root script.
Now product-specific root actions would be performed.
Using configuration parameter file:/u01/11.2.0/grid/crs/install/crsconfig_params
User ignored prerequisites during installation
Installing Trace File Analyzer
OLR initialization-successful
Adding Clusterware entries to upstart
Crs-4402:the CSS Daemon was started in exclusive mode but found a active CSS daemon on node node1, number 1, and are term Inating
An active cluster is found during exclusive startup, restarting to join the cluster
Preparing Packages for installation ...
Cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Check resource information
Node 1Ora. Listener_scan1.lsnr
1 Online Online Node1
Ora.cvu
1 Online Online Node1
Ora.node1.vip
1 Online Online Node1
Ora.node2.vip
1 Online Online Node2
Ora.oc4j
1 Online Online Node1
Ora.scan1.vip
1 Online Online Node1
Ora. Listener_scan1.lsnr
1 Online Online Node1
Ora.cvu
1 Online Online Node1
Ora.node1.vip
1 Online Online Node1
Ora.node2.vip
1 Online Online Node2
Ora.oc4j
1 Online Online Node1
Ora.scan1.vip
1 Online Online Node1
View Disk group information and mount it manually if it is not mounted:
Sql> select Name,state from V$asm_diskgroup;
4. Add resources (listening, database, instance, etc.)
Add Listener
[Email protected] ~]$ srvctl add listener-l Listener
View Monitoring
[Email protected] ~]$ SRVCTL Config listener
Adding DB and instance
[Email protected] ~]$ srvctl add database-h
[Email protected] ~]$ srvctl add database-d orcl-o/u01/app/oracle/product/11.2.0/db_1-c RAC
[Email protected] ~]$ srvctl add instance-h
[Email protected] ~]$ srvctl add instance-d orcl-i orcl1-n node1
[Email protected] ~]$ srvctl add instance-d orcl-i orcl2-n node2
[Email protected] ~]$ srvctl config database-d orcl
5, the resources have been added, restart the cluster
[Email protected] grid]# Crsctl stop Cluster-all
[Email protected] grid]# crsctl start Cluster-all
After the addition is complete, there may be an issue where the database does not start automatically. Attempt to execute the following statement:
[Email protected] ~]$ Srvctl enable database-d ORCL
[Email protected] ~]$ srvctl enable instance-d orcl-i ORCL1
[Email protected] ~]$ srvctl enable instance-d orcl-i Orcl2
[Email protected] ~]$ srvctl start database-d ORCL
Oracle11gr2-rac OCR No backup case damage recovery
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
