, the 11.2.0.1 RAC that is installed on the virtual machine is chosen 11.2.0.1 because of problems with public IP and private network segments. During the installation instance, the computer crashes and restarts, and CRS fails to start.
[Root@rac1 bin]#./crsctlstart CRS
Crs-4124:oracle highavailability Services startup failed.
Crs-4000:command startfailed, or completed with errors.
[Root@rac1 bin]# Ps-ef|grep has
Root 8081 1 0 03:14? 00:00:00/u01/app/grid/11.2.0/bin/ohasd.bin reboot
Root 8137 4230 1 03:23 pts/0 00:00:00 grep has
[Root@rac1 bin]# kill-9 8081
[Root@rac1 bin]#./crsctl start CRS
Crs-4124:oracle High Availability Servicesstartup failed.
Crs-4000:command Start failed, orcompleted with errors.
View log:
[Grid@rac2 rac2]$ LL
Total 72
Drwxr-x---2 grid oinstall 4096 Nov 2100:38 admin
Drwxrwxr-t 4 root oinstall 4096 Nov 2100:38 Agent
-rw-rw-r--1 rootroot 9693 Nov 02:26 alertrac2.log
Drwxr-x---2 grid oinstall 4096 Nov 2100:43 Client
Drwxr-x---2 root oinstall 4096 Nov 2100:42 CRSD
Drwxr-x---2 grid oinstall 4096 Nov 2100:39 CSSD
Drwxr-x---2 root oinstall 4096 Nov 2100:41 CTSSD
Drwxr-x---2 grid oinstall 4096 Nov 2100:39 DiskMon
Drwxr-x---2 grid oinstall 4096 Nov 2100:42 EVMD
Drwxr-x---2 grid oinstall 4096 Nov 2100:38 GIPCD
Drwxr-x---2 root oinstall 4096 Nov 2100:38 GNSD
Drwxr-x---2 grid oinstall 4096 Nov 2100:40 GPNPD
Drwxr-x---2 grid oinstall 4096 Nov 2100:38 MDNSD
Drwxr-x---2 root oinstall 4096 Nov 2100:39 OHASD
Drwxrwxr-t 5 grid oinstall 4096 Nov 2100:38 RACG
Drwxr-x---2 grid oinstall 4096 Nov 2100:42 SRVM
Other files are not updated except that the Alertrac2.log is updated during downtime. Reboot to Node 1:
[Root@rac1 client]# LL
Total 124
-rw-r--r--1 root root 193 Nov 00:31 Clscfg.log
-rw-rw-rw-1 root root 28635 Nov 00:32 crsctl.log
-rw-r--r--1 root root 114 Nov 00:32 CRSCTL.TRC
-rw-r--r--1 Gridoinstall 663 Nov 03:08 Css.log
-rw-r--r--1 grid oinstall 1051 Nov 00:28 Gpnptool_11653.log
-rw-r--r--1 grid oinstall 114 Nov 00:28 GPNPTOOL_11653.TRC
-rw-r--r--1 grid oinstall 1461 Nov 00:28 Gpnptool_11660.log
-rw-r--r--1 grid oinstall 114 Nov 00:28 GPNPTOOL_11660.TRC
-rw-r--r--1 grid oinstall 551 Nov 00:35 Oclskd.log
-rw-r-----1 Root 6100 Nov 00:27 Ocrconfig_11312.log
-rw-r--r--1 root root 3170 Nov 00:31 ocrconfig_12191.log
-rw-r-----1 root 342 Nov 00:37 Ocrconfig_13798.log
-rw-r--r--1 grid oinstall 33862 Nov 2100:45 oifcfg.log
-rw-r--r--1 grid oinstall 114 Nov 00:45 OIFCFG.TRC
-rw-r--r--1 root root 1067 Nov 00:36 Olsnodes.log
-rw-r--r--1 grid oinstall 114 Nov 00:37 OLSNODES.TRC
Only the following errors are--css.log:
[Root@rac1 client]# Cat Css.log
Oracle Database 11g clusterware release11.2.0.1.0-production Copyright 1996, 2009 Oracle. All rights reserved.
2012-11-21 03:08:22.764: [CSSCLNT][4171966208]CLSSSCCONNECT:GIPC request failed with (0x13)
2012-11-21 03:08:22.764: [Cssclnt][4171966208]clsssinitnative:connect failed, RC 29
2012-11-21 03:08:28.140: [CSSCLNT][4171966208]CLSSSCCONNECT:GIPC request failed with (0x13)
2012-11-21 03:08:28.140: [Cssclnt][4171966208]clsssinitnative:connect failed, RC 29
2012-11-21 03:08:37.908: [CSSCLNT][4171966208]CLSSSCCONNECT:GIPC request failed with (0x13)
2012-11-21 03:08:37.908:[Cssclnt][4171966208]clsssinitnative:connect failed, RC 29
According to MOS description:
How Totroubleshoot Grid infrastructure Startup Issues [ID 1050908.1]
http://www.cndba.cn/Dave/article/575
1. OCSSD is fully up
If Ocssd.bin is isn't fully up, crsd.log'll show messages like following:
2010-02-03 22:37:51.638: [CSSCLNT][1548456880]CLSSSCCONNECT:GIPC request failed with (0x16)
2010-02-03 22:37:51.638: [cssclnt][1548456880]clsssinitnative:connect FAILED,RC 29
2010-02-03 22:37:51.639: [crsrti][1548456880] CSS is not ready. Receivedstatus 3 from CSS. Waiting for good status.
Is that the OCSSD process cannot start. Then why is the OCSS process unable to start. We strace the OHASD process:
[Root@rac1 client]# Ps-ef|grep has
Root 12192 1 012:44? 00:00:00/u01/app/grid/11.2.0/bin/ohasd.bin reboot
Root 12281 8085 0 13:05 pts/2 00:00:00 grep has
[Root@rac1 client]# strace-p 12192-o dave.log
Process 12192 Attached-interrupt to quit
Quit
Process 12192 Detached
[Root@rac1 client]#
[Root@rac1 client]# ls
Clscfg.log dave.log gpnptool_11660.trc Ocrconfig_13798.log olsnodes.trc
Crsctl.log Gpnptool_11653.log Oclskd.log Oifcfg.log
CRSCTL.TRC gpnptool_11653.trc Ocrconfig_11312.log OIFCFG.TRC
Css.log Gpnptool_11660.log Ocrconfig_12191.log Olsnodes.log
[Root@rac1 client]# Cat Dave.log
Open ("/VAR/TMP/.ORACLE/NPOHASD", O_wronly <unfinished ...>
Here is a very important piece of information. This is the file here, this file, we will also encounter when installing 11.2.0.1 RAC, which should be said to be a 11.2.0.1 bug.
Reference:
Oracle 11gRAC OHASD failed to start at/u01/app/11.2.0/grid/crs/install/rootcrs.pl line443 solution
http://www.cndba.cn/Dave/article/464
So before you start CRS, specify the DD command at 2 nodes:
[Root@rac1 client]#/bin/ddif=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1
Then start, there's no problem:
[Root@rac1 bin]#./crsctlstart CRS
Crs-4123:oracle High Availability Serviceshas been started.
[Root@rac2 bin]#./crsctlstart CRS
Crs-4123:oracle High Availability Serviceshas been started.
[Root@rac2 bin]#./crsctl Check CRS
Crs-4638:oracle High Availabilityservices is online
Crs-4535:cannot Communicate with Clusterready Services
Crs-4530:communications failure contactingcluster Synchronization Services Daemon
Crs-4534:cannotcommunicate with Event Manager
[Root@rac1 bin]#./crsctlcheck CRS
Crs-4638:oracle High Availability servicesis Online
Crs-4535:cannot Communicate with Clusterready Services
Crs-4530:communications failure contactingcluster Synchronization Services Daemon
Crs-4534:cannot Communicate with EventManager
[Root@rac1 bin]#./crsctlstart Cluster-all
Crs-5702:resource ' ORA.CRSD ' is alreadyrunning on ' Rac1 '
Crs-5702:resource ' ORA.CRSD ' is alreadyrunning on ' Rac2 '
[Root@rac1 bin]#./crsctlcheck CRS
Crs-4638:oracle High Availability servicesis Online
Crs-4535:cannot Communicate with Clusterready Services
Crs-4529:cluster Synchronization Servicesis Online
Crs-4533:event Manager is online
[Root@rac2 bin]#./crsctlcheck CRS
Crs-4638:oracle High Availability servicesis Online
Crs-4535:cannot Communicate with Clusterready Services
Crs-4529:cluster Synchronization Servicesis Online
Crs-4533:event Manager is online
--View the process, all pulled up. Note that the 11g process starts somewhat slowly, wait a little more.
[Root@rac2 u01]# SH crs_stat.sh
Name Target State Host
------------------------------ ------------------- -------
Ora. DATA.DG Online Online Rac1
Ora. FRA.DG Online Online Rac1
Ora. Listener.lsnr Online Online Rac1
Ora. Listener_scan1.lsnr Online Online Rac2
Ora. OCRVOTING.DG Online Online Rac1
Ora.asm Online Online Rac1
Ora.dave.db OFFLINE OFFLINE
Ora.eons Online Online Rac1
ORA.GSD OFFLINE OFFLINE
Ora.net1.network Online Online Rac1
ora.oc4j OFFLINE OFFLINE
Ora.ons Online Online Rac1
Ora.rac1.ASM1.asm Online Online Rac1
Ora.rac1.LISTENER_RAC1.lsnr Online Online Rac1
ORA.RAC1.GSD OFFLINE OFFLINE
Ora.rac1.ons Online Online Rac1
ORA.RAC1.VIP Online Online Rac1
Ora.rac2.ASM2.asm Online Online Rac2
Ora.rac2.LISTENER_RAC2.lsnr Online Online Rac2
ORA.RAC2.GSD OFFLINE OFFLINE
Ora.rac2.ons Online Online Rac2
ORA.RAC2.VIP Online Online Rac2
ORA.SCAN1.VIP Online Online Rac2
We can now process our example and then upgrade to 11.2.0.3.4. Lest you encounter this problem every time.
---------------------------------------------------------------------------------------
Copyright, the article is allowed to reprint, but must be linked to the source address, otherwise held legal responsibility!
qq:492913789
Email:ahdba@qq.com
Blog:http://www.cndba.cn/dave
Weibo:http://weibo.com/tianlesoftware
Twitter:http://twitter.com/tianlesoftware
Facebook:http://www.facebook.com/tianlesoftware
Linkedin:http://cn.linkedin.com/in/tianlesoftware