Oracle 11.2.0.1 RAC GRID failed to start: Oracle High Availability Services startup failed____oracle

, the 11.2.0.1 RAC that is installed on the virtual machine is chosen 11.2.0.1 because of problems with public IP and private network segments. During the installation instance, the computer crashes and restarts, and CRS fails to start.

[Root@rac1 bin]#./crsctlstart CRS

Crs-4124:oracle highavailability Services startup failed.

Crs-4000:command startfailed, or completed with errors.

[Root@rac1 bin]# Ps-ef|grep has

Root 8081 1 0 03:14? 00:00:00/u01/app/grid/11.2.0/bin/ohasd.bin reboot

Root 8137 4230 1 03:23 pts/0 00:00:00 grep has

[Root@rac1 bin]# kill-9 8081

[Root@rac1 bin]#./crsctl start CRS

Crs-4124:oracle High Availability Servicesstartup failed.

Crs-4000:command Start failed, orcompleted with errors.

View log:

[Grid@rac2 rac2]$ LL

Total 72

Drwxr-x---2 grid oinstall 4096 Nov 2100:38 admin

Drwxrwxr-t 4 root oinstall 4096 Nov 2100:38 Agent

-rw-rw-r--1 rootroot 9693 Nov 02:26 alertrac2.log

Drwxr-x---2 grid oinstall 4096 Nov 2100:43 Client

Drwxr-x---2 root oinstall 4096 Nov 2100:42 CRSD

Drwxr-x---2 grid oinstall 4096 Nov 2100:39 CSSD

Drwxr-x---2 root oinstall 4096 Nov 2100:41 CTSSD

Drwxr-x---2 grid oinstall 4096 Nov 2100:39 DiskMon

Drwxr-x---2 grid oinstall 4096 Nov 2100:42 EVMD

Drwxr-x---2 grid oinstall 4096 Nov 2100:38 GIPCD

Drwxr-x---2 root oinstall 4096 Nov 2100:38 GNSD

Drwxr-x---2 grid oinstall 4096 Nov 2100:40 GPNPD

Drwxr-x---2 grid oinstall 4096 Nov 2100:38 MDNSD

Drwxr-x---2 root oinstall 4096 Nov 2100:39 OHASD

Drwxrwxr-t 5 grid oinstall 4096 Nov 2100:38 RACG

Drwxr-x---2 grid oinstall 4096 Nov 2100:42 SRVM

Other files are not updated except that the Alertrac2.log is updated during downtime. Reboot to Node 1:

[Root@rac1 client]# LL

Total 124

-rw-r--r--1 root root 193 Nov 00:31 Clscfg.log

-rw-rw-rw-1 root root 28635 Nov 00:32 crsctl.log

-rw-r--r--1 root root 114 Nov 00:32 CRSCTL.TRC

-rw-r--r--1 Gridoinstall 663 Nov 03:08 Css.log

-rw-r--r--1 grid oinstall 1051 Nov 00:28 Gpnptool_11653.log

-rw-r--r--1 grid oinstall 114 Nov 00:28 GPNPTOOL_11653.TRC

-rw-r--r--1 grid oinstall 1461 Nov 00:28 Gpnptool_11660.log

-rw-r--r--1 grid oinstall 114 Nov 00:28 GPNPTOOL_11660.TRC

-rw-r--r--1 grid oinstall 551 Nov 00:35 Oclskd.log

-rw-r-----1 Root 6100 Nov 00:27 Ocrconfig_11312.log

-rw-r--r--1 root root 3170 Nov 00:31 ocrconfig_12191.log

-rw-r-----1 root 342 Nov 00:37 Ocrconfig_13798.log

-rw-r--r--1 grid oinstall 33862 Nov 2100:45 oifcfg.log

-rw-r--r--1 grid oinstall 114 Nov 00:45 OIFCFG.TRC

-rw-r--r--1 root root 1067 Nov 00:36 Olsnodes.log

-rw-r--r--1 grid oinstall 114 Nov 00:37 OLSNODES.TRC

Only the following errors are--css.log:

[Root@rac1 client]# Cat Css.log

Oracle Database 11g clusterware release11.2.0.1.0-production Copyright 1996, 2009 Oracle. All rights reserved.

2012-11-21 03:08:22.764: [CSSCLNT][4171966208]CLSSSCCONNECT:GIPC request failed with (0x13)

2012-11-21 03:08:22.764: [Cssclnt][4171966208]clsssinitnative:connect failed, RC 29

2012-11-21 03:08:28.140: [CSSCLNT][4171966208]CLSSSCCONNECT:GIPC request failed with (0x13)

2012-11-21 03:08:28.140: [Cssclnt][4171966208]clsssinitnative:connect failed, RC 29

2012-11-21 03:08:37.908: [CSSCLNT][4171966208]CLSSSCCONNECT:GIPC request failed with (0x13)

2012-11-21 03:08:37.908:[Cssclnt][4171966208]clsssinitnative:connect failed, RC 29

According to MOS description:

How Totroubleshoot Grid infrastructure Startup Issues [ID 1050908.1]

http://www.cndba.cn/Dave/article/575

1. OCSSD is fully up

If Ocssd.bin is isn't fully up, crsd.log'll show messages like following:

2010-02-03 22:37:51.638: [CSSCLNT][1548456880]CLSSSCCONNECT:GIPC request failed with (0x16)
2010-02-03 22:37:51.638: [cssclnt][1548456880]clsssinitnative:connect FAILED,RC 29
2010-02-03 22:37:51.639: [crsrti][1548456880] CSS is not ready. Receivedstatus 3 from CSS. Waiting for good status.

Is that the OCSSD process cannot start. Then why is the OCSS process unable to start. We strace the OHASD process:

[Root@rac1 client]# Ps-ef|grep has

Root 12192 1 012:44? 00:00:00/u01/app/grid/11.2.0/bin/ohasd.bin reboot

Root 12281 8085 0 13:05 pts/2 00:00:00 grep has

[Root@rac1 client]# strace-p 12192-o dave.log

Process 12192 Attached-interrupt to quit

Quit

Process 12192 Detached

[Root@rac1 client]#

[Root@rac1 client]# ls

Clscfg.log dave.log gpnptool_11660.trc Ocrconfig_13798.log olsnodes.trc

Crsctl.log Gpnptool_11653.log Oclskd.log Oifcfg.log

CRSCTL.TRC gpnptool_11653.trc Ocrconfig_11312.log OIFCFG.TRC

Css.log Gpnptool_11660.log Ocrconfig_12191.log Olsnodes.log

[Root@rac1 client]# Cat Dave.log

Open ("/VAR/TMP/.ORACLE/NPOHASD", O_wronly <unfinished ...>

Here is a very important piece of information. This is the file here, this file, we will also encounter when installing 11.2.0.1 RAC, which should be said to be a 11.2.0.1 bug.

Reference:

Oracle 11gRAC OHASD failed to start at/u01/app/11.2.0/grid/crs/install/rootcrs.pl line443 solution

http://www.cndba.cn/Dave/article/464

So before you start CRS, specify the DD command at 2 nodes:

[Root@rac1 client]#/bin/ddif=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1

Then start, there's no problem:

[Root@rac1 bin]#./crsctlstart CRS

Crs-4123:oracle High Availability Serviceshas been started.

[Root@rac2 bin]#./crsctlstart CRS

Crs-4123:oracle High Availability Serviceshas been started.

[Root@rac2 bin]#./crsctl Check CRS

Crs-4638:oracle High Availabilityservices is online

Crs-4535:cannot Communicate with Clusterready Services

Crs-4530:communications failure contactingcluster Synchronization Services Daemon

Crs-4534:cannotcommunicate with Event Manager

[Root@rac1 bin]#./crsctlcheck CRS

Crs-4638:oracle High Availability servicesis Online

Crs-4535:cannot Communicate with Clusterready Services

Crs-4530:communications failure contactingcluster Synchronization Services Daemon

Crs-4534:cannot Communicate with EventManager

[Root@rac1 bin]#./crsctlstart Cluster-all

Crs-5702:resource ' ORA.CRSD ' is alreadyrunning on ' Rac1 '

Crs-5702:resource ' ORA.CRSD ' is alreadyrunning on ' Rac2 '

[Root@rac1 bin]#./crsctlcheck CRS

Crs-4638:oracle High Availability servicesis Online

Crs-4535:cannot Communicate with Clusterready Services

Crs-4529:cluster Synchronization Servicesis Online

Crs-4533:event Manager is online

[Root@rac2 bin]#./crsctlcheck CRS

Crs-4638:oracle High Availability servicesis Online

Crs-4535:cannot Communicate with Clusterready Services

Crs-4529:cluster Synchronization Servicesis Online

Crs-4533:event Manager is online

--View the process, all pulled up. Note that the 11g process starts somewhat slowly, wait a little more.

[Root@rac2 u01]# SH crs_stat.sh

Name Target State Host

------------------------------ -------------------  -------

Ora. DATA.DG Online Online Rac1

Ora. FRA.DG Online Online Rac1

Ora. Listener.lsnr Online Online Rac1

Ora. Listener_scan1.lsnr Online Online Rac2

Ora. OCRVOTING.DG Online Online Rac1

Ora.asm Online Online Rac1

Ora.dave.db OFFLINE OFFLINE

Ora.eons Online Online Rac1

ORA.GSD OFFLINE OFFLINE

Ora.net1.network Online Online Rac1

ora.oc4j OFFLINE OFFLINE

Ora.ons Online Online Rac1

Ora.rac1.ASM1.asm Online Online Rac1

Ora.rac1.LISTENER_RAC1.lsnr Online Online Rac1

ORA.RAC1.GSD OFFLINE OFFLINE

Ora.rac1.ons Online Online Rac1

ORA.RAC1.VIP Online Online Rac1

Ora.rac2.ASM2.asm Online Online Rac2

Ora.rac2.LISTENER_RAC2.lsnr Online Online Rac2

ORA.RAC2.GSD OFFLINE OFFLINE

Ora.rac2.ons Online Online Rac2

ORA.RAC2.VIP Online Online Rac2

ORA.SCAN1.VIP Online Online Rac2

We can now process our example and then upgrade to 11.2.0.3.4. Lest you encounter this problem every time.

---------------------------------------------------------------------------------------

Copyright, the article is allowed to reprint, but must be linked to the source address, otherwise held legal responsibility!

qq:492913789

Email:ahdba@qq.com

Blog:http://www.cndba.cn/dave

Weibo:http://weibo.com/tianlesoftware

Twitter:http://twitter.com/tianlesoftware

Facebook:http://www.facebook.com/tianlesoftware

Linkedin:http://cn.linkedin.com/in/tianlesoftware