Oracle RAC root. Sh error timed out waiting for the CRS stack to start Solution

 

I. Problem Description

 

Install RAC of 11.2.0.1 on Oracle Linux 6.1 and report time out when root. Sh is executed on the second node, as shown below:

[Root @ rac2 ~] #/U01/APP/11.2.0/GRID/root. Sh

Running Oracle 11g root. Sh script...

 

The following environment variables are Setas:

Oracle_owner = Oracle

ORACLE_HOME =/u01/APP/11.2.0/Grid

 

Enter the full pathname of the local bindirectory: [/usr/local/bin]:

Copying dbhome to/usr/local/bin...

Copying oraenv to/usr/local/bin...

Copying coraenv to/usr/local/bin...

 

Entries will be added to the/etc/oratabfile as needed

Database Configuration assistant when adatabase is created

Finished running generic part of root. shscript.

Now product-specific root actions will beperformed.

2012-06-27 14: 46: 35: parsing the Host Name

2012-06-27 14: 46: 35: checking for superuser privileges

2012-06-27 14: 46: 35: user has super userprivileges

Using configuration parameter file:/u01/APP/11.2.0/GRID/CRS/install/crsconfig_params

Creating trace directory

Local add Mode

Creating OCR keys for user 'root', privgrp 'root '..

Operation successful.

Adding daemon to inittab

CRS-4123: Oracle High Availability serviceshas been started.

Ohasd is starting

Advm/ACFs is not supported onoraclelinux-release-6Server-1.0.2.x86_64

 

 

 

The CRS-4402: the CSS Daemon was started inexclusive mode but found an active CSS daemon on node Rac1, Number 1, and isterminating

An active cluster was found duringexclusive startup, restarting to join the Cluster

CRS-2672: attempting to start 'ora. mdnsd' on 'rac2'

CRS-2676: Start of 'ora. mdnsd' on 'rac2' succeeded

CRS-2672: attempting to start 'ora. gipcd 'on 'rac2'

CRS-2676: Start of 'ora. gipcd' on 'rac2' succeeded

CRS-2672: attempting to start 'ora. gpnpd 'on 'rac2'

CRS-2676: Start of 'ora. gpnpd 'on 'rac2' succeeded

CRS-2672: attempting to start'ora.css dmonitor 'on 'rac2'

CRS-2676: Start of 'ora.css dmonitor 'on 'rac2' succeeded

CRS-2672: attempting to start 'ora.css d' on 'rac2'

CRS-2672: attempting to start 'ora. diskmon 'on 'rac2'

CRS-2676: Start of 'ora. diskmon' on 'rac2' succeeded

CRS-2676: Start of 'ora.css d' on 'rac2' succeeded

CRS-2672: attempting to start 'ora. ctssd 'on 'rac2'

CRS-2676: Start of 'ora. ctssd 'on 'rac2' succeeded

CRS-2672: attempting to start 'ora. ASM 'on 'rac2'

CRS-2676: Start of 'ora. ASM 'on 'rac2' succeeded

CRS-2672: attempting to start 'ora. crsd' on 'rac2'

CRS-2676: Start of 'ora. crsd' on 'rac2' succeeded

CRS-2672: attempting to start 'ora. evmd 'on 'rac2'

CRS-2676: Start of 'ora. evmd 'on 'rac2' succeeded

Timed outwaiting for the CRS stack to start.

 

 

View the related status:

[Oracle @ Rac1 bin] $./crsctl check cluster-all

**************************************** **********************

Rac1:

CRS-4537: Cluster ready services is online

CRS-4529: Cluster synchronization servicesis online

CRS-4533: eventmanager is online

 

[Oracle @ rac2 bin] $./crsctl check cluster-all

**************************************** **********************

Rac2:

CRS-4535: cannot communicate with clusterready services

CRS-4529: Cluster synchronization servicesis online

CRS-4533: eventmanager is online

 

[Oracle @ Rac1 bin] $./crs_stat-T-V

Name type R/ra f/ft target State host

----------------------------------------------------------------------

Ora. Data. DG ora... Up. Type 0/5 0/online Rac1

Ora... n1.lsnr ora... er. Type 0/5 0/0 online Rac1

Ora. ASM ora. ASM. Type 0/5 0/online Rac1

Ora. Eons ora. Eons. Type 0/3 0/online Rac1

Ora. GSD ora. GSD. Type 0/5 0/offline

Ora... network ora... rk. Type 0/5 0/online Rac1

Ora. Solaris ora. productname. Type 0/5 0/0 offline

Ora. ONS ora. ons. Type 0/3 0/online Rac1

Ora... sm1.asm application 0/5 0/0 online Rac1

Ora. rac1.gsd application 0/5 0/0 offline

Ora. rac1.ons application 0/3 0/0 online Rac1

Ora. rac1.vip ora... t1.type 0/0 0/0 online Rac1

Ora. scan1.vip ora... IP. Type 0/0 0/0 online Rac1

 

[Oracle @ rac2 bin] $./crs_stat-T-V

CRS-0184: cannot communicate with the crsdaemon.

 

[Oracle @ rac2 bin] $

 

The command on node 2 is not successfully executed.

 

 

Ii. Description on MOS

Root. Sh failing with 'prom _ RPC: CLSC sendfailure .. RET code 6' [ID 745215.1]

 

2.1 symptoms

During CRS install root. Sh fails on thelast node with follow message:

Waiting for the Oracle crsd and evmd tostart
Waiting for the Oracle crsd and evmd to start
Waiting for the Oracle crsd and evmd to start
Waiting for the Oracle crsd and evmd to start
Waiting for the Oracle crsd and evmd to start
Timed out waiting for the CRS stack to start.

The crsd. log on the first node shows:

21:04:55. 087: [ocrmsg] [1325496672] prom_rpc: CLSC send failure .. RET
Code 6
21:04:55. 087: [ocrmsg] [1325496672] prom_rpc: Possible ocrretry
Scenario
21:04:55. 087: [ocrsrv] [1325496672] proas_forward_request: prom_tim
E_out or master fail
21:04:55. 296: [commcrs] [2540957248] clscsendx: (0xc43bd0) Connection
Not active

21:04:55. 296: [ocrmsg] [2540957248] prom_rpc: CLSC send failure .. RET
Code 6
21:04:55. 296: [ocrmsg] [2540957248] prom_rpc: Possible ocrretry
Scenario
21:04:55. 296: [ocrcli] [2540957248] proac_open_key: [system. CRS. debug.
Ist4-db1-3-sfm.COMMNS]: writer failed. retval [203]

The ocssd. log on the first node shows:

20:33:01. 626: [ocrapi] [2540958848] procr_open: node failure.
Attempting retry #0
20:33:02. 628: [ocrapi] [2540958848] procr_open: node failure.
Attempting retry #1
20:33:03. 631: [ocrapi] [2540958848] procr_open: node failure.
Attempting retry #2

The ocssd. log on the last node shows:

[Cssd] 04:46:42. 457 [1241577824]> trace: clssnmrcfgmgrthread:
Lastleader (1) Unique (1224650474)
[Cssd] 04:46:43. 219 [1168148832]> trace: clssnmsendvoteinfo:
Node (1) syncseqno (4)
[Cssd] 04:46:56. 338 [2537955008]> error: clssgmstartnmmon:
Timed out waiting on nested nm reconfig. Self-sacrifle icing to kick othersawake.
[Cssd] 04:46:56. 338 [2537955008]> error: startcmmon ():
Clssnmnmdetach failed-2
[Cssd] 04:46:56. 338 [2537955008]> trace: clssscctx: dump
0x0x5d2360, Len 3792

2.2 cause

This is due to a failure of communicationbetween crsd. Bin on nodes.

 

2.3 Solution

Check the network for the following items:

-Check to see that there is no firewall between the nodes
-Make sure that the MTU size is same.
-If MTU is larger than 1500, then the switch must be able to support largermtu size.
-Make sure that you have disabled SELinux
-Make sure that values are using full duplex and not Auto Negotiate.
-Misconfiguration on the switches will also cause this issue.

 

 

 

Iii. Solution

The cause prompted in the mos document is related to the firewall, time, SELinux, and nic type. It can be determined that the cause is related to the NIC, my reason is that the network card names of the two nodes are inconsistent, so after modifying the network card names to be consistent, try to run the root command again. sh command.

 

That is, before modification, Rac1 is eth0 and eth1, and node 2 is eth5 and eth6. how to modify the NIC name? This is a Google record, which is not described here.

 

Run the following command to uninstall an ECS instance:

/U01/APP/11.2.0/GRID/CRS/install/rootcrs. pl-deconfig-verbose-Force

 

Note the differences between oracle11g and 10g commands.

 

-- Uninstall:

[Root @ rac2 ~] #/U01/APP/11.2.0/GRID/CRS/install/rootcrs. pl-deconfig-verbose-Force

2012-06-27 15: 12: 30: parsing the Host Name

2012-06-27 15: 12: 30: checking for superuser privileges

2012-06-27 15: 12: 30: user has super userprivileges

Using configuration parameter file:/u01/APP/11.2.0/GRID/CRS/install/crsconfig_params

PRCR-1035: failed to look up CRS resourceora. cluster_vip.type for 1

PRCR-1068: failed to query Resources

Cannot communicate with crsd

PRCR-1070: failed to check if resourceora. GSD is registered

Cannot communicate with crsd

PRCR-1070: failed to check if resourceora. ONS is registered

Cannot communicate with crsd

PRCR-1070: failed to check if resourceora. Eons is registered

Cannot communicate with crsd

 

Advm/ACFs is not supported onoraclelinux-release-6Server-1.0.2.x86_64

 

ACFS-9201: not supported

CRS-2791: Starting shutdown of Oracle highavailability services-managed resources on 'rac2'

CRS-2673: attempting to stop 'ora. mdnsd' on 'rac2'

CRS-2673: attempting to stop 'ora. gpnpd 'on 'rac2'

CRS-2673: attempting to stop'ora.css dmonitor 'on 'rac2'

CRS-2673: attempting to stop 'ora. ctssd 'on 'rac2'

CRS-2673: attempting to stop 'ora. evmd 'on 'rac2'

CRS-2673: attempting to stop 'ora. ASM 'on 'rac2'

CRS-2677: Stop of 'ora.css dmonitor 'on 'rac2' succeeded

CRS-2677: Stop of 'ora. gpnpd 'on 'rac2' succeeded

CRS-2677: Stop of 'ora. mdnsd' on 'rac2' succeeded

CRS-2677: Stop of 'ora. evmd 'on 'rac2' succeeded

CRS-2677: Stop of 'ora. ctssd 'on 'rac2' succeeded

CRS-2677: Stop of 'ora. ASM 'on 'rac2' succeeded

CRS-2673: attempting to stop 'ora.css d' on 'rac2'

CRS-2677: Stop of 'ora.css d' on 'rac2' succeeded

CRS-2673: attempting to stop 'ora. diskmon 'on 'rac2'

CRS-2673: attempting to stop 'ora. gipcd 'on 'rac2'

CRS-2677: Stop of 'ora. gipcd' on 'rac2' succeeded

CRS-2677: Stop of 'ora. diskmon' on 'rac2' succeeded

CRS-2793: shutdown of Oracle highavailability services-managed resources on 'rac2' has completed

CRS-4133: Oracle High Availability serviceshas been stopped.

Error: Package cvuqdisk Is Not Installed

Successfully deconfigured oracleclusterware stack on this node

 

-- Re-execute root. Sh. This operation is successful.

 

[Root @ rac2 ~] #/U01/APP/11.2.0/GRID/root. Sh

Running Oracle 11g root. Sh script...

 

The following environment variables are Setas:

Oracle_owner = Oracle

ORACLE_HOME =/u01/APP/11.2.0/Grid

 

Enter the full pathname of the local bindirectory: [/usr/local/bin]:

The file "dbhome" already existsin/usr/local/bin. Overwrite it? (Y/N)

[N]:

The file "oraenv" already existsin/usr/local/bin. Overwrite it? (Y/N)

[N]:

The file "coraenv" already existsin/usr/local/bin. Overwrite it? (Y/N)

[N]:

 

Entries will be added to the/etc/oratabfile as needed

Database Configuration assistant when adatabase is created

Finished running generic part of root. shscript.

Now product-specific root actions will beperformed.

2012-06-27 16: 21: 25: parsing the Host Name

2012-06-27 16: 21: 25: checking for superuser privileges

2012-06-27 16: 21: 25: user has super userprivileges

Using configuration parameter file:/u01/APP/11.2.0/GRID/CRS/install/crsconfig_params

Local add Mode

Creating OCR keys for user 'root', privgrp 'root '..

Operation successful.

Adding daemon to inittab

CRS-4123: Oracle High Availability serviceshas been started.

Ohasd is starting

Advm/ACFs is not supported onoraclelinux-release-6Server-1.0.2.x86_64

 

 

 

The CRS-4402: the CSS Daemon was started inexclusive mode but found an active CSS daemon on node Rac1, Number 1, and isterminating

An active cluster was found duringexclusive startup, restarting to join the Cluster

CRS-2672: attempting to start 'ora. mdnsd' on 'rac2'

CRS-2676: Start of 'ora. mdnsd' on 'rac2' succeeded

CRS-2672: attempting to start 'ora. gipcd 'on 'rac2'

CRS-2676: Start of 'ora. gipcd' on 'rac2' succeeded

CRS-2672: attempting to start 'ora. gpnpd 'on 'rac2'

CRS-2676: Start of 'ora. gpnpd 'on 'rac2' succeeded

CRS-2672: attempting to start'ora.css dmonitor 'on 'rac2'

CRS-2676: Start of 'ora.css dmonitor 'on 'rac2' succeeded

CRS-2672: attempting to start 'ora.css d' on 'rac2'

CRS-2672: attempting to start 'ora. diskmon 'on 'rac2'

CRS-2676: Start of 'ora. diskmon' on 'rac2' succeeded

CRS-2676: Start of 'ora.css d' on 'rac2' succeeded

CRS-2672: attempting to start 'ora. ctssd 'on 'rac2'

CRS-2676: Start of 'ora. ctssd 'on 'rac2' succeeded

CRS-2672: attempting to start 'ora. ASM 'on 'rac2'

CRS-2676: Start of 'ora. ASM 'on 'rac2' succeeded

CRS-2672: attempting to start 'ora. crsd' on 'rac2'

CRS-2676: Start of 'ora. crsd' on 'rac2' succeeded

CRS-2672: attempting to start 'ora. evmd 'on 'rac2'

CRS-2676: Start of 'ora. evmd 'on 'rac2' succeeded

 

Rac2 16:25:16/u01/APP/11.2.0/GRID/CDATA/rac2/backup_20120627_162516.olr

Preparing packages for installation...

Cvuqdisk-1.0.7-1

Configure Oracle grid infrastructure for acluster... succeeded

Updating inventory properties forclusterware

Starting Oracle universal installer...

 

Checking swap space: must be greater than500 mb. Actual 999 MB passed

The inventory pointer is located at/etc/orainst. Loc

The inventory is located at/u01/APP/orainventory

[Root @ rac2 ~] #

 

 

Verification:

 

[Oracle @ rac2 bin] $./crsctl check cluster-all

**************************************** **********************

Rac1:

CRS-4537: Cluster ready services is online

CRS-4529: Cluster synchronization servicesis online

CRS-4533: event manager is online.

**************************************** **********************

Rac2:

CRS-4537: Cluster ready services is online

CRS-4529: Cluster synchronization servicesis online

CRS-4533: event manager is online.

**************************************** **********************

[Oracle @ rac2 bin] $./crs_stat-T-V

Name type R/ra f/ft target State host

----------------------------------------------------------------------

Ora. Data. DG ora... Up. Type 0/5 0/online Rac1

Ora... n1.lsnr ora... er. Type 0/5 0/0 online Rac1

Ora. ASM ora. ASM. Type 0/5 0/online Rac1

Ora. Eons ora. Eons. Type 0/3 0/online Rac1

Ora. GSD ora. GSD. Type 0/5 0/offline

Ora... network ora... rk. Type 0/5 0/online Rac1

Ora. Solaris ora. productname. Type 0/5 0/0 offline

Ora. ONS ora. ons. Type 0/3 0/online Rac1

Ora... sm1.asm application 0/5 0/0 online Rac1

Ora. rac1.gsd application 0/5 0/0 offline

Ora. rac1.ons application 0/3 0/0 online Rac1

Ora. rac1.vip ora... t1.type 0/0 0/0 online Rac1

Ora... sm2.asm application 0/5 0/0 online rac2

Ora. rac2.gsd application 0/5 0/0 offline

Ora. rac2.ons application 0/3 0/0 online rac2

Ora. rac2.vip ora... t1.type 0/0 0/0 online rac2

Ora. scan1.vip ora... IP. Type 0/0 0/0 online Rac1

[Oracle @ rac2 bin] $

 

 

Root. Sh is successfully executed.

 

 

 

Bytes -------------------------------------------------------------------------------------------------------

All rights reserved. reprinted articles are allowed, but source addresses must be indicated by links. Otherwise, the documents will be held legally responsible!

Skype: tianlesoftware

QQ: tianlesoftware@gmail.com

Email: tianlesoftware@gmail.com

Blog: http://www.tianlesoftware.com

WEAVER: http://weibo.com/tianlesoftware

Twitter: http://twitter.com/tianlesoftware

Facebook: http://www.facebook.com/tianlesoftware

LinkedIn: http://cn.linkedin.com/in/tianlesoftware

 

 

------- Add a group to describe the relationship between Oracle tablespace and data files in the remarks section. Otherwise, reject the application ----

Dba1 group: 62697716 (full); dba2 group: 62697977 (full) dba3 group: 62697850 (full)

Super DBA group: 63306533 (full); dba4 group: 83829929 dba5 group: 142216823

Dba6 group: 158654907 dba7 group: 172855474 DBA group: 104207940