Oracle RAC Series: Redhat 5.4 RAC No listener at (address= (PROTOCOL=IPC) (Key=system.evm.acceptor.auth)) __oracle

The recent RP value was a little low, and yesterday began to install a 10g RAC. Encountered n many problems. After resolving the problem with the raw device, the error follows when the second node executes root.sh:

[Root@rac2 ~]#/u01/app/oracle/product/crs/root.sh

Warning:directory '/u01/app/oracle/product ' isn't owned by root

Warning:directory '/u01/app/oracle ' isn't owned by root

Warning:directory '/u01/app ' isn't owned by root

Warning:directory '/u01 ' isn't owned by root

Checking to-if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory

Setting Up NS Directories

Oracle Cluster Registry Configuration upgraded successfully

Warning:directory '/u01/app/oracle/product ' isn't owned by root

Warning:directory '/u01/app/oracle ' isn't owned by root

Warning:directory '/u01/app ' isn't owned by root

Warning:directory '/u01 ' isn't owned by root

Clscfg:existing configuration version 3 detected.

Clscfg:version 3 is 10G Release 2.

Assigning default hostname Rac1 for node 1.

Assigning default hostname RAC2 for node 2.

Successfully accumulated necessary OCR keys.

Using ports:css=49895 crs=49896 evmc=49898 and evmr=49897.

Node <nodenumber>: <nodename> <private interconnect name>

Node 1:rac1 rac1-priv Rac1

Node 2:rac2 rac2-priv RAC2

Clscfg:arguments Check out successfully.

NO KEYS were written. Supply-force parameter to override.

-force is destructive and would destroy any previous cluster

Configuration.

Oracle Cluster Registry for Cluster has already been initialized

Startup would be queued to init within seconds.

Adding daemons to Inittab

Expecting the CRS daemons to being up within seconds.

CSS is active on these nodes.

Rac1

Rac2

The CSS is active on the all nodes.

Waiting for the Oracle CRSD and EVMD to start

Waiting for the Oracle CRSD and EVMD to start

.....

Waiting for the Oracle CRSD and EVMD to start

Timed out waiting for the CRS stack to start.

The following information is found in the Crsd.log of Node 2:

[Root@rac2 crsd]# cat Crsd.log |more

Oracle Database 10g CRS release 10.2.0.1.0 Production Copyright 1996, Oracle. All Rights Reser

Ved.

2010-11-28 20:11:12.645: [default][1116368][enter]0

Oracle Database 10g CRS release 10.2.0.1.0 Production Copyright 1996, Oracle. All Rights Rese

RVed

2010-11-28 20:11:12.645: [Default][1116368]0crs Daemon starting

2010-11-28 20:11:12.690: [crsmain][1116368]0checking the OCR device

2010-11-28 20:11:12.994: [crsmain][1116368]0connecting to the CSS Daemon

2010-11-28 20:11:13.636: [Commcrs][60492688]clsc_connect: (0X8B937E0) No listener at (address= (PROTOCOL=IPC) (KEY= OCSSD_LL_RAC2_CRS))

2010-11-28 20:11:13.637: [Cssclnt][1116368]clsssinitnative:connect failed, RC 9

2010-11-28 20:11:13.640: [Crsrti][1116368]0css is not ready. Received Status 3 from CSS. Waiting for good status.

2010-11-28 20:11:17.062: [Commcrs][60492688]clsc_connect: (0X8C283E0) No listener at (address= (PROTOCOL=IPC) (KEY= OCSSD_LL_RAC2_CRS))

2010-11-28 20:11:17.062: [Cssclnt][1116368]clsssinitnative:connect failed, RC 9

2010-11-28 20:11:17.063: [Crsrti][1116368]0css is not ready. Received Status 3 from CSS. Waiting for good status.

2010-11-28 20:11:18.361: [Commcrs][60492688]clsc_connect: (0X8B94C30) No listener at (address= (PROTOCOL=IPC) (KEY= OCSSD_LL_RAC2_CRS))

2010-11-28 20:11:18.361: [Cssclnt][1116368]clsssinitnative:connect failed, RC 9

2010-11-28 20:11:18.361: [Crsrti][1116368]0css is not ready. Received Status 3 from CSS. Waiting for good status.

2010-11-28 20:11:19.642: [Commcrs][60492688]clsc_connect: (0x8c28840) No listener at (address= (PROTOCOL=IPC) (KEY= OCSSD_LL_RAC2_CRS))

2010-11-28 20:11:19.642: [Cssclnt][1116368]clsssinitnative:connect failed, RC 9

2010-11-28 20:11:19.642: [Crsrti][1116368]0css is not ready. Received Status 3 from CSS. Waiting for good status.

2010-11-28 20:11:26.540: [Crsd][1116368]0daemon version:10.2.0.1.0 Active version:10.1.0.2.0

2010-11-28 20:11:26.540: [Crsd][1116368]0active version is less than Software version

2010-11-28 20:11:26.557: [crsd][1116368]0registered in CSS group Crs_version

2010-11-28 20:11:26.557: [crsmain][1116368]0initializing OCR

2010-11-28 20:11:26.617: [crsd][104029072]0monitoring the Crs_version Group for AV Change notification

2010-11-28 20:11:26.617: [crsd][104029072]0doing grpstat on Crs_version Group

2010-11-28 20:11:26.617: [crsd][104029072]0returned from Grpstat with event 1

2010-11-28 20:11:26.617: [crsd][104029072]0doing grpstat on Crs_version Group

2010-11-28 20:11:26.827: [ocrraw][1116368]proprioo:for Disk 0 (/DEV/RAW/RAW1), id match (1), my ID set (1669906634,1882 63131) Total ID sets (1), 1st set (1669906634,188263131), 2nd set (0,0) I votes (1), Total votes (2)

2010-11-28 20:11:26.828: [ocrraw][1116368]proprioo:for Disk 1 (/DEV/RAW/RAW2), id match (1), my ID set (1669906634,1882 63131) Total ID sets (1), 1st set (1669906634,188263131), 2nd set (0,0) I votes (1), Total votes (2)

2010-11-28 20:11:28.715: [crsd][1116368]0env Logging level for Module:allcomp 0

2010-11-28 20:11:29.563: [crsd][1116368]0env Logging level for Module:default 0

2010-11-28 20:11:29.622: [crsd][1116368]0env Logging level for Module:commcrs 0

2010-11-28 20:11:30.671: [crsd][1116368]0env Logging level for module:commns 0

2010-11-28 20:11:31.620: [crsd][104029072]0returned from Grpstat with event 1

2010-11-28 20:11:31.620: [crsd][104029072]0doing grpstat on Crs_version Group

2010-11-28 20:11:31.620: [crsd][104029072]0returned from Grpstat with event 1

2010-11-28 20:11:31.620: [crsd][104029072]0doing grpstat on Crs_version Group

2010-11-28 20:11:31.620: [crsd][104029072]0returned from Grpstat with event 8

2010-11-28 20:11:31.620: [crsd][104029072]0recieved grppriv Event

2010-11-28 20:11:31.632: [Crsd][104029072]0av got from version group:10.2.0.1.0

2010-11-28 20:11:31.632: [crsd][104029072]0stopped Monitoring the version group

2010-11-28 20:11:31.632: [Crsd][104029072]0new Active version:10.2.0.1.0

2010-11-28 20:11:31.632: [crsd][104029072]0active Version changed to 10.2.0.1.0

2010-11-28 20:11:32.105: [crsd][1116368]0env Logging level for Module:crsui 0

...

2010-11-28 20:11:47.616: [crsd][1116368]0env Logging level for Module:ocrmas 0

2010-11-28 20:11:47.616: [Crsmain][1116368]0filename IS/U01/APP/ORACLE/PRODUCT/CRS/CRS/INIT/RAC2.P

Id

[CLSDMT] [104029072] Listening to (address= (PROTOCOL=IPC) (KEY=RAC2DBG_CRSD))

2010-11-28 20:11:48.124: [CRSOCR][1116368]0OCR API Procr_open_key failed for key SYSTEM.crs.updflag. OCR error code = 4 OCR Error MSG:PROC-4: The cluster registry key to is operated on does not exist.

2010-11-28 20:11:49.284: [CRSOCR][1116368]0OCR API Procr_delete_key failed for key SYSTEM.crs.updflag. OCR error code = 0 OCR Error msg:

2010-11-28 20:11:49.294: [crsmain][1116368]0using authorizer Location:/u01/app/oracle/product/crs/crs/auth/

2010-11-28 20:11:49.518: [crsmain][1116368]0initializing RTI

2010-11-28 20:11:49.519: [Crstimer][2823719824]0timer Thread starting.

2010-11-28 20:11:49.524: [Crsres][1116368]0parameter security = 1, running in USER Mode

2010-11-28 20:11:49.524: [crsmain][1116368]0initializing evmmgr

2010-11-28 20:11:49.636: [Commcrs][2813229968]clsc_connect: (0x918fc48) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))

2010-11-28 20:11:50.151: [Commcrs][2813229968]clsc_connect: (0x90fed98) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))

2010-11-28 20:11:50.444: [Commcrs][2813229968]clsc_connect: (0x918fe78) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))

2010-11-28 20:11:51.198: [Commcrs][2813229968]clsc_connect: (0X918FFB8) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))

2010-11-28 20:11:51.702: [Commcrs][2813229968]clsc_connect: (0x918f278) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))

2010-11-28 20:11:52.961: [Commcrs][2813229968]clsc_connect: (0x918f5f0) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))

2010-11-28 20:11:53.474: [Commcrs][2813229968]clsc_connect: (0x918fd88) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))

2010-11-28 20:11:54.726: [Commcrs][2813229968]clsc_connect: (0x918fd88) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))

[Root@rac1 cssd]# cat Ocssd.log |more

Oracle Database 10g CRS release 10.2.0.1.0 Production Copyright 1996, Oracle. All rights reserved.

[Cssd]2010-11-28 20:07:53.219 >user:oracle Database 10g CSS release 10.2.0.1.0 Production Copyright 1996, 2004  Oracle. All rights reserved.

[cssd]2010-11-28 20:07:53.219 >user:css Daemon Log for node Rac1, number 1, in cluster CRS

[Cssd]2010-11-28 20:07:53.257 [1277920] >trace:clssscmain:local-only set to False

[CLSDMT] Listening to (address= (PROTOCOL=IPC) (KEY=RAC1DBG_CSSD))

[Cssd]2010-11-28 20:07:53.369 [1277920] >trace:clssnmreadnodeinfo:added node 1 (RAC1) to cluster

[Cssd]2010-11-28 20:07:53.450 [1277920] >trace:clssnmreadnodeinfo:added Node 2 (RAC2) to cluster

[Cssd]2010-11-28 20:07:53.525 [38079376] >TRACE:CLSSNM_SKGXNMON:SKGXN init failed, RC 1

[Cssd]2010-11-28 20:07:53.525 [1277920] >trace:clssnm_skgxnonline:using vacuous SKGXN Monitor

[Cssd]2010-11-28 20:07:53.584 [1277920] >trace:clssnmdiskstatechange:state from 1 to 2 disk (0//DEV/RAW/RAW3)

[Cssd]2010-11-28 20:07:53.602 [1277920] >trace:clssnmdiskstatechange:state from 1 to 2 disk (1//DEV/RAW/RAW4)

[Cssd]2010-11-28 20:07:53.633 [1277920] >trace:clssnmdiskstatechange:state from 1 to 2 disk (2//DEV/RAW/RAW5)

[Cssd]2010-11-28 20:07:55.640 [65649552] >trace:clssnmdiskstatechange:state from 2 to 4 disk (1//DEV/RAW/RAW4)

[Cssd]2010-11-28 20:07:55.719 [38079376] >trace:clssnmdiskstatechange:state from 2 to 4 disk (0//DEV/RAW/RAW3)

[Cssd]2010-11-28 20:07:55.724 [76139408] >trace:clssnmdiskstatechange:state from 2 to 4 disk (2//DEV/RAW/RAW5)

[Cssd]2010-11-28 20:07:55.821 [1277920] >trace:clssscsclsfatal:read value of disable

[Cssd]2010-11-28 20:07:55.822 [1277920] >trace:clssscsclsfatal:read value of disable

[Cssd]2010-11-28 20:07:55.825 [114346896] >trace:clssnmfatalthread:spawned

[Cssd]2010-11-28 20:07:55.825 [3086044048] >trace:clssnmconnect:connecting to Node 1, flags 0x0001, connector 1

[Cssd]2010-11-28 20:07:56.024 [3086044048] >trace:clssnmconnect:connecting to node 0, flags 0x0000, connector 1

[Cssd]2010-11-28 20:07:56.025 [3086044048] >trace:clssnmclusterlistener:probing node (2)

[Cssd]2010-11-28 20:07:56.102 [3086044048] >trace:clsc_send_msg: (0x8c250e0) NS err (12571, 12560), Transport (5 30, 111, 0)

[Cssd]2010-11-28 20:07:56.102 [3086044048] >error:clssnminitialmsg:send failed, con (0x8c25528), RC 3

[Cssd]2010-11-28 20:07:56.121 [3075554192] >trace:clssgmclientlsnr:listening on (address= (PROTOCOL=IPC) (KEY=Ora Cle_css_lcllstnr_crs_1))

[Cssd]2010-11-28 20:07:56.122 [3075554192] >trace:clssgmclientlsnr:listening on (address= (PROTOCOL=IPC) (KEY=OCS SD_LL_RAC1_CRS))

[Cssd]2010-11-28 20:07:56.211 [3032476560] >trace:clssnmpollingthread:connection complete

[Cssd]2010-11-28 20:07:56.211 [3011496848] >trace:clssnmrcfgmgrthread:connection complete

[Cssd]2010-11-28 20:07:56.211 [3011496848] >trace:clssnmrcfgmgrthread:local Join

[Cssd]2010-11-28 20:07:56.211 [3011496848] >trace:clssnmdosyncupdate:initiating sync 1

[Root@rac2 client]# cat Css.log |more

Oracle Database 10g CRS release 10.2.0.1.0 Production Copyright 1996, Oracle. All rights reserved.

2010-11-28 20:10:00.188: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9

2010-11-28 20:10:02.359: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9

2010-11-28 20:10:05.369: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9

2010-11-28 20:10:08.821: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9

2010-11-28 20:10:10.073: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9

2010-11-28 20:10:11.613: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9

2010-11-28 20:10:12.765: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9

Start the CRS report with the following error:

[Root@rac1 bin]#./crsctl Check CRS

CSS appears healthy

Cannot communicate with CRS

Cannot communicate with EVM

Correlation analysis of the problem:

1. Firewall reason

A similar situation on Oracle Metalink is due to the firewall. But my firewall shuts down when the system is installed.

Problem performance, ping private IP is normal, but with tracert private IP. There will be the following error:

# traceroute 192.168.0.2
Traceroute to 192.168.0.2 (192.168.0.2), hops max, byte packets
1 Rac2prv (192.168.0.2) 0.201 Ms!<10> 0.198 Ms!<10> 0.109 MS!<10>

If this is the case, it's OK to turn off the firewall

# Service Iptables Stop
# chkconfig iptables off.

2. Permissions issues for RAW devices

In contrast, there is no problem with raw permissions. Because the raw configuration is configured according to the official Oracle documentation. So my raw problem here is not big.

[Root@rac2 ~]# cd/dev/raw/

[Root@rac2 raw]# LL

Total 0

Crw-r-----1 Root oinstall 162, 1 Nov 19:14 RAW1

Crw-r-----1 Root oinstall 162, 2 Nov 19:14 raw2

crw-r--r--1 Oracle Oinstall 162, 3 Nov 20:15 raw3

crw-r--r--1 Oracle Oinstall 162, 4 Nov 20:15 raw4

crw-r--r--1 Oracle Oinstall 162, 5 Nov 20:15 raw5

3. Permissions issues for related directories

CRS needs to write some information to the relevant files, and if these folders have permission problems, the files cannot be written. This may also happen. I found a few examples on the Internet. After they re empowered the files, CRS starts up normally.

Several related directories:/var/tmp/.oracle,/tmp/.oracle and $crs_home/log/sid/

Oracle will write a few sockets and log information to these files. If it cannot be written, it will cause CRS to fail to start.

How to determine if this problem causes CRS to fail to start is simple. is to empty these 2 folders first. At the start of CRS. If there is a file generation, there is no problem with permissions.

The thing to be aware of is to turn off CRS first. If CRS is running, forcing the deletion of these 2 folders may cause CRS to hang.

Try emptying these 2 directories. The root.sh command is then rerun and the operation is as follows:

1. Stop CRS with CRSCTL stop CRS command

2. Delete/etc/init.* several files. Rm-f/etc/init.*

3. Kill related Process

Ps-ef|grep CSS

Ps-ef|grep CRS

Ps-ef|grep EVM

End the process with the Kill-9 ID, based on the ID found by PS.

These processes are not killed if the related files are not deleted in the second part.

4. Delete the/etc/oracle/scls_scr/rac1/oracle/cssfatal files on each machine

If you do not delete this file, you will get an error when running the root.sh script.

Reference:

RAC root.sh Oracle CRS solution already configured and'll be running under init (1M)

Http://blog.csdn.net/tianlesoftware/archive/2010/02/21/5314804.aspx

5. Condition OCR of 2 raw devices

[Root@rac1 bin]# dd If=/dev/zero of=/dev/raw/raw1 bs=1m count=195

195+0 Records in

195+0 Records out

204472320 bytes (204 MB) copied, 23.5725 seconds, 8.7 mb/s

[Root@rac1 bin]# dd If=/dev/zero of=/dev/raw/raw2 bs=1m count=195

195+0 Records in

195+0 Records out

204472320 bytes (204 MB) copied, 28.1755 seconds, 7.3 mb/s

6. Rerun the/u01/app/oracle/product/crs/root.sh script.

The same error occurs after you have done the above. In the cup ...

Because the system has a RAC of Oracle 11GR2 installed. Without a successful installation, delete the relevant files and install a 10g RAC directly. It is estimated that some places have not been removed cleanly. Clusterware is also very strange. The system was eventually reset and the 10g RAC was installed.

A friend on the web is a normal RAC environment and cannot start CRS after rebooting. When this error occurs, the related directory is properly empowered and then started normally. I this is in the process of installation. Good toss. If the production environment is in trouble.

Original address: http://blog.csdn.net/tianlesoftware/article/details/6048651