The recent RP value was a little low, and yesterday began to install a 10g RAC. Encountered n many problems. After resolving the problem with the raw device, the error follows when the second node executes root.sh:
[Root@rac2 ~]#/u01/app/oracle/product/crs/root.sh
Warning:directory '/u01/app/oracle/product ' isn't owned by root
Warning:directory '/u01/app/oracle ' isn't owned by root
Warning:directory '/u01/app ' isn't owned by root
Warning:directory '/u01 ' isn't owned by root
Checking to-if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting Up NS Directories
Oracle Cluster Registry Configuration upgraded successfully
Warning:directory '/u01/app/oracle/product ' isn't owned by root
Warning:directory '/u01/app/oracle ' isn't owned by root
Warning:directory '/u01/app ' isn't owned by root
Warning:directory '/u01 ' isn't owned by root
Clscfg:existing configuration version 3 detected.
Clscfg:version 3 is 10G Release 2.
Assigning default hostname Rac1 for node 1.
Assigning default hostname RAC2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports:css=49895 crs=49896 evmc=49898 and evmr=49897.
Node <nodenumber>: <nodename> <private interconnect name>
Node 1:rac1 rac1-priv Rac1
Node 2:rac2 rac2-priv RAC2
Clscfg:arguments Check out successfully.
NO KEYS were written. Supply-force parameter to override.
-force is destructive and would destroy any previous cluster
Configuration.
Oracle Cluster Registry for Cluster has already been initialized
Startup would be queued to init within seconds.
Adding daemons to Inittab
Expecting the CRS daemons to being up within seconds.
CSS is active on these nodes.
Rac1
Rac2
The CSS is active on the all nodes.
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
.....
Waiting for the Oracle CRSD and EVMD to start
Timed out waiting for the CRS stack to start.
The following information is found in the Crsd.log of Node 2:
[Root@rac2 crsd]# cat Crsd.log |more
Oracle Database 10g CRS release 10.2.0.1.0 Production Copyright 1996, Oracle. All Rights Reser
Ved.
2010-11-28 20:11:12.645: [default][1116368][enter]0
Oracle Database 10g CRS release 10.2.0.1.0 Production Copyright 1996, Oracle. All Rights Rese
RVed
2010-11-28 20:11:12.645: [Default][1116368]0crs Daemon starting
2010-11-28 20:11:12.690: [crsmain][1116368]0checking the OCR device
2010-11-28 20:11:12.994: [crsmain][1116368]0connecting to the CSS Daemon
2010-11-28 20:11:13.636: [Commcrs][60492688]clsc_connect: (0X8B937E0) No listener at (address= (PROTOCOL=IPC) (KEY= OCSSD_LL_RAC2_CRS))
2010-11-28 20:11:13.637: [Cssclnt][1116368]clsssinitnative:connect failed, RC 9
2010-11-28 20:11:13.640: [Crsrti][1116368]0css is not ready. Received Status 3 from CSS. Waiting for good status.
2010-11-28 20:11:17.062: [Commcrs][60492688]clsc_connect: (0X8C283E0) No listener at (address= (PROTOCOL=IPC) (KEY= OCSSD_LL_RAC2_CRS))
2010-11-28 20:11:17.062: [Cssclnt][1116368]clsssinitnative:connect failed, RC 9
2010-11-28 20:11:17.063: [Crsrti][1116368]0css is not ready. Received Status 3 from CSS. Waiting for good status.
2010-11-28 20:11:18.361: [Commcrs][60492688]clsc_connect: (0X8B94C30) No listener at (address= (PROTOCOL=IPC) (KEY= OCSSD_LL_RAC2_CRS))
2010-11-28 20:11:18.361: [Cssclnt][1116368]clsssinitnative:connect failed, RC 9
2010-11-28 20:11:18.361: [Crsrti][1116368]0css is not ready. Received Status 3 from CSS. Waiting for good status.
2010-11-28 20:11:19.642: [Commcrs][60492688]clsc_connect: (0x8c28840) No listener at (address= (PROTOCOL=IPC) (KEY= OCSSD_LL_RAC2_CRS))
2010-11-28 20:11:19.642: [Cssclnt][1116368]clsssinitnative:connect failed, RC 9
2010-11-28 20:11:19.642: [Crsrti][1116368]0css is not ready. Received Status 3 from CSS. Waiting for good status.
2010-11-28 20:11:26.540: [Crsd][1116368]0daemon version:10.2.0.1.0 Active version:10.1.0.2.0
2010-11-28 20:11:26.540: [Crsd][1116368]0active version is less than Software version
2010-11-28 20:11:26.557: [crsd][1116368]0registered in CSS group Crs_version
2010-11-28 20:11:26.557: [crsmain][1116368]0initializing OCR
2010-11-28 20:11:26.617: [crsd][104029072]0monitoring the Crs_version Group for AV Change notification
2010-11-28 20:11:26.617: [crsd][104029072]0doing grpstat on Crs_version Group
2010-11-28 20:11:26.617: [crsd][104029072]0returned from Grpstat with event 1
2010-11-28 20:11:26.617: [crsd][104029072]0doing grpstat on Crs_version Group
2010-11-28 20:11:26.827: [ocrraw][1116368]proprioo:for Disk 0 (/DEV/RAW/RAW1), id match (1), my ID set (1669906634,1882 63131) Total ID sets (1), 1st set (1669906634,188263131), 2nd set (0,0) I votes (1), Total votes (2)
2010-11-28 20:11:26.828: [ocrraw][1116368]proprioo:for Disk 1 (/DEV/RAW/RAW2), id match (1), my ID set (1669906634,1882 63131) Total ID sets (1), 1st set (1669906634,188263131), 2nd set (0,0) I votes (1), Total votes (2)
2010-11-28 20:11:28.715: [crsd][1116368]0env Logging level for Module:allcomp 0
2010-11-28 20:11:29.563: [crsd][1116368]0env Logging level for Module:default 0
2010-11-28 20:11:29.622: [crsd][1116368]0env Logging level for Module:commcrs 0
2010-11-28 20:11:30.671: [crsd][1116368]0env Logging level for module:commns 0
2010-11-28 20:11:31.620: [crsd][104029072]0returned from Grpstat with event 1
2010-11-28 20:11:31.620: [crsd][104029072]0doing grpstat on Crs_version Group
2010-11-28 20:11:31.620: [crsd][104029072]0returned from Grpstat with event 1
2010-11-28 20:11:31.620: [crsd][104029072]0doing grpstat on Crs_version Group
2010-11-28 20:11:31.620: [crsd][104029072]0returned from Grpstat with event 8
2010-11-28 20:11:31.620: [crsd][104029072]0recieved grppriv Event
2010-11-28 20:11:31.632: [Crsd][104029072]0av got from version group:10.2.0.1.0
2010-11-28 20:11:31.632: [crsd][104029072]0stopped Monitoring the version group
2010-11-28 20:11:31.632: [Crsd][104029072]0new Active version:10.2.0.1.0
2010-11-28 20:11:31.632: [crsd][104029072]0active Version changed to 10.2.0.1.0
2010-11-28 20:11:32.105: [crsd][1116368]0env Logging level for Module:crsui 0
...
2010-11-28 20:11:47.616: [crsd][1116368]0env Logging level for Module:ocrmas 0
2010-11-28 20:11:47.616: [Crsmain][1116368]0filename IS/U01/APP/ORACLE/PRODUCT/CRS/CRS/INIT/RAC2.P
Id
[CLSDMT] [104029072] Listening to (address= (PROTOCOL=IPC) (KEY=RAC2DBG_CRSD))
2010-11-28 20:11:48.124: [CRSOCR][1116368]0OCR API Procr_open_key failed for key SYSTEM.crs.updflag. OCR error code = 4 OCR Error MSG:PROC-4: The cluster registry key to is operated on does not exist.
2010-11-28 20:11:49.284: [CRSOCR][1116368]0OCR API Procr_delete_key failed for key SYSTEM.crs.updflag. OCR error code = 0 OCR Error msg:
2010-11-28 20:11:49.294: [crsmain][1116368]0using authorizer Location:/u01/app/oracle/product/crs/crs/auth/
2010-11-28 20:11:49.518: [crsmain][1116368]0initializing RTI
2010-11-28 20:11:49.519: [Crstimer][2823719824]0timer Thread starting.
2010-11-28 20:11:49.524: [Crsres][1116368]0parameter security = 1, running in USER Mode
2010-11-28 20:11:49.524: [crsmain][1116368]0initializing evmmgr
2010-11-28 20:11:49.636: [Commcrs][2813229968]clsc_connect: (0x918fc48) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))
2010-11-28 20:11:50.151: [Commcrs][2813229968]clsc_connect: (0x90fed98) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))
2010-11-28 20:11:50.444: [Commcrs][2813229968]clsc_connect: (0x918fe78) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))
2010-11-28 20:11:51.198: [Commcrs][2813229968]clsc_connect: (0X918FFB8) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))
2010-11-28 20:11:51.702: [Commcrs][2813229968]clsc_connect: (0x918f278) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))
2010-11-28 20:11:52.961: [Commcrs][2813229968]clsc_connect: (0x918f5f0) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))
2010-11-28 20:11:53.474: [Commcrs][2813229968]clsc_connect: (0x918fd88) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))
2010-11-28 20:11:54.726: [Commcrs][2813229968]clsc_connect: (0x918fd88) No listener at (address= (PROTOCOL=IPC) (KEY= SYSTEM.evm.acceptor.auth))
[Root@rac1 cssd]# cat Ocssd.log |more
Oracle Database 10g CRS release 10.2.0.1.0 Production Copyright 1996, Oracle. All rights reserved.
[Cssd]2010-11-28 20:07:53.219 >user:oracle Database 10g CSS release 10.2.0.1.0 Production Copyright 1996, 2004 Oracle. All rights reserved.
[cssd]2010-11-28 20:07:53.219 >user:css Daemon Log for node Rac1, number 1, in cluster CRS
[Cssd]2010-11-28 20:07:53.257 [1277920] >trace:clssscmain:local-only set to False
[CLSDMT] Listening to (address= (PROTOCOL=IPC) (KEY=RAC1DBG_CSSD))
[Cssd]2010-11-28 20:07:53.369 [1277920] >trace:clssnmreadnodeinfo:added node 1 (RAC1) to cluster
[Cssd]2010-11-28 20:07:53.450 [1277920] >trace:clssnmreadnodeinfo:added Node 2 (RAC2) to cluster
[Cssd]2010-11-28 20:07:53.525 [38079376] >TRACE:CLSSNM_SKGXNMON:SKGXN init failed, RC 1
[Cssd]2010-11-28 20:07:53.525 [1277920] >trace:clssnm_skgxnonline:using vacuous SKGXN Monitor
[Cssd]2010-11-28 20:07:53.584 [1277920] >trace:clssnmdiskstatechange:state from 1 to 2 disk (0//DEV/RAW/RAW3)
[Cssd]2010-11-28 20:07:53.602 [1277920] >trace:clssnmdiskstatechange:state from 1 to 2 disk (1//DEV/RAW/RAW4)
[Cssd]2010-11-28 20:07:53.633 [1277920] >trace:clssnmdiskstatechange:state from 1 to 2 disk (2//DEV/RAW/RAW5)
[Cssd]2010-11-28 20:07:55.640 [65649552] >trace:clssnmdiskstatechange:state from 2 to 4 disk (1//DEV/RAW/RAW4)
[Cssd]2010-11-28 20:07:55.719 [38079376] >trace:clssnmdiskstatechange:state from 2 to 4 disk (0//DEV/RAW/RAW3)
[Cssd]2010-11-28 20:07:55.724 [76139408] >trace:clssnmdiskstatechange:state from 2 to 4 disk (2//DEV/RAW/RAW5)
[Cssd]2010-11-28 20:07:55.821 [1277920] >trace:clssscsclsfatal:read value of disable
[Cssd]2010-11-28 20:07:55.822 [1277920] >trace:clssscsclsfatal:read value of disable
[Cssd]2010-11-28 20:07:55.825 [114346896] >trace:clssnmfatalthread:spawned
[Cssd]2010-11-28 20:07:55.825 [3086044048] >trace:clssnmconnect:connecting to Node 1, flags 0x0001, connector 1
[Cssd]2010-11-28 20:07:56.024 [3086044048] >trace:clssnmconnect:connecting to node 0, flags 0x0000, connector 1
[Cssd]2010-11-28 20:07:56.025 [3086044048] >trace:clssnmclusterlistener:probing node (2)
[Cssd]2010-11-28 20:07:56.102 [3086044048] >trace:clsc_send_msg: (0x8c250e0) NS err (12571, 12560), Transport (5 30, 111, 0)
[Cssd]2010-11-28 20:07:56.102 [3086044048] >error:clssnminitialmsg:send failed, con (0x8c25528), RC 3
[Cssd]2010-11-28 20:07:56.121 [3075554192] >trace:clssgmclientlsnr:listening on (address= (PROTOCOL=IPC) (KEY=Ora Cle_css_lcllstnr_crs_1))
[Cssd]2010-11-28 20:07:56.122 [3075554192] >trace:clssgmclientlsnr:listening on (address= (PROTOCOL=IPC) (KEY=OCS SD_LL_RAC1_CRS))
[Cssd]2010-11-28 20:07:56.211 [3032476560] >trace:clssnmpollingthread:connection complete
[Cssd]2010-11-28 20:07:56.211 [3011496848] >trace:clssnmrcfgmgrthread:connection complete
[Cssd]2010-11-28 20:07:56.211 [3011496848] >trace:clssnmrcfgmgrthread:local Join
[Cssd]2010-11-28 20:07:56.211 [3011496848] >trace:clssnmdosyncupdate:initiating sync 1
[Root@rac2 client]# cat Css.log |more
Oracle Database 10g CRS release 10.2.0.1.0 Production Copyright 1996, Oracle. All rights reserved.
2010-11-28 20:10:00.188: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9
2010-11-28 20:10:02.359: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9
2010-11-28 20:10:05.369: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9
2010-11-28 20:10:08.821: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9
2010-11-28 20:10:10.073: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9
2010-11-28 20:10:11.613: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9
2010-11-28 20:10:12.765: [Cssclnt][1501280]clsssinitnative:connect failed, RC 9
Start the CRS report with the following error:
[Root@rac1 bin]#./crsctl Check CRS
CSS appears healthy
Cannot communicate with CRS
Cannot communicate with EVM
Correlation analysis of the problem:
1. Firewall reason
A similar situation on Oracle Metalink is due to the firewall. But my firewall shuts down when the system is installed.
Problem performance, ping private IP is normal, but with tracert private IP. There will be the following error:
# traceroute 192.168.0.2
Traceroute to 192.168.0.2 (192.168.0.2), hops max, byte packets
1 Rac2prv (192.168.0.2) 0.201 Ms!<10> 0.198 Ms!<10> 0.109 MS!<10>
If this is the case, it's OK to turn off the firewall
# Service Iptables Stop
# chkconfig iptables off.
2. Permissions issues for RAW devices
In contrast, there is no problem with raw permissions. Because the raw configuration is configured according to the official Oracle documentation. So my raw problem here is not big.
[Root@rac2 ~]# cd/dev/raw/
[Root@rac2 raw]# LL
Total 0
Crw-r-----1 Root oinstall 162, 1 Nov 19:14 RAW1
Crw-r-----1 Root oinstall 162, 2 Nov 19:14 raw2
crw-r--r--1 Oracle Oinstall 162, 3 Nov 20:15 raw3
crw-r--r--1 Oracle Oinstall 162, 4 Nov 20:15 raw4
crw-r--r--1 Oracle Oinstall 162, 5 Nov 20:15 raw5
3. Permissions issues for related directories
CRS needs to write some information to the relevant files, and if these folders have permission problems, the files cannot be written. This may also happen. I found a few examples on the Internet. After they re empowered the files, CRS starts up normally.
Several related directories:/var/tmp/.oracle,/tmp/.oracle and $crs_home/log/sid/
Oracle will write a few sockets and log information to these files. If it cannot be written, it will cause CRS to fail to start.
How to determine if this problem causes CRS to fail to start is simple. is to empty these 2 folders first. At the start of CRS. If there is a file generation, there is no problem with permissions.
The thing to be aware of is to turn off CRS first. If CRS is running, forcing the deletion of these 2 folders may cause CRS to hang.
Try emptying these 2 directories. The root.sh command is then rerun and the operation is as follows:
1. Stop CRS with CRSCTL stop CRS command
2. Delete/etc/init.* several files. Rm-f/etc/init.*
3. Kill related Process
Ps-ef|grep CSS
Ps-ef|grep CRS
Ps-ef|grep EVM
End the process with the Kill-9 ID, based on the ID found by PS.
These processes are not killed if the related files are not deleted in the second part.
4. Delete the/etc/oracle/scls_scr/rac1/oracle/cssfatal files on each machine
If you do not delete this file, you will get an error when running the root.sh script.
Reference:
RAC root.sh Oracle CRS solution already configured and'll be running under init (1M)
Http://blog.csdn.net/tianlesoftware/archive/2010/02/21/5314804.aspx
5. Condition OCR of 2 raw devices
[Root@rac1 bin]# dd If=/dev/zero of=/dev/raw/raw1 bs=1m count=195
195+0 Records in
195+0 Records out
204472320 bytes (204 MB) copied, 23.5725 seconds, 8.7 mb/s
[Root@rac1 bin]# dd If=/dev/zero of=/dev/raw/raw2 bs=1m count=195
195+0 Records in
195+0 Records out
204472320 bytes (204 MB) copied, 28.1755 seconds, 7.3 mb/s
6. Rerun the/u01/app/oracle/product/crs/root.sh script.
The same error occurs after you have done the above. In the cup ...
Because the system has a RAC of Oracle 11GR2 installed. Without a successful installation, delete the relevant files and install a 10g RAC directly. It is estimated that some places have not been removed cleanly. Clusterware is also very strange. The system was eventually reset and the 10g RAC was installed.
A friend on the web is a normal RAC environment and cannot start CRS after rebooting. When this error occurs, the related directory is properly empowered and then started normally. I this is in the process of installation. Good toss. If the production environment is in trouble.
Original address: http://blog.csdn.net/tianlesoftware/article/details/6048651