Oracle 10g RAC 故障之--CRS啟動失敗

系統內容：

OS: Linux EL5

CRS： CRS 10.2.0.1

Oracle：Oracle 10.2.0.1

1、crs 啟動失敗

[root@node2 ~]# crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM

2、查看系統日誌

[root@node2 ~]# tail /var/log/messages

Nov  2 09:45:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2928.

Nov  2 09:46:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2962.

Nov  2 09:46:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.3172.

Nov  2 09:46:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2928.

Nov  2 09:47:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2962.

Nov  2 09:47:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2928.

Nov  2 09:47:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.3172.

Nov  2 09:48:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2962.

Nov  2 09:48:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2928.

Nov  2 09:48:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.3172.

[root@node2 ~]# cat /tmp/crsctl.3172

OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [Permission denied] [13]

[root@node2 ~]#

 

3、ocr 對應的裸裝置許可權出錯

[root@node2 ~]# ls -l /dev/raw

total 0

crw------- 1 root root 162, 1 Nov  2 09:41 raw1

crw------- 1 root root 162, 2 Nov  2 09:41 raw2

[root@node2 ~]#

4、修改raw 許可權設定檔，系統啟動時自動應用許可權

 

[root@node2 ~]# cat /etc/udev/rules.d/65-raw-permissions.rules

# Set permissions of raw bindings to Oracle Clusterware devices

KERNEL=="raw1", OWNER="oracle", GROUP="dba", MODE="660"

KERNEL=="raw2", OWNER="oracle", GROUP="dba", MODE="660"

[root@node2 ~]#

 

5、重新啟動udev 服務

[root@node1 rules.d]# start_udev

Starting udev: [  OK  ]

[root@node1 rules.d]# ls -l /dev/raw

total 0

crw-rw---- 1 oracle dba 162, 1 Nov  2 09:55 raw1

crw-rw---- 1 oracle dba 162, 2 Nov  2 09:55 raw2

[root@node1 rules.d]#

 

6、重新啟動crs ，系統正常

[root@node2 ~]# crsctl stop crs

Stopping resources.

Successfully stopped CRS resources

Stopping CSSD.

Shutting down CSS daemon.

Shutdown request successfully issued.

[root@node2 ~]# crsctl start crs

Attempting to start CRS stack

The CRS stack will be started shortly

 

[root@node2 ~]# tail /var/log/messages

Nov  2 09:52:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2928.

Nov  2 09:52:08 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.3172.

Nov  2 09:53:08 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2962.

Nov  2 09:53:08 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.3172.

Nov  2 09:53:08 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in /tmp/crsctl.2928.

Nov  2 09:54:10 node2 logger: Cluster Ready Services completed waiting on dependencies.

Nov  2 09:54:10 node2 last message repeated 2 times

Nov  2 09:55:10 node2 logger: Running CRSD with TZ =

Nov  2 09:56:01 node2 logger: Oracle CSSD graceful shutdown

Nov  2 09:56:32 node2 logger: Cluster Ready Services completed waiting on dependencies.

 

[root@node2 ~]# crsctl check crs

CSS appears healthy

CRS appears healthy

EVM appears healthy

[root@node2 ~]# crs_stat -t

Name          Type          Target    State    Host       

------------------------------------------------------------

ora....SM1.asm application    ONLINE    ONLINE    node1     

ora....E1.lsnr application    ONLINE    ONLINE    node1     

ora.node1.gsd  application    ONLINE    ONLINE    node1     

ora.node1.ons  application    ONLINE    ONLINE    node1     

ora.node1.vip  application    ONLINE    ONLINE    node1     

ora....SM2.asm application    ONLINE    OFFLINE             

ora....E2.lsnr application    ONLINE    ONLINE    node2     

ora.node2.gsd  application    ONLINE    ONLINE    node2     

ora.node2.ons  application    ONLINE    ONLINE    node2     

ora.node2.vip  application    ONLINE    ONLINE    node2     

ora.prod.db    application    ONLINE    ONLINE    node2     

ora....d1.inst application    ONLINE    ONLINE    node1     

ora....d2.inst application    ONLINE    OFFLINE             

ora...._taf.cs application    OFFLINE  OFFLINE             

ora....od1.srv application    OFFLINE  OFFLINE             

ora....od2.srv application    OFFLINE  OFFLINE             

[root@node2 ~]#

Linux 中Oracle 10g RAC 關閉crs開機自啟動