Oracle 10g RAC fault-CRS startup failure

System Environment:

OS: Linux EL5

CRS: CRS 10.2.0.1

Oracle: Oracle 10.2.0.1

1. crs startup failed

[Root @ node2 ~] # Crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM

2. view system logs

[Root @ node2 ~] # Tail/var/log/messages

Nov 2 09:45:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2928.

Nov 2 09:46:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2962.

Nov 2 09:46:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.3172.

Nov 2 09:46:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2928.

Nov 2 09:47:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2962.

Nov 2 09:47:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2928.

Nov 2 09:47:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.3172.

Nov 2 09:48:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2962.

Nov 2 09:48:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2928.

Nov 2 09:48:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.3172.

[Root @ node2 ~] # Cat/tmp/crsctl.3172

OCR initialization failed accessing OCR device: Error while accessing the physical storage Operating System error [Permission denied] [13]

[Root @ node2 ~] #

 

3. An error occurred while authorizing the bare device corresponding to ocr.

[Root @ node2 ~] # Ls-l/dev/raw

Total 0

Crw ------- 1 root 162, 1 Nov 2 raw1

Crw ------- 1 root 162, 2 Nov 2 raw2

[Root @ node2 ~] #

4. Modify the raw permission configuration file and automatically apply the permission when the system starts.

 

[Root @ node2 ~] # Cat/etc/udev/rules. d/65-raw-permissions.rules

# Set permissions of raw bindings to Oracle Clusterware devices

KERNEL = "raw1", OWNER = "oracle", GROUP = "dba", MODE = "660"

KERNEL = "raw2", OWNER = "oracle", GROUP = "dba", MODE = "660"

[Root @ node2 ~] #

 

5. Restart the udev service.

[Root @ node1 rules. d] # start_udev

Starting udev: [OK]

[Root @ node1 rules. d] # ls-l/dev/raw

Total 0

Crw-rw ---- 1 oracle dba 162, 1 Nov 2 raw1

Crw-rw ---- 1 oracle dba 162, 2 Nov 2 raw2

[Root @ node1 rules. d] #

 

6. Restart crs. The system is normal.

[Root @ node2 ~] # Crsctl stop crs

Stopping resources.

Successfully stopped CRS resources

Stopping CSSD.

Shutting down CSS daemon.

Shutdown request successfully issued.

[Root @ node2 ~] # Crsctl start crs

Attempting to start CRS stack

The CRS stack will be started shortly

 

[Root @ node2 ~] # Tail/var/log/messages

Nov 2 09:52:07 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2928.

Nov 2 09:52:08 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.3172.

Nov 2 09:53:08 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2962.

Nov 2 09:53:08 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.3172.

Nov 2 09:53:08 node2 logger: Cluster Ready Services waiting on dependencies. Diagnostics in/tmp/crsctl.2928.

Nov 2 09:54:10 node2 logger: Cluster Ready Services completed waiting on dependencies.

Nov 2 09:54:10 node2 last message repeated 2 times

Nov 2 09:55:10 node2 logger: Running CRSD with TZ =

Nov 2 09:56:01 node2 logger: Oracle CSSD graceful shutdown

Nov 2 09:56:32 node2 logger: Cluster Ready Services completed waiting on dependencies.

 

[Root @ node2 ~] # Crsctl check crs

CSS appears healthy

CRS appears healthy

EVM appears healthy

[Root @ node2 ~] # Crs_stat-t

Name Type Target State Host

------------------------------------------------------------

Ora... SM1.asm application ONLINE node1

Ora... E1.lsnr application ONLINE node1

Ora. node1.gsd application ONLINE node1

Ora. node1.ons application ONLINE node1

Ora. node1.vip application ONLINE node1

Ora... SM2.asm application ONLINE OFFLINE

Ora... E2.lsnr application ONLINE node2

Ora. node2.gsd application ONLINE node2

Ora. node2.ons application ONLINE node2

Ora. node2.vip application ONLINE node2

Ora. prod. db application ONLINE node2

Ora... d1.inst application ONLINE node1

Ora... d2.inst application ONLINE OFFLINE

Ora... _ taf. cs application OFFLINE

Ora... od1.srv application OFFLINE

Ora... od2.srv application OFFLINE

[Root @ node2 ~] #

In Linux, Oracle 10g RAC is used to disable crs from startup.