Oracle rac ocr device: PROC-26: Error [Device or resource busy] [16]

[Oracle @ node1 crsd] $ crs_stat-t

CRS-0184: Cannot communicate with the CRS daemon.

 

[Oracle @ node1 crsd] $ crsctl check crs

Failure 1 contacting CSS daemon

Cannot communicate with CRS

Cannot communicate with EVM

 

[Root @ node1 crs] # ps-ef | grep crs

Root 3926 1 0 :46? 00:00:00/bin/sh/etc/init. d/init. crsd run

Root 29408 25855 0 00:00:00 pts/1 grep crs

 

[Root @ node1 bin] #./racgvip

There is no VIP name

 

 

[Root @ node1 crsd] #/etc/init. d/init. crs stop

Shutting down Oracle Cluster Ready Services (CRS ):

OCR initialization failed accessing OCR device: Error while accessing the physical storage operating system error [device or resource busy] [16]

Shutdown has begun. The daemons shoshould exit soon.

 

 

[Root @ node1 crsd] # raw-Qa

/Dev/raw/raw1: bound to major 8, minor 17

/Dev/raw/raw2: bound to major 8, minor 33

 

 

[Root @ node1 crsd] # ls-Al/dev/raw/raw2

CrW-RW ---- 1 Oracle DBA 162, 2 September 15 17:45/dev/raw/raw2

 

 

[Root @ node1 bin] #./crsctl query CSS votedisk

OCR initialization failed accessing OCR device: Error while accessing the physical storage operating system error [device or resource busy] [16]

 

 

[Root @ node1 bin] #./ocrcheck

PROT-602: Failed to retrieve data from the cluster registry

 

[Root @ node1 ~] # Ll/etc/oracle/ocr. loc

-Rw-r -- 1 root oinstall 45 2012-01-17/etc/oracle/ocr. loc

 

[Root @ node1 bin] # more/etc/oracle/ocr. loc

Ocrconfig_loc =/dev/raw/raw1

Local_only = FALSE

 

[Root @ node1 ~] # Dd if =/dev/raw/raw1 of =/opt/oracle/ocr_raw.bak

Dd: Open '/dev/raw/raw1': the device or resource is busy.

 

Lsof | grep/dev/raw/raw1

No one occupies

 

To format the partition corresponding to RAW1, we found that SDB1 is 10.7 GB instead of 100 M for bare devices.

 

 

The system administrator is here to help,

Fdisk sdb causes a problem when the file system is started. Therefore, after the root user password is entered, You can restart fdisk sdb.

Then, the sdb 10.7 GB is partitioned into sdb1, the raw device is partitioned into sdc1, and then mkfs. ext3/dev/sdb1 is formatted.

 

In this way, the system is accessed, and the connection of/etc/sysconfig/rawdevices is modified.

After restarting the system again, it is found that DD can back up/DEV/RAW/RAW1 and no error is reported.

 

[Root @ node1 tmp] # dd if =/dev/zero of =/dev/raw/raw1 bs = 512 count = 2048

Read 2048 + 0 blocks

Output 2048 + 0 blocks

[Root @ node1 tmp] # dd if =/dev/zero of =/dev/raw/raw2 bs = 512 count = 2048

Read 2048 + 0 blocks

Output 2048 + 0 blocks

The bare device is in normal use...

 

/Tmp does not produce new errors

 

Disable CRS

[Root @ node1 ~] #/Etc/init. d/init. crs stop

Shutting down Oracle Cluster Ready Services (CRS ):

OCR initialization failed accessing OCR device: Error while accessing the physical storage

Shutdown has begun. The daemons shoshould exit soon.

 

Perform OCR recovery

Ocrconfig-Restore/opt/Oracle/crshome/product/10.2.0/db_1/CDATA/CRS/backup00.ocr

No response

 

View OCR logs

CD/opt/Oracle/crshome/product/10.2.0/db_1/log/node1/client

[Root @ node1 client] # Cat ocrconfig_6090.log

Oracle Database 10g CRS release 10.2.0.1.0 production copyright 1996,200 5 oracle. All rights reserved.

10:51:08. 056: [ocrconf] [3086915264] ocrconfig starts...

10:51:08. 109: [ocrosd] [3086915264] utopen: 12: not enough space in the backing store

10:51:08. 109: [ocrosd] [3086915264] utopen: 10: none of the OCR devices are usable

10:51:08. 109: [ocrraw] [3086915264] phy_rec: 1: cocould not open OCR Device

10:51:08. 109: [ocrconf] [3086915264] failed to restore OCR from [/opt/Oracle/crshome/product/10.2.0/db_1/CDATA/CRS/backup00.ocr]

10:51:08. 109: [ocrconf] [3086915264] Exiting [Status = failed]...

 

Probably permission issues

[Root @ node1 client] # ll/dev/raw *

CrW-RW ---- 1 root disk 162, September 18 18:41/dev/raw/raw1

CrW-RW ---- 1 root disk 162, September 18 18:41/dev/raw/raw2

 

The permission is modified only to prevent the failure to read the bare device after OCR is run.

 

Temporarily shield crsd from startup

[Root @ node1 opt] # vi/etc/inittab

# Run xdm in runlevel 5

X: 5: respawn:/etc/X11/preofdm-nodaemon

# H1: 35: respawn:/etc/init. d/init. evmd run>/dev/null 2> & 1 </dev/null

# H2: 35: respawn:/etc/init. d/init.css D fatal>/dev/null 2> & 1 </dev/null

# H3: 35: respawn:/etc/init. d/init. crsd run>/dev/null 2> & 1 </dev/null

 

My colleague reminded me that the partition still has problems.

Disk/dev/sdc: 107 MB, 107374080 bytes

64 heads, 32 sectors/track, 102 cylinders

Units = cylinders of 2048*512 = 1048576 bytes

Device Boot Start End Blocks Id System

/Dev/sdc1 102 102 1024 83 Linux

Disk/dev/sdd: 107 MB, 107374080 bytes

64 heads, 32 sectors/track, 102 cylinders

Units = cylinders of 2048*512 = 1048576 bytes

Device Boot Start End Blocks Id System

/Dev/sdd1*1 102 104432 83 Linux

 

Repartition fdisk/dev/SDC

Re-import the raw1.file exported from the raw file.

Re-ocrconfig-Restore/opt/Oracle/crshome/product/10.2.0/db_1/CDATA/CRS/backup00.ocr

No response. No!

It's useless to redo the system ....

 

The next day, I want to get started with node 2. Because Node 2 also reports the same error, that is, adding the disk to the scsci0 bus causes the drive letter to change.

So it has never experienced the best practices of two colleagues.

Node 2 started

Modify/etc/sysconfig/rawdevices

[Root @ node2 ~] # Cat/etc/sysconfig/rawdevices

# This file and interface are deprecated.

# Applications needing raw device access shoshould Open Regular

# Block devices with o_direct.

# Raw device bindings

# Format: <rawdev> <major> <minor>

# <Rawdev> <blockdev>

# Example:/dev/raw/raw1/dev/sda1

#/Dev/raw/raw2 8 5

 

/Dev/raw/raw1/dev/sdc1

/Dev/raw/raw2/dev/sdd1

[Root @ node2 ~] # Service rawdevices restart

The OCR is not valid. restart the system.

The three Ocrconfig check crs items are OK.

Crs_stat-t Node 2 is OK.

 

Originally, I wanted to use node 2 to automatically restore the content of the OCR disk. Node 1's OCR can read the correct content and start successfully.

Node 2 disabled

Crsctl stop crs VM busy

 

When Node 1 is enabled, everything is the same. The old OCR logs are not written in the/TMP and client directories, and the CRS logs are not.

Isn't it true that the OCR program has been damaged? I Don't Want To Start Node 2 to compare the files one by one.

Ll/dev/raw * permission

Cat/etc/sysconfig/rawdevices drive letter.

 

Today, let's talk about RAC. This book goes to Chapter 1 OCR part of the tool 6th page. You can see whether the configuration CRS stack is automatically started.

The crsctl disable crs Command actually modifies the following file

/Etc/oracle/scls_scr/dbp/root/crsstart

Note that dbp is replaced with node1

Compare the two node files. Node 2 is enable Node 1 is disable.

I remember my colleague told me to disable this operation for node 1 CRS. Okay, change it to enable and restart section 1.

 

PS. Check whether/etc/init. d/init. crsd run is a bunch

[Root @ node1 ~] # Ps-ef | grep crs *

Root 3392 1 0 :38? 00:00:00 crond

Root 3427 1 0 :38? 00:00:00 anacron-s

Root 4045 1 0 :38? 00:00:00/bin/su-l oracle-c sh-c 'ulimit-c unlimited; cd/opt/oracle/crshome/product/10.2.0/db_1/log/node1/evmd; exec/opt/oracle/crshome/product/10.2.0/db_1/bin/evmd'

Root 4052 1? 00:00:08/opt/oracle/crshome/product/10.2.0/db_1/bin/crsd. bin reboot

Oracle 4773 4045 0? 00:00:01/opt/oracle/crshome/product/10.2.0/db_1/bin/evmd. bin

Root 4890 4752 0? 00:00:00/bin/su-l oracle-c/bin/sh-c 'ulimit-c unlimited; cd/opt/oracle/crshome/product/10.2.0/db_1/log/node1/cssd;

 

 

[Root @ node1 ~] # Crsctl check CRS

CSS appears healthy

CRS appears healthy

EVM appears healthy

[Root @ node1 ~] # Su-Oracle

[Oracle @ node1 ~] $ Crs_stat-T

Name type target State host

------------------------------------------------------------

Ora... c1.inst application online node1

Ora... c2.inst application online node2

Ora. MYRAC. db application ONLINE node2

Ora... SM1.asm application ONLINE node1

Ora... E1.lsnr application ONLINE node1

Ora. node1.gsd application ONLINE node1

Ora. node1.ons application ONLINE node1

Ora. node1.vip application ONLINE node1

Ora... sm2.asm application online node2

Ora... e2.lsnr application online node2

Ora. node2.gsd application online node2

Ora. node2.ons application online node2

Ora. node2.vip application online node2

 

Summary

1. When adding a disk, be careful when the drive letter is changed.

2. Note that when you create a partition by using the partition command start and end, two values are displayed: 1.

3. The OCR program is started before CRS. The OCR program cannot start CRS or

4. The two colleagues are familiar with the commands, which is fast and easy to hide details of information.

5. Remember not to use the try-and-error method to modify the CRS settings, especially when the problem is not located accurately.

6. Any changes should be manually recorded in the book or word, because constant modification and trial and error may cause environmental damage.

7. This BUG has been around for one week, asking for advice from many people. The two good colleagues can play a role and provide effective help. the Group provides commands and files to familiarize themselves with some linux commands and file configurations. therefore, when a person cannot solve the problem, he can wash his hair, sleep, or ask others. as the saying goes, the public will be fascinated by the audience. after a long time, people will become dizzy and visually fatigued, so that important information and prompts may be easily missed.

8 fortunately, this is a virtual machine. If it is a production system, it will take a short time to handle the problem. It is estimated that the problem cannot be solved under noisy, pressure, and sultry conditions. the trial and error methods may cause more problems under pressure.