11gR2 RAC exclusive mode replace votedisk encounters PROC-26, restore ocr encounters CRS-4000, PROT-35, proc-26crs-4000
Link: http://blog.itpub.net/23135684/viewspace-748816/
The storage data of the 11gR2 RAC system is completely lost. The software of all nodes is installed on the local disk, and the local disk retains the OCR backup, the following describes how to restore the normal operation of Clusterware through replace votedisk and restore ocr:
1. Start CRS to the exclusive mode. [Root @ rac1 bin] #./crsctl stop has-f [root @ rac1 bin] #./crsctl start crs-excl-nocrs
2. Create a disk group for storing OCR and Votedisk. [Grid @ rac1 ~] $ Sqlplus/as sysasm
SQL * Plus: Release 11.2.0.3.0 Production on Sun Oct 21 22:04:50 2012
Copyright (c) 1982,201 1, Oracle. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0-64bit ProductionWith the Real Application Clusters and Automatic Storage Management options
SQL> CREATE DISKGROUP OCR NORMAL REDUNDANCY 2 DISK 'orcl: OCRVDISK1 ', 'orcl: ocrvdisk2', 'orcl: OCRVDISK3 ';
Diskgroup created.
3. replace votedisk and restore ocr.
Perform the following replace votedisk and restore ocr operations based on the disk group created above, and receive the following error:
[Root @ rac1 bin] #./crsctl replace votedisk + OCROracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage
[Root @ rac1 bin] #. /ocrconfig-restore/u01/app/grid/11.2.0/cdata/backup00.ocrCRS-4000: Command Replace failed, or completed with errors. the PROT-35: The configured Oracle Cluster Registry locations are not accessible
4. Rebuild the disk group. DROP the disk group and create a new disk group (add the compatible. asm attribute settings): [grid @ rac1 ~] $ Sqlplus/as sysasm
SQL * Plus: Release 11.2.0.3.0 Production on Sun Oct 21 22:04:50 2012
Copyright (c) 1982,201 1, Oracle. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0-64bit ProductionWith the Real Application Clusters and Automatic Storage Management options
SQL> drop diskgroup ocrvdisk;
Diskgroup dropped.
SQL> CREATE DISKGROUP OCR NORMAL REDUNDANCY 2 DISK 'orcl: OCRVDISK1 ', 'orcl: OCRVDISK2', 'orcl: OCRVDISK3 '3 ATTRIBUTE 'compatible. asm '= '11. 2.0.0.0 ';
Diskgroup created.
[Root @ rac2 bin] #./crsctl replace votedisk + OCR Oracle Cluster Registry initialization failed accessing Oracle Cluster Registry device: PROC-26: Error while accessing the physical storage
The correct sequence should be to restore ocr first, and then replace votedisk.
[Root @ rac2 bin] #. /ocrconfig-restore/u01/app/grid/11.2.0/cdata/backup00.ocr [root @ rac2 bin] #. /crsctl replace votedisk + OCR Successful addition of voting disk 640c1dbe94ed4fe6bffe70b8ea7544b9. successful addition of voting disk 6f45ec9c09c24fb7bf02bed03de2b8a4. successful addition of voting disk a5a2c30e82f14fe9bfe23029864912cc. successfully replaced voting disk group with + OCR.CRS-4266: Voting file (s) successfully replaced
Pay attention to the above two points in red and bold, votedisk and ocr can be restored smoothly.
5. Configure the ASM parameter file.
After completing the preceding operations, the asm spfile is not created.
[Root @ rac2 bin] #./crsctl stop has-f ......
CRS-4133: Oracle High Availability Services has been stopped. [root @ rac2 bin] #. /crsctl start hasCRS-4123: Oracle High Availability Services has been started. [root @ rac2 bin] # su-grid [grid @ rac2 ~] $ Sqlplus/as sysdba
SQL * Plus: Release 11.2.0.3.0 Production on Thu Nov 8 23:49:28 2012
Copyright (c) 1982,201 1, Oracle. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0-64bit ProductionWith the Real Application Clusters and Automatic Storage Management options
SQL> show parameter spfile
Name type value ------------------------------------ ----------- -------------------------------- spfile string
Follow these steps to manually create an asm spfile:
1). Create the asm pfile file:
[Root @ rac2 ~] # Add the following parameters to vi/tmp/asm_pfile.txt :*. asm_power_limit = 1 *. diagnostic_dest = '/u01/app/grid/11.2.0/log '*. instance_type = 'asm '*. large_pool_size = 12 M *. remote_login_passwordfile = 'clusive'
2) create a SPFILE file: [root @ rac2 ~] # Su-grid [grid @ rac2 ~] $ Sqlplus/as sysasm
SQL * Plus: Release 11.2.0.3.0 Production on Fri Nov 9 00:03:22 2012
Copyright (c) 1982,201 1, Oracle. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0-64bit ProductionWith the Real Application Clusters and Automatic Storage Management options
SQL> create spfile = '+ OCR' from pfile = '/tmp/asm_pfile.txt ';
File created.
SQL> show parameter spfile
Name type value ------------------------------------ ----------- -------------------------------- spfile string
3) restart the HAS service and verify the SPFILE parameter: [root @ rac2 ~] # Cd/u01/app/grid/11.2.0/bin [root @ rac2 bin] #. /crsctl stop has ...... CRS-4133: Oracle High Availability Services has been stopped. [root @ rac2 bin] #. /crsctl start hasCRS-4123: Oracle High Availability Services has been started. [root @ rac2 bin] #. /crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is OnlineCRS-4533: Event Manager is online [root @ rac2 bin] # su-grid [grid @ rac2 ~] $ Sqlplus/as sysasm
SQL * Plus: Release 11.2.0.3.0 Production on Fri Nov 9 00:32:25 2012
Copyright (c) 1982,201 1, Oracle. All rights reserved.
Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0-64bit ProductionWith the Real Application Clusters and Automatic Storage Management options
SQL> show parameter spfile
Name type value ------------------------------------ ----------- ------------------------------ spfile string + OCR/wstcluster/asmparameterfile/registry.253.798854653
SPFILE is successfully configured.
References for the entire process:
How to Restore OCR After the 1st ASM Diskgroup is Lost on Windows [ID 1294915.1]
Modification time: 2012-3-6 type: HOWTO status: PUBLISHED Priority: 3
In this Document
Goal
Solution
References
Applies to: Oracle Server-Enterprise Edition-Version: 11.2.0.1 to 11.2.0.2 [Release: 11.2 to 11.2]
Microsoft Windows (32-bit)
Microsoft Windows x64 (64-bit)
GoalThe goal of this document is to provide the steps necessary to restore the OCR after the 1st ASM diskgroup, which contains the OCR, Voting Disks, and asm spfile is lost.
The general process of restoring the OCR from backup is already ented here: Restoring Oracle Cluster Repository. solutionWhen using an ASM disk group for CRS there are typically 3 different types of files located in the disk group that potentially need to be restored/recreated:
- The Oracle Cluster Registry file (OCR)
- The Voting file (s)
- The shared SPFILE for the ASM instance (s)
1. Locate the latest automatic ic ocr backup
When using a non-shared CRS home, automatic OCR backups can be located on any node of the cluster, consequently all nodes need to be checked for the most recent backup:
$ CRS_HOME \ bin \ ocrconfig-showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
RACNODE01 2010/08/20 12:25:00 C: \ app \ 11.2.0 \ grid \ cdata \ cluster_name \ backup00.ocr
RACNODE01 2010/08/20 08:24:58 C: \ app \ 11.2.0 \ grid \ cdata \ cluster_name \ backup01.ocr
RACNODE01 2010/08/20 04:24:55 C: \ app \ 11.2.0 \ grid \ cdata \ cluster_name \ backup02.ocr
RACNODE01 2010/08/19 04:24:41 C: \ app \ 11.2.0 \ grid \ cdata \ cluster_name \ day. ocr
RACNODE01 2010/08/09 04:22:07 C: \ app \ 11.2.0 \ grid \ cdata \ cluster_name \ week. ocr
2. Clear the partitions of the disks that were used in the 1st diskgroup:
$ CRS_HOME \ bin \ logpartformat/q \. \ ORCLDISKOCRVOTE0
$ CRS_HOME \ bin \ logpartformat/q \. \ ORCLDISKOCRVOTE1
$ CRS_HOME \ bin \ logpartformat/q \. \ ORCLDISKOCRVOTE2
3. Re-Stamp the 1st diskgroup's (+ OCRVOTE) disks using ASMTOOLG:
4. Open the server services console: Start --> Run --> services. msc
5. Manually stop the OracleOHService service
6. Start the CRS stack in exclusive mode:
11.2.0.1:
$ CRS_HOME \ bin \ crsctl start crs-excl
...
CRS-2672: Attempting to start 'ora. asm 'on 'racnode1'
CRS-2676: Start of 'ora. asm 'on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora. crsd' on 'racnode1'
CRS-2676: Start of 'ora. crsd' on 'racnode1' succeeded
11.2.0.2:
$ CRS_HOME \ bin \ crsctl start crs-excl-nocrs
...
CRS-2672: Attempting to start 'ora. cluster_interconnect.haip 'on 'racnode1'
CRS-2672: Attempting to start 'ora. ctssd 'on 'racnode1'
CRS-2676: Start of 'ora. drivers. acfs 'on 'racnode1' succeeded
CRS-2676: Start of 'ora. ctssd 'on 'racnode1' succeeded
CRS-2676: Start of 'ora. cluster_interconnect.haip 'on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora. asm 'on 'racnode1'
CRS-2676: Start of 'ora. asm 'on 'racnode1' succeeded
IMPORTANT:
A new option '-nocrs' has been introduced with 11.2.0.2, which prevents the start of the ora. crsd resource. it is vital that this option is specified, otherwise the failure to start the ora. crsd resource will tear down ora. cluster_interconnect.haip, which in turn will cause ASM to crash.
7. Re-create the OCRVOTE diskgroup via sqlplus
The disk group can now be (re-) created via sqlplus from the grid user. The compatible. asm attribute must be set to 11.2 in order for the disk group to be used by CRS:
Sqlplus/as sysasm
SQL * Plus: Release 11.2.0.1.0 Production on Thu Sep 2 08:39:18 2010
Copyright (c) 1982,201 0, Oracle. All rights reserved.
Connected:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0-64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> CREATE DISKGROUP OCRVOTE NORMAL REDUNDANCY
DISK '\. \ ORCLDISKOCRVOTE0', '\. \ ORCLDISKOCRVOTE1', '\. \ ORCLDISKOCRVOTE2'
ATTRIBUTE 'compatible. asm '= '11. 2.0.0.0 ';
Diskgroup created.
SQL> exit
8. Restore the latest OCR backup
Now that the CRS disk group is created & mounted the OCR can be restored:
$ CRS_HOME \ bin \ ocrconfig-restore C: \ app \ 11.2.0 \ grid \ cdata \ cluster_name \ backup00.ocr
9. Start the CRS daemon on the current node (11.2.0.1 only !)
Now that the OCR has been restored the CRS daemon can be started, this is needed to recreate the Voting file. Skip this step for 11.2.0.2.0.
$ CRS_HOME \ bin \ crsctl start res ora. crsd-init
CRS-2672: Attempting to start 'ora. crsd' on 'racnode1'
CRS-2676: Start of 'ora. crsd' on 'racnode1' succeeded
10. Recreate the Voting file
The Voting file needs to be initialized in the CRS disk group:
$ CRS_HOME \ bin \ crsctl replace votedisk + OCRVOTE
Successful addition of voting disk 401738d9d75e4f7abf60d6bac7f9119c.
Successful addition of voting disk 7449ffaed19b4f22bf643553ea0000683.
Successful addition of voting disk 5be68de758a74fbbbf9b8a4240d5e32d.
Successfully replaced voting disk group with + OCRVOTE.
CRS-4266: Voting file (s) successfully replaced
11. Recreate the SPFILE for ASM
Prepare a pfile (e.g. c: \ tmp \ asm_pfile.ora) with the ASM startup parameters-these may vary from the example below. if in doubt consult the ASM alert log as the ASM instance startup shocould list all non-default parameter values. please note the last startup of ASM (in step 2 via CRS start) will not have used an SPFILE, so a startup prior to the loss of the CRS disk group wocould need to be located.
*. Asm_power_limit = 1
*. Diagnostic_dest = 'C: \ app \ 11.2.0 \ grid'
*. Instance_type = 'asm'
*. Large_pool_size = 12 M
*. Remote_login_passwordfile = 'clusive'
Now the SPFILE can be created using this PFILE:
Sqlplus/as sysasm
SQL * Plus: Release 11.2.0.1.0 Production on Tue Mar 30 11:52:39 2010
Copyright (c) 1982,200 9, Oracle. All rights reserved.
Connected:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0-Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create spfile = '+ ocrvote' from pfile = 'C: \ tmp \ asm_pfile.ora ';
File created.
SQL> exit
12. shutdown CRS
Since CRS is running in exclusive mode, it needs to be shutdown to allow CRS to run on all nodes again. Use of the force (-f) option may be required:
$ CRS_HOME/bin/crsctl stop crs-f
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'auw2k3 'has completed
CRS-4133: Oracle High Availability Services has been stopped.
13. start CRS
As the root user submit the CRS startup on all cluster nodes:
# $ GRID_HOME/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
** This step can take several minutes to run as a result of the changes we have made to the RAC configuration. please wait about 5 minutes, and then check that all the CRS services are running:
$ CRS_HOME \ bin \ crsctl status resource-t
14. Verify CRS
$ CRS_HOME \ bin \ crsctl check cluster-all
$ CRS_HOME \ bin \ crsctl status resource-t
Errors:
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage ASM error [SLOS: cat = 8, pn = kgfolclcpi1, dep = 210, loc = kgfokge
AMDU-00210: No disks found in diskgroup OCRVOTE
AMDU-00210: No disks found in diskgroup OCRVOTE
ReferencesNOTE: 1062983.1-How to restore ASM based OCR after complete loss of the CRS diskgroup on Linux/Unix systems
-- End --
