The biggest tragedy is that the database is broken and the backup is gone. It reminds me of a popular phrase in the past few years: the biggest tragedy is that people are living and money is gone. In short, backup is more important than everything. Scenario: In the 11G ocr, the asm disk is accidentally deleted by the user and all the backups are deleted by mistake, and the olr is damaged. So far, the tragedy has been repeated. Recovery Process: (in fact, root is used.
The biggest tragedy is that the database is broken and the backup is gone. It reminds me of a popular phrase in the past few years: the biggest tragedy is that people are living and money is gone. In short, backup is more important than everything. Scenario: In the 11G ocr, the asm disk is accidentally deleted by the user and all the backups are deleted by mistake, and the olr is damaged. So far, the tragedy has been repeated. Recovery Process: (in fact, root is used.
The biggest tragedy is that the database is broken and the backup is gone. It reminds me of a popular saying in the past few years: "the biggest tragedy is that people are living and money is gone ". In short, backup is more important than everything.
Scenario introduction:
In the 11G ocr, the asm disk is accidentally deleted by the user and all backups are deleted by mistake. The olr is also damaged. So far, the tragedy has been repeated.
Recovery Process: (in fact, it is the process of rebuilding ocr using root. sh. After reconstruction, you may need to re-register related resources, such as listener or database instances !)
1. Clear the configuration information of all nodes in the cluster. (The rootcrs. sh script exists in the 11G. This script exists in $ GRID_HOME/crs/install by default ).
Note: This script must be executed under root; otherwise, an error message will be prompted.
?
1
2
3
4
5
6
[Grid @ rac-one install] $./rootcrs. pl
You must be logged in as root to run this script.
Log in as root and rerun this script.
2014-04-01 17: 08: 12: Not running as authorized user
Insufficient privileges to execute this script.
Root or administrative privileges needed to run the script.
For more information about the script function, see-help. Such as upgrade, rollback, and reconfiguration wait
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
[Root @ rac-two install] #./rootcrs. pl-help
Usage:
Rootcrs. pl [-verbose] [-upgrade [-force] |-patch]
[-Paramfile ]
[-Deconfig [-deinstall] [-keepdg] [-force] [-lastnode]
[-Downgrade-oldcrshome -Version [-Force] [-lastnode]
[-Unlock [-crshome ] [-Nocrsstop]
[-Init]
Options:
-Verbose Run this script in verbose mode
-Upgrade Oracle HA is being upgraded from previous version
-Patch Oracle HA is being upgraded to a patch version
-Paramfile Complete path of file specifying HA parameter values
-Lastnode Force the node this script is executing on to be considered
As the last node of deconfiguration or downgrade, and perform
Actions associated with demo-ing or downgrading
Last node
-Downgrade Downgrade the clusterware
-Version For use with downgrade; special handling is required if
Downgrading to 9i. This is the old crs version in the format
A. B .C.D. E (e. g 11.1.0.6.0 ).
-Deconfig Remove Oracle Clusterware to allow it to be uninstalled or reinstalled
-Force Force the execution of steps in delete or dwongrade that cannot
Be verified to be safe
-Deinstall Reset the permissions on CRS home during de-configuration
-Keepdg Keep existing diskgroups during de-configuration
-Unlock Unlock CRS home
-Crshome Complete path of crs home. Use with unlock option
-Oldcrshome For use with downgrade. Complete path of the old crs home
-Nocrsstop used with unlock option to reset permissions on an inactive grid home
-Init Reset the permissions of all files and directories under CRS home
If neither-upgrade nor-patch is supplied, a new install is already med
To see the full manpage for this program, execute:
Perldoc rootcrs. pl
[Root @ rac-two install] #
Clear all node cluster information under root:
Node 2:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
[Grid @ rac-one install] $ su
Password:
[Root @ rac-one install] #./rootcrs. pl-deconfig-force
Using configuration parameter file:./crsconfig_params
Network exists: 1/192. 168.4.0/255.255.255.0/eth0, type static
VIP exists:/rac-one-vip/192.168.4.113/192.168.4.0/255.255.255.0/eth0, hosting node rac-one
VIP exists:/rac-two-vip/192.168.4.114/192.168.4.0/255.255.255.0/eth0, hosting node rac-two
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2673: Attempting to stop 'ora. registry. acfs 'on 'rac-one'
CRS-2677: Stop of 'ora. registry. acfs 'on 'rac-one' succeeded
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac-one'
CRS-2673: Attempting to stop 'ora. crsd' on 'rac-one'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac-one'
CRS-2673: Attempting to stop 'ora. c1' on 'rac-one'
CRS-2673: Attempting to stop 'ora. GIDG. dg 'on 'rac-one'
CRS-2673: Attempting to stop 'ora. DATADG. dg 'on 'rac-one'
CRS-2677: Stop of 'ora. DATADG. dg 'on 'rac-one' succeeded
CRS-2677: Stop of 'ora. GIDG. dg 'on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora. asm 'on 'rac-one'
CRS-2677: Stop of 'ora. * 'on 'rac-one' succeeded
CRS-2672: Attempting to start 'ora. c1' on 'rac-two'
CRS-2677: Stop of 'ora. asm 'on 'rac-one' succeeded
CRS-2676: Start of 'ora. brax' on 'rac-two' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac-one' has completed
CRS-2677: Stop of 'ora. crsd' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora. crf' on 'rac-one'
CRS-2673: Attempting to stop 'ora. ctssd 'on 'rac-one'
CRS-2673: Attempting to stop 'ora. evmd 'on 'rac-one'
CRS-2673: Attempting to stop 'ora. asm 'on 'rac-one'
CRS-2673: Attempting to stop 'ora. drivers. acfs 'on 'rac-one'
CRS-2673: Attempting to stop 'ora. mdnsd' on 'rac-one'
CRS-2677: Stop of 'ora. crf' on 'rac-one 'succeeded
CRS-2677: Stop of 'ora. evmd 'on 'rac-one' succeeded
CRS-2677: Stop of 'ora. mdnsd' on 'rac-one' succeeded
CRS-2677: Stop of 'ora. ctssd 'on 'rac-one' succeeded
CRS-2677: Stop of 'ora. asm 'on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora. cluster_interconnect.haip 'on 'rac-one'
CRS-2677: Stop of 'ora. cluster_interconnect.haip 'on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora.css d' on 'rac-one'
CRS-2677: Stop of 'ora.css d' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora. gipcd 'on 'rac-one'
CRS-2677: Stop of 'ora. drivers. acfs 'on 'rac-one' succeeded
CRS-2677: Stop of 'ora. gipcd' on 'rac-one' succeeded
CRS-2673: Attempting to stop 'ora. gpnpd 'on 'rac-one'
CRS-2677: Stop of 'ora. gpnpd 'on 'rac-one' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac-one' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
[Root @ rac-one install] #
Node 1. Because my RAC is dual-node, the lastnode parameter is used to clear the last one:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
[Grid @ rac-two crs] $ cd install/
[Grid @ rac-two install] $ su
Password:
[Root @ rac-two install] #./rootcrs. pl-deconfig-force-lastnode
Using configuration parameter file:./crsconfig_params
CRS resources for listeners are still configured
Network exists: 1/192. 168.4.0/255.255.255.0/eth0, type static
VIP exists:/rac-two-vip/192.168.4.114/192.168.4.0/255.255.255.0/eth0, hosting node rac-two
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2673: Attempting to stop 'ora. registry. acfs 'on 'rac-two'
CRS-2677: Stop of 'ora. registry. acfs 'on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora. crsd' on 'rac-two'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora. c1' on 'rac-two'
CRS-2673: Attempting to stop 'ora. GIDG. dg 'on 'rac-two'
CRS-2673: Attempting to stop 'ora. DATADG. dg 'on 'rac-two'
CRS-2677: Stop of 'ora. DATADG. dg 'on 'rac-two' succeeded
CRS-2677: Stop of 'ora. brax' on 'rac-two' succeeded
CRS-2677: Stop of 'ora. GIDG. dg 'on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora. asm 'on 'rac-two'
CRS-2677: Stop of 'ora. asm 'on 'rac-two' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac-two' has completed
CRS-2677: Stop of 'ora. crsd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora. ctssd 'on 'rac-two'
CRS-2673: Attempting to stop 'ora. evmd 'on 'rac-two'
CRS-2673: Attempting to stop 'ora. asm 'on 'rac-two'
CRS-2677: Stop of 'ora. evmd 'on 'rac-two' succeeded
CRS-2677: Stop of 'ora. asm 'on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora. cluster_interconnect.haip 'on 'rac-two'
CRS-2677: Stop of 'ora. cluster_interconnect.haip 'on 'rac-two' succeeded
CRS-2677: Stop of 'ora. ctssd 'on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.css d' on 'rac-two'
CRS-2677: Stop of 'ora.css d' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.css dmonitor 'on 'rac-two'
CRS-2676: Start of 'ora.css dmonitor 'on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.css d' on 'rac-two'
CRS-2672: Attempting to start 'ora. diskmon 'on 'rac-two'
CRS-2676: Start of 'ora. diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.css d' on 'rac-two' succeeded
CRS-4611: Successful deletion of voting disk + GIDG.
ASM de-configuration Tracing file location:/tmp/asmcadc_clean2014-04-01_05-14-52-PM.log
ASM Clean Configuration START
ASM Clean Configuration END
ASM with SID + ASM1 deleted successfully. Check/tmp/asmcadc_clean2014-04-01_05-14-52-PM.log for details.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac-two'
CRS-2673: Attempting to stop 'ora. mdnsd' on 'rac-two'
CRS-2673: Attempting to stop 'ora. ctssd 'on 'rac-two'
CRS-2673: Attempting to stop 'ora. asm 'on 'rac-two'
CRS-2677: Stop of 'ora. mdnsd' on 'rac-two' succeeded
CRS-2677: Stop of 'ora. ctssd 'on 'rac-two' succeeded
CRS-2677: Stop of 'ora. asm 'on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora. cluster_interconnect.haip 'on 'rac-two'
CRS-2677: Stop of 'ora. cluster_interconnect.haip 'on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora.css d' on 'rac-two'
CRS-2677: Stop of 'ora.css d' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora. crf' on 'rac-two'
CRS-2677: Stop of 'ora. crf' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora. gipcd 'on 'rac-two'
CRS-2677: Stop of 'ora. gipcd' on 'rac-two' succeeded
CRS-2673: Attempting to stop 'ora. gpnpd 'on 'rac-two'
CRS-2677: Stop of 'ora. gpnpd 'on 'rac-two' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac-two' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
[Root @ rac-two install] #
2. re-build ocr and olr and use the root. sh script to re-build. In fact, this is the script executed in the installation of RAC. The default location is $ GRID_HOME /;
Eg:
Node 1:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
[Root @ rac-two grid] #./root. sh
Padding Ming root user operation for Oracle 11g
The following environment variables are set:
ORACLE_OWNER = grid
ORACLE_HOME =/u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the/etc/oratab file as needed
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be saved med.
Using configuration parameter file:/u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
CRS-2672: Attempting to start 'ora. mdnsd' on 'rac-two'
CRS-2676: Start of 'ora. mdnsd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora. gpnpd 'on 'rac-two'
CRS-2676: Start of 'ora. gpnpd 'on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.css dmonitor 'on 'rac-two'
CRS-2672: Attempting to start 'ora. gipcd 'on 'rac-two'
CRS-2676: Start of 'ora.css dmonitor 'on 'rac-two' succeeded
CRS-2676: Start of 'ora. gipcd' on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora.css d' on 'rac-two'
CRS-2672: Attempting to start 'ora. diskmon 'on 'rac-two'
CRS-2676: Start of 'ora. diskmon' on 'rac-two' succeeded
CRS-2676: Start of 'ora.css d' on 'rac-two' succeeded
ASM created and started successfully.
Disk Group GIDG created successfully.
Clscfg:-install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root '..
Operation successful.
Successful addition of voting disk 498646ba39604f86bf697c9748a67697.
Successful addition of voting disk 2e1bd16f9e6d4f36bf93550dc8268725.
Successful addition of voting disk 3fbd31a0b2634feabfa1115a504cbbe6.
Successfully replaced voting disk group with + GIDG.
CRS-4266: Voting file (s) successfully replaced
# STATE File Universal Id File Name Disk group
------------------------------------------
1. ONLINE 498646ba39604f86bf697c9748a67697 (/dev/asm-diske) [GIDG]
2. ONLINE 2e1bd16f9e6d4f36bf93550dc8268725 (/dev/asm-diskd) [GIDG]
3. ONLINE 3fbd31a0b2634feabfa1115a504cbbe6 (/dev/asm-diskf) [GIDG]
Located 3 voting disk (s ).
CRS-2672: Attempting to start 'ora. asm 'on 'rac-two'
CRS-2676: Start of 'ora. asm 'on 'rac-two' succeeded
CRS-2672: Attempting to start 'ora. GIDG. dg 'on 'rac-two'
CRS-2676: Start of 'ora. GIDG. dg 'on 'rac-two' succeeded
Preparing packages for installation...
Cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster... succeeded
[Root @ rac-two grid] #
In addition, if the asm disk is damaged, the disk needs to be repaired again, and then the previous diskgroup will be automatically re-created.
Now olr and ocr are created successfully.
Node 2:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
[Root @ rac-one grid] #./root. sh
Padding Ming root user operation for Oracle 11g
The following environment variables are set:
ORACLE_OWNER = grid
ORACLE_HOME =/u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the/etc/oratab file as needed
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be saved med.
Using configuration parameter file:/u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization-successful
Adding Clusterware entries to upstart
The CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac-two, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Preparing packages for installation...
Cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster... succeeded
[Root @ rac-one grid] #
3. Check Resource Information:
Now we are using the srvctl and crsctl tools, which are frequently used by oracle RAC. Another tool is oifcfg to configure Nic information. Not introduced.
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
[Root @ rac-one bin] #./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online.
[Root @ rac-one bin] #./crsctl stat res-t
--------------------------------------------------------------------------------