Rebuilding a node of RAC

Problem description:

Hacmp5.5 + AIX 5.3.10 + concurrentvg + Oracle 10.2.0.4 RAC

 

The test database is running normally on RAC.

Because a node of RAC is always unable to join the dlpar.

I handed it over to the IBM 2-line engineer. No result was returned. I said I had to send it to the 3-line engineer to solve the problem.

Because the Test Library is the object of stream replication. It takes too much to reconstruct the entire target RAC.

Is there a way to directly remove a node and restore the entire application after the OS is reinstalled?

Finally, we decided to use the following method to restore the entire database.

 

 
|
Clear relevant log information on the first node, focusing on error logs during debugging
Errclear 0
|

|
Pre backup ocr_disk vote_disk,
# Ls-l vote_disk
CrW-r -- 1 Oracle oinstall 40, 3 Mar 25 :01 vote_disk
# Ls-l ocr_disk
CrW-r ----- 1 root oinstall 40, 2 Mar 25 12:01 ocr_disk
#
#
# Dd If =/dev/vote_disk of =/home/Oracle/vote_disk.bak

# Dd If =/dev/vote_disk of =/home/Oracle/vote_disk.bak
2096640 + 0 records in.
2096640 + 0 records out.
# Dd If =/dev/ocr_disk of =/home/Oracle/ocr_disk.bak
2096640 + 0 records in.
2096640 + 0 records out.

# Dd If =/dev/ocr_disk of =/home/Oracle/ocr_disk.bak

# Dd If =/dev/raw/raw2 F =/home/Oracle/vote_disk.bak
# Dd If =/dev/raw/raw2 F =/home/Oracle/ocr_disk.bak
|
 

Scan the array on zhyw2 and confirm that the two nodes in the powerpath path are consistent.
[Root @ zhyw2] #./symdev list PD

Symmetrix ID: 000290103435

Device Name directors Device
-----------------------------------------------------------------------------
Cap
Sym physical Sa da: It config attribute STS (MB)
-----------------------------------------------------------------------------

0022/dev/rhdisk3 09B: 1 01a: C0 2-way mir N/GRP 'd vcm wd 3
0035/dev/rhdiskpower64 08B: 1 16D: D7 2-way mir N/GRP 'd RW 3
0036/dev/rhdiskpower65 08B: 1 16C: C7 2-way mir N/GRP 'd RW 3
0065/dev/rhdiskpower0 09B: 1 01b: DC 2-way mir N/GRP 'd RW 3
0066/dev/rhdiskpower1 09B: 1 16A: CB 2-way mir N/GRP 'd RW 3
0084/dev/rhdiskpower2 09B: 1 16A: D0 2-way mir N/GRP 'd RW 1024
0085/dev/rhdiskpower3 09B: 1 16b: D3 2-way mir N/GRP 'd RW 1024
013a/dev/rhdiskpower4 09B: 1 16b: Ce 2-way mir N/GRP 'd (m) RW 49140
013e/dev/rhdiskpower5 09B: 1 01c: D5 2-way mir N/GRP 'd (m) RW 49140
0142/dev/rhdiskpower6 09B: 1 16C: D6 2-way mir N/GRP 'd (m) RW 49140
0146/dev/rhdiskpower7 09B: 1 01c: D7 2-way mir N/GRP 'd (m) RW 49140
014a/dev/rhdiskpower8 09B: 1 16C: D8 2-way mir N/GRP 'd (m) RW 49140
014e/dev/rhdiskpower9 09B: 1 16A: C3 2-way mir N/GRP 'd (m) RW 49140
0152/dev/rhdiskpower10 09B: 1 01a: C2 2-way mir N/GRP 'd (m) RW 49140
0156/dev/rhdiskpower11 09B: 1 16A: C1 2-way mir N/GRP 'd (m) RW 49140
015a/dev/rhdiskpower12 09B: 1 01a: D9 2-way mir N/GRP 'd (m) RW 49140
015e/dev/rhdiskpower13 09B: 1 16A: D8 2-way mir N/GRP 'd (m) RW 49140
0162/dev/rhdiskpower14 09B: 1 01a: DB 2-way mir N/GRP 'd (m) RW 49140
0166/dev/rhdiskpower15 09B: 1 16A: DA 2-way mir N/GRP 'd (m) RW 49140
016a/dev/rhdiskpower16 09B: 1 01a: D5 2-way mir N/GRP 'd (m) RW 49140
016e/dev/rhdiskpower17 09B: 1 16A: D4 2-way mir N/GRP 'd (m) RW 49140
0172/dev/rhdiskpower18 09B: 1 01a: D7 2-way mir N/GRP 'd (m) RW 49140
0176/dev/rhdiskpower19 09B: 1 16A: D6 2-way mir N/GRP 'd (m) RW 49140
017a/dev/rhdiskpower20 09B: 1 16C: C3 2-way mir N/GRP 'd (m) RW 49140
017e/dev/rhdiskpower21 09B: 1 01c: C2 2-way mir N/GRP 'd (m) RW 49140
0182/dev/rhdiskpower22 09B: 1 16C: C1 2-way mir N/GRP 'd (m) RW 49140
0186/dev/rhdiskpower23 09B: 1 01c: C0 2-way mir N/GRP 'd (m) RW 49140
018a/dev/rhdiskpower24 09B: 1 16C: D0 2-way mir N/GRP 'd (m) RW 49140
018e/dev/rhdiskpower25 09B: 1 16C: DA 2-way mir N/GRP 'd (m) RW 49140
0192/dev/rhdiskpower26 09B: 1 01c: D9 2-way mir N/GRP 'd (m) RW 49140
0196/dev/rhdiskpower27 09B: 1 16C: DC 2-way mir N/GRP 'd (m) RW 49140
019a/dev/rhdiskpower28 09B: 1 01c: DB 2-way mir N/GRP 'd (m) RW 49140
019e/dev/rhdiskpower29 09B: 1 16b: Ce 2-way mir N/GRP 'd (m) RW 49140
01a2/dev/rhdiskpower30 09B: 1 01c: D5 2-way mir N/GRP 'd (m) RW 49140
01a6/dev/rhdiskpower31 09B: 1 16C: D6 2-way mir N/GRP 'd (m) RW 49140
01aa/dev/rhdiskpower32 09B: 1 01c: D7 2-way mir N/GRP 'd (m) RW 49140
01ae/dev/rhdiskpower33 09B: 1 16C: D8 2-way mir N/GRP 'd (m) RW 49140
01b2/dev/rhdiskpower34 09B: 1 16A: C3 2-way mir N/GRP 'd (m) RW 49140
01b6/dev/rhdiskpower35 09B: 1 01a: C2 2-way mir N/GRP 'd (m) RW 49140
01ba/dev/rhdiskpower36 09B: 1 16A: C1 2-way mir N/GRP 'd (m) RW 49140
01be/dev/rhdiskpower37 09B: 1 01a: D9 2-way mir N/GRP 'd (m) RW 49140
01c2/dev/rhdiskpower38 09B: 1 16A: D8 2-way mir N/GRP 'd (m) RW 49140
01c6/dev/rhdiskpower39 09B: 1 01a: DB 2-way mir N/GRP 'd (m) RW 49140
01ca/dev/rhdiskpower40 09B: 1 16A: DA 2-way mir N/GRP 'd (m) RW 49140
01ce/dev/rhdiskpower41 09B: 1 01a: D5 2-way mir N/GRP 'd (m) RW 49140
01d2/dev/rhdiskpower42 09B: 1 16A: D4 2-way mir N/GRP 'd (m) RW 49140
01d6/dev/rhdiskpower43 09B: 1 01a: D7 2-way mir N/GRP 'd (m) RW 49140
01da/dev/rhdiskpower44 09B: 1 16A: D6 2-way mir N/GRP 'd (m) RW 49140
01de/dev/rhdiskpower45 09B: 1 16C: C3 2-way mir N/GRP 'd (m) RW 49140
01e2/dev/rhdiskpower46 09B: 1 01c: C2 2-way mir N/GRP 'd (m) RW 49140
01e6/dev/rhdiskpower47 09B: 1 16C: C1 2-way mir N/GRP 'd (m) RW 49140
01ea/dev/rhdiskpower48 09B: 1 01c: C0 2-way mir N/GRP 'd (m) RW 49140
01ee/dev/rhdiskpower49 09B: 1 16C: D0 2-way mir N/GRP 'd (m) RW 49140
01f2/dev/rhdiskpower50 09B: 1 16C: DA 2-way mir N/GRP 'd (m) RW 49140
01f6/dev/rhdiskpower51 09B: 1 01c: D9 2-way mir N/GRP 'd (m) RW 49140
01fa/dev/rhdiskpower52 09B: 1 16C: DC 2-way mir N/GRP 'd (m) RW 49140
01fe/dev/rhdiskpower53 09B: 1 01c: DB 2-way mir N/GRP 'd (m) RW 49140
0202/dev/rhdiskpower54 09B: 1 16b: Ce 2-way mir N/GRP 'd (m) RW 49140
0206/dev/rhdiskpower55 09B: 1 01c: D5 2-way mir N/GRP 'd (m) RW 49140
020a/dev/rhdiskpower56 09B: 1 16C: D6 2-way mir N/GRP 'd (m) RW 49140
020e/dev/rhdiskpower57 09B: 1 01c: D7 2-way mir N/GRP 'd (m) RW 49140
0212/dev/rhdiskpower58 09B: 1 16C: D8 2-way mir N/GRP 'd (m) RW 49140
0216/dev/rhdiskpower59 09B: 1 16A: C3 2-way mir N/GRP 'd (m) RW 49140
021a/dev/rhdiskpower60 09B: 1 01a: C2 2-way mir N/GRP 'd (m) RW 49140
021e/dev/rhdiskpower61 09B: 1 16A: C1 2-way mir N/GRP 'd (m) RW 49140
0222/dev/rhdiskpower62 09B: 1 01a: D9 2-way mir N/GRP 'd (m) RW 49140
0226/dev/rhdiskpower63 09B: 1 16A: D8 2-way mir N/GRP 'd (m) RW 49140

Back up hacmp information on zhyw2
Back up RAC parameters related to zhyw2
|
1. Back up lpar zhyw2_old, oracle_base, and corresponding startup parameters.
Ls-L/etc/init.css D/etc/init. CRS/etc/init. crsd/etc/init. evmd/etc/rc. d/rc2.d/k96init. CRS
Ls-L/etc/rc. d/rc2.d/s96init. CRS/etc/Oracle/scls_scr/etc/Oracle/oprocd/etc/inittab. CRS
Ls-L/etc/inittab
Ls-L/etc/Oracle

/Etc/Oracle/scls_scr/<node name>/Oracle/cssfatal
$ Oracle_crs_home/install/paramfile. CRS

[Root @ zhyw2] # ls-L/etc/init.css D/etc/init. CRS/etc/init. crsd/etc/init. evmd/etc/rc. d/rc2.d/k96init. CRS
-R-XR-x 1 root system 2236 Mar 13/etc/init. CRS
-R-XR-x 1 root system 4944 Mar 13/etc/init. crsd
-R-XR-x 1 root system 54038 Mar 13/etc/init.css d
-R-XR-x 1 root system 3226 Mar 13/etc/init. evmd
Lrwxrwxrwx 1 root system 13 mar 13 16:02/etc/rc. d/rc2.d/k96init. CRS->/etc/init. CRS

Lrwxrwxrwx 1 root system 13 mar 13 16:02/etc/rc. d/rc2.d/s96init. CRS->/etc/init. CRS
/Etc/Oracle/oprocd:
Total 0
Drwxrwx --- 2 root system 256 Mar 13 check
Drwxrwx --- 2 root system 256 Mar 13 Fatal
Drwxrwx --- 2 root system 256 Mar 13 stop
/Etc/Oracle/scls_scr:
Total 0
Drwxr-XR-x 4 root system 256 Mar 13 zhyw2
[Root @ zhyw2] #
[Root @ zhyw2] # ls-L/etc/inittab
-RW-r -- 1 root system 4014 Mar 25/etc/inittab

|

2. Create a profile for HMC, zhyw2_new, add a disk to the profile, and add CDROM/RMT to the profile.

3. Perform mksysb on zhyw1 and restore the backup tape to zhyw2_new.

4. Remove the etherchannel corresponding to zhyw2_new and delete the adapter.
(Confirmation 1 is successful)
4-1> Add the adapter back (HbA + Nic)
4-2> re-scan the disk (using Mgr-V,/usr/symcli/bin/symcfg DIS, powermt config, powermt save)
4-3> Add IP address and Host Name
Confirm that the powerpath path is correct. Perform mknod operations on the corresponding vote and OCR.
0084/dev/rhdiskpower2 09B: 1 16A: D0 2-way mir N/GRP 'd RW 1024
0085/dev/rhdiskpower3 09B: 1 16b: D3 2-way mir N/GRP 'd RW 1024
Ls-L/dev/* hdiskpower2
Ls-L/dev/* hdiskpower3
BRW-RW ---- 1 root system 40, 2 Mar 25 19:38/dev/hdiskpower2
CrW-RW ---- 1 root system 40, 2 Mar 25 19:43/dev/rhdiskpower2

BRW-RW ---- 1 root system 40, 3 Mar 25 19:38/dev/hdiskpower3
CrW-RW ---- 1 root system 40, 3 Mar 25 19:43/dev/rhdiskpower3

Mknod/dev/ocr_disk C 40 2
Mknod/dev/vote_disk C 40 3

# Ls-L/dev/* _ Disk
CrW-r -- 1 root system 40, 2 Mar 25 :45/dev/ocr_disk
CrW-r -- 1 root system 40, 3 Mar 25 19:45/dev/vote_disk
# Chown ORACLE: oinstall * _ Disk

4-4> perform the dlpar and confirm that the new node can join the dlpar group.
4-5> adjust ha
| |
Add the zhyw2_old memory card and Nic
Add PVID
Chdev-l hdiskpower64-A Pv = Yes
Chdev-l hdiskpower65-A Pv = Yes
Chdev-l hdiskpower64-A reserve_lock = No
Chdev-l hdiskpower65-A reserve_lock = No
Mknod operation
CD/dev/
Ls-L/dev/* hdiskpower64
BRW-RW ---- 1 root system 39, 64 Mar 09/dev/hdiskpower64
CrW-RW ---- 1 root system 39, 64 Mar 09/dev/rhdiskpower64
Mknod/dev/ocr_disk C 39 64 LS-L/dev/* hdiskpower65 BRW-RW ---- 1 root system 39, 65 Mar 09 :12/dev/hdiskpower65
CrW-RW ---- 1 root system 39, 65 Mar 09 :24/dev/rhdiskpower65
Mknod/dev/vote_disk C 39 65
| |

4-3> kill oracle_base and return the backed up oracle_base tar.
Modify. Profile and replace instance-ID with the ID of 2nd nodes.
4-4> restore these files
/Etc/init.css D/etc/init. CRS/etc/init. crsd/etc/init. evmd/etc/rc. d/rc2.d/k96init. CRS
/Etc/rc. d/rc2.d/s96init. CRS/etc/Oracle/scls_scr/etc/Oracle/oprocd/etc/inittab. CRS
/Etc/inittab
/Etc/Oracle/scls_scr/<node name>/Oracle/cssfatal
$ Oracle_crs_home/install/paramfile. CRS
4-5> reconnect to zhyw2_new and try to start CRS
5> stop zhyw1, stop ha, concurrentvg
[Root @ zhyw1] # ls-Al * VG *
CrW-RW ---- 1 root system 10, 0 Mar 02 :03 ipl_rootvg
CrW ------- 1 root system 10, 0 Mar 08 14:08 _ vg10
CrW ------- 1 root system 100, 0 Mar 15 _ vg100
CrW ------- 1 root system 101, 0 Mar 15 _ vg101
CrW ------- 1 root system 102, 0 Mar 16 _ vg102
CrW ------- 1 root system 103, 0 Mar 16 _ vg103
CrW ------- 1 root system 104, 0 Mar 16 _ vg104
CrW ------- 1 root system 105, 0 Mar 16 _ vg105
CrW ------- 1 Root System 106, 0 Mar 16 _ vg106
CrW ------- 1 root system 42, 0 Mar 16 00:12 _ vg42
CrW-RW ---- 1 root system 42, 0 Mar 16 00:11 archvg
CrW-RW ---- 1 root system 100, 0 Mar 15 oravg
CrW-RW ---- 1 root system 101, 0 Mar 15 oravg2
CrW-RW ---- 1 root system 102, 0 Mar 16 oravg3
CrW-RW ---- 1 root system 103, 0 Mar 16 oravg4
CrW-RW ---- 1 root system 104, 0 Mar 16 oravg5
CrW-RW ---- 1 root system 105, 0 Mar 16 oravg6
CrW-RW ---- 1 Root System 106, 0 Mar 16 oravg7
CrW-RW ---- 1 root system 10, 0 Mar 02 rootvg

Importvg-y oravg-V 100 hdiskpower63
Importvg-y oravg2-V 101 hdiskpower59
Importvg-y oravg3-V 102 hdiskpower48
Importvg-y oravg4-V 103 hdiskpower40
Importvg-y oravg5-V 104 hdiskpower32
Importvg-y oravg6-V 105 hdiskpower24
Importvg-y oravg7-V 106 hdiskpower16

6> debug Ha on zhyw1 and start hacmp
7> Start the instance on zhyw2_new