[VIP] solution to abnormal VIP2 transfer in RAC

VIP2 is unavailable, so that node2 does not have a new connection, the load is all on node1, and node1 does not have the Failover function. To make node2 available

VIP2 is unavailable, so that node2 does not have a new connection, the load is all on node1, and node1 does not have the Failover function. To make node2 available

Environment: AIX 5.3 + 10.2.0.5 RAC

Scenario Description: In a set of RAC, VIP2 of node2 can be abnormally moved to node1 and 2 cannot be transferred back to node2. at the same time, no fault is found on node2. Even if the server is restarted, VIP2 cannot be restored to normal.

VIP2 is unavailable, so that node2 does not have a new connection, the load is all on node1, and node1 does not have the Failover function, in order to make node2 available, if the original VIP2 problem cannot be solved, the final solution is: Disable VIP2 on node1 and then add a VIP on node2.

1. View Resource Status

Oracle @ gisdb1:/oracle $ crs_stat-t
Name Type Target State Host
------------------------------------------------------------
Ora... B1.lsnr application ONLINE gisdb1
Ora. gisdb1.gsd application ONLINE gisdb1
Ora. gisdb1.ons application ONLINE gisdb1
Ora. gisdb1.vip application ONLINE gisdb1
Ora... B2.lsnr application ONLINE OFFLINE
Ora. gisdb2.vip application ONLINE OFFLINE
Ora. gisdb2.ons application ONLINE gisdb2
Ora. gisdb2.gsd application ONLINE gisdb2
Ora. nxgis. db application ONLINE gisdb2
Ora... s1.inst application ONLINE gisdb1
Ora... s2.inst application ONLINE gisdb2

At this time, the ora. gisdb2.vip resource cannot be started on node node2. If you use crs_start to start the resource, it is automatically started on node1.
Of course, the resource ora. gisdb2.listener _ gisdb2.lsnr on node node2 is dependent on vip2 and cannot be started.

2. During the problem solving process, I tried to track the Startup Process of vip2 multiple times, but there was still no effective result. Then view the ocr information:
Oracle @ gisdb1> ocrdump
OCRDUMPFILE
Oracle @ gisdb1> vi OCRDUMPFILE
No information about vip2 is found. We know that the information in ocr is messy. (Previously executed: srvctl remove nodeapps-f, where the-f parameter indicates forced execution. This parameter may cause ocr information disorder and is not recommended)

3. When there is a problem with ora. gisdb2.vip information in ocr, you should first re-register ora. gisdb2.vip.
The so-called re-registration is to delete the information about ora. gisdb2.vip in ocr, and then recreate the vip. You can use VIPCA or crs_profile, crs_register, or a series of crs _ * commands to recreate the VIP.

4. Run the following command to remove ora. gisdb2.vip:
Run the following command as the root user:
Root @ gisdb2 :. /srvctl remove nodeapps-n gisdb2 (this command will delete vip, ons, and gsd, so you will need to recreate vip and ons later. gsd can choose not to recreate gsd to maintain backward compatibility)
PRKO-02112: "Some or all node applications are not removed successfully onnode: gisdb2"
Obviously, an error is reported here, And nodeapps on node2 cannot be deleted.
Then, view the crs resources:
Oracle @ gisdb1:/oracle $ crs_stat-t
Name Type Target State Host
------------------------------------------------------------
Ora... B1.lsnr application ONLINE gisdb1
Ora. gisdb1.gsd application ONLINE gisdb1
Ora. gisdb1.ons application ONLINE gisdb1
Ora. gisdb1.vip application ONLINE gisdb1
Ora... B2.lsnr application ONLINE OFFLINE
Ora. gisdb2.vip application ONLINE OFFLINE
Ora. nxgis. db application ONLINE gisdb2
Ora... s1.inst application ONLINE gisdb1
Ora... s2.inst application ONLINE gisdb2
We can see that ora. gisdb2.vip still exists, but ora. gisdb2.ong and ora. gisdb2.gsd resources are unavailable.

5. To remove ora. gisdb2.vip from ocr and use crs_unregister, this command deletes the information of the corresponding resource from the ocr file.
Run the following command as the root user:
Root @ gisdb2: // #./crs_unresgiter ora. gisdb2.vip
Cann't unregister 'ora. gisdb2.vip 'because it is required by other resources.
CRS-0214: cocould not unregister resource 'ora. gisdb2.vip'
An error is reported again because ora. gisdb2.vip cannot be registered because it is associated with other resources. To cancel vip settings, you may need to register lsnr and database (not tested)
In short, we failed. At this time, the problem is in a dilemma. If the command deleted by VIP2 fails, we cannot update the VIP2 information in OCR. VIP2 is unavailable on node2,
RAC becomes a single node and loses high availability.