Oracle RAC migration under VMware causes CRS-1006/CRS-0215/CRS-0233

Recently, the Oracle 10g RAC migration under the virtual machine, after the migration is completed, the Oracle cluster resource VIP cannot start normally, received the CRS-0233: resource or relatives are currently involved with another operation error prompt. Why? It turns out that the migrated address has changed. You have to use the new address in your house ....

1. Environment Description Oracle 10g RAC + SuSE 10 note: After the RAC virtual machine is migrated, we usually choose to copy it when adding the Virtual Machine (move this one before, I don't know if there is any problem.) Secondly, because of the selection of replication, the virtual machine needs to generate a new UUID (UUID refers to the number generated on a machine, it guarantees the uniqueness of machines in the same virtual environment ). At the same time, the MAC address and network interface name will also change accordingly (the original eth0 and eth1 of the virtual machine are not available for the first time), which usually needs to be modified. Different Linux systems have different processing methods for new network interfaces. Oracle Linux and RedHat can directly Delete the original network interface and rename the new network interface name to the original one. Suselinux is a little troublesome. For details, refer to: slave # modify the NIC and restart two nodes. # The resource VIP status is offline Oracle @ bo2dbp: ~>. /Crs_stat.sh | grep bo2dbp Resource Name target State -------------- ------ ----- ora. bo2dbp. asm1.asm online on bo2dbp ora. bo2dbp. listener_bo2dbp.lsnr online offline ora. bo2dbp. listener_ora10g_bo2dbp.lsnr online offline ora. bo2dbp. GSD online on bo2dbp ora. bo2dbp. ONS online offline ora. bo2dbp. VIP online offline ora. ora10g. DB online on bo2dbp ora. ora10g. ora10g1. inst online onlin E on bo2dbp # Try to manually start ons Oracle @ bo2dbp: ~> Crs_start ora. bo2dbp. ONS attempting to start' ora. bo2dbp. ONS 'on member 'bo2dbp 'start of 'ora. bo2dbp. ONS 'on member 'bo2dbp' failed. CRS-1006: No more Members to consider CRS-0215: cocould not start resource 'ora. bo2dbp. ONS '. # onsctl-based startup also receives failure messages. Oracle @ bo2dbp: ~> Onsctl start Number of onsconfiguration retrieved, numcfg = 2 onscfg [0] {node = bo2dbp.2gotrade.com, Port = 6200} adding remote host failed: 6200 onscfg [1] {node = bo2dbs.2gotrade.com, port = 6200} adding remote host bo2dbs.2gotrade.com: 6200 Number of onsconfiguration retrieved, numcfg = 2 onscfg [0] {node = bo2dbp.2gotrade.com, Port = 6200} adding remote host failed: 6200 o Nscfg [1] {node = bo2dbs.2gotrade.com, Port = 6200} adding remote host bo2dbs.2gotrade.com: 6200 onsctl: ONS failed to start # Try to manually start the VIP and receive the CRS-0233 error Oracle @ bo2dbp: ~> Crs_start ora. bo2dbp. VIP CRS-0233: resource or relatives are currently involved with another operation.3. Analyze the fault # view the IP address on the node bo2dbp host Oracle @ bo2dbp: ~> Ifconfig # The Network Interface Name of the current system is eth2, eth5 eth2 link encap: Ethernet hwaddr 00: 0C: 29: 4A: 66: 28 Inet ADDR: 192.168.7.51 bcast: 192.168.7.255 mask: 255.255.255.0 eth5 link encap: Ethernet hwaddr 00: 0C: 29: 4A: 66: 32 Inet ADDR: 10.10.7.51 bcast: 10.10.7.255 mask: 255.255.255.0 # view the cluster network layer configuration on the node bo2dbp, the network interface name is the same as the actual Nic name. Oracle @ bo2dbp: ~> Oifcfg iflist eth2 192.168.7.0 eth5 10.10.7.0 Oracle @ bo2dbp: ~> Oifcfg getif-Global # The Network Interface Name is inconsistent with the actual Nic name. eth3 192.168.7.0 global public eth4 10.10.7.0 global cluster_interconnect # view the IP address on the node bo2dbs host Oracle @ bo2dbs: ~> Ifconfig eth5 link encap: Ethernet hwaddr 00: 0C: 29: 27: 43: EB Inet ADDR: 10.10.7.52 bcast: 10.10.7.255 mask: 255.255.255.0 eth6 link encap: Ethernet hwaddr 00: 0C: 29: 27: 43: E1 Inet ADDR: 192.168.7.52 bcast: 192.168.7.255 mask: 255.255.255.255.0 # view the cluster network layer configuration on the node bo2dbp. The Network Interface Name is consistent with the actual network adapter name. Oracle @ bo2dbs: ~> Oifcfg iflist eth6 192.168.7.0 eth5 10.10.7.0 Oracle @ bo2dbs: ~> Oifcfg getif-Global # The Network Interface Name is also found to be inconsistent with the actual Nic name. It should be the original network interface name eth3 192.168.7.0 global public eth4 10.10.7.0 global cluster_interconnect # From the above situation, the network interface names of each node are inconsistent, and the configuration information of the network cluster layer uses the original interface name, which should be updated # to unify the network interface Name, the following uses the unified name bond1, bond24, solve the fault # rename the NIC, the method reference: http://blog.csdn.net/robinson_0612/article/details/8131771 # The following is the result of the rename Oracle @ bo2dbp: ~> Oifcfg iflist bond1 192.168.7.0 bond2 10.10.7.0 Oracle @ bo2dbs: ~> Oifcfg iflist bond1 192.168.7.0 bond2 10.10.7.0 # In the following query, the public and cluster_interconnect of the cluster layer are still the previous configuration information # It should be changed to the same one. We will not change it here, see what will happen Oracle @ bo2dbp: ~> Oifcfg getif-Global eth3 192.168.7.0 global public eth4 10.10.7.0 global cluster_interconnect Oracle @ bo2dbs: ~> Oifcfg getif-Global eth3 192.168.7.0 global public eth4 10.10.7.0 global cluster_interconnect # restart CRS Oracle @ bo2dbp: ~> Sudo-S/u01/Oracle/CRS/bin/crsctl start CRS root ''s password: attempting to start CRS stack the CRS stack will be started shortly # The following query indicates that the CRS background process is normal Oracle @ bo2dbp: ~> Crsctl check crs css appears healthy CRS appears healthy EVM appears healthy # The following query results are still the same as before:. /Crs_stat.sh | grep bo2dbp Resource Name target State -------------- ------ ----- ora. bo2dbp. asm1.asm online on bo2dbp ora. bo2dbp. listener_bo2dbp.lsnr online offline ora. bo2dbp. listener_ora10g_bo2dbp.lsnr online offline ora. bo2dbp. GSD online on bo2dbp ora. bo2dbp. ONS online offline ora. bo2dbp. VIP online offline ora. ora10g. DB online on bo2dbp ora. ora10g. ora10g1. inst online onlin E on bo2dbp # Shut down all resources. Oracle @ bo2dbp: ~> Crs_stop-all # Use oifcfg to modify the cluster layer network configuration Oracle @ bo2dbp: ~> Oifcfg delif-Global Oracle @ bo2dbp: ~> Oifcfg getif-Global Oracle @ bo2dbp: ~> Oifcfg setif-Global bond1/192.168.7.0: Public Oracle @ bo2dbp: ~> Oifcfg setif-Global bond2/10.10.7.0: cluster_interconnect Oracle @ bo2dbp: ~> Oifcfg getif-Global bond1 192.168.7.0 global public bond2 10.10.7.0 global cluster_interconnect # After reboot, the resource status remains the same as before # Check the VIP log information first. Let's first solve the VIP problem bo2dbp: /u01/Oracle/CRS/log/bo2dbp/racg # tail-50 ora. bo2dbp. VIP. log 11:25:13. 783: [racg] [2151948784] [16581] [2151948784] [ora. bo2dbp. VIP]: clsrcexecut: ENV oracle_config_home =/u01/Oracle/CRS 11:25:13. 783: [racg] [2151948784] [16581] [21 51948784] [ora. bo2dbp. VIP]: clsrcexecut: cmd =/u01/Oracle/CRS/bin/racgeut-E _ usr_ora_debug = 0 54/u01/Oracle/CRS/bin/racgvip start bo2dbp 11:25:13. 783: [racg] [2151948784] [16581] [2151948784] [ora. bo2dbp. VIP]: clsrcexecut: rc = 1, time = 3.220 s 11:25:16. 979: [racg] [2151948784] [16581] [2151948784] [ora. bo2dbp. VIP]: clsrcexecut: ENV oracle_config_home =/u01/Oracle/CRS 2012-12-28 11: 25: 16.979: [racg] [2151948784] [16581] [2151948784] [ora. bo2dbp. VIP]: clsrcexecut: cmd =/u01/Oracle/CRS/bin/racgeut-E _ usr_ora_debug = 0 54/u01/Oracle/CRS/bin/racgvip check bo2dbp 11:25:16. 979: [racg] [2151948784] [16581] [2151948784] [ora. bo2dbp. VIP]: clsrcexecut: rc = 1, time = 3.190 s 11:25:16. 979: [racg] [2151948784] [16581] [2151948784] [ora. bo2dbp. VIP]: End for resource = ora. B O2dbp. VIP, Action = start, status = 1, time = 6.430 s 11:25:23. 807: [racg] [695611888] [17488] [695611888] [ora. bo2dbp. VIP]: eth3: Error fetching Interface Information: device not found # the error message eth3 is not found. We want to use bond1 checkif: interface eth3 is down invalid parameters, or failed to bring up VIP (host = bo2dbp) 11:25:23. 807: [racg] [695611888] [17488] [695611888] [ora. bo2dbp. VIP]: clsrcexe Cut: ENV oracle_config_home =/u01/Oracle/CRS 11:25:23. 807: [racg] [695611888] [17488] [695611888] [ora. bo2dbp. VIP]: clsrcexecut: cmd =/u01/Oracle/CRS/bin/racgeut-E _ usr_ora_debug = 0 54/u01/Oracle/CRS/bin/racgvip start bo2dbp 11:25:23. 807: [racg] [695611888] [17488] [695611888] [ora. bo2dbp. VIP]: clsrcexecut: rc = 1, time = 3.220 s 11:25:27. 018: [racg] [695611888] [17488] [6 95611888] [ora. bo2dbp. VIP]: clsrcexecut: ENV oracle_config_home =/u01/Oracle/CRS 11:25:27. 018: [racg] [695611888] [17488] [695611888] [ora. bo2dbp. VIP]: clsrcexecut: cmd =/u01/Oracle/CRS/bin/racgeut-E _ usr_ora_debug = 0 54/u01/Oracle/CRS/bin/racgvip check bo2dbp 11:25:27. 018: [racg] [695611888] [17488] [695611888] [ora. bo2dbp. VIP]: clsrcexecut: rc = 1, time = 3.210 s 2012-12-28 11: 25: 27.018: [racg] [695611888] [17488] [695611888] [ora. bo2dbp. VIP]: End for resource = ora. bo2dbp. VIP, Action = start, status = 1, time = 6.450 s 11:25:33. 822: [racg] [3600347632] [18308] [3600347632] [ora. bo2dbp. VIP]: eth3: Error fetching Interface Information: device not found # the error message "checkif: interface eth3 is down invalid Parameters" is displayed again, or failed to bring up VIP (host = bo2dbp) 2012-12- 28 11:25:33. 822: [racg] [3600347632] [18308] [3600347632] [ora. bo2dbp. VIP]: clsrcexecut: ENV oracle_config_home =/u01/Oracle/CRS 11:25:33. 822: [racg] [3600347632] [18308] [3600347632] [ora. bo2dbp. VIP]: clsrcexecut: cmd =/u01/Oracle/CRS/bin/racgeut-E _ usr_ora_debug = 0 54/u01/Oracle/CRS/bin/racgvip start bo2dbp 11:25:33. 822: [racg] [3600347632] [18308] [3600347632] [ora. bo2dbp. VIP]: CL Srcexecut: rc = 1, time = 3.210 s 11:25:37. 063: [racg] [3600347632] [18308] [3600347632] [ora. bo2dbp. VIP]: clsrcexecut: ENV oracle_config_home =/u01/Oracle/CRS 11:25:37. 063: [racg] [3600347632] [18308] [3600347632] [ora. bo2dbp. VIP]: clsrcexecut: cmd =/u01/Oracle/CRS/bin/racgeut-E _ usr_ora_debug = 0 54/u01/Oracle/CRS/bin/racgvip check bo2dbp 11:25:37. 063: [racg] [1, 3600347632] [18308] [3600347632] [ora. bo2dbp. VIP]: clsrcexecut: rc = 1, time = 3.240 s 11:25:37. 063: [racg] [3600347632] [18308] [3600347632] [ora. bo2dbp. VIP]: End for resource = ora. bo2dbp. VIP, Action = start, status = 1, time = 6.490 s # From the log above, we can see that the network interface name is still a problem # You can also see that action = start, status = 1, time = 6.490 s this place should be where the target is online, and in fact the State is offline # the network interface layer has also changed, that is, this eth3 is not updated in OCR, next, update bo2dbp:/u01/Oracle/C. RS/bin #./srvctl modify nodeapps-N bo2dbp-A 192.168.7.61/255.255.255.0/bond1 # use the same method to update bo2dbs on the second node :~ #/U01/Oracle/CRS/bin/srvctl modify nodeapps-N bo2dbs-A 192.168.7.62/255.255.255.0/bond1 # VIP Oracle @ bo2dbp is successfully started at this time: ~> Crs_start ora. bo2dbp. VIP attempting to start' ora. bo2dbp. VIP 'on member 'bo2dbp 'start of 'ora. bo2dbp. VIP 'on member 'bo2dbp 'succeeded. # view ons log information: Oracle @ bo2dbp:/u01/Oracle/CRS/log/bo2dbp/racg> tail-20 ora. bo2dbp. ONS. log ............ onscfg [0] {node = bo2dbp.2gotrade.com, Port = 6200} adding remote host bo2dbp.2gotrade.com: 6200 onscfg [1] {node = bo2dbs.2gotrade.com, Port = 6200} adding remot E HOST bo2dbs.2gotrade.com: 6200 ONS is n 11:00:49. 345: [racg] [2554102256] [19071] [2554102256] [ora. bo2dbp. ONS]: ot running... 11:00:49. 345: [racg] [2554102256] [19071] [2554102256] [ora. bo2dbp. ONS]: clsrcexecut: ENV oracle_config_home =/u01/Oracle/CRS 11:00:49. 345: [racg] [2554102256] [19071] [2554102256] [ora. bo2dbp. ONS]: clsrcexecut: cmd =/u01/Oracle/CRS/bin/racgeut -E _ usr_ora_debug = 0 540/u01/Oracle/CRS/bin/onsctl Ping 11:00:49. 345: [racg] [2554102256] [19071] [2554102256] [ora. bo2dbp. ONS]: clsrcexecut: rc = 1, time = 0.210 s 11:00:49. 346: [racg] [2554102256] [19071] [2554102256] [ora. bo2dbp. ONS]: End for resource = ora. bo2dbp. ONS, Action = start, status = 1, time = 7.560 s 11:00:55. 661: [racg] [368746992] [19812] [368746992] [ora. Bo2dbp. ONS]: onsctl: Shutting Down ons daemon... connect: Connection refused Number of onsconfiguration retrieved, numcfg = 2 onscfg [0] {node = success, Port = 6200} adding remote host bo2dbp.2gotrade.com: 6200 onscfg [1] {node = bo2dbs.2gotrade.com, 11:00:55. 661: [racg] [368746992] [19812] [368746992] [ora. bo2dbp. ONS]: Port = 6200 }............... # For This ons error, remember that there was a similar situation before, which was previously installed This is because there is no local loop. # This problem occurs again, because the original configuration copy/etc/hosts was accidentally commented out, Khan ...... # refer to the http://blog.csdn.net/robinson_0612/article/details/6303583 # Try to start ons successfully Oracle @ bo2dbp: ~> Crs_start ora. bo2dbp. ONS attempting to start' ora. bo2dbp. ONS 'on member 'bo2dbp 'start of 'ora. bo2dbp. ONS 'on member 'bo2dbp 'succeeded. # Author: Robinson # blog: http://blog.csdn.net/?son_06125] summary A. After migrating a virtual machine in the RAC environment, you must first change the path of all disks in the configuration files of each Virtual Machine (Local disk, asmdisk, OCR, votingdisk) B. When the VM is added, the network adapter changes when the replication mode is used (the Mobile mode is uncertain). This mainly ensures that the MAC address is unique. C. You need to reconfigure the network, if you want to use the original network interface name, rename or modify the configuration file in the command line to retain the original network interface name D. If you use the X Window Method to modify the network configuration, check whether the hosts file is modified accordingly. E. If a new network interface name or IP address is used, the cluster network layer F should be reconfigured, And the OCR g and the last sentence of the new network interface name or the new IP Address should also be updated. It is the source to analyze and solve the problem from the log, is the best way to quickly locate Problems

More references

For more information about Oracle RAC, see
Use crs_setperm to modify the resource owner and permissions of RAC.
Use crs_profile to manage RAC resource configuration files
RAC database startup and Shutdown
Oracle RAC services
Services in Oracle Database 10g
Migrate datbase from single instance to Oracle RAC
Connect Oracle RAC to a specified instance
Oracle RAC load balancing test (combined with server and client)
Oracle RAC server connection Load Balance)
Load Balance)
Non-Default port listening configuration in Oracle RAC (listener. ora tnsnames. ora)
Oracle RAC Listener Configuration (listener. ora tnsnames. ora)
Configure RAC load balancing and Failover
CRS-1006, CRS-0215 fault case
Installing Oracle 10g RAC Based on Linux (RHEL 5.5)
Use runcluvfy to verify the Oracle RAC installation environment

For more information about the basics and concepts of Oracle network configuration, see:
Configure dynamic service registration for non-default ports
Configure sqlnet. ora to restrict IP Access to Oracle
Configure and manage Oracle listener logs
Set the Oracle listener password (listener)
Configure the Oracle client to connect to the database

For more information about user-managed backup and recovery, see
Oracle cold backup
Oracle Hot Backup
Concept of Oracle backup recovery
Oracle instance recovery
Oracle recovery based on user management
System tablespace management and Backup Recovery
Sysaux tablespace management and recovery
Oracle backup control file recovery (unsing backup controlfile)

For information on RMAN backup recovery and management, see
RMAN overview and architecture
RMAN configuration, Monitoring and Management
Detailed description of RMAN backup
RMAN restoration and recovery
Create and use RMAN catalog
Create RMAN storage script based on catalog
Catalog-based RMAN backup and recovery
RMAN backup path confusion
Use RMAN for recovery from different machine backups (WIN platform)
Use RMAN to migrate a file system database to ASM
Linux RMAN backup shell script
Use RMAN to migrate the database to a different machine

For the Oracle architecture, see
Oracle tablespace and data files
Oracle Password File
Oracle parameter file
Oracle online redo log file)
Oracle Control File)
Oracle archiving logs
Oracle rollback and undo)
Oracle database instance startup and Shutdown Process
Automated Management of Oracle 10g SGA
Oracle instances and Oracle databases (Oracle Architecture)