VIP cannot be started properly, error CRS-1006

VIP cannot be started normally
Description: Our environment is 2 node RAC, and node 1 is down due to physical faults.
In this case, I want to start the VIP of Node 1 from node 2 so that the single node is transparent to the user program.

[Oracle @ UNID02 ~] $ Crs_start ora. unid01.vip
Attempting to start 'ora. unid01.vip 'on member 'unid02'
Start of 'ora. unid01.vip 'on member 'unid02' failed.
CRS-1006: No more members to consider

CRS-0215: cocould not start resource 'ora. unid01.vip '.

[Oracle @ UNID02 ~] $
But the error CRS-1006: No more members to consider is reported at startup.
Check the VIP log (at $ CRS_HOME/log/<NODENAME>/racg) and report Nic errors:
09:50:26. 877: [RACG] [2345793472] [16627] [2345793472] [ora. unid01.vip]: checkIf: interface eth0 is down
Invalid parameters, or failed to bring up VIP (host = UNID02) ==================================>

09:50:26. 877: [RACG] [2345793472] [16627] [2345793472] [ora. unid01.vip]: clsrcexecut: cmd =/oracle/app/11gR1/crs/bin/racgeut-e _ USR_ORA_DEBUG = 0 54/oracle/app/11gR1/crs/bin/racgvip start unid01

09:50:26. 877: [RACG] [2345793472] [16627] [2345793472] [ora. unid01.vip]: clsrcexecut: rc = 1, time = 3.130 s

09:50:30. 010: [RACG] [2345793472] [16627] [2345793472] [ora. unid01.vip]: clsrcexecut: cmd =/oracle/app/11gR1/crs/bin/racgeut-e _ USR_ORA_DEBUG = 0 54/oracle/app/11gR1/crs/bin/racgvip check unid01

09:50:30. 010: [RACG] [2345793472] [16627] [2345793472] [ora. unid01.vip]: clsrcexecut: rc = 1, time = 3.130 s

09:50:30. 010: [RACG] [2345793472] [16627] [2345793472] [ora. unid01.vip]: end for resource = ora. unid01.vip, action = start, status = 1, time = 6.280 s

013-12-10 01:17:41. 966: [COMMCRS] [1472985408] clsc_receive: (0x2aaaac1428c0) error 2

09:50:23. 702: [CRSRES] [1538058560] startRunnable: setting CLI values
09:50:23. 705: [CRSRES] [1538058560] Attempting to start 'ora. unid01.vip 'on member 'unid02'
09:50:30. 012: [CRSAPP] [1538058560] StartResource error for ora. unid01.vip error code = 1
09:50:33. 198: [CRSRES] [1538058560] Start of 'ora. unid01.vip 'on member 'unid02' failed.
09:50:33. 204: [CRSRES] [1538058560] CRS-1006: No more members to consider

Through srvctl view found that the UNID02-vip bound network card is eth2, And the unid01-vip bound network card is eth0.
[Oracle @ UNID02 ~] $ Srvctl config nodeapps-n UNID02-a-g-s-l
VIP exists.:/UNID02-vip/10.0.15.176/255.255.255.0/eth2
GSD exists.
ONS daemon exists.
Listener exists.
[Oracle @ UNID02 ~] $ Srvctl config nodeapps-n unid01-a-g-s-l
VIP exists.:/unid01-vip/10.0.15.175/255.255.255.0/eth0
GSD exists.
ONS daemon exists.
Listener exists.

Ifconfig check found that eth0 is not enabled
[Oracle @ UNID02 ~] $

[Root @ UNID02 bin] # ifconfig
Eth1 Link encap: Ethernet HWaddr A4: BA: DB: 13: EA: C2
Inet addr: 192.168.127.102 Bcast: 192.168.127.255 Mask: 255.255.255.0
Inet6 addr: fe80: a6ba: dbff: fe13: eac2/64 Scope: Link
Up broadcast multicast mtu: 1500 Metric: 1
RX packets: 53 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 43 errors: 0 dropped: 0 overruns: 0 carrier: 0
Collisions: 0 FIG: 1000
RX bytes: 8246 (8.0 KiB) TX bytes: 6848 (6.6 KiB)
Interrupt: 122 Memory: d8000000-d8012800

Eth2 Link encap: Ethernet HWaddr A4: BA: DB: 13: EA: C4
Inet addr: 10.0.15.172 Bcast: 10.0.15.255 Mask: 255.255.255.0
Inet6 addr: fe80: a6ba: dbff: fe13: eac4/64 Scope: Link
Up broadcast running multicast mtu: 1500 Metric: 1
RX packets: 5778770 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 2798242 errors: 0 dropped: 0 overruns: 0 carrier: 0
Collisions: 0 FIG: 1000
RX bytes: 1493987596 (1.3 GiB) TX bytes: 1004608379 (958.0 MiB)
Interrupt: 130 Memory: da000000-da012800

Eth2: 1 Link encap: Ethernet HWaddr A4: BA: DB: 13: EA: C4
Inet addr: 10.0.15.176 Bcast: 10.0.15.255 Mask: 255.255.255.0
Up broadcast running multicast mtu: 1500 Metric: 1
Interrupt: 130 Memory: da000000-da012800
Lo Link encap: Local Loopback
Inet addr: 127.0.0.1 Mask: 255.0.0.0
Inet6 addr: 1/128 Scope: Host
Up loopback running mtu: 16436 Metric: 1
RX packets: 921339 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 921339 errors: 0 dropped: 0 overruns: 0 carrier: 0
Collisions: 0 txqueuelen: 0
RX bytes: 417953992 (398.5 MiB) TX bytes: 417953992 (398.5 MiB)

[Root @ UNID02 bin] #

Consult the system engineer and tell the machine that the Public IP Address used the eth0 Nic. Later, the eth0 Nic failed and switched to the eth2 Nic.
There are two solutions:
1. Change unid01-vip to eth2
[Root @ UNID02 ~] $ Srvctl modify nodeapps-n unid01-A 10.0.15.175/255.255.255.0/eth2
Start again and start successfully.
[Oracle @ UNID02 ~] $ Crs_start ora. unid01.vip
Attempting to start 'ora. unid01.vip 'on member 'unid02'
Start of 'ora. unid01.vip 'on member 'unid02' succeeded.

2. Because crs_start will call the racgvip script to start the vip, you can directly modify the environment variable and then directly execute sh racgvip start ora. unid01.vip.
[Root @ UNID02 ~] # Export _ USR_ORA_VIP = 10.0.15.175
[Root @ UNID02 ~] # Export _ USR_ORA_NETMASK = 255.255.255.0
[Root @ UNID02 ~] # Export _ USR_ORA_IF = eth2
[Root @ UNID02 ~] # Export _ CAA_NAME = ora. unid01.vip
[Root @ UNID02 bin] # sh-x racgvip start ora. unid01.vip
+ IFCONFIG =/sbin/ifconfig
+ GREP =/bin/grep
+ SED =/bin/sed
+ RM =/bin/rm
+ MV =/bin/mv
+ UNIQ =/usr/bin/uniq
+ PING =/bin/ping
+ WC =/usr/bin/wc
+ NETSTAT =/bin/netstat
+ AWK =/bin/awk
+ WHOAMI =/usr/bin/whoami
+ CAT =/bin/cat
+ UNAME =/bin/uname
+ SLEEP =/bin/sleep
+ SORT =/bin/sort
+ EXPR =/usr/bin/expr
+ DATE =/bin/date
+ RENICE =/usr/bin/renice
+ MIITOOL =/sbin/mii-tool
+ ARPING =/sbin/arping
+ IPCMD = '/sbin/ip-f inet'
+ LANG = C
+ LC_ALL = C
+ Export LANG LC_ALL
+ FAIL_WHEN_ALL_LINK_DOWN = 1
+ FAIL_WHEN_DEFAULTGW_NOT_FOUND = 1
+ DEFAULTGW =
+/Usr/bin/renice-20-p 15145
++/Bin/hostname
+ HOSTNAME = UNID02
+ PING_TIMEOUT = '-w 3-c 1'
+ PING_COUNT = 10
+ LOCKED = 0
+ CRS_STAT =/bin/crs_stat
+ CHECK_TIMES = 2
+ SUCCESS = 0
+ ERROR = 1
+ DEFAULT_TIMEOUT = 60
+ IP = 10.0.15.175
+ MASK = 255.255.255.0
+ IF = eth2
+ OP = start
++/Usr/bin/whoami
+ USER = root
++ Uname
+ [[Linux! = Linux]
+ Listif_result =
+ '['Root '! = 'Root-a start '! = 'LIST']'
+ '['-N 10.0.15.175-a-n 255.255.255.0 ']'
++ IFS =.
++ Set 10 0 15 175 255 255 255 0
++ Echo 10.0.15.255
+ BROADCAST = 10.0.15.255
+ Logx 'broadcast = 10.0.15.255'
+ '['-N']'
+ '['Start = list']'
++ Echo ora. unid01.vip
+ +/Bin/sed '-es/^ ora \. //; s/\. vip $ //'
+ VIP_NAME = unid01
+ NAME = ora. unid01.vip
+ '['-Z ora. unid01.vip ']'
+ IF_USING =
+ '['-N 10.0.15.175 ']'
+ Logx Checking interface existance
+ '['-N']'
+ Logx 'calling getifbyip'
+ '['-N']'
++ Getifbyip 10.0.15.175
++ _ LOCAL_IP = 10.0.15.175
++ Gf_retif =
+ Logx 'getifbyip: started for 10.0.15.175'
+ '['-N']'
++/Sbin/ip-f inet-o addr
++/Bin/grep 'inet 10.0.15.175 /'
++/Bin/awk '{print $ NF }'
++ Gf_retif =
+ Logx 'getifbyip: returning ip'
+ '['-N']'
+ '['-Z''']'
+ LI =
+ Logx Completed getifbyip
+ '['-N']'
+ Logx 'calling getifbyip-'
+ '['-N']'
++ Getifbyip 10.0.15.175-
++ _ LOCAL_IP = 10.0.15.175
++ Gf_retif =
+ Logx 'getifbyip: started for 10.0.15.175'
+ '['-N']'
++/Sbin/ip-f inet-o addr
++/Bin/grep 'inet 10.0.15.175 /'
++/Bin/awk '{print $ NF }'
++ Gf_retif =
+ Logx 'getifbyip: returning ip'
+ '['-N']'
+ '['-Z-a']'
+ '['-N']'
+ LI_A =
+ Logx Completed getifbyip
+ '['-N']'
+ '[''''! = ''']'
+ Echo''
+/Bin/grep-q:
+ '['1-ne 0']'
+ '['Start = stop']'
+ Ping_vip 10.0.15.175
+ Logx 'Ping _ vip 10.0.15.175 started'
+ '['-N']'
+ '['-N 10.0.15.175 ']'
+ _ Count = 1
+ '['1-le 10']'
+/Bin/ping 10.0.15.175-w 3-c 1
+ '['1-ne 0']'
+ Logx 'Ping _ vip: 10.0.15.175 is not pingable, _ count = 1'
+ '['-N']'
+ Return 1
+ '['1-eq 0']'
+ Logx 'completed with initial interface Test'
+ '['-N']'
+ Case $ OP in
+ '['Start = check']'
+ '['Start = check']'
+ '['-N 10.0.15.175-a-n Limit 255.255.0-a-n eth2 ']'
+ '['-N']'
+ Logx 'interface exists'
+ '['-N']'
++ Echo eth2
+ +/Bin/sed '-es/| // G'
+ IF = eth2
+ For I in '$ if'
+ '['Eth2 = ''']'
+ CheckIf eth2
+ _ IF = eth2
+ _ RET = 0
+ _ LINK_STAT =
+ Logx 'checkif: start for if = eth2'
+ '['-N']'
+ '['-Z eth2 ']'
+/Sbin/ifconfig eth2
+/Bin/grep-q-w UP
+ '['0-ne 0']'
+ '['-X/sbin/mii-tool']'
++/Sbin/mii-tool eth2
+ _ LINK_STAT = 'eth2: negotiated 100baseTx-FD flow-control, link OK'
+ '['0-eq 0']'
+ Echo 'eth2: negotiated 100baseTx-FD flow-control, link OK'
+/Bin/grep-q 'link OK'
+ '['0-eq 0']'
+ Logx 'checkif: mii-tool checked if = eth2 OK'
+ '['-N']'
+ _ RET = 0
+ '['-Z' eth2: negotiated 100baseTx-FD flow-control, link OK '']'
+ '['0-eq 1']'
+ Logx 'checkif: end for if = eth2'
+ '['-N']'
+ Return 0
+ '['0-eq 0']'
+ Getnextli eth2
+ _ LOCAL_IF = eth2
+ Nextli =
+ _ LIN =
+ Logx 'getnextli: started for if = eth2'
+ '['-N']'
++ Listif
+ Logx 'listif: starting'
+ '['-N']'
+ '['-Z''']'
++/Sbin/ip-f inet-o addr
++/Bin/grep eth2:
+ +/Bin/sed '-es/^ .*://'
++/Bin/awk '{print $ NF }'
++/Bin/sort-n
++/Bin/grep-vw lo
++ Listif_result = 'eth1'
Eth2
Eth2: 1'
+ Logx 'listif: completed with eth1
Eth2
Eth2: 1'
+ '['-N']'
+ Echo 'eth1'
Eth2
Eth2: 1'
+ _ LIN = 1
+ I = 1
+ '['1-le 256']'
+ _ Found = 0
+ For j in '$ {_ LIN }'
+ '['1-eq 0']'
+ '['1-eq 1']'
+ _ Found = 1
+ Break
+ '['1-eq 0']'
+ I = 2
+ '['2-le 256']'
+ _ Found = 0
+ For j in '$ {_ LIN }'
+ '['1-eq 0']'
+ '['2-eq 1']'
+ '['0-eq 0']'
+ Get_lock eth2_2
+ TOUCH =/bin/touch
+ LS =/bin/ls
+ KILL =/bin/kill
+ LOCK =/var/tmp/vip_eth2_2_UNID02.lock
+/Bin/touch/var/tmp/vip_eth2_2_UNID02.lock.15145
+ '['0-ne 0']'
++/Bin/ls/var/tmp/vip_eth2_2_UNID02.lock.15145
++/Usr/bin/wc-l
+ '['1-eq 1']'
+ Logx 'get _ lock: lock file/var/tmp/vip_eth2_2_UNID02.lock.15145 is created'
+ '['-N']'
+ LOCKED = 1
+ Return 0
+ '['0-eq 0']'
+ Listif_result =
+ Listif
+ Logx 'listif: starting'
+ '['-N']'
+ '['-Z''']'
++/Sbin/ip-f inet-o addr
+/Bin/grep-w eth2: 2
+ +/Bin/awk '{print $ NF }'
++/Bin/grep-vw lo
+ Listif_result = 'eth1'
Eth2
Eth2: 1'
+ Logx 'listif: completed with eth1
Eth2
Eth2: 1'
+ '['-N']'
+ Echo 'eth1'
Eth2
Eth2: 1'
+ '['1-ne 0']'
+ Break
+ '['2-eq 256']'
+ Nextli = eth2: 2
+ Logx 'getnextli: completed with nextli = eth2: 2'
+ '['-N']'
+ Return 2
+ LI = eth2: 2
+/Sbin/ifconfig eth2: 2 10.0.15.175 netmask 255.255.255.0 broadcast 10.0.15.255 up
+ '['0-ne 0']'
+ Logx 'success exit 1'
+ '['-N']'
+ '['-N']'
+/Sbin/arping-q-U-c 3-I eth2 10.0.15.175
+ Release_lock
+ '['1 = 1']'
+/Bin/rm-f/var/tmp/vip_eth2_2_UNID02.lock.15145
+ Logx 'release _ lock: remove lock file/var/tmp/vip_eth2_2_UNID02.lock.15145'
+ '['-N']'
+ LOCKED = 0
+ Exit 0 -- the return value is 0, and start is successful.

[Root @ UNID02 bin] # ifconfig
Eth1 Link encap: Ethernet HWaddr A4: BA: DB: 13: EA: C2
Inet addr: 192.168.127.102 Bcast: 192.168.127.255 Mask: 255.255.255.0
Inet6 addr: fe80: a6ba: dbff: fe13: eac2/64 Scope: Link
Up broadcast multicast mtu: 1500 Metric: 1
RX packets: 53 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 43 errors: 0 dropped: 0 overruns: 0 carrier: 0
Collisions: 0 FIG: 1000
RX bytes: 8246 (8.0 KiB) TX bytes: 6848 (6.6 KiB)
Interrupt: 122 Memory: d8000000-d8012800

Eth2 Link encap: Ethernet HWaddr A4: BA: DB: 13: EA: C4
Inet addr: 10.0.15.172 Bcast: 10.0.15.255 Mask: 255.255.255.0
Inet6 addr: fe80: a6ba: dbff: fe13: eac4/64 Scope: Link
Up broadcast running multicast mtu: 1500 Metric: 1
RX packets: 5778770 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 2798242 errors: 0 dropped: 0 overruns: 0 carrier: 0
Collisions: 0 FIG: 1000
RX bytes: 1493987596 (1.3 GiB) TX bytes: 1004608379 (958.0 MiB)
Interrupt: 130 Memory: da000000-da012800

Eth2: 1 Link encap: Ethernet HWaddr A4: BA: DB: 13: EA: C4
Inet addr: 10.0.15.176 Bcast: 10.0.15.255 Mask: 255.255.255.0
Up broadcast running multicast mtu: 1500 Metric: 1
Interrupt: 130 Memory: da000000-da012800

Eth2: 2 Link encap: Ethernet HWaddr A4: BA: DB: 13: EA: C4
Inet addr: 10.0.15.175 Bcast: 10.0.15.255 Mask: 255.255.255.0
Up broadcast running multicast mtu: 1500 Metric: 1
Interrupt: 130 Memory: da000000-da012800

Lo Link encap: Local Loopback
Inet addr: 127.0.0.1 Mask: 255.0.0.0
Inet6 addr: 1/128 Scope: Host
Up loopback running mtu: 16436 Metric: 1
RX packets: 921339 errors: 0 dropped: 0 overruns: 0 frame: 0
TX packets: 921339 errors: 0 dropped: 0 overruns: 0 carrier: 0
Collisions: 0 txqueuelen: 0
RX bytes: 417953992 (398.5 MiB) TX bytes: 417953992 (398.5 MiB)

[Root @ UNID02 bin] #

Ifconfig: Check that the vip has been started successfully.

Return the outputs if still not working. Then refer the script/u01/app/11.2.0/grid/bin/racgvip
If there is more than one interfaces, remove the cable on the interface
Which VIP is set and run check action, the VIP shoshould be set to another interface.

 

#1. becomes root user
#2. set environment variables
#-_ USR_ORA_VIP for VIP address
#-_ USR_ORA_NETMASK for netmask address
#-_ USR_ORA_IF for interface names, they are separated by '| 'character
#-_ CAA_NAME for the VIP resource name, ora. <nodename>. vip
#3. Test list command
# Sh racgvip list
#4. Test start command
# Sh racgvip start
# Echo $?
# Ifconfig (to check if the VIP is set)
#5. Test check command
# Sh racgvip check
# Echo $?
#6. Test stop command
# Sh racgvip stop
# Echo $?
# Ifconfig (to check if the VIP is unset)
#7. If there is more than one interfaces, remove the cable on the interface
# Which VIP is set and run check action, the VIP shocould be set to another
# Interface.
# Note: if cables are pulled from all interfaces or there is only one
# Interface, VIP will stay on the original interface and
# The script returns success. This behavior is to keep VIP resource
# From failover if there is a network brown out.
#
# Sh racgvip check
# Echo $?
# Ifconfig (to check if the VIP is set to another interface)

VIP is brought up using/u01/app/11.2.0/grid/bin/racgvip. from the script, it will check the status of the insterface. if it is down then VIP can not be up.

Reviewed the scripts in/u01/app/11.2.0/grid/bin/racgvip:
If [-z "$ _ IF"]
Then
Echo "checkIf: interface name is NULL"
Return 1
Fi

# Check if ther interface is up
$ IFCONFIG $ _ IF | $ GREP-q-w UP
If [$? -Ne 0]
Then
Echo "checkIf: interface $ _ IF is down"
Return 1
Fi