OracleRACDatabase11.1.0.6 listener fault case

After receiving the call, the customer's core set of javaslerac databases cannot be connected. When connected, no listening program is reported. The customer's javaslerac version is 11.1.0.6 and the platform is AIX6.1.05,

After receiving a call, the customer's core Oracle RAC database cannot be connected. When connected, no listening program is available. The customer's Oracle RAC version is 11.1.0.6 and the platform is AIX 6.1.05,

After receiving a call, the customer's core Oracle RAC database cannot be connected, and no listening program is available at the time of connection. The customer's Oracle RAC version is 11.1.0.6 and the platform is AIX 6.1.05, using ibm hacmp 5.5.0.8.

When I remotely passed, I found that node 2 had no oracle user processes, and the concurrent vg was not activated, and the HACMP service was offline.

The instance of another Oracle node is normal, and some server processes are still working, but the local listener fails, resulting in a new connection failure to connect to the instance, through crs_stat-t, we can see that both instances are in OFFLINE status.

No LISTENER process is found on the node, and all server processes are manually killed. When the oracle user starts the LISTENER, the following error is returned:

$ Lsnrctl start listener_cdfy740a

LSNRCTL for IBM/aix risc System/6000: Version 11.1.0.6.0-Production on 20-NOV-2014 20:09:09

Copyright (c) 1991,200 7, Oracle. All rights reserved.

Starting/oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for IBM/aix risc System/6000: Version 11.1.0.6.0-Production
System parameter file is/oracle/app/oracle/product/11.1.0/db_1/network/admin/listener. ora
Log messages written to/oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log. xml
Listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = tcp) (HOST = 10.107.64.1) (PORT = 1521 )))
Error listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.1) (PORT = 1521) (IP = FIRST )))
TNS-12542: TNS: address already in use
TNS-12560: TNS: protocol adapter error
TNS-00512: Address already in use
IBM/aix risc System/6000 Error: 67: Address already in use

Listener failed to start. See the error message (s) abve...

10.107.64.1 is the vip address of the node. The hosts configuration in the RAC environment is as follows:

10.107.64.1 vip1
10.107.64.2 vip2
10.107.64.3 cdfy740a
10.107.64.4 cdfy740b
172..201.201.1 prv1
172..201.201.2 prv2

Manually stop the node's nodeapps service:

Cdfy740a @ root [/oracle/app/11.1.0/crs/bin]./srvctl stop nodeapps-n cdfy740a

After the VIP is successfully stopped, it disappears at the host level:

Cdfy740a @ root [/oracle/app/11.1.0/crs/bin] ifconfig-a | more
En0: flags = 1e080863, c0
Inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255
Tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
En1: flags = 1e080863, c0
Inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255
Tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
En4: flags = 5e080863, c0
Inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255
Tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
Lo0: flags = e08084b, c0
Inet 127.0.0.1 netmask 0xff000000 broadcast 127.0000255
Inet6: 1% 1/0
Tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

Restart node nodeapps:

Cdfy740a @ root [/oracle/app/11.1.0/crs/bin]./srvctl start nodeapps-n cdfy740a
CRS-1006: No more members to consider
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: LSNRCTL for IBM/aix risc System/6000: Version 11.1.0.6.0-Production on 20-NOV-2014 20:13:07
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Copyright (c) 1991,200 7, Oracle. All rights reserved.
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Starting/oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNSLSNR for IBM/aix risc System/6000: Version 11.1.0.6.0-Production
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: System parameter file is/oracle/app/oracle/product/11.1.0/db_1/network/admin/listener. ora
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Log messages written to/oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log. xml
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = tcp) (HOST = 10.107.64.1) (PORT = 1521 )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Error listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.1) (PORT = 1521) (IP = FIRST )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12542: TNS: address already in use
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12560: TNS: protocol adapter error
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-00512: Address already in use
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: IBM/aix risc System/6000 Error: 67: Address already in use
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Listener failed to start. See the error message (s) abve...
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: LSNRCTL for IBM/aix risc System/6000: Version 11.1.0.6.0-Production on 20-NOV-2014 20:13:08
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Copyright (c) 1991,200 7, Oracle. All rights reserved.
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Connecting to (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = vip1) (PORT = 1521 )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12541: TNS: no listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12560: TNS: protocol adapter error
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-00511: No listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: IBM/aix risc System/6000 Error: 79: Connection refused
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Connecting to (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.1) (PORT = 1521) (IP = FIRST )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12541: TNS: no listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12560: TNS: protocol adapter error
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-00511: No listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: IBM/aix risc System/6000 Error: 79: Connection refused
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Connecting to (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.3) (PORT = 1521) (IP = FIRST )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12541: TNS: no listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12560: TNS: protocol adapter error
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-00511: No listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: IBM/aix risc System/6000 Error: 79: Connection refused
CRS-0215: cocould not start resource 'ora. cdfy740a. LISTENER_CDFY740A.lsnr '.

Previously, when you used lsnrctl status listener_cdfy740a to view the listener status, you also received the Connection refused error.