Oracle RAC Database 11.1.0.6 listening fault case

Oracle RAC Database 11.1.0.6 listening fault case

After receiving a call, the customer's core Oracle RAC database cannot be connected, and no listening program is available at the time of connection. The customer's Oracle RAC version is 11.1.0.6 and the platform is AIX 6.1.05, using ibm hacmp 5.5.0.8.

When I remotely passed, I found that node 2 had no oracle user processes, and the concurrent vg was not activated, and the HACMP service was offline.

The instance of another Oracle node is normal, and some server processes are still working, but the local listener fails, resulting in a new connection failure to connect to the instance, through crs_stat-t, we can see that both instances are in OFFLINE status.

No LISTENER process is found on the node, and all server processes are manually killed. When the oracle user starts the LISTENER, the following error is returned:

$ Lsnrctl start listener_cdfy740a

LSNRCTL for IBM/aix risc System/6000: Version 11.1.0.6.0-Production on 20-NOV-2014 20:09:09

Copyright (c) 1991,200 7, Oracle. All rights reserved.

Starting/oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for IBM/aix risc System/6000: Version 11.1.0.6.0-Production
System parameter file is/oracle/app/oracle/product/11.1.0/db_1/network/admin/listener. ora
Log messages written to/oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log. xml
Listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = tcp) (HOST = 10.107.64.1) (PORT = 1521 )))
Error listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.1) (PORT = 1521) (IP = FIRST )))
TNS-12542: TNS: address already in use
TNS-12560: TNS: protocol adapter error
TNS-00512: Address already in use
IBM/aix risc System/6000 Error: 67: Address already in use

Listener failed to start. See the error message (s) abve...

10.107.64.1 is the vip address of the node. The hosts configuration in the RAC environment is as follows:

10.107.64.1 vip1
10.107.64.2 vip2
10.107.64.3 cdfy740a
10.107.64.4 cdfy740b
172..201.201.1 prv1
172..201.201.2 prv2

Manually stop the node's nodeapps service:

Cdfy740a @ root [/oracle/app/11.1.0/crs/bin]./srvctl stop nodeapps-n cdfy740a

After the VIP is successfully stopped, it disappears at the host level:

Cdfy740a @ root [/oracle/app/11.1.0/crs/bin] ifconfig-a | more
En0: flags = 1e080863, c0 <UP, BROADCAST, NOTRAILERS, RUNNING, SIMPLEX, MULTICAST, GROUPRT, 64BIT, CHECKSUM_OFFLOAD (ACTIVE), LARGESEND, CHAIN>
Inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255
Tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
En1: flags = 1e080863, c0 <UP, BROADCAST, NOTRAILERS, RUNNING, SIMPLEX, MULTICAST, GROUPRT, 64BIT, CHECKSUM_OFFLOAD (ACTIVE), LARGESEND, CHAIN>
Inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255
Tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
En4: flags = 5e080863, c0 <UP, BROADCAST, NOTRAILERS, RUNNING, SIMPLEX, MULTICAST, GROUPRT, 64BIT, CHECKSUM_OFFLOAD (ACTIVE), mongog, LARGESEND, CHAIN>
Inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255
Tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
Lo0: flags = e08084b, c0 <UP, BROADCAST, LOOPBACK, RUNNING, SIMPLEX, MULTICAST, GROUPRT, 64BIT, LARGESEND, CHAIN>
Inet 127.0.0.1 netmask 0xff000000 broadcast 127.0000255
Inet6: 1% 1/0
Tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

Restart node nodeapps:

Cdfy740a @ root [/oracle/app/11.1.0/crs/bin]./srvctl start nodeapps-n cdfy740a
CRS-1006: No more members to consider
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: LSNRCTL for IBM/aix risc System/6000: Version 11.1.0.6.0-Production on 20-NOV-2014 20:13:07
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Copyright (c) 1991,200 7, Oracle. All rights reserved.
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Starting/oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNSLSNR for IBM/aix risc System/6000: Version 11.1.0.6.0-Production
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: System parameter file is/oracle/app/oracle/product/11.1.0/db_1/network/admin/listener. ora
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Log messages written to/oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log. xml
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = tcp) (HOST = 10.107.64.1) (PORT = 1521 )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Error listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.1) (PORT = 1521) (IP = FIRST )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12542: TNS: address already in use
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12560: TNS: protocol adapter error
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-00512: Address already in use
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: IBM/aix risc System/6000 Error: 67: Address already in use
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Listener failed to start. See the error message (s) abve...
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: LSNRCTL for IBM/aix risc System/6000: Version 11.1.0.6.0-Production on 20-NOV-2014 20:13:08
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Copyright (c) 1991,200 7, Oracle. All rights reserved.
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr:
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Connecting to (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = vip1) (PORT = 1521 )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12541: TNS: no listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12560: TNS: protocol adapter error
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-00511: No listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: IBM/aix risc System/6000 Error: 79: Connection refused
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Connecting to (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.1) (PORT = 1521) (IP = FIRST )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12541: TNS: no listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12560: TNS: protocol adapter error
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-00511: No listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: IBM/aix risc System/6000 Error: 79: Connection refused
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: Connecting to (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.3) (PORT = 1521) (IP = FIRST )))
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12541: TNS: no listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-12560: TNS: protocol adapter error
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: TNS-00511: No listener
Cdfy740a: ora. cdfy740a. LISTENER_CDFY740A.lsnr: IBM/aix risc System/6000 Error: 79: Connection refused
CRS-0215: cocould not start resource 'ora. cdfy740a. LISTENER_CDFY740A.lsnr '.

Previously, when you used lsnrctl status listener_cdfy740a to view the listener status, you also received the Connection refused error.

Check that the VIP address has been successfully bound to the host layer:
Cdfy740a @ root [/oracle/app/11.1.0/crs/bin] ifconfig-a | more
En0: flags = 1e080863, c0 <UP, BROADCAST, NOTRAILERS, RUNNING, SIMPLEX, MULTICAST, GROUPRT, 64BIT, CHECKSUM_OFFLOAD (ACTIVE), LARGESEND, CHAIN>
Inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255
Tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
En1: flags = 1e080863, c0 <UP, BROADCAST, NOTRAILERS, RUNNING, SIMPLEX, MULTICAST, GROUPRT, 64BIT, CHECKSUM_OFFLOAD (ACTIVE), LARGESEND, CHAIN>
Inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255
Tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
En4: flags = 5e080863, c0 <UP, BROADCAST, NOTRAILERS, RUNNING, SIMPLEX, MULTICAST, GROUPRT, 64BIT, CHECKSUM_OFFLOAD (ACTIVE), mongog, LARGESEND, CHAIN>
Inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255
Inet 10.107.64.1 netmask 0xffffff00 broadcast 10.107.64.255
Tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0
Lo0: flags = e08084b, c0 <UP, BROADCAST, LOOPBACK, RUNNING, SIMPLEX, MULTICAST, GROUPRT, 64BIT, LARGESEND, CHAIN>
Inet 127.0.0.1 netmask 0xff000000 broadcast 127.0000255
Inet6: 1% 1/0
Tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

Try to manually start the local listener again:

Cdfy740a @ root [/] su-oracle
$ Lsnrctl start listener_cdfy740a

LSNRCTL for IBM/aix risc System/6000: Version 11.1.0.6.0-Production on 20-NOV-2014 20:18:37

Copyright (c) 1991,200 7, Oracle. All rights reserved.

Starting/oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for IBM/aix risc System/6000: Version 11.1.0.6.0-Production
System parameter file is/oracle/app/oracle/product/11.1.0/db_1/network/admin/listener. ora
Log messages written to/oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log. xml
Listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = tcp) (HOST = 10.107.64.1) (PORT = 1521 )))
Error listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.1) (PORT = 1521) (IP = FIRST )))
TNS-12542: TNS: address already in use
TNS-12560: TNS: protocol adapter error
TNS-00512: Address already in use
IBM/aix risc System/6000 Error: 67: Address already in use

Listener failed to start. See the error message (s) abve...

Startup still failed.

Check the Listener Configuration file:
$ Cat listener. ora
# Listener. ora. cdfy740a Network Configuration File:/oracle/app/oracle/product/11.1.0/db_1/network/admin/listener. ora. cdfy740a
# Generated by Oracle configuration tools.

LISTENER_CDFY740A =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP) (HOST = vip1) (PORT = 1521 ))
(ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.1) (PORT = 1521) (IP = FIRST ))
(ADDRESS = (PROTOCOL = TCP) (HOST = 10.107.64.3) (PORT = 1521) (IP = FIRST ))
)
)

In the Listener Configuration File, vip1 and 10.107.64.1 are duplicate addresses. After you manually remove the rows of 10.107.64.1, the listener can operate normally.

Then, the HACMP service of Node 2 is restored, and Oracle RAC returns to normal.

In addition, it is found that the customer's listening log has been filled in a large size, which is about GB. A large listening log file will also cause unstable listeners, here, the listening logs of the two nodes are renamed.

Oracle Database listening is very slow, and hang basically handles faults

Dynamic and Static monitoring features of Oracle listeners

Non-default listeners and port configurations for placing an instance in the Oracle 11g RAC Environment

Configure and manage Oracle listener logs

Oracle error-ORA-12514: TNS: No listener

ORA-12514 listening error solved

Six connection problems and solutions for Oracle listeners

Oracle LISTENER does not listen to Oracle instances. Problem solved

Set the Oracle LISTENER password (LISTENER)