Oracle11g ORA-609 TNS 12537 TNS 12560

Wed Dec 18 08:25:58 2013***********************************************************************Fatal NI connect error 12537, connecting to: (LOCAL=NO)  VERSION INFORMATION:TNS for Linux: Version 11.2.0.1.0 - ProductionOracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.1.0 - ProductionTCP/IP NT Protocol Adapter for Linux: Version 11.2.0.1.0 - Production  Time: 18-DEC-2013 08:25:58  Tracing not turned on.  Tns error struct:    ns main err code: 12537    TNS-12537: TNS:connection closed    ns secondary err code: 12560    nt main err code: 0    nt secondary err code: 0    nt OS err code: 0opiodr aborting process unknown ospid (16596) as a result of ORA-609Wed Dec 18 08:28:25 2013Thread 1 advanced to log sequence 22477 (LGWR switch)

e.g.

Sqlnet.ora: SQLNET.INBOUND_CONNECT_TIMEOUT=180
Listener.ora: INBOUND_CONNECT_TIMEOUT_listener_name=120

These settings are in seconds.  Again, the default is 60.  A setting of 120 seconds should resolve most if not all ORA-609 errors.

If the issue persists and inbound connect does not have any effect, the following steps are intended to help locate  the client that may be causing the errors.

1)  Suppress the TNS errors in the alert.log by setting the following sqlnet.ora file parameter:  

sqlnet.ora file (on the server): DIAG_ADR_ENABLED = OFF

This will cause the TNS errors to be posted to the ORACLE_HOME/network/log/sqlnet.log file that is local to the database and may yield useful information about the client's address.

For example, here's a snippet from a server side sqlnet.log where client address info was posted:

Production Time: 15-FEB-2010 07:15:01

Fatal NI connect error 12537, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(Host=yourhost)(Port=1521))(CONNECT_DATA=(SID=PROD1DR)(CID=(PROGRAM=sqlplus)(HOST=client_host)(USER=client))))

Observe the PROGRAM and HOST fields on the last line.  This is where the connection originated.
Be sure to match timestamps in the sqlnet.log with the timestamps of the alert.log errors.  Once you've located the offending client, you can enable client tracing to try and determine the cause:

TRACE_LEVEL_CLIENT=16
TRACE_DIRECTORY_CLIENT=<dir location>
TRACE_TIMESTAMP_CLIENT=TRUE
DIAG_ADR_ENABLED=off   <<<
If you need assistance with client or server tracing, please open an SR with Global Customer Support.

2)  Check the listener.log for client connections that were logged at timestamps that match the ORA-609 timestamps as they appear in the alert.log.  The client information is recorded in each listener.log entry.  Since this error occurs AFTER the listener has handled the connection, do not expect to see errors in the listener.log.

Here's an example snippet of an incoming client connection that was posted to the listener.log:

20-JAN-2009 17:08:45 (CONNECT_DATA=(SID=orcl)(CID=(PROGRAM=D:\oracle\product\10.1.0\Db_1\perl\5.6.1\bin\MSWin32-x86\perl.exe)(HOST=myclient)

Note that the exact timestamp, program name and client host will often be recorded.  Again, once you've located the offending client, enable tracing (see above) to try to capture the connection failure.

3)  Enable server side Oracle Net tracing and capture the TNS error along with the incoming connection.
Match the PID that accompanies the ORA-609 to the server trace label.  e.g.  

ORA-609 : opiodr aborting process unknown ospid (4799_1)  *Note the PID

This PID would correspond to server trace labeled:  svr_4799.trc.  Check the server trace for either TNS error (the 609 will not appear) and try to locate the originating client address.  If assistance is needed for this investigation, please open an SR with Oracle Support.

See below for instuctions on enabling Oracle Net server tracing.

Mon Nov 16 22:39:22 2009
ORA-609 : opiodr aborting process unknown ospid (nnnn)

Enabled Oracle Net server tracing:

TRACE_LEVEL_SERVER=16
TRACE_DIRECTORY_SERVER=<dir location>
TRACE_TIMESTAMP_SERVER=TRUE
DIAG_ADR_ENABLED=off

Reloaded listener and wait for error to appear again.:

ORA-609 : opiodr aborting process unknown ospid  (5233_1)

Note that the server trace file set that corresponded to this event was named svr_5233*.trc.
Of course the timestamps of the alert.log event and the server trace creation matched as well.

A review of the server trace showed only an EOF failure and the  TNS-12537 error:

Read unexpected EOF ERROR 
nserror: nsres: id=0, op=68, ns=12537

In this particular case, there was no information about the client in the trace. This is atypical for a server trace.   It may be that the client aborted before all the client information was posted to the file.  However, there was post in the listener.log for an emagent connection that was established at the same point in time.

Here's an excerpt from a listener.log entry where an emagent establishes a connection:

PROGRAM=D:\oracle\product\10.1.0\Db_1\bin\emagent.exe)

Checked the EM Agent traces and logs and discovered the following entry:

Fatal NI connect error 12547, connecting to:
(LOCAL=NO)

VERSION INFORMATION:
TNS for Solaris: Version 11.1.0.7.0 - Production
Oracle Bequeath NT Protocol Adapter for Solaris: Version 11.1.0.7.0 - Production
TCP/IP NT Protocol Adapter for Solaris: Version 11.1.0.7.0 - Production
Time: 16-NOV-2009 22:39:22

****Tracing to file: /backup/sid_traces/sqlnetlog/ svr_5233.trc 

Tns error struct:

ns main err code: 12547
TNS-12547: TNS:lost contact
ns secondary err code: 12560
nt main err code: 0
nt secondary err code: 0
nt OS err code: 0

****Note the name of the server trace which contains the PID:  svr_5233.trc 
Also, the timestamp of the agent event matches the timestamp of the alert.log error.