ORA-609 Error Analysis and Solution

A customer database in the inspection of the alert Log will appear occasionally ORA-609 error, because of the ORA-609, ospid (xxxx) process is aborting, but also

A customer database in the inspection of the alert Log will appear occasionally ORA-609 error, because of the ORA-609, ospid (xxxx) process is aborting, but also

A customer database in the inspection of the alert Log will occasionally appear in the ORA-609 error, the general content is as follows:

**************************************** *******************************

Fatal NI connect error 12537, connecting:
(LOCAL = NO)

Version information:
TNS for HPUX: Version 11.2.0.3.0-Production
Oracle Bequeath NT Protocol Adapter for HPUX: Version 11.2.0.3.0-Production
TCP/IP NT Protocol Adapter for HPUX: Version 11.2.0.3.0-Production
Time: 19-OCT-2014 20:24:16
Tracing not turned on.
Tns error struct:
Ns main err code: 12537

TNS-12537: TNS: connection closed
Ns secondary err code: 12560
Nt main err code: 0
Nt secondary err code: 0
Nt OS err code: 0
Opiodr aborting process unknown ospid (2734) as a result of ORA-609
Sun Oct 19 21:27:24 2014

**************************************** *******************************

Because of the ORA-609, The ospid (xxxx) process is aborting, and with the TNS-12537 error, the connection is closed

I searched the MOS circle and found a document aiming at this error. The following is a description:

Applicable:

Oracle Net Services-Version 11.2.0.1 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.

Symptoms:

The above similar content appears in the alert Log (omitted)

Changes:

Changes in database server load, client connect descriptor, changes in network infrastructure (firewall configuration ).

Cause:

First, the "opiodr aborting process unknown ospid (2734) as a result of ORA-609" message only shows that the Oracle database dedicated process is disabled due to the ORA-609

Here is a description:

ORA-609 means "cocould not attach to incoming connection" so the database process was 'aborted' (closed) because it couldn't attach to the incoming connection passed to it by the listener.

The ORA-609 means you cannot append it to an upcoming connection through a listener, so the server process is terminated (disabled)

The reason for this is found in the sqlnet error stack, in our case is:
TNS-12537: TNS: connection closed.
Basically the dedicated process didn't have a client connection anymore to work.

There are six steps to connect the client:

Client initiates a connection to the database so it connects to the listener
Listener starts (fork) a dedicated database process that will receive this connection (session)
After this dedicated process is started, the listener passes the connection from the client to this process
The server process takes the connection from the listener to continue the handshake with the client
Server process and client exchange information required for establishing a session (ASO, Two Task Common, User logon)
Session is opened

In the case of the above error the connection from the client was closed somewhere between 3. and 4. So when the dedicated process tries to communicate with the client it finds that connection closed.

In view of the preceding error, the client connection is closed when the VPC tries to connect to the client, and the connection is closed.

To determine the client which hit this problem we can try to match the timestamp of the error from alert log with an entry in listener. log, but this might be difficult in case of a loaded listener with writable incoming connections per second.
Server sqlnet trace will not provide any information about the client.

To identify the client that encountered the problem, we can try to match the time stamp of the error in the alert Log and there are corresponding entries in the listening log, however, it is very difficult to judge when the load listener has many connections per second. The sqlnet trace on the server does not provide any information about the client.

We can enable sqlnet server trace to catch the error (the match is done based on the ospid found in sqlnet server trace file name and the line with ORA-609 error ):

You can also enable capture of ORA-609 errors in the trace of sqlnet server, matching successful Based on the ospid in the sqlnet server trace file name and ORA-609 error message

Nscon: doing connect handshake...
Nscon: recving a packet
Nsprecv: entry
Nsprecv: reading from transport...
Nttrd: entry
Nttrd: exit
Ntt2err: entry
Ntt2err: Read unexpected eof error on 15 <error
Ntt2err: exit
Nsprecv: error exit
Nserror: entry
Nserror: nsres: id = 0, op = 68, ns = 12537, NS2. nt [0] = 12560, nt [1] = 0, nt [2] = 0; ora [0] = 0, ora [1] = 0, ora [2] = 0
Nscon: error exit
Nsdo: nsctxrnk = 0
Nsdo: error exit
Nsinh_hoff: error recving request

Possible causes:

Several possible situations can cause this to happen:

Client changed its mind and closed the connection immediately after initiating it
Client crashed
Firewall kills the connection
Some oracle timeout set on client
Solution:

Because the entry from listener. log contains only CONNECT_DATA and CID related information we need to check the client configuration for any sqlnet timeouts: