ORA-3136錯誤分析——WARNING Inbound Connection Timed Out

概要說明

　

WARNING: inbound connection timed out (ORA-3136)

這個錯誤我以前一直沒有遇到過，今天早上客戶在MSN上，他的一台Oracle Server很忙，alert中頻繁的出現這個錯誤提示，導致串連無法成功。下面是alert 檔案的錯誤提示：

......

Wed Feb 27 09:03:02 2008 Completed checkpoint up to RBA [0x184d.2.10], SCN: 1203810646 Wed Feb 27 09:03:27 2008 WARNING: inbound connection timed out (ORA-3136) Wed Feb 27 09:03:27 2008 WARNING: inbound connection timed out (ORA-3136) Wed Feb 27 09:03:27 2008 WARNING: inbound connection timed out (ORA-3136) Wed Feb 27 09:03:27 2008 WARNING: inbound connection timed out (ORA-3136) Wed Feb 27 09:03:27 2008 WARNING: inbound connection timed out (ORA-3136) Wed Feb 27 09:03:27 2008 WARNING: inbound connection timed out (ORA-3136) Wed Feb 27 09:03:27 2008 WARNING: inbound connection timed out (ORA-3136) Wed Feb 27 09:03:27 2008 WARNING: inbound connection timed out (ORA-3136) Wed Feb 27 09:04:30 2008 Incremental checkpoint up to RBA [0x184d.e5a6.0], current log tail at RBA [0x184d.43aaa.0] Wed Feb 27 09:05:02 2008

......

　

這個WARNING我以前沒接觸過，但從字面上可以看到應該是connect time out，以前常見的是ORA-12170。所以很納悶。

　

初步分析

　

1）檢查listener.ora和sqlnet.ora的參數設定，未發現其他異常

    $ cat listener.ora    ################    # Filename......: listener.ora    # Name..........:    # Date..........:    ################    ADMIN_RESTRICTIONS_LISTENER = on    LISTENER =        (ADDRESS_LIST =           (ADDRESS =             (PROTOCOL = IPC)             (KEY = HS5.WORLD)           )           (ADDRESS=             (PROTOCOL = IPC)             (KEY = HS5)           )           (ADDRESS =             (COMMUNITY = SAP.WORLD)             (PROTOCOL = TCP)             (HOST = GVSHS5DB)             (PORT = 1527)          )       )    STARTUP_WAIT_TIME_LISTENER = 0    CONNECT_TIMEOUT_LISTENER = 10    TRACE_LEVEL_LISTENER = OFF    SID_LIST_LISTENER =      (SID_LIST =        (SID_DESC =           (SID_NAME = HS5)           (ORACLE_HOME = /oracle/HS5/102_64)        )      )    $ cat sqlnet.ora    ################    # Filename......: sqlnet.ora    ################    AUTOMATIC_IPC = ON    TRACE_LEVEL_CLIENT = OFF    NAMES.DEFAULT_DOMAIN = WORLD    # 05.01.06 unsorported parameter now     #NAME.DEFAULT_ZONE = WORLD    # 05.01.06 set the default to 10     SQLNET.EXPIRE_TIME = 10    # 05.01.06 set to default     #TCP.NODELAY=YES     # 05.01.06 set to 32768     DEFAULT_SDU_SIZE=32768    $

2）通過topas、vmstat可以看到當前系統的負載很高，cpu基本上是100%（略）

所以初步可以斷定是因為系統負載過重導致串連timeout。

　

進一步分析

　

因為這個WARNING我以前沒見過，所以就直接查閱了Oracle 相關資料。原來這是10gR2上新加的一個屬性，可以通過SQLNET.INBOUND_CONNECT_TIMEOUT來設定，預設情況下是60秒。

　

導致這個WARNING出現的主要原因可能是：

1）Server gets a connection request from a malicious(惡意) client which is not supposed to connect to the database , in which case the error thrown is the correct behavior. You can get the client address for which the error was thrown via sqlnet log file.
2）The server receives a valid client connection request but the client takes a long time to authenticate more than the default 60 seconds.
3）The DB server is heavily loaded due to which it cannot finish the client logon within the timeout specified.
　

那麼如何定位導致這個WARNING出現的呢？

　

The default value of 60 seconds is good enough in most conditions for the database server to authenticate a client connection. If its taking longer period, then its worth checking all the below points before going for the workadound:

1. Check whether local connection on the database server is sucessful & quick.
2. If local connections are quick ,then check for underlying network delay with the help of your network administrator.
3. Check whether your Database performance has degraded by anyway.
4. Check alert log for any critical errors for eg, ORA-600 or ORA-7445 and get them resolved first.
These critical errors might have triggered the slowness of the database server.

As a workaround to avoid only this warning messages, you can set the parameters SQLNET.INBOUND_CONNECT_TIMEOUT
and INBOUND_CONNECT_TIMEOUT_<listenername> to the value more than 60.

For e.g 120. So that the client will have more time to provide the authentication information to the database. You may have to further tune these parameter values according to your setup.

To set these parameter
1. In server side sqlnet.ora file add SQLNET.INBOUND_CONNECT_TIMEOUT
For e.g
SQLNET.INBOUND_CONNECT_TIMEOUT = 120
2. In listener.ora file - INBOUND_CONNECT_TIMEOUT_<listenername> = 110
For e.g if the listener name is LISTENER then -
INBOUND_CONNECT_TIMEOUT_LISTENER = 110

Note:From Oracle version 10.2.0.3 onwards the default value of INBOUND_CONNECT_TIMEOUT_<listenername> is 60 seconds. For previous releases it is zero by default.

How to check whether inbound timout is active for the listener and database server

For eg. INBOUND_CONNECT_TIMEOUT_<listener_name> =4

You can check whether the parameter is active or not by simply doing telnet to the listener port.
$ telnet <database server IP> <listener port>

for eg.
$ telnet 192.168.12.13 1521
The telnet session should disconnect after 4 seconds which indicates that the inbound connection timeout for the listener is active.

To check whether database server sqlnet.inbound_connect_timeout is active:
Eg. sqlnet.inbound_connect_timeout =5

a. For Dedicated server setup, enable the support level sqlnet server tracing will show the timeout value as below:

niotns: Enabling CTO, value=5000 (milliseconds) <== 5 seconds niotns: Not enabling dead connection detection. niotns: listener bequeathed shadow coming to life...

b. For shared Server setup,
$ telnet <database server IP> <dispatcher port>
For eg.
$ telnet 192.168.12.13 51658
The telnet session should disconnect after 5 seconds which indicates that the sqlnet.inbound_connection_timeout is active.

轉自：http://tomszrp.itpub.net/post/11835/467438