Linux oracle10.2.0.1 lsnrctl cannot be started

Fault description: lsnrctl does not respond after stat, while SQLPLUS is normal:
[Oracle @ wlserver3 ~] $ Lsnrctl stat
LSNRCTL for Linux: Version 10.2.0.1.0-Production on 13-SEP-2011 23:16:11
Copyright (c) 1991,200 5, Oracle. All rights reserved.
Connecting to (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = localhost. localdomain) (PORT = 1521 )))
Try ps-ef | grep lsnrctl and kill-9 to kill the process, but the problem also occurs when the process is restarted after the kill.
Try ps-ef | grep lsnr (note the difference with the above command is less than ctl), kill-9 to kill the process, and then restart successfully
[Root @ wlserver3 admin] # ps-ef | grep lsnr
Oracle 3067 16807 0 Sep10? 00:00:01/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER-inherit
Oracle 16807 1 0 Jul08? 00:02:07/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER-inherit
Oracle 18378 18350 0 Sep13 pts/1 00:00:00 lsnrctl stat
Oracle 18386 18350 0 Sep13 pts/1 00:00:00 lsnrctl stat
Oracle 18423 18350 0 Sep13 pts/1 00:00:00 lsnrctl stat
Root 19686 18480 0 00:00:00 pts/3 grep lsnr
[Root @ wlserver3 admin] # kill-9 3067
[Root @ wlserver3 admin] # kill-9 16807
Reboot successful:
Oracle @ wlserver3 bdump] $ lsnrctl start
LSNRCTL for Linux: Version 10.2.0.1.0-Production on 14-SEP-2011 01:21:35
Copyright (c) 1991,200 5, Oracle. All rights reserved.
Starting/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 10.2.0.1.0-Production
System parameter file is/oracle/product/10.2.0/db_1/network/admin/listener. ora
Log messages written to/oracle/product/10.2.0/db_1/network/log/listener. log
Listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = tcp) (HOST = localhost. localdomain) (PORT = 1521 )))
Listening on: (DESCRIPTION = (ADDRESS = (PROTOCOL = ipc) (KEY = EXTPROC0 )))
Connecting to (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = localhost. localdomain) (PORT = 1521 )))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 10.2.0.1.0-Production
Start Date 14-SEP-2011 01:21:37
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File/oracle/product/10.2.0/db_1/network/admin/listener. ora
Listener Log File/oracle/product/10.2.0/db_1/network/log/listener. log
Listening Endpoints Summary...
(DESCRIPTION = (ADDRESS = (PROTOCOL = tcp) (HOST = localhost. localdomain) (PORT = 1521 )))
(DESCRIPTION = (ADDRESS = (PROTOCOL = ipc) (KEY = EXTPROC0 )))
Services Summary...
Service "PLSExtProc" has 1 instance (s ).
Instance "PLSExtProc", status UNKNOWN, has 1 handler (s) for this service...
The command completed successfully
[Oracle @ wlserver3 bdump] $
Supplement: I saw a BUG in the 10.2.0.1 environment during the processing and fixed it together.
10g Listener: High CPU Utilization-Listener May be Hang [ID 284602.1]

Symptoms
1) If the listener is hanging, a sqlnet trace (level 16) will show that the last operation was a fork.

[000068 28-SEP-2004 14: 16: 17: 398] sntpcall: entry
[000068 28-SEP-2004 14: 16: 17: 401] sntpcall: detaching from parent with additional fork 68 bytes to NS buffer

2) netstat-na reports numerous (maybe thousands) of sockets associated with port 6100 in a TIME_WAIT status

127.0.0.1.6100 127.0.0.1.55228 49152 0 49152 0 TIME_WAIT
127.0.0.1.6100 127.0.0.1.55229 49152 0 49152 0 TIME_WAIT
127.0.0.1.6100 127.0.0.1.55230 49152 0 49152 0 TIME_WAIT
127.0.0.1.6100 127.0.0.1.55231 49152 0 49152 0 TIME_WAIT
127.0.0.1.6100 127.0.0.1.5520.49152 0 49152 0 TIME_WAIT

3) "sdtprocess" (Solaris), or "top" show that the listener process (tnslsnr) is using a large amount of cpu.

ID Name Owner CPU % RAM Size Started Parent
Bytes -----------------------------------------------------------------------------------------------
21145 tnslsnr mseibt 44 9440 22368 13:36:03 1/u01/app/oracle/product/10.1.0/bin/tnslsnr LISTENER-inherit

4) The 10g (iAS) $ ORACLE_HOME/opmn/log on the same system shows the following repeating error in the ons logs:

04/10/01 13:30:43 [4] Local connection 0, 127.0.0.1, 6100 missing form factor
04/10/01 13:30:43 [4] Local connection 0, 127.0.0.1, 6100 missing form factor
04/10/01 13:30:43 [4] Local connection 0, 127.0.0.1, 6100 missing form factor
04/10/01 13:30:43 [4] Local connection 0, 127.0.0.1, 6100 missing form factor
04/10/01 13:30:43 [4] Local connection 0, 127.0.0.1, 6100 missing form factor

Changes
-Both 10g (iAS) and Oracle 10g (rdbms) are installed on the same system.
-Oracle 10g (non-RAC) is installed on a system with or without other Oracle installations.

Cause
The Oracle Notification Server (ONS) running in the 10g (iAS) home uses the ports defined in $ ORACLE_HOME/opmn/conf/ons. config. the database listener, which is an ONS client, also utilizes the ONS ports identified in its own $ ORACLE_HOME/opmn/conf/ons. config. when 10g (iAS) and 10g (rdbms) are installed on the same box, the installer will mistakenly configure identical ONS ports in both homes which creates an operational conflict when both the 10g (iAS) ONS and 10g (rdbms) listener services are running.

In non-RAC environments where Oracle 10g is the only Oracle product on the system, the existence of the ons. config may still cause a hang condition in the listener.

Solution
This issue is corrected in versions 10.2.0.3 and higher.
(Or apply backport patch <4518443>)

You may also use either of the following 2 workarounds to address this problem:

1) In the 10g (rdbms) home, rename the $ ORACLE_HOME/opmn/conf/ons. config file (so that the listener does not find or use it ).

For example:

Cd $ ORACLE_HOME/opmn/conf
Mv ons. config ons. config. orig

(Then restart the listener)

-Or-

2) Explicitly disable the 10g TNS Listener from subscribing to the ONS.

Edit the active 10g listener. ora file and add the parameter below:

SUBSCRIBE_FOR_NODE_DOWN_EVENT _ <listener_name> = OFF

Where <listener_name> wocould be replaced with the actual listener name configured in the listener. ora file.

For example, if the listener name is LISTENER (default), the parameter wocould be:

SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER = OFF

(Then restart the listener)

References
BUG: 3881276-listener hangs intermittantly after applying 10.1.0.3 PATCHSET
BUG: 6017068-PMON SPINNING IN NETWORKING CODE
 
Author: "mgewu"