was cluster series (2): Database connection Low-level error-Network connectivity issues

Environment

Project point	indicator
was version	7.0
Application server operating system	windows 2008
Application Server System bits	64bit
database server operating system	centos 5.6
database version	oracle 11.0.2.1

After the was cluster was built today, the release of the application package and the launch were successful, but when testing the database connection, it is not successful, it is recommended to view the KVM log.

Then go to the log path to view the relevant log (referred to as D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\server1\SystemOut.log here), the information is as follows:

[14-9-3 16:34:08:609 CST] 0000001d datasourcecon E dsra8040i: Unable to connect to DataSource. Encountered "": Java.sql.SQLException:The Network Adapter could not establish the Connectiondsra0010e:sql status: 61000, error code: 20. At Oracle.jdbc.driver.T4CConnection.logon (t4cconnection.java:412) at oracle.jdbc.driver.physicalconnection.< Init> (physicalconnection.java:531) at oracle.jdbc.driver.t4cconnection.<init> (T4CConnection.java:221) at Oracle.jdbc.driver.T4CDriverExtension.getConnection (t4cdriverextension.java:32) at Oracle.jdbc.driver.OracleDriver.connect (oracledriver.java:503) at Oracle.jdbc.pool.OracleDataSource.getPhysicalConnection (oracledatasource.java:280) at Oracle.jdbc.pool.OracleDataSource.getConnection (oracledatasource.java:207) at Oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection (oracleconnectionpooldatasource.java:139) At Oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection (oracleconnectionpooldatasource.java:88) At Oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledconnection (oracleconnectionpooldatasource.java:70) at Com.ibm.ws.rsadapter.dsconfigurationhelper$2.run ( dsconfigurationhelper.java:1704) at Com.ibm.ws.security.auth.ContextManagerImpl.runAs (Contextmanagerimpl.java : 5263) at Com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem (contextmanagerimpl.java:5441) at Com.ibm.ws.security.core.SecurityContext.runAsSystem (securitycontext.java:255) at Com.ibm.ws.rsadapter.dsconfigurationhelper$3.run (dsconfigurationhelper.java:1720) at Com.ibm.ws.security.util.AccessController.doPrivileged (accesscontroller.java:118) at Com.ibm.ws.rsadapter.DSConfigurationHelper.getPooledConnection (dsconfigurationhelper.java:1732) at Com.ibm.ws.rsadapter.DSConfigurationHelper.getPooledConnection (dsconfigurationhelper.java:1627) at Com.ibm.ws.rsadapter.DSConfigurationHelper.getConnectionFromDSOrPooledDS (dsconfigurationhelper.java:3370) at Com.ibm.ws.rsadapter.DSConfigurationHelper.getConnectionFromDSOrPooledDS (dsconfigurationhelper.java:3245) at Com.ibm.ws.rsadapter.DSConFigurationhelper.testconnectionforgui (dsconfigurationhelper.java:4105) at SUN.REFLECT.NATIVEMETHODACCESSORIMPL.INVOKE0 (Native Method) at Sun.reflect.NativeMethodAccessorImpl.invoke ( nativemethodaccessorimpl.java:48) at Sun.reflect.DelegatingMethodAccessorImpl.invoke ( DELEGATINGMETHODACCESSORIMPL.JAVA:25) at Java.lang.reflect.Method.invoke (method.java:600) at Com.ibm.ws.management.DataSourceConfigHelperMBean.testConnectionToDataSource2 (Datasourceconfighelpermbean.java : 556) at Com.ibm.ws.management.DataSourceConfigHelperMBean.testConnection (datasourceconfighelpermbean.java:484) At Sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at Sun.reflect.NativeMethodAccessorImpl.invoke ( nativemethodaccessorimpl.java:48) at Sun.reflect.DelegatingMethodAccessorImpl.invoke ( DELEGATINGMETHODACCESSORIMPL.JAVA:25) at Java.lang.reflect.Method.invoke (method.java:600) at Sun.reflect.misc.Trampoline.invoke (methodutil.java:37) at Sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at Sun.reflect.NativeMethoDaccessorimpl.invoke (nativemethodaccessorimpl.java:48) at Sun.reflect.DelegatingMethodAccessorImpl.invoke ( DELEGATINGMETHODACCESSORIMPL.JAVA:25) at Java.lang.reflect.Method.invoke (method.java:600) at Sun.reflect.misc.MethodUtil.invoke (methodutil.java:244) at Javax.management.modelmbean.RequiredModelMBean.invokeMethod (requiredmodelmbean.java:1074) at Javax.management.modelmbean.RequiredModelMBean.invoke (requiredmodelmbean.java:955) at Com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke (defaultmbeanserverinterceptor.java:836) at Com.sun.jmx.mbeanserver.JmxMBeanServer.invoke (jmxmbeanserver.java:761) at Com.ibm.ws.management.AdminServiceImpl $1.run (adminserviceimpl.java:1331) at com.ibm.ws.security.util.AccessController.doPrivileged ( accesscontroller.java:118) at Com.ibm.ws.management.AdminServiceImpl.invoke (adminserviceimpl.java:1224) at Com.ibm.ws.management.connector.AdminServiceDelegator.invoke (adminservicedelegator.java:181) at Com.ibm.ws.management.connector.ipc.CallRouter.route (Callrouter. java:242) at Com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink.doWork (Ipcconnectorinboundlink.java : 353) at Com.ibm.ws.management.connector.ipc.ipcconnectorinboundlink$ipcconnectorreadcallback.complete ( ipcconnectorinboundlink.java:595) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted ( aioreadcompletionlistener.java:165) at Com.ibm.io.async.AbstractAsyncFuture.invokeCallback ( abstractasyncfuture.java:217) at Com.ibm.io.async.AsyncChannelFuture.fireCompletionActions ( asyncchannelfuture.java:161) at com.ibm.io.async.AsyncFuture.completed (asyncfuture.java:138) at Com.ibm.io.async.ResultHandler.complete (resulthandler.java:204) at Com.ibm.io.async.ResultHandler.runEventProcessingLoop (resulthandler.java:775) at com.ibm.io.async.resulthandler$ 2.run (resulthandler.java:905) at Com.ibm.ws.util.threadpool$worker.run (threadpool.java:1613) Caused by: Oracle.net.ns.NetException:The Network Adapter could not establish the Connectionat Oracle.net.nt.ConnStrategy.execute (Connstrategy.java:359) at Oracle.net.resolver.AddrResolution.resolveAndExecute (addrresolution.java:422) at Oracle.net.ns.NSProtocol.establishConnection (nsprotocol.java:672) at Oracle.net.ns.NSProtocol.connect ( nsprotocol.java:237) at Oracle.jdbc.driver.T4CConnection.connect (t4cconnection.java:1042) at Oracle.jdbc.driver.T4CConnection.logon (t4cconnection.java:301) ... morecaused by:java.net.ConnectException:Connection Refused:connectat Java.net.PlainSocketImpl.doConnect ( plainsocketimpl.java:370) at Java.net.PlainSocketImpl.connectToAddress (plainsocketimpl.java:232) at Java.net.PlainSocketImpl.connect (plainsocketimpl.java:219) at Java.net.SocksSocketImpl.connect ( sockssocketimpl.java:366) at Java.net.Socket.connect (socket.java:528) at Oracle.net.nt.TcpNTAdapter.connect ( tcpntadapter.java:141) at Oracle.net.nt.ConnOption.connect (connoption.java:123) at Oracle.net.nt.ConnStrategy.execute (connstrategy.java:337) ... More. [14-9-2 16:34:27:921 CST] 0000005f nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:35:27:921 CST] 00000060 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:36:27:921 CST] 00000061 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:37:27:937 CST] 00000062 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:38:27:937 CST] 00000063 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:39:27:937 CST] 00000064 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:40:27:921 CST] 00000065 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:41:27:953 CST] 00000066 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:42:27:921 CST] 00000067 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:43:27:937 CST] 00000068 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:44:27:937 CST] 00000069 nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:45:27:921 CST] 0000006a nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:46:27:937 CST] 0000006b nodesynctask A adms0003i: Configuration synchronization completed successfully. [14-9-2 16:47:27:921 CST] 0000006c nodesynctask A adms0003i: Configuration synchronization completed successfully.

Although the log information is so verbose, but the problem is locked to the first line, "The Network Adapter could not establish the connectiondsra0010e", the problem is arguably a regular problem, But because of the various configuration settings in the afternoon, the head was a bit dizzy. After seeing the problem, the whole idea ran off. Now record it, warning.

The wrong thinking:

1, see "The Network Adapter", is definitely a problem, think of IS is cluster two server network card problem, database server and cluster server to ping each other test, ping pass state is normal;

2, view was data source configuration information, start to check the node services, restart the node services and HTTP services, and so on, some of the random attempt to try, the result is my train of thought was completely messed up;

3, then, did a more ridiculous thing, look at was cluster configuration information, suspected data source connection problem, the data source format from the original "Jdbc:oracle:thin:@10.53.105.64:1521:hyl" to "Jdbc:oracle:thin: @ (DESCRIPTION = (Address_list = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.53.105.64) (PORT =1521))) (Connect_data = (service_name =hyl)) ". This is because of network connectivity problems, suspected that the cluster server could not complete the connection, may be the resolution of the problem, and then modified into a full format. Testing the database connection is still unsuccessful. Here the problem arises, ignoring the simple question and complicating the problem more.

Corrected thinking:

In fact, starting from the 1th above, some run off, Ping Pass test, should be the first to think of is to see the database monitoring not. And I completely ignore this, for the simple reason that this database server was connected to a single-node was, the connection test is not a problem, it is assured that the monitoring is not a problem. In such a small place, oneself have not personally to verify, and cause the complication of the problem. When it was discovered that the database listener was not up, the problem was that the database server was restarted when the single instance was stopped and the listener on that server was not set to start automatically. It's so easy to say. So I started listening, I restored the previously changed test back to the general setting state after the configuration was completed, the was data source test connection, the result two node test connection pass.

Summary:

1, when dealing with problems, the correct way of thinking than the random attempt is important, only the direction of the will be "charm", otherwise it may be "fuel", the idea of their own chaos.

2, in the technology, remember "the Network Adapter couldnot establish ..." When the appearance, remember the ping test, check the listening status, confirm and then do further analysis.

*********************************************** Declaration ************************************************

original works, from the "Blue Blog" blog, Welcome to reprint, please be sure to indicate the source ( Http://blog.csdn.net/huangyanlong ).

Please leave a message or email ([email protected]) to indicate the error.

************************** ***************************************************************************

was cluster series (2): Database connection Low-level error-Network connectivity issues