WAS cluster series (2): database connection low-level error network connection problems

Environment Project metric WAS version 7.0 application server operating system Windows2008 Application Server System bits 64bit database server operating system CentOS5.6 database version Oracle11.0.2.1 after the WAS cluster is built today, the application package is released and started successfully, however, when testing the database connection, the system prompts that the connection fails. We recommend that you check the KVM day.

Environment Project metric WAS version 7.0 Application Server Operating System Windows 2008 Application Server System bits 64bit database server operating system CentOS 5.6 database version Oracle 11.0.2.1 after building a WAS cluster today, the application package is released and started successfully, but the system prompts that the connection to the database is not successful. We recommend that you check the KVM

Environment

Project point	Metrics
WAS version	7.0
Application Server Operating System	Windows 2008
Number of application server systems	64bit
Database Server Operating System	CentOS 1, 5.6
Database Version	Oracle 11.0.2.1

After the WAS cluster is set up today, the application package is released and started successfully. However, the system prompts that the connection to the database is not successful. We recommend that you view the KVM log.

View related logs in the log Path (see D: \ IBM \ WebSphere \ AppServer \ profiles \ Alibaba rv01 \ logs \ server1 \ SystemOut. log). The information is as follows:

[14-9-3 16: 34: 08: 609 CST] 10000001d performancecon E DSRA8040I: Unable to connect to DataSource. ": Java. SQL. SQLException: The Network Adapter cocould not establish the connectionDSRA0010E: SQL status: 61000, error code: 20. At oracle. jdbc. driver. T4CConnection. logon (T4CConnection. java: 412) at oracle. jdbc. driver. PhysicalConnection.
 
  
(PhysicalConnection. java: 531) at oracle. jdbc. driver. T4CConnection.
  
   
(T4CConnection. java: 221) at oracle. jdbc. driver. t4CDriverExtension. getConnection (T4CDriverExtension. java: 32) at oracle. jdbc. driver. oracleDriver. connect (OracleDriver. java: 503) at oracle. jdbc. pool. oracleDataSource. getPhysicalConnection (OracleDataSource. java: 280) at oracle. jdbc. pool. oracleDataSource. getConnection (OracleDataSource. java: 207) at oracle. jdbc. pool. oracleConnectionPoolDataSource. getPhysicalConnec Tion (OracleConnectionPoolDataSource. java: 139) at oracle. jdbc. pool. oracleConnectionPoolDataSource. getPooledConnection (OracleConnectionPoolDataSource. java: 88) at oracle. jdbc. pool. oracleConnectionPoolDataSource. getPooledConnection (OracleConnectionPoolDataSource. java: 70) at com. ibm. ws. rsadapter. DSConfigurationHelper $ 2.run( DSConfigurationHelper. java: 1704) at com. ibm. ws. security. auth. contextManagerImpl. runA S (ContextManagerImpl. java: 5263) at com. ibm. ws. security. auth. contextManagerImpl. runAsSystem (ContextManagerImpl. java: 5441) at com. ibm. ws. security. core. securityContext. runAsSystem (SecurityContext. java: 255) at com. ibm. ws. rsadapter. DSConfigurationHelper $ 3.run( DSConfigurationHelper. java: 1720) at com. ibm. ws. security. util. accessController. doPrivileged (AccessController. java: 118) at com. ibm. ws. rsadapter. DSConfig UrationHelper. getPooledConnection (DSConfigurationHelper. java: 1732) at com. ibm. ws. rsadapter. DSConfigurationHelper. getPooledConnection (DSConfigurationHelper. java: 1627) at com. ibm. ws. rsadapter. DSConfigurationHelper. getConnectionFromDSOrPooledDS (DSConfigurationHelper. java: 3370) at com. ibm. ws. rsadapter. DSConfigurationHelper. getConnectionFromDSOrPooledDS (DSConfigurationHelper. java: 3245) at com. ibm. ws. rsadap Ter. DSConfigurationHelper. testConnectionForGUI (DSConfigurationHelper. java: 4105) at sun. reflect. nativeMethodAccessorImpl. invoke0 (Native Method) at sun. reflect. nativeMethodAccessorImpl. invoke (NativeMethodAccessorImpl. java: 48) at sun. reflect. delegatingMethodAccessorImpl. invoke (DelegatingMethodAccessorImpl. java: 25) at java. lang. reflect. method. invoke (Method. java: 600) at com. ibm. ws. management. performanceconfi GHelperMBean. testConnectionToDataSource2 (performanceconfighelpermbean. java: 556) at com. ibm. ws. management. performanceconfighelpermbean. testConnection (performanceconfighelpermbean. java: 484) at sun. reflect. nativeMethodAccessorImpl. invoke0 (Native Method) at sun. reflect. nativeMethodAccessorImpl. invoke (NativeMethodAccessorImpl. java: 48) at sun. reflect. delegatingMethodAccessorImpl. invoke (DelegatingMethodAccessor Impl. java: 25) at java. lang. reflect. method. invoke (Method. java: 600) at sun. reflect. misc. trampoline. invoke (MethodUtil. java: 37) at sun. reflect. nativeMethodAccessorImpl. invoke0 (Native Method) at sun. reflect. nativeMethodAccessorImpl. invoke (NativeMethodAccessorImpl. java: 48) at sun. reflect. delegatingMethodAccessorImpl. invoke (DelegatingMethodAccessorImpl. java: 25) at java. lang. reflect. method. invoke (Method. java: 60 0) at sun. reflect. misc. methodUtil. invoke (MethodUtil. java: 244) at javax. management. modelmbean. requiredModelMBean. invokeMethod (RequiredModelMBean. java: 1074) at javax. management. modelmbean. requiredModelMBean. invoke (RequiredModelMBean. java: 955) at com. sun. jmx. interceptor. defaultMBeanServerInterceptor. invoke (DefaultMBeanServerInterceptor. java: 836) at com. sun. jmx. mbeanserver. jmxMBeanServer. invoke (JmxMBeanSer Ver. java: 761) at com. ibm. ws. management. adminServiceImpl $ 1.run( AdminServiceImpl. java: 1331) at com. ibm. ws. security. util. accessController. doPrivileged (AccessController. java: 118) at com. ibm. ws. management. adminServiceImpl. invoke (AdminServiceImpl. java: 1224) at com. ibm. ws. management. connector. adminServiceDelegator. invoke (AdminServiceDelegator. java: 181) at com. ibm. ws. management. connector. ipc. callRouter. route (C AllRouter. java: 242) at com. ibm. ws. management. connector. ipc. IPCConnectorInboundLink. doWork (IPCConnectorInboundLink. java: 353) at com. ibm. ws. management. connector. ipc. IPCConnectorInboundLink $ IPCConnectorReadCallback. complete (IPCConnectorInboundLink. java: 595) at com. ibm. ws. tcp. channel. impl. aioReadCompletionListener. futureCompleted (AioReadCompletionListener. java: 165) at com. ibm. io. async. abstractAsyncFuture. InvokeCallback (AbstractAsyncFuture. java: 217) at com. ibm. io. async. asyncChannelFuture. fireCompletionActions (AsyncChannelFuture. java: 161) at com. ibm. io. async. asyncFuture. completed (AsyncFuture. java: 138) at com. ibm. io. async. resultHandler. complete (ResultHandler. java: 204) at com. ibm. io. async. resultHandler. runEventProcessingLoop (ResultHandler. java: 775) at com. ibm. io. async. resultHandler $ 2.run( ResultHandler. java 905) at com. ibm. ws. util. threadPool $ Worker. run (ThreadPool. java: 1613) Caused by: oracle.net. ns. netException: The Network Adapter cocould not establish the connectionat oracle.net.nt.ConnStrategy.exe cute (ConnStrategy. java: 359) at oracle.net. resolver. addrResolution. resolveAndExecute (AddrResolution. java: 422) at oracle.net. ns. NSProtocol. establishConnection (NSProtocol. java: 672) at oracle.net. ns. NSProtocol. conn Ect (NSProtocol. java: 237) at oracle. jdbc. driver. t4CConnection. connect (T4CConnection. java: 1042) at oracle. jdbc. driver. t4CConnection. logon (T4CConnection. java: 301 )... 55 moreCaused by: java.net. connectException: Connection refused: connectat java.net. plainSocketImpl. doConnect (PlainSocketImpl. java: 370) at java.net. plainSocketImpl. connectToAddress (PlainSocketImpl. java: 232) at java.net. plainSocketImpl. connec T (PlainSocketImpl. java: 219) at java.net. socksSocketImpl. connect (SocksSocketImpl. java: 366) at java.net. socket. connect (Socket. java: 528) at oracle.net. nt. tcpNTAdapter. connect (TcpNTAdapter. java: 141) at oracle.net. nt. connOption. connect (ConnOption. java: 123) at oracle.net.nt.ConnStrategy.exe cute (ConnStrategy. java: 337 )... 60 more. [14-9-2 16: 34: 27: 921 CST] 127005f NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 35: 27: 921 CST] 00000060 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 36: 27: 921 CST] 00000061 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 37: 27: 937 CST] 00000062 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 38: 27: 937 CST] 00000063 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 39: 27: 937 CST] 00000064 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 40: 27: 921 CST] 00000065 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 41: 27: 953 CST] 00000066 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 42: 27: 921 CST] 00000067 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 43: 27: 937 CST] 00000068 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 44: 27: 937 CST] 00000069 NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 45: 27: 921 CST] 10000006a NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 46: 27: 937 CST] 10000006b NodeSyncTask A ADMS0003I: the synchronization is successfully configured. [14-9-2 16: 47: 27: 921 CST] 10000006c NodeSyncTask A ADMS0003I: the synchronization is successfully configured.
  
 

Despite The lengthy log information, the problem is locked to The first line, "the Network Adapter cocould not establish The connectionDSRA0010E", which is a general problem, however, due to a variety of configuration settings in the afternoon, my head was dizzy. After seeing this problem, the whole idea is biased. Now let's take it as a warning.

Wrong thinking:

1. When you see The "The Network Adapter", it must be a Network problem. If The two WAS cluster servers have Nic problems, ping The database servers and cluster servers to each other for testing, the ping status is normal;

2. Check the WAS data source configuration information, start to check the services of each node, restart the services of each node and HTTP services, etc, as a result, my ideas were completely disrupted by myself;

3. Then, I did something more ridiculous. I checked the configuration information in the WAS cluster and suspected the data source connection problem. The data source format WAS changed from "jdbc: oracle: thin: @ 10.53.105.64: 1521: hyl "to" jdbc: oracle: thin: @ (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP) (HOST = 10.53.105.64) (PORT = 1521 ))) (CONNECT_DATA = (SERVICE_NAME = hyl )))". This is because of network connection problems. It is suspected that the cluster server cannot complete the connection. It may be due to a resolution problem, so it is changed to the complete format. The test database connection still fails. Here the problem arises, ignoring the simple Problem and increasing the complexity of the problem.

Thoughts after correction:

In fact, from the first point above, there was some deviation. After the ping test, the first thing that should come to mind was to check whether the database was listening. I ignore this because the database server is connected to a single-node WAS. The connection test is normal, and the listening is fixed. In such a small place, do you have to verify it yourself, which leads to complicated problems. I realized the problem when I found that the database listening WAS unavailable. It turned out that when I stopped a single instance WAS, the database server WAS restarted, in addition, the listener on the server is not set to automatically start. It is so simple. So I started the listener and restored the previously modified test to the normal setting status after the configuration WAS completed. The WAS data source tested the connection. As a result, the two nodes passed the test connection.

Summary:

1. When dealing with the problem, the correct thinking is more important than the random attempt. Only when the direction is right will the medicine get rid of the disease. Otherwise, the fuel may be added to the fire, and all of your ideas may be disrupted.

2. Technically, remember "The Network Adapter couldnot establish ..." When it appears, remember to check the listening status after the ping test and confirm the status before further analysis.

**************************************** ******************************** ****************

Original works, from "Deep Blue blog" blog, welcome to reprint, reprint please be sure to indicate the source (http://blog.csdn.net/huangyanlong ).

Please leave a message or email (hyldba@163.com) indicating that there is an error. Thank you very much.

****************************************************************** ***********************************