The httpclient connection pool throws a large number of connectionpooltimeoutexception: timeout waiting for connection troubleshooting

Today, an httpclient exception is solved. Khan, a httpclient can be destroyed if it is used accidentally.

Here is a previously handled server exception caused by improper route configuration: http://blog.csdn.net/shootyou/article/details/6415248

The httpconnectionmanager implementation is what I use here.

Problem:

A large number of exceptions are found in Tomcat background logs.

org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection

After a long period of time, Tomcat cannot continue to process other requests, from being suspended to actually dead.

Linux:

netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

It is found that the number of close_wait is always above 400, and it has never been downgraded.

Problem Analysis:

At first, I was confident in my httpclient usage process. I don't think the exception is from here.

So I started to start with the TCP connection status and guessed the possible causes of exceptions. In the past, we often encountered a server exception caused by a large number of time_wait instances, which is easy to solve. Modify sysctl and you will be OK. But this is close_wait, which is a completely different concept.

For the difference and Exception Handling between time_wait and close_wait, I will write an article to describe my understanding in detail.

To put it simply, the excessive number of close_wait is caused by improper handling of passive closed connections.

In a scenario, server a requests Apache on server B to obtain file resources. Normally, if the request is successful, after capturing the resource, server a will send a request to close the connection. In this case, the connection is closed. The connection status is time_wait. What if an exception occurs? Assume that the requested resource server B does not exist. In this case, server B will send a request to close the connection. server a passively closes the connection, if Server A does not release the connection after it closes the connection passively, it will cause the close_wait status.

So it is obvious that the problem is still in the program.

Let's take a look at my httpconnectionmanager implementation:

Public class httpconnectionmanager {Private Static httpparams; Private Static clientconnectionmanager connectionmanager;/*** maximum number of connections */public final static int max_total_connections = 800; /*** get the maximum wait time for the connection */public final static int wait_timeout = 60000;/*** maximum number of connections per route */public final static int max_route_connections = 400; /*** connection timeout */public final static int connect_timeout = 10000;/***** read timeout */public final static int read_timeout = 10000; static {httpparams = new basichttpparams (); // you can specify the maximum number of connmanagerparams. setmaxtotalconnections (httpparams, max_total_connections); // you can specify the maximum wait time for a connection. setTimeout (httpparams, wait_timeout); // set the maximum number of connections of each route. connperroutebean connperroute = new connperroutebean (max_route_connections); connmanagerparams. setmaxconnectionsperroute (httpparams, connperroute); // you can specify the connection timeout value for httpconnectionparams. setconnectiontimeout (httpparams, connect_timeout); // you can specify the read timeout value for httpconnectionparams. setsotimeout (httpparams, read_timeout); schemeregistry registry = new schemeregistry (); registry. register (New Scheme ("HTTP", plainsocketfactory. getsocketfactory (), 80); registry. register (New Scheme ("HTTPS", sslsocketfactory. getsocketfactory (), 443); connectionmanager = new threadsafeclientconnmanager (httpparams, registry);} public static httpclient gethttpclient () {return New defaulthttpclient (connectionmanager, httpparams );}}

We can see that max_route_connections is exactly 400, which is very close to close_wait. Is it a coincidence? Continue.

Then let's look at the code that calls it:

public static String readNet (String urlPath){StringBuffer sb = new StringBuffer ();HttpClient client = null;InputStream in = null;InputStreamReader isr = null;try{client = HttpConnectionManager.getHttpClient();HttpGet get = new HttpGet();            get.setURI(new URI(urlPath));            HttpResponse response = client.execute(get);            if (response.getStatusLine ().getStatusCode () != 200) {                return null;            }            HttpEntity entity =response.getEntity();                        if( entity != null ){            in = entity.getContent();            .....            }            return sb.toString ();}catch (Exception e){e.printStackTrace ();return null;}finally{if (isr != null){try{isr.close ();}catch (IOException e){e.printStackTrace ();}}if (in != null){try{in.close ();}catch (IOException e){e.printStackTrace ();}}}}

It is very simple, that is, a method to remotely read Chinese pages. It is worth noting that this piece of code was added by a later employee. It seems that there is no problem. It is used for non-200 state exception handling:

if (response.getStatusLine ().getStatusCode () != 200) {                return null;            }

The code itself is fine, but the problem is that the location is wrong. If this is the case, there will be no problem:

client = HttpConnectionManager.getHttpClient();HttpGet get = new HttpGet();            get.setURI(new URI(urlPath));            HttpResponse response = client.execute(get);                        HttpEntity entity =response.getEntity();                        if( entity != null ){            in = entity.getContent();            ..........            }                        if (response.getStatusLine ().getStatusCode () != 200) {                return null;            }            return sb.toString ();

See the problem. In this article (httpclient4.x upgrade entry + HTTP connection pool usage), I mentioned that httpclient4 uses our frequently-used inputstream. close () to confirm that the connection is closed. inputstream in is not assigned a value at all, which means that once a non-200 connection occurs, the connection will always be stuck in the connection pool, that's terrible... So we can see that the number of clost_wait is 400, because the connections to a route have been completely occupied by dead connections...

In fact, the above Code still has a problem, and the exception handling is not rigorous enough. So I finally changed the code to this:

public static String readNet (String urlPath){StringBuffer sb = new StringBuffer ();HttpClient client = null;InputStream in = null;InputStreamReader isr = null;HttpGet get = new HttpGet();try{client = HttpConnectionManager.getHttpClient();            get.setURI(new URI(urlPath));            HttpResponse response = client.execute(get);            if (response.getStatusLine ().getStatusCode () != 200) {            get.abort();                return null;            }            HttpEntity entity =response.getEntity();                        if( entity != null ){            in = entity.getContent();            ......            }            return sb.toString ();}catch (Exception e){get.abort();e.printStackTrace ();return null;}finally{if (isr != null){try{isr.close ();}catch (IOException e){e.printStackTrace ();}}if (in != null){try{in.close ();}catch (IOException e){e.printStackTrace ();}}}}

Show the abort that calls httpget. This will stop the connection directly. When an exception occurs, we should display the call. Who can ensure that the exception is thrown after the value of inputstream in.

Now, the analysis is complete. We will summarize the differences between close_wait and time_wait tomorrow.