HttpClient Connection pool throws a large number of connectionpooltimeoutexception:timeout waiting for connection exception troubleshooting

Today solved a httpclient abnormal, Khan Ah, a httpclient use a little careless will be destroyed level AH.

Here are a few previous processing of server exceptions due to improper route configuration: http://blog.csdn.net/shootyou/article/details/6415248

The Httpconnectionmanager implementation inside is the implementation that I use here.

Problem Performance:

Tomcat background log finds a large number of exceptions

Org.apache.http.conn.ConnectionPoolTimeoutException:Timeout Waiting for connection

For a long time Tomcat cannot continue to process other requests, from suspended animation to real death.

Linux running:

Netstat-n | awk '/^tcp/{++s[$NF]} end {for (a in S) print A, s[a]} '

found that the number of close_wait is always above 400, has not been down.

Problem Analysis:

At first I was convinced of my httpclient use and I don't think the anomaly came from here.

So I started with the TCP connection state to guess what might have caused the exception. Before often encountered time_wait caused by the server anomaly, it is easy to solve, modify the sysctl on OK. But this time it's close_wait, it's a completely different concept.

About the difference and exception handling of time_wait and close_wait I'll have a separate article to say my understanding in detail.

Simply speaking, the close_wait number is too large due to improper passive shutdown of the connection.

I said a scene, server A will go to request Server B above Apache to obtain file resources, under normal circumstances, if the request succeeds, then after crawling resources Server A will actively send off the connection request, this time is the active shutdown connection, the connection state we can see is time_wait. What if an exception occurs. Assuming that the requested resource Server B does not exist, this time will be the Server B issued a shutdown request, server A is passively shut down the connection, if the server a passively shut down the connection after they did not release the connection, it will cause close_wait state.

So obviously, the problem is still in the process.

First look at my Httpconnectionmanager implementation:

public class Httpconnectionmanager {private static httpparams httpparams;

	private static Clientconnectionmanager ConnectionManager;
	/** * Maximum Connection number * * Public final static int max_total_connections = 800;
	/** * Get the maximum waiting time for the connection/public final static int wait_timeout = 60000;
	/** * The maximum number of connections per route * * Public final static int max_route_connections = 400;
	/** * Connection Timeout/public final static int connect_timeout = 10000;

	/** * Read Timeout/public final static int read_timeout = 10000;
		static {httpparams = new basichttpparams ();
		Set the maximum number of connections connmanagerparams.setmaxtotalconnections (Httpparams, max_total_connections);
		Sets the maximum wait time connmanagerparams.settimeout (Httpparams, wait_timeout) to get the connection;
		Set the maximum number of connections per route Connperroutebean Connperroute = new Connperroutebean (max_route_connections);
		Connmanagerparams.setmaxconnectionsperroute (Httpparams,connperroute);
		Set the connection timeout httpconnectionparams.setconnectiontimeout (httpparams, connect_timeout); Set the Read timeout httpconnEctionparams.setsotimeout (Httpparams, read_timeout);
		Schemeregistry Registry = new Schemeregistry ();
		Registry.register (New Scheme ("http", Plainsocketfactory.getsocketfactory (), 80));

		Registry.register (New Scheme ("https", Sslsocketfactory.getsocketfactory (), 443));
	ConnectionManager = new Threadsafeclientconnmanager (Httpparams, registry);
	public static HttpClient Gethttpclient () {return new defaulthttpclient (ConnectionManager, httpparams); }

}

See no max_route_connections happened to be 400, with close_wait very close Ah, it is a coincidence. Keep looking down. Then look at the code that calls it:

public static string Readnet (String urlpath) {StringBuffer sb = new StringBuffer ();
		HttpClient client = null;
		InputStream in = null;
		InputStreamReader ISR = null;
			try {client = Httpconnectionmanager.gethttpclient ();
            HttpGet get = new HttpGet ();
            Get.seturi (New URI (URLPath));
            HttpResponse response = Client.execute (get);
            if (Response.getstatusline (). Getstatuscode ()!=) {return null;
            
            } httpentity entity =response.getentity ();
            	if (entity!= null) {in = Entity.getcontent ();
            .....
            }
			
		return sb.tostring ();
			catch (Exception e) {e.printstacktrace ();
		return null;
				Finally {if (ISR!= null) {try {isr.close ();
				catch (IOException e) {e.printstacktrace ();
				} if (in!= null) {try {in.close ();
	catch (IOException e) {				E.printstacktrace (); }
			}
		}
	}

Very simple, is a remote reading of the Chinese page method. It is worth noting that this piece of code is later and so classmate plus go, it seems that there is no problem, is for the exception of the 200 state of processing:

if (Response.getstatusline (). Getstatuscode ()!=) {return
                null;
            }

The code itself is not a problem, but the problem is misplaced. If that's the case, there's no problem:

Client = Httpconnectionmanager.gethttpclient ();
			HttpGet get = new HttpGet ();
            Get.seturi (New URI (URLPath));
            HttpResponse response = Client.execute (get);
            
            Httpentity entity =response.getentity ();
            
            if (entity!= null) {in
            	= Entity.getcontent ();
            ..........
            }
            
            if (Response.getstatusline (). Getstatuscode ()!=) {return
                null;
            }
            return sb.tostring ();

Let's see what's wrong. In this primer (httpclient4.x upgrade introduction + HTTP connection pool usage) I mentioned that HttpClient4 use our usual inputstream.close () to confirm the connection is off, the way the previous wording InputStream in Will not be assigned at all, which means that once a non 200 connection is made, the connection will be permanently dead in the connection pool. So we see that the number of clost_wait is 400, because the connection to a route has been completely filled with zombie connections ...

In fact, the above code there is not a good place to deal with, exception handling is not rigorous, so finally I changed the code to this:

public static string Readnet (String urlpath) {StringBuffer sb = new StringBuffer ();
		HttpClient client = null;
		InputStream in = null;
		InputStreamReader ISR = null;
		HttpGet get = new HttpGet ();
            try {client = Httpconnectionmanager.gethttpclient ();
            Get.seturi (New URI (URLPath));
            HttpResponse response = Client.execute (get);
                if (Response.getstatusline (). Getstatuscode ()!=) {get.abort ();
            return null;
            
            } httpentity entity =response.getentity ();
            	if (entity!= null) {in = Entity.getcontent ();
            ......
            }
			
		return sb.tostring ();
			catch (Exception e) {get.abort ();
			E.printstacktrace ();
		return null;
				Finally {if (ISR!= null) {try {isr.close ();
				catch (IOException e) {e.printstacktrace (); } if (in!= null) {try {in.close ();
				catch (IOException e) {e.printstacktrace (); }
			}
		}
	}

Shows the call to the HttpGet abort, which will abort the connection directly, we should show the call when we encounter an exception, because who can guarantee that the exception is thrown after inputstream in assignment.

Well, the analysis is finished, tomorrow is ready to summarize the difference between close_wait and time_wait.