httpclient4.x Upgrade entry + HTTP connection pool usage

Reprint please indicate the source, thank you ~

Http://blog.csdn.net/shootyou/archive/2011/05/12/6415248.aspx

During the troubleshooting of a server exception (the process of Troubleshooting server exceptions), we decided to use httpclient4.x instead of httpclient3.x or httpconnection.

Why use HTTPCLIENT4. The main httpconnection is the concept of no connection pool, how many IO the number of requests will be, and the server's IO may be depleted in the event of a huge amount of traffic.

HTTPCLIENT3 also has a connection to the pool of things in the inside, using Multithreadedhttpconnectionmanager, the approximate process is as follows:

Multithreadedhttpconnectionmanager ConnectionManager = new Multithreadedhttpconnectionmanager ();
HttpClient client = new HttpClient (ConnectionManager);.//On a thread.
GetMethod get = new GetMethod ("http://jakarta.apache.org/");
try {
client.executemethod (get);//print response to stdout
System.out.println Get.getresponsebodyasstream ( ));
} finally {
//Be sure the connection is released the connection 
managerget.releaseconnection ();
}

As you can see, it's in the same way that the JDBC connection pool is used, and I find it rather uncomfortable to manually invoke Releaseconnection to release the connection. Each httpclient.executemethod must have a method.releaseconnection () to match it.

HTTPCLIENT4 has improved on this point by using our usual inputstream.close () to confirm that the connection is closed (use Entity.consumecontent () before version 4.1 to confirm that the content has been consumed and closed). The specific methods are as follows:

... HttpClient client = Null;inputstream in = null;
try{
client = Httpconnectionmanager.gethttpclient ();
HttpGet get = new HttpGet ();
Get.seturi (New URI (URLPath));
HttpResponse response = Client.execute (get);
Httpentity entity =response.getentity ();
if (entity!= null) {in 
 = Entity.getcontent ();
 ....
} catch (Exception e) {
...
} finally{
if (in!= null) {
try{in.close ();} catch (IOException e) {
e.printstacktrace ();}}}

2012-03-06 Update:

A netizen proposed to call In.close () will shut down the underlying socket, things like this:

Reply kangkang203: Thank you for this question.
first of all, the method In.close () It will trigger a connection to release the connection will be resumed by the connection Manager, the official website of the original said: "Closing the input stream would trigger connection Release...the underlying connection gets released back to the connection manager. But the bottom of the socket will be closed is not necessarily, I saw some source code (Eofsensorinputstream) found that most of the socket does not close, and whether to close the socket seems to be a watcher to decide. So the in.close call does not cause the socket to close. In
addition, since HTTP itself is considered a "short connection", it is not very important to open the socket after a single request interaction, after all, it is not as long as a connection can have many data interactions after it is established. The more significance of our trial Connection Manager is its management of connections.

We're done. The usage flow of the connection pool is now a few of the most important parameters for the connection pool in use. I implemented a simple Httpconnectionmanager with the 4.1 version, and the code was as follows:

public class Httpconnectionmanager {private static httpparams httpparams;

	private static Clientconnectionmanager ConnectionManager;
	/** * Maximum Connection number * * Public final static int max_total_connections = 800;
	/** * Get the maximum waiting time for the connection/public final static int wait_timeout = 60000;
	/** * The maximum number of connections per route * * Public final static int max_route_connections = 400;
	/** * Connection Timeout/public final static int connect_timeout = 10000;

	/** * Read Timeout/public final static int read_timeout = 10000;
		static {httpparams = new basichttpparams ();
		Set the maximum number of connections connmanagerparams.setmaxtotalconnections (Httpparams, max_total_connections);
		Sets the maximum wait time connmanagerparams.settimeout (Httpparams, wait_timeout) to get the connection;
		Set the maximum number of connections per route Connperroutebean Connperroute = new Connperroutebean (max_route_connections);
		Connmanagerparams.setmaxconnectionsperroute (Httpparams,connperroute);
		Set the connection timeout httpconnectionparams.setconnectiontimeout (httpparams, connect_timeout); Set the Read timeout httpconnEctionparams.setsotimeout (Httpparams, read_timeout);
		Schemeregistry Registry = new Schemeregistry ();
		Registry.register (New Scheme ("http", Plainsocketfactory.getsocketfactory (), 80));

		Registry.register (New Scheme ("https", Sslsocketfactory.getsocketfactory (), 443));
	ConnectionManager = new Threadsafeclientconnmanager (Httpparams, registry);
	public static HttpClient Gethttpclient () {return new defaulthttpclient (ConnectionManager, httpparams); }

}

Maximum number of connections, maximum wait time for a connection, read timeout these configurations should be easier to understand, and the general connection pool will have these configurations, and more specifically, the maximum number of connections per route (route).

What is a route.

Here the concept of route can be understood as a line running the environment machine to the target machine. For example, we use the implementation of HttpClient to request www.baidu.com resources and www.bing.com resources respectively, so he generates two route.

Why specifically mention the route maximum connection number This parameter, because the default value of this parameter is 2, if you do not set this parameter value by default for the same target machine maximum concurrent connection only 2. This means that if you are performing a crawl task for a particular target machine, even if you set the maximum number of connections for the connection pool to 200, there are actually only 2 connections working, and the remaining 198 connections are waiting for other target machines.

How does the egg ache, I have already had the blood the lesson, in the switch to HttpClient4.1 at the beginning did not notice this configuration, finally causes the service to withstand the pressure to be inferior to formerly, therefore here specially reminds everybody to notice.

httpclient4.x Tutorial Downloads:

http://svn.apache.org/repos/asf/httpcomponents/httpclient/trunk/httpclient-contrib/docs/translated-tutorial/ Httpclient-tutorial-simplified-chinese.pdf

about the version of the supplement:

Netizen w2449008821 Reminder I only then found in httpclient4.1+ version connmanagerparams has been deprecated.

I was writing this log when the httpclient version is 4.0.3, from the 4.0 version Connmanagerparams was deprecated, did not expect a small version of the upgrade will have such a big change.

The new connection pool settings are illustrated in the official website tutorial:

Schemeregistry schemeregistry = new Schemeregistry ();
Schemeregistry.register (
         New Scheme ("http", Plainsocketfactory.getsocketfactory ());
Schemeregistry.register (
         new Scheme ("https", 443, Sslsocketfactory.getsocketfactory ());

Threadsafeclientconnmanager cm = new Threadsafeclientconnmanager (schemeregistry);
Increase Max total connection to
cm.setmaxtotalconnections (MB);
Increase default Max connection per route to
Cm.setdefaultmaxperroute;
Increase Max connections
for localhost:80 to httphost localhost = new Httphost ("Locahost");
Cm.setmaxforroute (new Httproute (localhost), m);
 
HttpClient httpclient = new defaulthttpclient (cm);

The Connmanagerparams function was moved to the Threadsafeclientconnmanager and httpconnectionparams two classes:

Static Connperroute	Getmaxconnectionsperroute (Httpparams params) deprecated. Use Threadsafeclientconnmanager.getmaxforroute (Org.apache.http.conn.routing.HttpRoute)
static int	getmaxtotalconnections (Httpparams params) deprecated. Use Threadsafeclientconnmanager.getmaxtotal ()
Static long	GetTimeout (Httpparams params) deprecated. Use Httpconnectionparams.getconnectiontimeout (httpparams)
static void	Setmaxconnectionsperroute (Httpparams params, Connperroute connperroute) deprecated. Use Threadsafeclientconnmanager.setmaxforroute (org.apache.http.conn.routing.HttpRoute, int)
static void	setmaxtotalconnections (httpparams params, int maxtotalconnections) deprecated. Use Threadsafeclientconnmanager.setmaxtotal (int)
static void	settimeout (Httpparams params, long timeout) deprecated. Use Httpconnectionparams.setconnectiontimeout (httpparams, int)

Reference: http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/params/ConnManagerParams.html

http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d4e638