Sample Code of HttpClient encapsulation in spring boot and httpclient sample code

Sample Code of HttpClient encapsulation in spring boot and httpclient sample code

I recently used HttpClient and read the official document: HttpClient implementations are expected to be thread safe. it is recommended that the same instance of this class is reused for multiple request executions, you can use the same instance to execute multiple requests. In this case, we should think of the need to encapsulate HttpClient. Because spring boot is used, the following uses spring boot to encapsulate HttpClient.

1. Request retry handler (Request retry processing)

To make the custom exception mechanism take effect, you need to implement the HttpRequestRetryHandler interface. The Code is as follows:

Import java. io. IOException; import java. io. interruptedIOException; import java.net. unknownHostException; import javax.net. ssl. SSLException; import javax.net. ssl. SSLHandshakeException; import org. apache. http. httpEntityEnclosingRequest; import org. apache. http. httpRequest; import org. apache. http. noHttpResponseException; import org. apache. http. client. httpRequestRetryHandler; import org. apache. http. clie Nt. protocol. httpClientContext; import org. apache. http. conn. connectTimeoutException; import org. apache. http. protocol. httpContext; import org. springframework. beans. factory. annotation. value; import org. springframework. context. annotation. bean; import org. springframework. context. annotation. configuration;/*** Description: HttpClient retry Processing Mechanism */@ Configuration public class MyhttpRequestRetryHandler {@ Value ("$ {htt Pclient. config. retryTime} ") // @ ConfigurationProperties (prefix =" httpclient. private int retryTime; @ Bean public HttpRequestRetryHandler httpRequestRetryHandler () {// request to retry final int retryTime = this. retryTime; return new HttpRequestRetryHandler () {public boolean retryRequest (IOException exception, int executionCount, HttpContext context) {// Do not retry if over max retry cou Nt. if the number of retries exceeds retryTime, the request if (executionCount> = retryTime) {return false;} // the server is disconnected from the client. if (exception instanceof NoHttpResponseException) {return true;} // time out retry if (exception instanceof InterruptedIOException) {return true;} // Unknown host if (exception instanceof UnknownHostException) {return false ;} // Connection refused if (exception instanceof ConnectTimeoutException) {Return false;} // SSL handshake exception if (exception instanceof SSLException) {return false;} HttpClientContext clientContext = HttpClientContext. adapt (context); HttpRequest request = clientContext. getRequest (); if (! (Request instanceof HttpEntityEnclosingRequest) {return true ;}return false ;}};}}

2. Pooling connection manager (connection pool management)

PoolingHttpClientConnectionManager is used to manage the connection pool of the client and provides services for requests from multiple threads. The Code is as follows:

Import org. apache. http. config. registry; import org. apache. http. config. registryBuilder; import org. apache. http. conn. socket. connectionSocketFactory; import org. apache. http. conn. socket. layeredConnectionSocketFactory; import org. apache. http. conn. socket. plainConnectionSocketFactory; import org. apache. http. conn. ssl. SSLConnectionSocketFactory; import org. apache. http. impl. conn. poolingHttpClientConnectionManager; import org. springframework. beans. factory. annotation. value; import org. springframework. context. annotation. bean; import org. springframework. context. annotation. configuration; @ Configuration public class MyPoolingHttpClientConnectionManager {/*** maximum number of connections in the connection pool */@ Value ("$ {httpclient. config. connMaxTotal} ") private int connMaxTotal = 20;/*****/@ Value (" $ {httpclient. config. maxPerRoute} ") private int maxPerRoute = 20;/*** connection survival time, in the unit of s */@ Value (" $ {httpclient. config. timeToLive} ") private int timeToLive = 60; @ Bean public PoolingHttpClientConnectionManager poolingClientConnectionManager () {PoolingHttpClientConnectionManager poolHttpcConnManager = new timeout (60, TimeUnit. SECONDS); // The maximum number of connections poolHttpcConnManager. setMaxTotal (this. connMaxTotal); // The Routing Base poolHttpcConnManager. setdefamaxmaxperroute (this. maxPerRoute); return poolHttpcConnManager ;}}

Note: When the HttpClient instance is no longer needed and is about to be out of range, it is important to close its connection manager to ensure that all connections maintained by the manager are closed, and release the system resources allocated by these connections.

The constructor of the PoolingHttpClientConnectionManager class is as follows:

public PoolingHttpClientConnectionManager(final long timeToLive, final TimeUnit tunit) {     this(getDefaultRegistry(), null, null ,null, timeToLive, tunit);   }  private static Registry<ConnectionSocketFactory> getDefaultRegistry() {     return RegistryBuilder.<ConnectionSocketFactory>create()         .register("http", PlainConnectionSocketFactory.getSocketFactory())         .register("https", SSLConnectionSocketFactory.getSocketFactory())         .build();   } 

In the PoolingHttpClientConnectionManager configuration, there are two maximum connections, respectively controlling the total maximum number of connections and the maximum number of connections for each route. If no explicit setting is set, only two connections are allowed for each route by default, and the total number of connections cannot exceed 20. This value is not enough for many applications with high concurrency. You must set an appropriate value based on the actual situation. The idea is similar to the thread pool size setting method, if all the connection requests are to the same url, you can set the value of MaxPerRoute to be consistent with that of MaxTotal, so that the connection can be reused more efficiently.

Note: to reuse a connection, you must release the system resources it occupies as follows:

If you use outputStream, ensure that the entire entity is written out. If you use inputStream, remember to call inputStream. close (). You can also use EntityUtils. consume (entity) or EntityUtils. consumeQuietly (entity) to completely exhaust the entity (the latter does not throw an exception) for this purpose. EntityUtils has a toString method, which is also very convenient (the inputStream will be closed automatically after this method is called, but in the actual test process, the connection will not be released ), however, it can be used only when it is determined that the received entity is not very large. If the entire entity is not fully consumed, the connection cannot be reused, soon it will be because no available connections can be obtained in the connection pool timeout or blocked here (because the connection status will always be leased, that is, the status being used ). Therefore, if you want to reuse the connection, you must remember to remove entity fully consume. As long as the stream eof is detected, the releonholder releaseConnection method will be automatically called for processing.

Iii. Connection keep alive strategy (keep Connection Policy)

The HTTP specification does not specify the possibility of persistent connections and how long they should survive. Some HTTP servers use non-standard Keep-Alive headers to communicate with the client. They intend to maintain the connection period on the server side (in seconds ). HttpClient can use this information. If the response does not contain the Keep-Alive header, HttpClient assumes that the connection can be active indefinitely. However, many HTTP servers are configured to delete persistent connections after an inactive status, saving system resources without notifying the client. If the Default policy is too optimistic, you may need to provide a custom persistence strategy. The Code is as follows:

Import org. apache. http. headerElement; import org. apache. http. headerElementIterator; import org. apache. http. httpResponse; import org. apache. http. conn. connectionKeepAliveStrategy; import org. apache. http. message. basicHeaderElementIterator; import org. apache. http. protocol. HTTP; import org. apache. http. protocol. httpContext; import org. springframework. beans. factory. annotation. value; import org. springframe Work. context. annotation. bean; import org. springframework. context. annotation. configuration;/*** Description: Connection persistence policy * @ author chhliu */@ Configuration public class MyconnectionKeepAliveStrategy {@ Value ("$ {httpclient. config. keepAliveTime} ") private int keepAliveTime = 30; @ Bean (" connectionKeepAliveStrategy ") public ConnectionKeepAliveStrategy connectionKeepAliveStrategy () {return new ConnectionKeepAliveSt Rategy () {public long getKeepAliveDuration (HttpResponse response, HttpContext context) {// Honor 'keep-alive' header HeaderElementIterator it = new BasicHeaderElementIterator (response. headerIterator (HTTP. CONN_KEEP_ALIVE); while (it. hasNext () {HeaderElement he = it. nextElement (); String param = he. getName (); String value = he. getValue (); if (value! = Null & param. equalsIgnoreCase ("timeout") {try {return Long. parseLong (value) * 1000;} catch (NumberFormatException ignore) {}} return 30*1000 ;}};}}

Note: persistent connections are not used in all cases. In particular, most of the current systems are deployed on multiple servers and have the Server Load balancer function, persistent connections are always maintained. Once the server goes down, the client will be affected. At the same time, the Server Load balancer feature cannot be fully utilized. On the contrary, transient connections are more advantageous, these need to be determined based on specific needs, rather than simply generalization.

Iv. HttpClient proxy configuration (proxy configuration)

The Code is as follows:

Import org. apache. http. httpHost; import org. apache. http. impl. conn. defaultProxyRoutePlanner; import org. springframework. beans. factory. annotation. value; import org. springframework. context. annotation. bean; import org. springframework. context. annotation. configuration;/*** Description: HttpClient proxy * @ author chhliu */@ Configuration public class MyDefaultProxyRoutePlanner {// proxy host address @ Value ("$ {httpclient. config. proxyhost} ") private String proxyHost; // proxy port number @ Value (" $ {httpclient. config. proxyPort} ") private int proxyPort = 8080; @ Bean public DefaultProxyRoutePlanner defaultProxyRoutePlanner () {HttpHost proxy = new HttpHost (this. proxyHost, this. proxyPort); return new DefaultProxyRoutePlanner (proxy );}}

HttpClient not only supports simple direct connection, complex routing policies and proxies. HttpRoutePlanner is a route computing policy from the client to the server based on the http context. If there is no proxy, you do not need to set this policy. Here is a key concept-Route: In HttpClient, a Route refers to a line of the host of the running environment machine-> the target machine, that is, if the host of the target url is the same, so their route is the same.

5. RequestConfig

The Code is as follows:

import org.apache.http.client.config.RequestConfig; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration;  @Configuration public class MyRequestConfig {   @Value("${httpclient.config.connectTimeout}")   private int connectTimeout = 2000;      @Value("${httpclient.config.connectRequestTimeout}")   private int connectRequestTimeout = 2000;      @Value("${httpclient.config.socketTimeout}")   private int socketTimeout = 2000;   @Bean   public RequestConfig config(){     return RequestConfig.custom()         .setConnectionRequestTimeout(this.connectRequestTimeout)         .setConnectTimeout(this.connectTimeout)         .setSocketTimeout(this.socketTimeout)         .build();   } } 

RequestConfig is the configuration of the request. There are three most important time-out periods. By default, all three time-out periods are 0 (if the Config of the request is not set, it will use the default parameter in getRequestConfig of HttpClientParamConfig during the execute Process), which means unlimited waiting, which can easily cause all requests to be blocked in this place for an indefinite wait. The three timeout periods are:

A. connectionRequestTimeout-get the connection timeout value from the connection pool

This time defines the timeout time of the connection from the connection pool managed by ConnectionManager. If no connection is available in the connection pool, the request will be blocked and the maximum waiting time for connectionRequestTimeout will be reached, if the service is not available, the ConnectionPoolTimeoutException exception is thrown and the system does not wait.

B. connectTimeout-connection timeout

This time defines the timeout time for establishing a connection with the server through the network, that is, the waiting time for a connection to connect to the target url after a connection in the connection pool is obtained. A ConnectionTimeoutException exception is thrown when a timeout occurs.

C. socketTimeout-request timeout

This time defines the timeout time for the socket to read data, that is, the time required to wait for the response data to be obtained from the server after the connection to the server, or connect to the previous url and wait for the response to be returned. A SocketTimeoutException exception is thrown when a timeout occurs.

6. instantiate HttpClient

Implement FactoryBean to instantiate HttpClient. The Code is as follows:

Import org. apache. http. client. httpRequestRetryHandler; import org. apache. http. client. config. requestConfig; import org. apache. http. conn. connectionKeepAliveStrategy; import org. apache. http. impl. client. closeableHttpClient; import org. apache. http. impl. client. httpClients; import org. apache. http. impl. conn. defaultProxyRoutePlanner; import org. apache. http. impl. conn. poolingHttpClientConnectionManager; impor T org. springframework. beans. factory. disposableBean; import org. springframework. beans. factory. factoryBean; import org. springframework. beans. factory. initializingBean; import org. springframework. beans. factory. annotation. autowired; import org. springframework. stereotype. service;/*** Description: HttpClient client encapsulation */@ Service ("httpClientManagerFactoryBen") public class HttpClientManagerFactoryBen implements Facto RyBean <strong>, InitializingBean, DisposableBean {/*** FactoryBean generated target object */private jsonclient; @ Autowired private ConnectionKeepAliveStrategy connectionKeepAliveStrategy; @ Autowired private HttpRequestRetryHandler listener; @ Autowired private defaproproxyrouteplanner proxyRoutePlanner; @ Autowired private PoolingHttpClientConnectionManager poolHttpcCo NnManager; @ Autowired private RequestConfig config; // destroy HttpClient instance @ Override public void destroy () throws Exception {/** call httpClient. close () will first shut down the connection manager, and then release all resources occupied by the HttpClient, * close all connections in use or idle including the underlying socket. Because the connection manager used is disabled here, * You need to create a new connection manager to build an HttpClient next time you request an http request, * connection manager cannot be a singleton if you need to close or create a Client. */if (null! = This. client) {this. client. close () ;}@ Override // initialize the public void afterPropertiesSet () throws Exception {/** we recommend that you use HttpClients here. instead of using HttpClientBuilder. the create () method is used to create HttpClientBuilder *. According to the official documentation, HttpClientBuilder is non-thread-safe, but HttpClients does Immutable. immutable object not only ensures that the object state is not changed, * The lock mechanism can be shared by other threads without being used */this. client = HttpClients. custom (). setConnectionManager (poolHttpc ConnManager ). setRetryHandler (httpRequestRetryHandler ). setKeepAliveStrategy (connectionKeepAliveStrategy ). setRoutePlanner (proxyRoutePlanner ). setDefaultRequestConfig (config ). build () ;}// type of the returned instance @ Override public CloseableHttpClient getObject () throws Exception {return this. client ;}@ Override public Class <?> GetObjectType () {return (this. client = null? CloseableHttpClient. class: this. client. getClass ();} // The constructed instance is a singleton () {return true ;}}

7. Add a configuration file

# Proxy host httpclient. config. proxyhost = xxx. xx. xx. xx # proxy port httpclient. config. proxyPort = 8080 # connection timeout or number of exceptional retries httpclient. config. retryTime = 3 # long connection persistence time, in the unit of s httpclient. config. keepAliveTime = 30 # maximum number of connections in the connection pool httpclient. config. connMaxTotal = 20 httpclient. config. maxPerRoute = 20 # connection timeout, in ms httpclient. config. connectTimeout = 2000 # request timeout httpclient. config. connectRequestTimeout = 2000 # sock timeout time httpclient. config. socketTimeout = 2000 # connection survival time, in seconds httpclient. config. timeToLive = 60

8. Test

The test code is as follows:

Import java. io. IOException; import java. util. concurrent. executorService; import java. util. concurrent. executors; import javax. annotation. resource; import org. apache. http. consts; import org. apache. http. parseException; import org. apache. http. client. clientProtocolException; import org. apache. http. client. methods. closeableHttpResponse; import org. apache. http. client. methods. httpGet; import org. apache. http. Impl. client. closeableHttpClient; import org. apache. http. util. entityUtils; import org. junit. test; import org. junit. runner. runWith; import org. springframework. boot. test. context. springBootTest; import org. springframework. test. context. junit4.SpringRunner; @ RunWith (SpringRunner. class) @ SpringBootTest public class HttpClientManagerFactoryBenTest {// inject HttpClient instance @ Resource (name = "httpClientManagerFa CtoryBen ") private CloseableHttpClient client; @ Test public void test () throws ClientProtocolException, IOException, InterruptedException {ExecutorService service = Executors. newFixedThreadPool (2); for (int I = 0; I <10; I ++) {service. submit (new Runnable () {@ Override public void run () {System. out. println ("the current thread is:" + Thread. currentThread (). getName (); HttpEntity entity = null; try {Http Get get = new HttpGet ("https: // localhost: 8080/testjson"); // submit the request through the execute Command of httpclient, use CloseableHttpResponse to accept the response information CloseableHttpResponse response = client.exe cute (get); System. out. println ("client object:" + client); entity = response. getEntity (); System. out. println ("=============" + EntityUtils. toString (entity, Consts. UTF_8) + "================"); EntityUtils. consumeQuietly (entity); // release connection} catch (Clien TProtocolException e) {e. printStackTrace ();} catch (ParseException e) {e. printStackTrace ();} catch (IOException e) {e. printStackTrace ();} finally {if (null! = Entity) {// release connection EntityUtils. consumeQuietly (entity) ;}}}) ;} Thread. sleep (60000 );}}

Through the above steps, we have basically completed the HttpClient encapsulation. If you need more detailed information, you can follow the above ideas and gradually improve it to encapsulate HttpClient into HttpClientTemplate, closeableHttpClient uses the callback mechanism internally, which is similar to JdbcTemplate or RedisTemplate until the service can be provided in spring boot starter mode.

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.