HTTP requests based on the thread pool and connection pool

Background: The latest project needs to invoke the HTTP interface, so I intend to use the latest HttpClient client to write a tool class, write well later in the actual application process encountered some problems, because the amount of data

Large, each processing about 600-700 requests, the average count down to 20 minutes, although the speed is running in the scheduled task, but also unbearable, so with this blog.

1. The first thought of the solution is the multi-line Cheng request, but this has a pit, the end will be explained in the results.

2. The code aspect is as follows

Executorservice executor = Executors.newfixedthreadpool (5); Futuretask<order> future;        for (Transactionrecord record:list) {            final String OrderNo = Record.getorderno ();            Future = new Futuretask<order> (new Ordertask (OrderNo));            Executor.submit (future);            Futurelist.add (future);        }    Class Ordertask implements callable<order> {        private String OrderNo;        Public Ordertask (String orderno) {            This.orderno = OrderNo;        }        @Override Public        Order call () throws Exception {            Order order = Orderservice.getorder (OrderNo); In GetOrder, call the following I-Encapsulated HTTP tool class            return order;}    }

This is a very simple multithreaded code, but there is a pit to be aware of, do not call the Future.get () method in the above loop directly, if the direct call will become blocked directly, and single-threaded

There is no difference, you can write a demo to test the efficiency.

3.http aspects of the code, can be all posted, as follows

   /** * Get */public static httpresultentry doPost (String URL, map<string, string> params,     String CharSet) throws Exception {httpresultentry resultentry = new Httpresultentry ();       Custom return result closeablehttpresponse response = null;        Returns the result, releasing the link list<namevaluepair> pairs = new arraylist<> ();        ;            Parameter if (Stringutils.isblank (URL)) {resultentry.setmsg ("Request address exception");            Resultentry.setstatus (404);            Resultentry.setdata ("");        return resultentry; } try {if (params! = null &&!params.isempty ()) {for (map.entry<string, Stri                    Ng> Entry:params.entrySet ()) {String value = Entry.getvalue ();                    if (value! = null) {Pairs.add (New Basicnamevaluepair (Entry.getkey (), value)); }}} HTTPPOST httppost = new HttpPost (URL);            Httppost.setentity (new urlencodedformentity (pairs, CharSet));     Response = Httpclient.execute (HttpPost);      Establish a link to get the returned result int statusCode = Response.getstatusline (). Getstatuscode ();                Returns the result code if (statusCode! =) {Httppost.abort ();                Resultentry.setmsg ("request Exception");                Resultentry.setstatus (StatusCode);                Resultentry.setdata ("");                Logger.info ("return exception: {}", resultentry);            return resultentry;            } httpentity httpentity = Response.getentity ();            String result = null;                if (httpentity = = null) {resultentry.setmsg ("return result exception");                Resultentry.setstatus (StatusCode);                Resultentry.setdata ("");            return resultentry;            } else {result = Entityutils.tostring (httpentity, CharSet);   } resultentry.setmsg ("Request Normal");         Resultentry.setstatus (StatusCode);            Resultentry.setdata (result);        Entityutils.consume (httpentity);            According to the official document: both released before the normal release of the link Response.close ();        return resultentry;            } catch (Exception e) {logger.error ("Request error: {}, error message: {}", E.getmessage (), E);        throw new Exception ("HTTP request exception");                } finally {if (response! = null) {try {response.close ();                } catch (IOException e) {logger.error ("Close stream exception: {}, error message: {}", E.getmessage (), E); }}}}/** * post */public static httpresultentry doget (String URL, map<string, St ring> params, String charset) throws Exception {Httpresultentry Resulten    try = new Httpresultentry ();       Custom return result closeablehttpresponse response = null; Return result, release link list<namevaluepair> pairs = new ARRAylist<> ();//Parameter if (Stringutils.isblank (URL)) {resultentry.setmsg ("Request address exception");            Resultentry.setstatus (404);            Resultentry.setdata ("");        return resultentry; } try {if (params! = null &&!params.isempty ()) {for (map.entry<string, Stri                    Ng> Entry:params.entrySet ()) {String value = Entry.getvalue ();                    if (value! = null) {Pairs.add (New Basicnamevaluepair (Entry.getkey (), value));            }} URL + = "?" + entityutils.tostring (new urlencodedformentity (pairs, CharSet));            } httpget HttpGet = new HttpGet (URL);     Response = Httpclient.execute (HttpGet);            Establish a link to get the returned result int statusCode = Response.getstatusline (). Getstatuscode ();                if (statusCode! =) {Httpget.abort ();                Resultentry.setmsg ("request Exception");Resultentry.setstatus (StatusCode);                Resultentry.setdata ("");                Logger.info ("return exception: {}", resultentry);            return resultentry;            } httpentity httpentity = Response.getentity ();            String result = null;                if (httpentity = = null) {resultentry.setmsg ("return result exception");                Resultentry.setstatus (StatusCode);                Resultentry.setdata ("");            return resultentry;            } else {result = Entityutils.tostring (httpentity, CharSet);            } resultentry.setmsg ("Request Normal");            Resultentry.setstatus (StatusCode);            Resultentry.setdata (result);        Entityutils.consume (httpentity);            According to the official document: both released before the normal release of the link Response.close ();        return resultentry;            } catch (Exception e) {logger.error ("Request error: {}, error message: {}", E.getmessage (), E);        throw new Exception ("HTTP request exception");       } finally {     if (response! = null) {try {response.close ();                } catch (IOException e) {logger.error ("Close stream exception: {}, error message: {}", E.getmessage (), E); }            }        }    }

The use of the HTTP connection pool, the code of the connection pool is simply not pasted, first use must pay attention to the final release work, must be Httpentry and Respose are released, according to official documents, the only way to really release the link, that is, this link can be reused.

Summary: Need to pay special attention to the other people's HTTP interface must not open too many threads, so that the other people's interface to hang up, think I learned, I have access to an HTTP interface when I opened 100 threads, 666 requests ran 3.7 seconds or so, is very soon I am also very happy, and then the database can not stand the pressure, led to the alarm finally directly opened the white list, embarrassed, so use these when must consider these, open 35 is enough, in addition if too many threads, Tomcat server may be suspended animation Also, do not do so !

HTTP requests based on the thread pool and connection pool