Using Java httpclient to implement multithreaded concurrency _java

Description: The following code is implemented based on httpclient4.5.2.

We want to use Java's httpclient implementation GET request Crawl page is a relatively easy to achieve work:

 public static String get (string url) {closeablehttpresponseresponse = null;
    BufferedReader in = null;
    String result = "";
      try {closeablehttpclienthttpclient = Httpclients.createdefault ();
      Httpgethttpget = new HttpGet (URL);
 
      Response = Httpclient.execute (HttpGet);
      in = new BufferedReader (New InputStreamReader (Response.getentity (). GetContent ()));
      STRINGBUFFERSB = new StringBuffer ("");
      String line = "";
      String NL = System.getproperty ("Line.separator");
      while (line = In.readline ())!= null) {Sb.append (line + NL);
      } in.close ();
    result = Sb.tostring ();
    catch (IOException e) {e.printstacktrace ();
      Finally {try {if (null!= response) Response.close ();
      catch (IOException e) {e.printstacktrace ();
  } return result; }

The method above is also available when you want multiple threads to execute a GET request. However, this multi-threaded request is based on the creation of a httpclient instance each time the Get method is invoked. Each httpclient instance is reclaimed once it is used. This is obviously not an optimal implementation.

HttpClient provides a multithreaded request scenario that allows you to view the section of the official document, "Pooling Connection manager." HttpClient implementation of multi-threaded requests is based on a built-in connection pool, with a key class called Poolinghttpclientconnectionmanager, which manages the httpclient connection pool. There are two key methods available in Poolinghttpclientconnectionmanager: Setmaxtotal and Setdefaultmaxperroute. Setmaxtotal sets the maximum number of connections for a connection pool, Setdefaultmaxperroute sets the number of default connections on each route. In addition, there is a way to set the maximum number of connections for a site individually, like this: setmaxperroute--

   Httphosthost = new Httphost ("Locahost");
   Cm.setmaxperroute (New Httproute (host), 50);

Our GET request implementation is slightly adjusted according to the document:

Package com.zhyea.robin;
Import Org.apache.http.client.methods.CloseableHttpResponse;
Import Org.apache.http.client.methods.HttpGet;
Import org.apache.http.impl.client.CloseableHttpClient;
Import org.apache.http.impl.client.HttpClients;
 
Import Org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
Import Java.io.BufferedReader;
Import java.io.IOException;
 
Import Java.io.InputStreamReader;
 
  public class Httputil {private static closeablehttpclienthttpclient;
    static {POOLINGHTTPCLIENTCONNECTIONMANAGERCM = new Poolinghttpclientconnectionmanager ();
    Cm.setmaxtotal (200);
    Cm.setdefaultmaxperroute (20);
    Cm.setdefaultmaxperroute (50);
  HttpClient = Httpclients.custom (). Setconnectionmanager (CM). Build ();
    public static string get (string url) {closeablehttpresponseresponse = null;
    Bufferedreaderin = null;
    String result = "";
      try {httpgethttpget = new HttpGet (URL);
 
      Response = Httpclient.execute (HttpGet); in = new BufferedReader (New InputStreamReader (Response.getentity (). GetContent ()));
      STRINGBUFFERSB = new StringBuffer ("");
      String line = "";
      String NL = System.getproperty ("Line.separator");
      while (line = In.readline ())!= null) {Sb.append (line + NL);
      } in.close ();
    result = Sb.tostring ();
    catch (IOException e) {e.printstacktrace ();
      Finally {try {if (null!= response) Response.close ();
      catch (IOException e) {e.printstacktrace ();
  } return result;
  public static void Main (string[] args) {System.out.println ("https://www.baidu.com/");
 }
}

That's almost it. But for me, I prefer the httpclient fluent implementation, such as the HTTP GET request we just implemented, which can be implemented in such a simple way:

Package com.zhyea.robin;
 
Import Org.apache.http.client.fluent.Request;
Import java.io.IOException;
 
public class Httputil {public
 
  static string get (string url) {
    string result = ' ";
    try {result
      = Request.get (URL)
          . ConnectTimeout (1000).
          sockettimeout (1000).
          execute (). Returncontent (). asstring ();
    } catch (IOException e) {
      e.printstacktrace ();
    }
    return result;
  }
 
  public static void Main (string[] args) {
    System.out.println ("https://www.baidu.com/");
  }

All we have to do is replace the previous httpclient dependencies with FLUENT-HC dependencies:

<dependency>
   <groupId>org.apache.httpcomponents</groupId>
   <artifactId> fluent-hc</artifactid>
   <version>4.5.2</version>
</dependency>

And this fluent implementation of natural is to use Poolinghttpclientconnectionmanager completed. It sets the values of Maxtotal and Defaultmaxperroute to be 200 and 100, respectively:

    Connmgr = new Poolinghttpclientconnectionmanager (SFR);
    Connmgr.setdefaultmaxperroute (m);
    Connmgr.setmaxtotal (200);

The only one by one thing that makes people uncomfortable is that executor doesn't provide a way to adjust these two values. But that's all it takes, and if it doesn't, consider rewriting the executor method and then executing the GET request directly using executor:

Executor.newinstance (). Execute (request.get (URL))
        . Returncontent (). asstring ();

That's it!