On the importance of tuning thread pool in Java Web application _java

Whether you care or not, Java Web applications use a thread pool more or less to process requests. The implementation details of the thread pool may be overlooked, but the use and tuning of the thread pool will need to be understood sooner or later. This article mainly describes the use of the Java thread pool and how to properly configure the thread pool.

Single Thread

Let's start with the basics. Regardless of which application server or framework you use (such as Tomcat, jetty, etc.), they all have a similar underlying implementation. The Web service is based on a socket (socket), which is responsible for listening to the port, waiting for TCP connections, and accepting TCP connections. Once the TCP connection is accepted, data can be read and sent from the newly created TCP connection.

In order to understand the above process, we do not use any application server directly, but build a simple Web service from scratch. This service is the epitome of most application servers. A simple single threaded Web service is probably like this:

ServerSocket listener = new ServerSocket (8080);
try {while
 (true) {
  socket socket = listener.accept ();
  try {
   handlerequest (socket);
  } catch (IOException e) {
   e.printstacktrace ();}}}
finally {
 listener.close ();
}

The code above creates a service-side socket (serversocket) , listens on port 8080, and then loops through the socket to see if there is a new connection. Once a new connection is accepted, the socket is passed into the HandleRequest method. This method parses the data stream into an HTTP request, responds, and writes the response data. In this simple example, the HandleRequest method simply implements the reading of the data stream and returns a simple response data. In a common implementation, the method can be much more complex, such as reading data from a database.

Final static String response =
  "http/1.0 ok/r/n" +
  "content-type:text/plain/r/n" +
  "/r/n" +
  "Hello W orld/r/n ";

public static void HandleRequest (socket socket) throws IOException {
 //Read the input stream, and return "OK" 
   
    try {
  BufferedReader in = new BufferedReader (
   new InputStreamReader (Socket.getinputstream ()));
  Log.info (In.readline ());

  OutputStream out = Socket.getoutputstream ();
  Out.write (Response.getbytes (standardcharsets.utf_8));
 } finally {
  socket.close ();
 }
}


   

Because there is only one thread to process the request, each request must wait for the previous request to be processed before it can be responded to. Assuming a request response time of 100 milliseconds, the server's response number per second (TPS) is only 10.

Multithreading

Although the HandleRequest method may block Io, the CPU can still handle more requests. But in a single thread, this is not possible. As a result, you can increase the parallel processing capabilities of the server by creating multithreaded methods.

public static class Handlerequestrunnable implements Runnable {

 final socket socket;

 Public handlerequestrunnable (socket socket) {
  this.socket = socket;
 }

 public void Run () {
  try {
   handlerequest (socket);
  } catch (IOException e) {
   e.printstacktrace ();
  }
 }
}

ServerSocket listener = new ServerSocket (8080);
try {while
 (true) {
  socket socket = listener.accept ();
  New Thread (new handlerequestrunnable (socket)). Start ();
finally {
 listener.close ();
}

Here, the Accept () method is still invoked in the main thread, but once the TCP connection is established, a new thread is created to handle the new request, which executes the HandleRequest method in the previous article in the new thread.

By creating a new thread, the main thread can continue to accept new TCP connections, and these letters can be processed in parallel. This is referred to as "each request for a thread". Of course, there are other ways to improve processing performance, such as the asynchronous event-driven model used by NGINX and node.js, but they do not use a thread pool and therefore are not covered in this article.

Creating a thread (and subsequent destruction) overhead is very expensive in each request for a thread implementation, because both the JVM and the operating system need to allocate resources. In addition, the above implementation also has the problem that the number of threads created is not controllable, which can cause system resources to be quickly depleted.

Resource Exhaustion

Each thread requires a certain amount of stack memory space. In the most recent 64-bit JVM, the default stack size is 1024KB. If the server receives a large number of requests, or the HandleRequest method executes slowly, the server may crash because it created a large number of threads. For example, with 1000 concurrent requests, the 1000 threads created need to use 1GB of JVM memory as a line stacks space. In addition, objects created during the execution of each thread code may also create objects on the heap. This worsens, will go beyond the JVM heap memory, and generate a lot of garbage collection operations, eventually causing a memory overflow (outofmemoryerrors).

Not only do these threads consume memory, they also use other limited resources, such as file handles, database connections, and so on. An uncontrolled creation thread can also cause other types of errors and crashes. Therefore, an important way to avoid resource depletion is to avoid the uncontrolled data structure.

By the way, because of the memory problem caused by the thread stack size, you can adjust the stack size by using the-XSS switch. After reducing the thread stack size, you can reduce the cost per thread, but it may cause a stack overflow (stackoverflowerrors). For a typical application, the default 1024KB is too rich, and it may be more appropriate to reduce it to 256KB or 512KB. The minimum allowable value for Java is 160KB.

Thread pool

To avoid continuous creation of new threads, you can pool the upper limit of a thread pool by using a simple line. The thread pool manages the thread, and if the number of threads has not reached the upper limit, line Cheng creates the thread to the upper bound and, as much as possible, uses the idle thread.

ServerSocket listener = new ServerSocket (8080);
Executorservice executor = Executors.newfixedthreadpool (4);
try {while
 (true) {
  socket socket = listener.accept ();
  Executor.submit (new handlerequestrunnable (socket));
 }
finally {
 listener.close ();
}

In this example, the thread is not created directly, but Executorservice is used. It commits the task that needs to be performed (requiring implementation of the Runnables interface) to the thread pool and executes the code using threads in the thread pool. example, all requests are processed using a fixed size thread pool with a number of threads of 4. This limits the number of threads that are processing requests and also limits the use of resources.

In addition to creating a fixed size thread pool through the Newfixedthreadpool method, the Executors class also provides a Newcachedthreadpool method. The reuse thread pool may also cause an uncontrollable number of threads, but it will use the idle threads previously created as much as possible. Typically, this type of thread pool is suitable for short tasks that are not blocked by external resources.

Work queues

With a fixed size thread pool, if all the threads are busy, what happens when a new request is made? Threadpoolexecutor uses a queue to hold requests waiting to be processed, and the fixed size thread pool uses unrestricted linked lists by default. Note that this can cause resource exhaustion problems, but it does not happen as long as the thread is processing faster than the queue is growing. Then, in the preceding example, each queued request holds a socket, which in some operating systems consumes the file handle. Because the operating system restricts the number of file handles open by the process, it is best to limit the size of the work queue.

public static Executorservice newboundedfixedthreadpool (int nthreads, int capacity) {return
 new Threadpoolexecutor (Nthreads, Nthreads,
   0L, Timeunit.milliseconds,
   new linkedblockingqueue<runnable> (capacity),
   New Threadpoolexecutor.discardpolicy ());
}

public static void Boundedthreadpoolserversocket () throws IOException {
 ServerSocket listener = new ServerSocket ( 8080);
 Executorservice executor = Newboundedfixedthreadpool (4);
 try {while
  (true) {
   socket socket = listener.accept ();
   Executor.submit (new handlerequestrunnable (socket));
  }
 finally {
  listener.close ();
 }
}

Instead of using the Executors.newfixedthreadpool method directly to create the thread pool, we built the Threadpoolexecutor object ourselves and limited the work queue length to 16 elements.

If all the threads are busy, the new task will be populated into the queue, because the queue limits the size to 16 elements, and if this limit is exceeded, it needs to be handled by the last argument that constructs the Threadpoolexecutor object. example, the Discard policy (discardpolicy) is used, which means that when the queue reaches the upper limit, the new task is discarded. In addition to the initial, there are abort policies (abortpolicy) and caller execution Policies (callerrunspolicy). The former throws an exception, and the latter executes the task in the caller's thread.

For Web applications, the optimal default policy should be to discard or abort the policy and return an error to the client (such as an HTTP 503 error). Of course, you can avoid discarding client requests by increasing the length of work queues, but user requests are generally reluctant to wait for long periods of time and will consume more server resources. The purpose of the Task Force column is not to respond to client requests indefinitely, but to smooth out sudden bursts of requests. Normally, the work queue should be empty.

Number of threads tuning

The previous example shows how to create and use a thread pool, but the core problem with a thread pool is how many threads should be used. First, we want to ensure that the thread cap is reached without causing the resource to run out. The resources here include memory (heap and stack), number of open file handles, number of TCP connections, number of remote database connections, and other limited resources. In particular, if the thread task is computationally intensive, the CPU core number is also one of the resource constraints, generally the number of threads does not exceed the CPU core number.

Because the selection of the number of threads depends on the type of application, it may take a lot of performance testing to get the best results. Of course, you can also improve the performance of your application by increasing the number of resources. For example, modify the JVM heap memory size, or modify the operating system file handle upper limit, and so on. Then these adjustments will eventually touch on the theoretical limits.

The Law of the Colt

The law of the Philistines describes the relationship between three variables in a stable system.

where l represents the average number of requests, λ represents the frequency of the request, and W represents the average time to respond to requests. For example, if the number of requests per second is 10, and each request processing time is 1 seconds, then 10 requests are being processed at any one time. Back to our topic, we need to use 10 threads for processing. If the processing time of a single request doubles, the number of threads processed is doubled to 20.

After understanding the effect of processing time on request processing efficiency, we will find that the theoretical upper limit may not be the best value for the size of the thread pool. The thread pool Cap also requires reference task processing time.

Assuming that the JVM can handle 1000 tasks in parallel, if the processing time for each request does not exceed 30 seconds, at worst, only 33.3 requests per second can be processed. However, if each request takes only 500 milliseconds, the application can process 2000 requests per second.

Split thread pool

In a micro-service or service-oriented architecture (SOA), it is often necessary to access multiple back-end services. If one of these services is degraded, it can cause thread pool threads to run out, which can affect requests for others.

An effective way to respond to the failure of a backend service is to isolate the thread pool used by each service. In this mode, there is still an assigned thread pool that assigns the task to a different backend request thread pool. The thread pool may be a slow backend without load, and the burden will be shifted to the thread pool at the slow back end of the request.

In addition, the multithreaded pool mode also needs to avoid deadlock problems. A deadlock occurs if each thread blocks on the result of waiting for an unhandled request. Therefore, in multithreaded pool mode, it is necessary to understand the tasks that each thread pool performs and the dependencies between them, so as to avoid deadlock problems as much as possible.

Summarize

Even if you do not use the thread pool directly in your application, they are likely to be used indirectly in an application server or framework. The frameworks for Tomcat, JBoss, Undertow, and Dropwizard provide the option to tune the thread pool (the thread pool that the servlet uses).

I hope this article can improve the understanding of the thread pool, for everyone to learn help.