Go: Importance of tuning thread pools in Java Web applications

Regardless of whether you are concerned, Java Web applications use a thread pool to handle requests more or less. The implementation details of the thread pool may be overlooked, but the use and tuning of the thread pool is something that needs to be understood sooner or later. This article mainly describes the use of the Java thread pool and how to properly configure the thread pools.

Single Thread

Let's start with the basics. Regardless of which application server or framework you use (such as Tomcat, Jetty, and so on), they all have a similar base implementation. The Web service is based on a socket socket, which is responsible for listening to the port, waiting for TCP connections, and accepting TCP connections. Once the TCP connection is accepted, the data can be read and sent from the newly created TCP connection.

To be able to understand the above process, we don't use any application server directly, but build a simple Web service from scratch. This service is the epitome of most application servers. A simple single-threaded Web service is probably like this:

ServerSocket listener = new ServerSocket (8080); try {while (true) {   socket socket = listener.accept ();   try {     handlerequest (socket),   } catch (IOException e) {     e.printstacktrace ();}}   } finally { Listener.close ();}

The code above creates a service-side socket (ServerSocket), listens on port 8080, and then loops through the socket to see if there is a new connection. Once a new connection is accepted, the socket is passed into the HandleRequest method. This method parses the data stream into an HTTP request, responds to it, and writes the response data. In this simple example, the HandleRequest method only implements the reading of the data stream, returning a simple response data. In a typical implementation, the method can be much more complex, such as reading data from a database.

Final static String response =   "http/1.0 ok\r\n" +   "content-type:text/plain\r\n" +   "\ r \ n" +   "Hello W" orld\r\n ";p ublic static void HandleRequest (socket socket) throws IOException {//Read the input stream, and return" K "try {   BufferedReader in = new BufferedReader (     new InputStreamReader (Socket.getinputstream ()));   Log.info (In.readline ());   OutputStream out = Socket.getoutputstream ();   Out.write (Response.getbytes (standardcharsets.utf_8)); } finally {   socket.close ();}}

Because there is only one thread to process the request, each request must wait for the previous request to be processed before it can be responded to. Assuming a request response time of 100 milliseconds, the server has only 10 responses per second (TPS).

Related Vendor Content

Using probe technology to implement Java application Self-protection new Java, new future distributed storage practice for containerized services Distributed relational database architecture explore the performance of internet finance micro-innovation, give you fantastic thinking!

Related Sponsors

Qcon Global Software Development Conference Shanghai Station, October 20, 2016-22nd, Shanghai Marriott Hotel, Bao Hua Marriott, exciting content first look!

Multithreading

Although the HandleRequest method may block on Io, the CPU can still handle more requests. But in a single-threaded scenario, this is not possible. Therefore, you can increase the parallel processing power of the server by creating multi-threaded methods.

public static class Handlerequestrunnable implements Runnable {final socket socket, public handlerequestrunnable (socket s Ocket) {   this.socket = socket;} public void Run () {   try {     handlerequest (socket)   } catch (IOException e) {     e.printstacktrace ();   } }}serversocket listener = new ServerSocket (8080); try {while (true) {   socket socket = listener.accept ();   New Thread (new handlerequestrunnable (socket)). Start (); }} finally {Listener.close ();}

Here, the Accept () method is still called in the main thread, but once the TCP connection is established, a new thread is created to process the new request, both in the new thread and the HandleRequest method in the previous article.

By creating new threads, the main thread can continue to accept new TCP connections, and these letters can be processed in parallel. This method is called "one thread per request (thread per requests)." There are, of course, other ways to improve processing performance, such as the asynchronous event-driven model used by Nginx and node. js, but they do not use the thread pool and therefore are not covered in this article.

In each request for a thread implementation, it is very expensive to create a thread (and subsequent destruction), because both the JVM and the operating system need to allocate resources. Also, the above implementation has the problem that the number of threads created is not controllable, which could cause system resources to be quickly exhausted.

Resource Exhaustion

Each thread requires a certain amount of stack memory space. In the nearest 64-bit JVM, the default stack size is 1024KB. If the server receives a large number of requests, or if the HandleRequest method executes slowly, the server may crash because a large number of threads were created. For example, there are 1000 parallel requests, and the 1000 threads created will need to use 1GB of JVM memory as the line stacks space. Additionally, objects that are created during each thread code execution may also be created on the heap. This deterioration will go beyond the JVM heap memory and generate a lot of garbage collection operations, eventually causing a memory overflow (outofmemoryerrors).

These threads not only consume memory, they also use other limited resources, such as file handles, database connections, and so on. An uncontrolled creation of threads can also cause other types of errors and crashes. Therefore, an important way to avoid resource exhaustion is to avoid an uncontrolled data structure.

By the way, the size of the stack can be adjusted by the-XSS switch because of the memory problems raised by the thread stack sizes. After shrinking the size of the thread stack, you can reduce the overhead for each thread, but may cause a stack overflow (stackoverflowerrors). For general applications, the default 1024KB is too rich, and a smaller 256KB or 512KB may be more appropriate. The minimum allowable value for Java is 160KB.

Thread pool

To avoid continuous creation of new threads, you can limit the thread pool by using a simple line pool. The thread pool manages the threads, and if the number of threads has not reached the upper limit, the line pool creates the thread to the upper limit and reuse the idle thread as much as possible.

ServerSocket listener = new ServerSocket (8080); Executorservice executor = Executors.newfixedthreadpool (4); try {while (true) {   socket socket = listener.accept (); C1/>executor.submit (new handlerequestrunnable (socket)); }} finally {Listener.close ();}

In this example, instead of creating a thread directly, Executorservice is used. It commits the tasks that need to be performed (the implementation of the Runnables interface) to the thread pool and executes the code using threads in the thread pools. example, a fixed-size thread pool of 4 threads is used to process all requests. This limits the number of threads that are processing requests and limits the use of resources.

In addition to creating a fixed-size thread pool through the Newfixedthreadpool method, the Executors class also provides a Newcachedthreadpool method. Reusing a thread pool can also lead to an uncontrolled number of threads, but it uses the idle threads that were previously created whenever possible. Typically, this type of thread pool is useful for short tasks that are not blocked by external resources.

Work queue

After using a fixed-size thread pool, what happens if all the threads are busy and a new request is made? Threadpoolexecutor uses a queue to hold requests waiting to be processed, and the fixed-size thread pool uses the unrestricted list by default. Note that this can also cause resource exhaustion issues, but will not occur as long as the thread processing speed is greater than the queue growth rate. Then in the previous example, each queued request holds a socket, and in some operating systems, it consumes the file handle. Because the operating system restricts the number of file handles open by the process, it is best to limit the size of the work queue.

public static Executorservice newboundedfixedthreadpool (int nthreads, int capacity) {return new Threadpoolexecutor ( Nthreads, Nthreads,     0L, Timeunit.milliseconds,     new linkedblockingqueue<runnable> (capacity),     new Threadpoolexecutor.discardpolicy ());} public static void Boundedthreadpoolserversocket () throws IOException {ServerSocket listener = new ServerSocket (8080); Executorservice executor = Newboundedfixedthreadpool (4, 16); try {   while (true) {     socket socket = listener.accept ();     Executor.submit (new handlerequestrunnable (socket));   } } finally {   listener.close ();}}

Instead of using the Executors.newfixedthreadpool method directly to create the thread pool, we built the Threadpoolexecutor object ourselves and limited the work queue length to 16 elements.

If all the threads are busy, the new task will be populated into the queue, because the queue limits the size to 16 elements, and if this limit is exceeded, it needs to be handled by the last parameter when the Threadpoolexecutor object is constructed. In the example, the Discard policy (discardpolicy) is used, that is, when the queue reaches the upper limit, the new task is discarded. For the first time, there is the Abort policy (abortpolicy) and the caller Execution policy (callerrunspolicy). The former throws an exception, and the latter executes the task in the caller's thread.

For Web applications, the best default policy should be to discard or abort the policy and return an error to the client (such as an HTTP 503 error). It is also possible to avoid discarding client requests by increasing the length of the work queue, but the user request is generally not willing to wait for a long time, and it consumes more server resources. The purpose of the task queue is not to respond to client requests without restrictions, but to smooth bursts of burst requests. Typically, the work queue should be empty.

Tuning the number of threads

The previous example shows how to create and use a thread pool, but the core problem with using the thread pool is how many threads should be used. First, we want to ensure that when the thread cap is reached, the resource exhaustion is not incurred. Resources here include memory (heap and stack), number of open file handles, number of TCP connections, number of remote database connections, and other limited resources. In particular, if the thread task is computationally intensive, the number of CPU cores is also one of the resource limits, and generally the number of threads does not exceed the number of CPU cores.

Because the number of threads selected depends on the type of application, it may take a lot of performance testing to get the best results. Of course, you can also improve the performance of your application by increasing the number of resources. For example, modify the JVM heap memory size, or modify the operating system's file handle upper limit, and so on. These adjustments will then eventually touch the theoretical limit.

The Law of Delphi

The Law of Delphi describes the relationship between three variables in a stable system.

where l represents the average number of requests, λ represents the frequency of the request, and W represents the average time to respond to the request. For example, if the number of requests per second is 10 and each request processing time is 1 seconds, then 10 requests are being processed at any one time. Back to our topic, it is necessary to use 10 threads for processing. If the processing time of a single request doubles, the number of threads processed will also be doubled to 20.

After understanding the effect of processing time on the efficiency of request processing, we will find that the theoretical upper limit may not be the best value for the thread pool size. The thread pool Cap also requires reference to task processing time.

Assuming that the JVM can handle 1000 tasks in parallel, if each request is processed for no more than 30 seconds, in the worst case, you can handle up to 33.3 requests per second. However, if each request only requires 500 milliseconds, the application can process 2000 requests per second.

Split thread pool

In microservices or service-oriented architectures (SOA), it is often necessary to access multiple back-end services. If one of these services degrades, it can cause thread pool threads to run out, affecting requests to other service.

An effective way to deal with back-end service failures is to isolate the thread pool used by each service. In this mode, there is still an assigned thread pool that dispatches the task to a different backend request thread pool. The thread pool may be transferred to the thread pools in the slow backend of the request because of a slow backend and no load.

In addition, the multithreaded pool mode also needs to avoid deadlock problems. Deadlocks can occur if each thread is blocked on the result of waiting for an unhandled request. Therefore, in multithreaded pool mode, it is necessary to understand the tasks performed by each thread pool and the dependencies between them, so as to avoid deadlock problems as much as possible.

Summarize

Even if the thread pool is not used directly in the application, they are most likely to be used indirectly by the application server or framework in the application. The framework for Tomcat, JBoss, Undertow, Dropwizard, and so on, provides the option to tune the thread pool (the thread pool used by the servlet).

Hopefully this article will improve the knowledge of the thread pool. By understanding the application's requirements, combining the maximum number of threads and the average response time, a suitable thread pool configuration can be obtained.

This article refers to: Http://www.infoq.com/cn/articles/the-importance-of-thread-pool-in-java-web-application

Go: Importance of tuning thread pools in Java Web applications