On the optimization of thread pool scale

How large should the thread pool be configured?

A previous friend asked me over Skype about running some problems with the 64-bit JVM cluster, which runs several 300,000 + thread tasks a day. With 300,000 + threads running, the core modules spend too much time managing them, causing the application to be extremely unstable. Obviously, the application needs a thread pool to ensure that the client can be killed, rather than allowing the client to crash the entire application.

The above example is a more extreme case, but it highlights the reason why we use the thread pool. Although the thread pool is reasonably used, users may still be annoyed by data loss or transaction failure. If our thread pool is defined too large or too small, it is possible for the application to be completely paralyzed. A properly sized thread pool allows for as many requests as possible, as long as the hardware and software support is reasonable. In other words, we do not want the request to wait in the queue when we are able to handle it, nor do we want the request to run beyond our management capabilities. How much should the thread pool be set to fit?

If we follow the "measure not to guess" allusion, we need to look at the technology related to the problem, ask the available metrics and how our system obtains them. We need to use some mathematical methods. We believe that the thread pool is one or more service providers connected by queues, so that this is a system that can be interpreted with the Little law. Let's take a closer look.

The law of the Little

The Little law is that the number of system requests equals the product of the requested arrival rate and the average time spent on each individual request. The whole law is common in our daily lives, and it is surprising that it was not brought up until the 50 's, and it was not proved until the 60. Here is an example of a form of the bill of laws in real life. Have you ever queued up and tried to figure out how long to wait? You might think about the number of people in line, and then quickly figure out how long it takes for the person in front of the service queue. At this point, you can multiply the two values to produce your estimate for the queue time. If you don't look at the length of the queue, you can remember how often the new person joins the queue, and then multiply the service time so that you know the average number of people in the queue or service.

There are many other similar games that apply to the Little law, which can answer other questions, such as "What is the average time spent on a person waiting for service in a queue?" "And so forth.

Fig. 1 (Little's Law)

Similarly, we can use the Little law to determine the size of the thread pool. We only need to calculate the request arrival rate and the average time required for processing. The above value is then put into the Little law to work out the average number of requests for the system. If the number of requests is smaller than the size of our thread pool, the size of the thread pool is reduced accordingly. Conversely, if the number of requests is greater than the thread pool size, things are a little more complicated.

When there are more requests to be processed, we first need to assess whether the system has sufficient capacity to support a larger pool of threads. The premise of an accurate assessment is that we have to assess which resources will limit the application's ability to scale. In this article, we will assume that the CPU is, and in practice, other resources. The simplest case is that we have enough space to increase the size of the thread pool. If not, you have to consider other options, such as software tuning, adding hardware, or tuning and adding hardware.

See more highlights of this column: http://www.bianceng.cnhttp://www.bianceng.cn/Programming/extra/