How to determine the thread pool size for Web apps

Thread pool (thread pool)
The size of the thread pool in the Web app determines the number of concurrent applications that can handle requests at any point in time. If the number of requests received by a system exceeds the size of the thread pool, the exceeded request either goes into the waiting queue or is rejected.
Note that concurrency and parallelism are different. Concurrent requests mean that at any one point in time, only a very small percentage of all processed requests occupy the CPU (the translator notes: Use the CPU in turn). Parallelism means that all requests processed at any one point in time are running on the CPU.
In non-blocking (no-blocking) applications (such as Nodejs), a single thread or process can handle multiple requests concurrently. In multi-core CPUs, parallel processing can be achieved by increasing the number of threads or processes.
In a blocking IO application, such as Java's SRINGMVC, a thread can handle only one concurrent request at a time. If you want to handle multiple requests concurrently, you can only do this by increasing the number of threads.

CPU-consuming applications
For CPU-consuming applications, the thread pool should be the same size as the number of CPUs on a single server. For such applications, increasing the number of threads due to thread context switching can hinder processing of requests and increase response time.
Non-blocking IO applications are CPU-intensive because they do not need to wait for request processing to complete when the request is processed.

io consumable application
Because IO consumption applications depend on the response time of the system on which the downstream traffic is located, And one thread will block until the other system responds, making it more difficult to determine the size of the thread pool for IO-consuming applications. For this type of application, as we said in the Block IO application article, increase the CPU utilization by increasing the number of threads.
Cotterfah (Little's Law)
Cotterfah is often used in non-technical areas, such as telling the bank counter teller how many customers are waiting for the request to be processed.
 
The following is the Wikipedia description of the special law, as follows:
The average number of threads in a system (threads) is equal average Web request Arrival Rate (Webrequests per sec), multiplied by the average response time (responsetime)
Translation: Average number of threads per system (number of threads) equals average request arrival Rate (number of requests per second) multiplied by the average response time (response time).
Formula: Number of threads = Number of requests per second X response time
Formula Description:
Number of threads that the system can handle
the number of requests per second that can be processed by requests
Response time the time it takes to process a request
of course, The above formula gives the number of threads required to process the number of requests, but does not take into account the CPU occupancy of the thread, nor does it indicate how many threads should be allocated for a single machine with multicore.

Determine thread pool size by testing
Allocating the appropriate size of the thread pool requires a balance between the two features of throughput and response time. Starting with the minimum number of threads per CPU (that is, the number of threads =cpu), the number of system threads is proportional to the average response time until the CPU usage is maximized or the response time is no longer reduced.
Describes the relationship between the number of requests, the CPU, and the response time.
The graph of CPU and requests shows how the CPU is being used as the load on the web system increases.
The response time and the number of requests are shown in the graph showing the impact of increased web system load on response time.
The green dots represent the most advantageous of throughput and response time.
Thread pool Size =CPU Core number

Shows what happens when the online pool size equals the number of CPU cores in a blocked IO consumption application. The thread is blocked because it waits for IO processing of the downstream traffic, and because the thread's blocking increases the response time, and even if the CPU is very low, all threads in the thread pool are blocked, and the application rejects the request.
A large thread pool

Shows the usage of a blocking IO consumption application under a large thread pool. Because of the large number of line cities, thread context switching has also become very frequent, and it is these unnecessary context switches that have not reached the maximum throughput of the CPU has reached the maximum occupancy rate. Request response times also grow rapidly due to frequent context switching.
Optimal thread pool Size

Shows the situation of the blocking IO consumption application under the optimal thread pool. The CPU has been used efficiently with high throughput and fewer threads switching above. At the same time, we note that good response times depend on efficient processing of requests with fewer threads being blocked (context switching).

Thread pool Isolation
In most applications, only a few types of requests are more time-consuming than other requests, but these few time-consuming requests can affect the performance of the entire system. There are two ways to solve this problem:
1) Isolate more time-consuming requests for specialized processing
2) allocate a single thread pool for time-consuming Web requests in the same application
Determining the optimal thread pool size for a blocking IO-consuming application is a difficult task, which usually requires multiple performance tests. If multiple thread pools are used in an application, the optimization of the thread pool is further complicated.

How to determine the thread pool size for Web apps