What is the number of threads reasonable?

First, The origin of demand

Web-server usually have a configuration, the maximum number of worker threads , the backend service generally has a configuration, the number of threads in the worker thread pool, the number of threads configured by different business architects has different experience values, some of the business is set to twice times the number of CPU cores, Some businesses are set to 8 times times the number of CPU cores, and some are set to 32 times times the number of CPU Cores.

The number of "worker threads" is based on what is set to how much to maximize CPU performance is the problem that this article will discuss.

second, Some common cognition

Before further in-depth discussion, we should agree on some common cognition in the way of asking Questions.

Question: does the number of worker threads be set as large as possible?

Answer: definitely Not.

1) one server CPU has a limited number of cores, while the number of concurrent threads is limited, 1-core CPU set 10,000 worker threads are meaningless

2) thread switching is overhead and can degrade performance if thread switching is too frequent

question: Call Sleep () function, does the thread always occupy the CPU ?

Answer: do not occupy, wait for the CPU will be sent out, to other CPU resources to use the thread

Not only call the sleep () function, but also make some blocking calls, such as blocking accept () "waiting for client connections" in network programming, and blocking recv () "waiting for downstream return packets" without consuming CPU resources

question: if the CPU is single core, set multithreading meaningful, can improve concurrency performance?

Answer: even a single core, using multithreading is also meaningful

1) multithreaded coding can make our Service/code clearer, some IO threads receive packages, Some worker threads perform task processing, and some timeout threads do time-out detection

2) If you have a task that has been using CPU resources for calculations, then increasing the thread at this point does not increase concurrency, such as a code

While (1) {i++;}

This code has been using CPU resources to calculate, which will make the CPU utilization rate of up to 100%

3) typically, worker threads do not always consume CPU compute, even if the CPU is a single core, increasing the worker thread can also increase concurrency because the thread will continue to work while it is resting

iii. Common service threading Model

Understanding the Common service threading model helps to understand the principle of service concurrency, and generally there are two types of service threading models available on the Internet

IO threading and worker threads decoupling class model by queue

For example, most web-server and service frameworks use such an "io thread to decouple from worker threads through queues" class threading Model:

1) There are a few IO threads listening to upstream requests and sending packets (producers).

2) There is one or more task Queues , which are asynchronous decoupled data transfer channels (critical resources) of the IO thread and the worker thread

3) There are multiple worker threads performing a true task (consumer)

This threading model is widely used and fits most scenarios, and the threading model is characterized by a synchronous blocking of tasks within the worker thread (recall How the Java program is executed in the Tomcat thread, how the task is performed in the Dubbo worker thread), As a result, you can increase the number of worker threads by increasing the concurrency capability, and the main focus today is "how much of this model worker thread is set to maximum concurrency".

Pure Asynchronous Threading Model

There is no blocking anywhere, this threading model only needs to set a very small number of threads to achieve very high throughput, lighttpd has a single-process single-threaded mode, concurrent processing power is very strong, is the use of this model. The disadvantages of this model are:

1) if using single-threaded mode, it is difficult to take advantage of MULTI-CPU multicore

2) programmers are more accustomed to writing synchronization code, callback the way to the readability of the code, the requirements of the programmer is higher

3) the framework is more complex and often requires Server-side transceiver components, server-side queues, client-side transceiver components, client-side queues, context-management components, finite-state Unit components, and timeout management component support

however, This model is not the focus of Today's Discussion.

Iv. Working mode of the worker thread

Understanding the working mode of a worker thread is helpful in quantifying the number of threads that are analyzed:

is a typical work-thread processing process, starting from processing start to ending processing end, the task has a total of 7 steps to Handle:

1) take the task from the work queue and perform some local initialization calculations, such as HTTP protocol parsing, parameter parsing, parameter checking, etc.

2) access the cache to get some data

3) after getting the data in the cache, perform some local calculations, which are related to the business logic

4) call downstream service via RPC to fetch some data, or let downstream service handle some related tasks

5) after the RPC call is over, do some local calculations, How the calculations and business logic are related

6) access to DB for some data manipulation

7) do some finishing work after the database has been completed, as well as these finishing touches are local calculations, and business logic related

Analyzing the timeline of the entire process, you will find:

1) where the 1,3,5,7 step is "medium pink timeline", the thread takes CPU when It calculates the local business logic

2) while the 2,4,6 step in the "orange time axis", access to the cache, service, db process in a waiting for the result of the state, do not need to occupy the CPU, further decomposition, the "wait for the result" time is divided into three parts:

2.1) request transfer to downstream cache, service, DB on the network

2.2) downstream cache, service, db for Task processing

2.3) cache, service, db upload messages back to worker threads on the network

five, Quantitative Analysis and reasonable set of working threads

The last one to answer the number of worker threads is set to how many reasonable questions.

Through the above analysis, the worker thread in the process of execution, there is a computation time to occupy the cpu, the other part of the wait time does not need to occupy the cpu, through quantitative analysis, such as logging statistics, you can count the entire worker thread during the execution of these two parts of the time ratio, for example:

1) time axis 1,3,5,7 "medium Pink timeline" is calculated execution time is 100ms

2) wait time for the time axis 2,4,6 "mid Orange timeline" is also 100ms

The result is that the thread calculates and waits for 1:1, that is, 50% of the time is calculated (cpu occupied), and 50% of the time is waiting (without cpu):

1) Assuming that this is a single core , then set to 2 worker threads can fully utilize the cpu, let the CPU run to 100%

2) Assuming that this is the n-core , then set to 2N work site can fully utilize the cpu, let the CPU run to n*100%

Conclusion:

N-core server, through the execution of the business of single-threaded analysis of the local computing time is x, waiting time is y, the number of worker threads (thread Pool Threads) is set to n (x+y)/x, can maximize the utilization of the CPU.

Experience:

In general, non-cpu-intensive services (encryption, decompression, search sequencing, and other services are cpu-intensive business), bottlenecks are in the backend database, the local CPU time is low, so it is possible to set dozens of or hundreds of worker threads.

Vi. Conclusion

N-core server, through the execution of the business of single-threaded analysis of the local computing time is x, waiting time is y, the number of worker threads (thread Pool Threads) is set to n (x+y)/x, can maximize the utilization of the CPU.

What is the number of threads reasonable?