How to improve the concurrency efficiency of Web server by asynchronous programming technology

As a web engineer who wants to make a web App that is more and more used by people, if the Web app We're doing goes down as the user grows, Then more and more people will become less and more, in order for our Web application to have more people to use, we have to improve the Web Application Server concurrency. So how do we do this, based on the existing concurrency technology we will have the following options:

The first approach: it is straightforward to open a thread for each client's request to the server, and the thread is destroyed when the request is processed, but it is rarely used in modern Web servers because a new thread is created, The cost of destroying a thread (overhead is the use of computer system resources such asCPU, memory, etc. ) is very large, it is often greater than the actual processing of the request itself overhead, so this method does not fully utilize the computer resources, improve concurrency efficiency is effective, If there is a thread-safe problem, using the locking mechanism of the thread, the data synchronization technology, the concurrency promotion will be subject to greater limitations; In addition, a request to open a thread, the number of threads without any control, which will easily lead to exhaustion of computer resources, for the web The stability of the service side is a big threat.

The second approach: In view of the above problem, we have a second way to improve the service side concurrency, first we are no longer a client request to open a new thread, the request processing completed the destruction of the thread, but the use of thread pool technology.

Thread pool technology is to create a batch of threads in advance, this batch of threads are put into a pool, when there is no request to reach the server, these threads are on standby, when the request arrives, the program will take a thread from the line constructor, the thread processing the incoming request, the request processing, the thread will not be destroyed, Instead, it is recycled by the thread pool, which uses threads to reduce the overhead of creating threads and destroying threads at will, while also controlling the number of service threads, typically one thread corresponding to a request, and controlling the number of concurrent requests, which improves the stability of the system over the first scheme (controlling the number of concurrent Preventing excessive concurrency results in a service outage), while also increasing the number of concurrent (because the cost of creating threads and destroying threads is reduced, and the system resources of the computer are more fully utilized).

But there are also a lot of problems with the two approaches, as follows:

The procedure two and the procedure one compares, the procedure two is much better, but this is only with the practice one ratio, if according to our design goal, the practice two is not perfect, the reason is as follows: The first procedure two will let many technology not solid people think the thread pool to open how many threads determines the system concurrency quantity, So for the system to handle more requests and take full advantage of computer resources, some people will start to set the number of threads constructor new thread to the maximum, a Web Application of concurrency in a certain time is a curve form, peak in a certain time frame is a few cases , so the maximum number of threads is turned on at first, which naturally wastes system resources for most of the time, and if those wasted idle computing resources can be used to process requests, they might be more efficient. In addition, a server exactly how many threads in advance, this standard is very difficult to control, and whether you use the thread pool technology or new threads, the number of processing requests and the number of threads is one by one corresponding relationship, if there is a point in time to come over the number of requests just beyond the line constructor thread number, For example, there is one more, then this request because can not find the corresponding thread is very likely to be abandoned by the program, in fact, this request is not more than the computer can withstand the load, but because our program design is unreasonable to be abandoned, this is certainly the developers do not want to happen .

Addressing these issues inJavaof theJDKThe thread pool provided is a good solution .(thread pooling technology is broad and profound, if we do not study the pool technology, or do not write their own, but with a ready-made), JDKThe thread pool size of the thread pool uses two parameters, one is the number of core threads, one is the maximum number of threads, the core thread is created when the system starts, and if the user request does not exceed the core threading capability, the thread pool no longer creates a new thread. If the number of core threads has not been processed, the thread pool will open a new thread, and after the new thread is first created, it is not destroyed immediately after use, it is also received in the thread pool, and the thread pool will no longer create new threads when the total number of threads constructor exceeds the maximum number of threads. This approach allows the number of threads to be adjusted according to the actual request, so as to achieve the full use of computer resources, but also to avoid the waste of system resources,JDKThread Pool has a time-out, and when threads outside the core thread have been unused for a certain amount of time, the threads will be destroyed and the resources will be freed so that the thread pool's threads are always in a reasonable range; If the request is too much, the thread constructor threads are temporarily out of the process. ,JDKThe thread pool also provides a queue mechanism for these requests to queue up, and when a thread finishes processing, the thread then pulls out a request for processing from the team joins, thus avoiding the loss of the request.JDKthread pool management of the queue there are a lot of strategies, interested in children's shoes can ask Niang, here I would like to say isJDKThe thread pool's security policy is well done, and if the capacity of the queue exceeds the processing power of the computer, the queue discards the unhandled request, which is also called the thread pool's deny policy.

Look at me so detailed description of procedure two, is not the practice of the second is a perfect solution? The answer, of course, is no, and the second is not the most efficient approach, and neither does it take full advantage of the computer's system resources.

I also have the practice here three, the specific practice is as follows:

First I want to ask a question, concurrent processing a task and single-threaded processing the same task, that way more efficient? Perhaps a lot of people will think of course is the concurrent processing task more efficient, two people do one thing is better than a person, the answer to this question is to look at the scene, in the single-core era, single-threaded processing of a task is often more efficient than the concurrency mode.

Why is it? Because multithreading is performed on a single core or a single CPU ,theCPU is not also capable of concurrent processing, andtheCPU can handle only one compute task at a time, so the concurrency task the CPU has a context-switching operation on the thread, and the overhead of the threading context is relatively large, so it is not necessarily more efficient to process concurrent requests on a single core than a single thread, but if a multi-core computer, the concurrency task is evenly distributed to each CPU , the efficiency of concurrent processing is much higher than for single-threaded processing, because you can avoid the switching of thread contexts at this time.

The processing of a network request is done by a total of two different types of operations, both of which areCPUthe calculation operations andIOoperation, if we judge these two operations in terms of processing efficiency,CPUoperating efficiency is the speed of light, andIOThe operation is not the same, the computerIOoperation is the operation of the storage of data media, the computer has the following several media can store data, which are:CPU-level cache, level two cache, memory, hard disk and network, the ability to store and read data at the first level is close to the speed of light, which is faster than the level two cache5Times to6times, but whether it is a primary cache or a level two cache, they store data too little, do not do anything big, the following is the memory, with the efficiency of the first-level cache to do the reference, one cache faster than the memory speed -multiple times, to hard disk storage and read data efficiency is even slower, first-class cache faster than the hard disk +many times, to the network on the slow more outrageous, first-class cache than the network faster than 100 million times, visible a request processing efficiency bottlenecks areIOcaused by, andCPUAlthough the processing is fast, butCPUThe calculation of the task is one after the other, if a request first waits for the processing of the network data in the process ofCPUoperation, then it must be slowing down.CPUthe overall efficiency of the processing, this slow is hundreds of billions of times, but in reality, a network request processing is a combination of these two operations. ForIOoperation inJavaThere are two ways in which a way is called blocking.IO, one way is called non-blockingIO, blocking theIOis to doIOoperation Time,CPUto waitIOoperation, which creates aCPUThe waste of computing resources, the degree of waste has been written in the above, is very scary, so we want to be a request for a thread to doIOoperation Time,CPUInstead of waiting for it and then dealing with other threads and requests, this approach is necessarily very efficient, when non-blockingIOIt's on the stage, non-blocking.IOcan be used in a threadIOoperation Time LetCPUto handle other threads, then non-blockingIOHow do you do that? Non-blockingIOoperation in the request andCPUadd a middle layer between the calculation, the request first sent to the middle tier, the middle tier to obtain a request to directly notify the request sender, the request received, note this time the middle tier nothing dry, just received the request, the real computing task has not begun oh, this time the middle layer if you want toCPUprocessing then letCPUprocessing, if the calculation process has to beIOoperation, the middle layer tellsCPUDon't wait for me, the middle layer let the request doIOoperation,CPUyou can handle other requests at this time, etc.IOThe operation is done, and the middle layer then gives the taskCPUProcessing , the middle tier sends the processing results back to the client, which makes full use of theCPUof computer resources, with non-blockingIOin fact, using a single thread can also develop multi-threaded tasks, even this single-threaded processing efficiency may be higher than multithreading, because it has no thread to create the cost of destruction, nor the cost of thread context switching. In fact, the implementation of a non-blocking request is a major issue, the use of a lot of advanced and complex technology such as: callback function and polling, for non-blocking development I am not good enough, and so I have a day fully mastered it I will write another article, but here to mention is likeJavainNettyTechnology,Nginx,PHPconcurrent processing uses the principle of this mechanism, especially now it's very hot.NodejsThe reason for this is that it relies on this non-blocking technology to write more efficientWebserver, can sayNodejsThis technology is used to the extreme, but here to correct, non-blocking is forIOoperation of the technology, forNodejs,Nettythe implementation mechanism has a better term description is the event-driven(In fact, using the callback function, the Observer pattern) and the asynchronousIOTechnology(which is non-blockingIOTechnology)。 Now we go back to the description of procedure three, the core idea of practice three is to make each thread resource utilization more efficient, and the third is based on the practice of two, using event-driven development ideas, using the non-blockingIOProgramming mode, when multiple client requests are sent to the server, the server can process these requests with just one thread, usingIOoperational performance bottlenecks, making the most ofCPUcomputing power, so that one thread can handle multiple requests with no more efficiency than multithreading, or even higher, while the enhancement of single-threaded processing power will lead to the entireWebimprove the performance of the service concurrency. You can think about it in this way under a multi-core server, if this server has8kernel, each of which opens a thread, which8threads may be able to host thousands of of concurrency, and also take advantage of eachCPUcomputing power, if we open more threads(Of course, the number of new threads is best8Multiple cores, which is better for multicore utilizationThen the efficiency of concurrency is higher, the increase is based on geometric multiples, we thinkNginx, it takes this pattern, so it just comes out when its concurrency processing power isApacheseveral times the server, and nowNginxhave been andApache, the event-driven asynchronous mechanism is the most common.

Well, the article is finished, today writing this article is a summary of my recent study of multithreading, but also my recent shift to research Nodejs ,nodejs has a perfect asynchronous programming model, But lately I've been skeptical about its concurrency, because I've never found the nodejs of asynchronous programming in Java , and now I find thatNodejs a more ingenious way to solve the problem of asynchronous development, and this approach is efficient, this point nodejs is too attractive, so it is worth studying and learning.

How to improve the concurrency efficiency of Web server by asynchronous programming technology