A time-out problem caused by Server using an asynchronous database client using a high concurrency high number of threads

Phenomenon

Today, when doing a project, the maxthreads of Tomcat was added to the 1024x768, and Tomcat provides services primarily to do some arithmetic, then insert Redis, query Redis, and finally return the task to the client

At the time of the pressure test, 1000 threads were opened concurrently, and the HTTP request was initiated to access the Tomcat service, resulting in a series of Redis query timeouts on the first visit to Tomcat, such as 1000 concurrent initiates 10W requests, and possibly the first 1W requests A redis timeout of about 2000 times causes the service to fail, but the Redis timeout is no longer present.

Solve

By observing the Vmstat, it was found that the R (running queue) was particularly high at the time of the timeout, reaching 12~25 for a duration of 2-3 seconds, and many redis timeouts occurred during this period. When there is no time-out, the R value remains at around 5. At this point, by reducing the number of tomcat threads to 200 and then re-testing, no timeouts occur.

Analysis

This is because my Redis client is an asynchronous client, but when I use it, I'm using an asynchronous client to emulate the synchronous client execution (call Future.get ()), which is implemented internally by the client, with only 4 or 8 worker threads Netty. When we just started TOMCA. T service, not warm-up, system performance is poor, at this time queued tasks, will allow the huge tomcat business threads (1024) occupy the CPU task queue, and Netty worker threads less, not CPU time execution, and finally caused the timeout.

The biggest problem here is that the asynchronous Redis client is used in the synchronized Tomcat business code, because the number of Tomcat business threads is large, taking up a lot of CPU time, leaving the Redis client thread out of execution and finally causing the problem.

If this is to change the business process of Tomcat to asynchronous processing (using Servlet 3.0), then you can reduce the number of Tomcat business threads a lot, which reduces business thread switching and allows the Redis client thread to get more execution time.

Extended

Aside from using the asynchronous Redis client, if you are using a traditional synchronous DB client, then the execution thread of the DB client is the business thread, and the CPU time of the thread is equal, and the timeout problem does not occur. So how do you define how many threads should be enabled?

When we have IO operations within our business threads, such as MySQL operations, if this operation is time consuming, such as the need to execute 2s, we should allocate more ground threads. Assuming that there are 1000 requests at this time, and only 8 business threads, the 8 business threads can only perform 8 tasks, and because the CPU is idle at this time because it is waiting for MySQL io to return, the other 992 requests are queued only because there are no threads to execute. So if you allocate more business threads at this point, you can make more efficient use of CPU resources.

And if the business process is CPU-intensive, fewer threads should be used. Because the CPU is busy at this point, adding more ground threads will not allow the thread to execute the time slice, but will increase the overhead of thread switching.

Summarize

The best way for my business scenario would be to reduce the business threads and change the business process to full async using Servlet 3.0. The reason is that because of the asynchronous Redis client, it does not leave the thread idle waiting, which can reduce thread switching and reasonably allocate the execution time slices of the Tomcat business thread and the Redis client thread.

A time-out problem caused by Server using an asynchronous database client using a high concurrency high number of threads