Tomcat is unresponsive because the Redis thread pool hangs

Last week, a sudden error on the online server, the Tomcat service inexplicably unresponsive, view memory and CPU, everything is OK, view Nginx log, found that always return 499 error code, it appears that there is an error in the program, resulting in Tomcat hangs. There's no good way. After the thread, heap dump comes out, restart the Tomcat service and the app returns to normal.

After analyzing the dump heap, we found that there were more than 500 redis connections hanging there, looked at the configuration of the Redis connection pool, found a pool of up to 500, and then looked at the code carefully, found that there is a problem with the code logic, when the application passed in a key, get a value from Redis, If the key does not exist, the processing in the code is returned directly to NULL, without releasing the connection, so that the connection takes up space until the connection pool is full.

Tomcat is unresponsive because the Redis thread pool hangs