A redis connection pool connection failure fault and Solution

A redis connection pool connection failure fault and Solution
Description of failures in connection pool connection failure and fault resolution in redis

• An error occurred while connecting the application to redis.

[ERROR]java.lang.NullPointerException        at redis.clients.util.Pool.getActivePool(Pool.java:101)

• Redis itself has no error

Fault Locating

Initial positioning should be at the network level

• Check Network

  Through zabbix monitoring, we can see that when a fault occurs, the speed of the redis host NIC reaches several hundred megabytes, And the other periods only have dozens of megabytes.

• Check the cause of network burst speed

  During the fault, the redis backup script starts to run.

Fault Analysis

Backup script analysis

The script uses redis-cli save to complete the backup. The save Command needs to be synchronized, so other data operations that submit updates during running will be blocked until the save command is complete. Therefore, the cause of this failure is h

Troubleshooting

Replace save with the bgsave command

Take more measures to avoid faults

• Migrate the backup script from the redis master to the slave

  Redis master only processes client requests and is no longer responsible for backup.

• Speed limit backup script

  In the script, scp copies the rdb file to the backup storage machine. scp adds the-l 163840 parameter and copies the file at a speed of 20 Mb/s to prevent the process from occupying too much network bandwidth.