Connection failure due to Redis high concurrency

Connection failure due to Redis high concurrency

When ELK collected logs today, it found that the collection failed, looked for various reasons, and finally found an error in the redis log:
[2, 2489] 02 Jun 10:43:42 # Error allocating resoures for the client

An error occurred while allocating resources to the client. view the number of redis connections:
# Redis-cli info | grep connected
Connected_clients: 5036
Connected_slaves: 0

The number of client connections reaches more than 5000.

View client connection information:
Redis 127.0.0.1: 6379> client list
Addr = 10.247.64.115: 52834 fd = 5 idle = 2144 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = rpush
Addr = 127.0.0.1: 38484 fd = 7 idle = 0 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = blpop
Addr = 10.105.1.2: 53125 fd = 8 idle = 2144 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = rpush
Addr = 10.105.1.3: 32639 fd = 9 idle = 2144 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = rpush
Addr = 10.105.1.4: 57134 fd = 10 idle = 2144 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = rpush
Addr = 10.105.1.5: 53126 fd = 11 idle = 2144 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = rpush
Addr = 10.105.1.7: 8887 fd = 12 idle = 2144 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = rpush

The meaning of each field is as follows:
Addr: client address and port
Fd: The file descriptor used by the socket
Idle: the idle duration in seconds.
Flags: client flag
Db: ID of the database in use by the client
Sub: Number of subscribed Channels
Psub: Number of subscribed instances
Multi: Number of commands executed in transactions
Qbuf: Query Buffer length (in bytes, 0 indicates no Query Buffer is allocated)
Qbuf-free: query the length of the remaining buffer space (in bytes, 0 indicates no remaining space)
Obl: length of the output buffer (in bytes, 0 indicates no output buffer is allocated)
Oll: number of objects contained in the output list (when there is no available space in the output buffer, the command reply will be queued into this queue as a string object)
Omem: total memory occupied by the output buffer and output list
Events: file descriptor event
Cmd: The Last Command executed

It is found that the idle of the client is idle for too long, and the connection pool maintains too many connections. You need to release unused connections in time:
Redis 127.0.0.1: 6379> config set timeout 30
OK

View the connection again:
# Redis-cli info | grep connected
Connected_clients: 1137
Connected_slaves: 0

A large decrease in connections

View connection details
Redis 127.0.0.1: 6379> client list
Addr = 127.0.0.1: 38484 fd = 7 idle = 0 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = blpop
Addr = 10.105.1.150: 39742 fd = 269 idle = 29 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = rpush
Addr = 10.105.1.226: 53291 fd = 5 idle = 31 flags = N db = 0 sub = 0 psub = 0 qbuf = 0 obl = 0 oll = 0 events = r cmd = rpush

The idle time has been limited to 30 seconds.

You may also like the following articles about Redis. For details, refer:

Install and test Redis in Ubuntu 14.04

Basic configuration of Redis master-slave Replication

Redis cluster details

Install Redis in Ubuntu 12.10 (graphic explanation) + Jedis to connect to Redis

Redis series-installation, deployment, and maintenance

Install Redis in CentOS 6.3

Learning notes on Redis installation and deployment

Redis. conf

Redis details: click here
Redis: click here

This article permanently updates the link address: