Redis primary node memory consumption is high

0. Basic information

Redis uses cluster mode, 560 primary nodes, 1:1 master-slave ratio, and 16 nodes on a single machine. Info memory observed that master Node a single Redis memory Used_memory_rss_human to 9.2G (set Maxmemory_human to 6G) exceeds the maximum setting.

1. View Client Connection Status

[Email protected] A ~]# redis-cli-p 9001 Info Clients

# clients

connected_clients:4705

client_longest_output_list:204928

client_biggest_input_buf:0

blocked_clients:0

A large number of output queues were found through client_longest_output_list, and the situation was further observed for all clients.

On the A machine, swipe the client output list:

client_a:9001# clients

client_longest_output_list:491335

client_a:9002# clients

client_longest_output_list:520840

client_a:9003# clients

client_longest_output_list:486498

client_a:9004# clients

client_longest_output_list:485387

client_a:9005# clients

client_longest_output_list:480000

client_a:9006# clients

client_longest_output_list:537211

client_a:9007# clients

client_longest_output_list:487594

client_a:9008# clients

client_longest_output_list:490037

client_a:9009# clients

client_longest_output_list:478734

client_a:9010# clients

client_longest_output_list:524717

client_a:9011# clients

client_longest_output_list:487200

client_a:9012# clients

client_longest_output_list:491687

client_a:9013# clients

client_longest_output_list:483900

client_a:9014# clients

client_longest_output_list:524557

client_a:9015# clients

client_longest_output_list:497967

client_a:9016# clients

client_longest_output_list:484250

2. Output memory consumes not 0 of content

[[email protected] A redis-node]# redis-cli-p 9001 Client List |grep-v "Omem=0"

id=843 addr=client_b:53475 fd=2248 name= age=582784 idle=0 flags=s db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll =456078 omem=7446059139 EVENTS=RW cmd=replconf

The cmd indicates that the master-slave synchronization consumes a lot of output memory

3. Check statistical values

There's a problem with client a from Client_b

[Email protected] 9001]# REDIS-CLI-P 9001

127.0.0.1:9001> Info Stats

# Stats

Total_connections_received:2

Total_commands_processed:25

instantaneous_ops_per_sec:0

total_net_input_bytes:1280055129

total_net_output_bytes:36461

instantaneous_input_kbps:192.22

instantaneous_output_kbps:0.00

rejected_connections:0

Normal machine C from client D

[[email protected] D redis-node]# redis-cli-p 9001

127.0.0.1:9001> Info Stats

# Stats

Total_connections_received:10

total_commands_processed:23220212769

instantaneous_ops_per_sec:18281

total_net_input_bytes:3405817622262

total_net_output_bytes:59530419

instantaneous_input_kbps:2882.54

instantaneous_output_kbps:0.03

rejected_connections:0

The input speed is 10 times times worse, other configurations are the same, suspected of being a network problem.

4. Check network Latency

From Client_b of Machine A

[Email protected] redis-node]# traceroute client_b

Traceroute to Client_b (client_b), hops max, byte packets

1 *.*.*.254 0.752 ms 0.809 ms 0.900 ms

2 *.*.*.133 2.164 ms 2.170 ms 2.157 ms

3 * * *

4 * * *

5 *.*.*.2 163.542 ms 163.940 ms 164.181 ms

6 Client_b (Client_b) 160.995 ms 156.202 ms 158.603 ms

Clinet C from client D

[[email protected] ~]# traceroute Client D

Traceroute to client D (client D), hops max, packets

1 *.*.*.254 0.732 ms 0.799 ms 0.897 ms

2 *.*.*.133 3.147 ms 3.182 ms 3.168 ms

3 * * *

4 * * *

5 *.*.*.2 5.537 ms 5.788 ms 5.961 ms

6 Client D (client D) 1.953 Ms 1.960 ms 1.956 ms

Test machine A to machine D performance

[Email protected] 9001]# traceroute Client D

Traceroute to client D (client D), hops max, packets

1 *.*.*.254 (10.180.128.254) 0.738 ms 0.800 ms 0.889 ms

2 *.*.*.133 (10.180.176.133) 2.156 ms 2.190 ms 2.177 ms

3 * * *

4 * * *

5 *.*.*.2 5.768 ms 5.942 ms 6.114 ms

6 Client D (client D) 1.940 Ms 1.964 ms 1.948 ms

Description is a problem with machine A to Client_b network

5. Workaround

Replace the slave machine to Client_b to Clinet E

9001 memory usage on machine a after startup

127.0.0.1:9001> Info Memory

# Memory

used_memory:2771776520

used_memory_human:2.58g

used_memory_rss:3403390976

used_memory_rss_human:3.17g

used_memory_peak:18154232496

used_memory_peak_human:16.91g

used_memory_peak_perc:15.27%

used_memory_overhead:758586508

used_memory_startup:8105584

used_memory_dataset:2013190012

used_memory_dataset_perc:72.84%

total_system_memory:540783263744

total_system_memory_human:503.64g

used_memory_lua:37888

used_memory_lua_human:37.00k

maxmemory:6442450944

maxmemory_human:6.00g

Maxmemory_policy:volatile-ttl

mem_fragmentation_ratio:1.23

mem_allocator:jemalloc-4.0.3

active_defrag_running:0

lazyfree_pending_objects:0

127.0.0.1:9001>

6. Post-settlement observation

Client e Memory Synchronization situation

127.0.0.1:9001> Info Stats

# Stats

Total_connections_received:37

total_commands_processed:5318696

instantaneous_ops_per_sec:19310

total_net_input_bytes:1711079396

total_net_output_bytes:4954958

instantaneous_input_kbps:2967.88

instantaneous_output_kbps:70.17

rejected_connections:0

sync_full:0

Modified 9001 output on machine a

[Email protected] redis-node]# REDIS-CLI-P 9001 Info Clients

# clients

connected_clients:4695

client_longest_output_list:0

client_biggest_input_buf:0

Blocked_clients:

7. Summary:

If the Redis settings are the same and the memory usage is not the same, you should check the info command to compare the output of the normal machine, give priority to the output of stats and the local Redis.log log.

Redis primary node memory consumption is high