Redis master-slave replication network transient disconnection

Redis master-slave replication network transient disconnection

Record several interesting usage first
1. Repeated execution
The-r option repeats the number of times specified by a command.
-I: Set the interval between command execution.
For example, you can view the commands (qps) executed by redis per second)

./Redis-cli-r 100-I 1 info stats | grep instantaneous_ops_per_sec
Instantaneous_ops_per_sec: 334
Instantaneous_ops_per_sec: 306
Instantaneous_ops_per_sec: 294
Instantaneous_ops_per_sec: 349
Instantaneous_ops_per_sec: 371
Instantaneous_ops_per_sec: 380
Instantaneous_ops_per_sec: 388

2. Obtain the rdb file of the specified redis instance and save it to the local device.
Redis-cli-h 192.168.44.16-p 6379 -- rdb 62.16.rdb

3. Simulate the commands received by slave from the master.
./Redis-cli -- slave

Unlike monitor, monitor receives all commands. slave only receives insert, update, and delete commands.

4. -- scan and -- pattern
Replacing keys IM *
./Redis-cli -- scan -- pattern 'im *'

This will not block redis for a long time and cause the command requests of other clients to remain blocked.

Recently, I often receive email alerts,
Generally, the Redis CPU usage reaches the threshold.
It is a Slave instance, and the qps is only a few hundred.
After troubleshooting, it is possible that the network is unstable, resulting in Slave disconnection and reconnection.

There are several parameters in the middle. You have to review them.

1. client-output-buffer-limit
Redis server processes the received requests in a single process. However, redis performs slow operations such as network I/O and disk I/O. Redis has made many optimizations to improve the response time for processing client requests. For example, network I/O and disk I/O are completed asynchronously, bgsave and bgrewriteaof are completed using background processes, and read buffer is provided to customers on the server.
Client buffer is a read buffer implemented on the server. After receiving a request from the client, the redis server writes the affected results to the client buffer instead of sending them directly to the client. After the server writes the result to the client buffer, it continues to process other client requests. This asynchronous processing method prevents the redis server from blocking the processing of other requests due to network reasons. In the previous article, replication buffer is also one of client buffer.

1) "client-output-buffer-limit"
2) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60"

The specific parameter meanings are as follows:
Class: client type, normal, slave, pubsub.
-Mormal: Common Client
-Slave: slave Database Replication Client
-Pub/sub: The Publish and subscribe Client
Hard limit: hard limit on the buffer size.
Soft limit: soft limit on buffer removal.
Soft seconds: The duration when the buffer size exceeds the soft limit value.
The client-output-buffer-limit parameter limits the size of the allocated buffer to prevent uncontrolled memory allocation. The default value of the parameter is 0, which means no limit is imposed.

Hard limit: When the buffer volume reaches the hard limit, redis immediately disconnects from the client.
Soft limit and soft seconds: When the buffer exceeds soft limit within soft seconds, redis will not close the client connection. If the buffer still exceeds the soft limit after soft seconds, redis immediately closes the client connection.

2. repl-timeout
The repl-timeout parameter value in redis is too small, which may cause the replication to fail.
The repl-timeout parameter in the redis configuration file is described as follows:
# The following option sets the replication timeout:
#
#1) Bulk transfer I/O during SYNC, from the point of view of slave.
#2) Master timeout from the point of view of slaves (data, pings ).
#3) Slave timeout from the point of view of masters (replconf ack pings ).
In three cases, replication times out:
1) from the slave angle, if the rdb snapshot data transmitted by master SYNC is not received within the repl-timeout time,
2) from the slave perspective, the repl-timeout does not receive the data packet or ping sent by the master.
3) from the master perspective, no repconf ack confirmation message is received during the repl-timeout time.
When redis detects a repl-timeout (60 s by default), the connection between the master and slave is closed, and redis slave initiates a request to re-establish the master-slave connection.
For systems with large memory datasets, you can increase the repl-timeout parameter.

3. slave ping period
Redis slave regularly sends ping commands from the master, with the interval specified by repl-ping-slave-period.
Therefore, when you set the parameter, repl-timeout> repl-ping-slave-period.
# Slaves send PINGs to server in a predefined interval. The default value is 10 seconds.
# Repl-ping-slave-period 10

# It is important to make sure that this value is greater than the values pecified for repl-ping-slave-period otherwise a timeout will be detected every time there is low traffic between the master and the slave.

4. repl-backlog-size
When the master server executes command propagation, the maser not only sends all data update commands to the replication buffer of all slave instances, but also writes the replication backlog. When the disconnected slave reconnects to the master, slave will send the psync command (including the offset of the copy) to request partial resync. If the requested offset does not exist, the full sync operation is equivalent to re-establishing master-slave replication.

To avoid full synchronization caused by network instability.
Modify the parameters as follows:
Config set repl-timeout 240
Config set repl-backlog-size 67108864

You may also like the following articles about Redis. For details, refer:

Install and test Redis in Ubuntu 14.04

Basic configuration of Redis master-slave Replication

Redis cluster details

Install Redis in Ubuntu 12.10 (graphic explanation) + Jedis to connect to Redis

Redis series-installation, deployment, and maintenance

Install Redis in CentOS 6.3

Learning notes on Redis installation and deployment

Redis. conf

Redis details: click here
Redis: click here

This article permanently updates the link address: