Detailed description of redis misconfiguration

When using redis for caching, applications often get very high performance. However, If you do not configure it properly, you will encounter many headaches, such as the replication buffer limit and replication timeout.

 

Redis provides many tools to improve and maintain efficient memory databases. Without the need to configure the application layer, the unique data type, command, and command Optimization of redis can meet the needs of applications, but the configuration is incorrect, more specifically, those out-of-host devices may cause operational troubles and performance problems. Although it causes some headaches, the solution exists, and the solution may be simpler than we expected.

This series of articles describes some of the headaches encountered when using redis and how to solve these problems. These are based on our real experiences when running thousands of redis database instances.

Replication buffer limit

The memory area in which data is stored when the master/slave server synchronizes data in the replication buffer. In a complete master-slave synchronization, the master server saves data changes in the replication buffer during initialization. After the initialization phase is complete, the buffered content is sent to the slave server. This process may encounter a buffer capacity limit. When the maximum capacity is reached, the replication starts again. To avoid this, the buffer must be initialized based on the type and quantity changed during the replication process. For example, a small buffer can store a small amount of changed data, but when there are many and large changes, we need a large buffer. A more complex solution will set the buffer in more detail to avoid lengthy and large complex processes consuming the buffer (if the buffer is too small ). Eventually, this solution requires fine-tuning specific databases.

When a MB hard limit is reached, or the soft limit is reached for 60 seconds, the default replication link will break (causing synchronization to start from scratch ). In many cases, especially when the write load is high and the bandwidth of the slave server is insufficient, the load process cannot end. This may result in infinite loops. The primary redis continuously copies the entire dataset to the hard disk, which can result in high-speed I/O operations, consuming up to three times of additional memory. In addition, the slave server cannot catch up with the master server and cannot be fully synchronized with the master server.

A simple solution is to increase the output from the buffer and set the hard and soft limits to 512 MB. This solution can quickly improve the results.

Because there are a lot of reconfiguration, be sure to understand:

1. Before increasing the size of the replication buffer, make sure there is sufficient memory on the machine.

2. redis memory usage computing does not consider the replication buffer capacity.

The above is the first question introduced in this article. As we mentioned above, despite the limitations of the replication buffer, suitable configurations can work well. Next we will talk about another issue of master-slave replication. We will discuss in depth the time required to complete the process and some configuration problems that may cause troubles.

Replication timeout

As we discussed earlier, the redis replication process includes two synchronization phases: the initialization phase and the ongoing phase. Despite the stability of the phase (as long as the link between the master and slave servers is maintained), the initialization phase is not easy to complete. Successful initialization and synchronization depend not only on the memory allocated by the replication buffer, but also on the time spent in this step.

You may have remembered that the initialization synchronization steps include background storage and data-driven transmission. Based on the database capacity and network connection quality, this may be a long process. If the stage takes too long, redis may reach the replication timeout setting, which may cause repeated operations in the initial stage. In this case, you will find that the redis log document is filled with this information:

 

[28618] 21 Jul 00:33:36.031 * Connecting to MASTER 10.60.228.106:25994 [28618] 21 Jul 00:33:36.032 * MASTER <-> SLAVE sync started [28618] 21 Jul 00:33:36.032 * Non blocking connect for SYNC fired the event. [28618] 21 Jul 00:33:36.032 * Master replied to PING, replication can continue... [28618] 21 Jul 00:33:36.032 * Partial resynchronization not possible (no cached master) [28618] 21 Jul 00:33:36.032 * Full resync from master: 549907b03661629665eb90846ea921f23de6c961:2537453 

 

The default value of redis replication timeout is 60 seconds (see the repl-timeout command in the redis. conf file, or run "Config get repl-Timeout" using redis-cli "). This time may be short, especially when you have:

 

• Low-speed Storage: If the master server or slave server is based on low-speed storage, the background process will take a lot of time if the master server is used; if the server disk reads/writes data, the time will be extended.
• Large Dataset: A larger dataset requires longer storage and transmission time.
• Network Performance: When the network links of the master server and slave server have limited bandwidth and high latency, This will directly affect the data transmission rate.

 

We can solve this problem by setting replication timeout to a more appropriate value. The first is the estimated time for an acceptable database copy. Step 1: Check redis through the bgsave command and check related lines (for example, "background saving started by PID nnn" indicates that the process starts and "background saving terminated with success" indicates that the process ends) the time it takes for log files to run background processes. Then, measure the time required by CN to copy the result RDB file on the master server to the hard disk of the slave server. Finally, measure the actual time consumed to load data from the hard disk (for example, restart redis and search for the "DB loaded from disk" line in the log file ). These methods can roughly estimate the replication timeout value. For the sake of insurance, we may need to add 10 ~ 20%.

After timeout is set based on the measured value, we can measure the actual time consumption of replication by allowing the slave server to perform complete synchronization and Check log files several times. If possible, repeat this operation at different daily times to ensure that the system performs well under different loads. Finally, remember that as the database grows, we need to regularly check the timeout value.

The above describes the redis replication problem. Replication is a powerful tool that keeps the database available and extends the readability of the database. However, pay attention to the default settings of replication to ensure that the database is configured according to actual usage. Next we will talk about the client buffer. In some cases, the client buffer may cause many problems.

Client Buffer

You probably know that redis is a memory database, which means that all data is directly managed and provided by Ram. Therefore, redis has excellent delivery performance. redis can process tens of thousands or hundreds of thousands of requests at a latency of several milliseconds. Ram is currently the fastest storage technology-to better understand the delay numbers, please refer to the following data:

 

Latency Comparison Numbers--------------------------L1 cache reference 0.5 nsBranch mispredict 5 nsL2 cache reference 7 ns 14x L1 cacheMutex lock/unlock 25 nsMain memory reference 100 ns 20x L2 cache, 200x L1 cacheCompress 1K bytes with Zippy 3,000 nsSend 1K bytes over 1 Gbps network 10,000 ns 0.01 msRead 4K randomly from SSD* 150,000 ns 0.15 msRead 1 MB sequentially from memory 250,000 ns 0.25 msRound trip within same datacenter 500,000 ns 0.5 msRead 1 MB sequentially from SSD* 1,000,000 ns 1 ms 4X memoryDisk seek 10,000,000 ns 10 ms 20x datacenter roundtripRead 1 MB sequentially from disk 20,000,000 ns 20 ms 80x memory, 20X SSDSend packet CA->Netherlands->CA 150,000,000 ns 150 ms Notes-----1 ns = 10-9 seconds1 ms = 10-3 seconds* Assuming ~1GB/sec SSD Credit------By Jeff Dean: http://research.google.com/people/jeff/Originally by Peter Norvig: http://norvig.com/21-days.html#answers Contributions-------------Some updates from: https://gist.github.com/2843375Great ‘humanized‘ comparison version:https://gist.github.com/2843375Visual comparison chart: http://i.imgur.com/k0t1e.pngNice animated presentation of the data: http://prezi.com/pdkvgys-r0y6/latency-numbers-for-programmers-web-development/- See more at:http://redislabs.com/blog/top-redis-headaches-for-devops-client-buffers#sthash.9rHluMPF.dpuf

 

Redis, like its name and design, is a mobile server. Clients (usually) connect to redis over the network. In this case, the return time of client requests is significantly longer than the time when redis CPU reads data from Ram. This means that if there is no client buffer, the main difference between redis and the response to the service during this period of time is related.

Client buffers constitute the memory space required for service customer requests. Each connection to redis has its own buffer space. After processing the request, redis copies the response data to the client buffer, and then continues to process the next request. At the same time, the request client reads the data through a network connection. The redis client buffer is configured in redis. CONF file, which is configured using the client-output-buffer-limit normal command (you can get the settings through the config get client-output-buffer-limit command at runtime ). The default redis. conf file is defined as follows:

client-output-buffer-limit normal 0 0 0

These values represent the buffer soft limit, hard limit, and timeout in seconds (similar to the replication buffer ). When redis terminates the connection, these values provide protection-the customer does not need to read the reply-when the buffer size reaches a) soft limit and remains in the status until the timeout B) Hard limit. Setting these values to 0 means disabling protection.

However, unlike the replication buffer, the client buffer comes from the redis data memory space. You can use the maxmemory command to set the total memory value of redis. When the limit is reached, redis will apply the configured eviction policy (defined by the maxmemory-policy command ). Therefore, low-performance customers or a large number of simultaneous connections may cause redis instances to prematurely evict keys or prohibit updates because the dataset size and Client Buffer reach the memory limit.

Due to the relative life cycle, a client does not need to degrade the performance, which may lead to this phenomenon. Because there is a large speed difference between Ram reading and Network reading, too many client buffers may exhaust redis memory, even in high-performance client and network connections. For example, consider the keys command. After this command is triggered, redis will copy the entire key namespace to the client buffer. If our database has many keys, this may lead to eviction.

Warning:Be cautious when using keys. Do not use keys in the production environment. Besides the eviction mentioned above, using keys may block redis for a long time.

Keys is not the only command that may cause this situation. Similar Commands include smembers, hgetall, lrange, and zrange (and commands related to these commands). The value (or range) is large enough, or when there are many public connections (each connection requires a separate buffer), these commands may lead to a similar phenomenon.

We strongly recommend that you use these commands with caution. We recommend that you use the scan command family to replace these commands. In v2.8, the scan command is added. The scan command not only allows redis to continue processing requests between subsequent scan calls, but also reduces the probability of depletion of the client buffer.

The client buffer is often ignored in redis memory requirements and management. The default settings of the client buffer are risky, which may result in memory depletion. We need to set the buffer threshold accordingly-consider "maxmemory" Settings, current and predicted memory usage, and application traffic mode. Careful use of previously mentioned commands can avoid headaches. Welcome to follow our subsequent articles.

Original article:

Top redis headaches for devops-client Buffers

Top redis headaches for devops-replication timeouts

Top redis headaches for devops-client Buffers)

 

Http://www.csdn.net/article/2014-07-31/2820964

Detailed description of redis misconfiguration