Experience and lessons learned from Redis-based building systems

Redis is a very fast and powerful storage (persistence) system, and its greatest feature is its support for rich data structures.

Due to the limitations of Redis itself, the data it can handle must be completely in memory, and the data on the hard disk is a mirror image of the memory data, so limit its capacity to not exceed the capacity of memory. Current server memory to 32G as a general situation, 96G is better, if a system to store 1T of data, then must use 10 servers, the hardware cost is very high. For a 1 million active users of the system, the average person produces 1K of data per day, it will require 1G of storage space, which is equivalent to each user only 10 micro-blog or 10 chat messages per day, the real popular system will be far more than this data size.

When the general data reaches hundreds of M or 1G, Redis must and can only turn on the aof operation log for the persistent mode of the asynchronous write drive, and the aof file must be shrunk (rewrite) periodically because the AoF file volume of the user records the data change log is more serious. The process of shrinking is actually the process of mirroring the memory data to the hard disk, and the Redis master process needs to fork a process out, although the operating system has a write-time copy function, but still has enough memory space for the fork-out process, so redis can only use 50% of the memory capacity.

When writing aof files, Redis has no speed control policy at all, often resulting in full disk reads and writes, and other processes that are involved in file manipulation will be blocked.

Redis itself supports master-slave mode, can easily do data backup, to avoid single-point failure caused data loss, but when the network jitter, can lead to a full-volume replication between the master and slave, which is a blow to network bandwidth.

Redis is a single-machine storage scheme, when the data exceeds the memory capacity of a single server, the software designer must design a data splitting scheme in the software logic layer.

Conclusion:

Redis is not suitable as a massive data storage solution. Redis is suitable for applications with small data sizes and high performance requirements.

Experience and lessons learned from Redis-based building systems