Example of splitting a large Redis database

Reproduced in: http://www.itxuexiwang.com/a/shujukujishu/redis/2016/0216/124.html?1455853509

The partner feature on the Mint App uses a lot of in-memory database Redis, and as the volume of data grows fast, Redis expands quickly and is close to the size of a single Redis instance. A single giant Redis instance has the following disadvantages:

1. First, a machine with a large memory is needed. Redis is a memory database, it needs to put all the requirements in memory, need to install a few GB of Redis instance, at least a few gigabytes of memory size of the machine, consider the reserved growth space, generally requires 12 * 1.5 GB of memory. Another factor to consider is that when Redis is storing hard disk data, the fork process consumes the same amount of memory, so a 12GB Redis instance requires approximately GB or so of memory, which makes the machine very demanding and often difficult to meet.

2. Then, Redis becomes a performance bottleneck easily. Redis's concurrency model is single-process single-threaded, which does not take full advantage of multicore CPUs and can be a performance bottleneck for a system when the number of requests is high, or when some request processing is slow (such as some large data ordering). There are ways to mitigate even this problem by building multiple Redis instances that are implemented through multiple redis connections.

3. In addition, a single large Redis instance can also increase the difficulty of data management, because such a large amount of data, whether copied, backup operations are slow, easy to impact on the online system.

Therefore, it is necessary to split a single large Redis instance into multiple small redis instances.

Using the Redis replication mechanism, redis instance segmentation can be smoothed online with little or no significant impact on the system.

The specific operation idea of the segmentation is as follows:

1. First, plan the Redis segmentation strategy, usually based on business division, for example, the Mint partner is based on business partitioning timeline, User_relationship, and other 3 Redis instances. After planning, you need to modify the Redis program code in your application according to the planning results, usually with a unified Redis link being modified to multiple redis connections, and different businesses using different connections.

2. Then, establish multiple Redis replicas with Redis replication to allow different Redis connections to use different Redis replicas to remove redundant data from Redis replicas. To delete a mode's keys in bulk, you can use the following shell command:

The code looks like this: #p # page Title #e#

REDIS-CLI KEYS "<pattern>" | Xargs redis-cli DEL

Change to a real pattern, such as

The code is as follows:

REDIS-CLI KEYS "User:*:followers" | Xargs redis-cli DEL

Represents the deletion of user followers data.

Finally, the Redis instance is fully segmented by switching back and forth and restarting the APSARADB.

Example of splitting a large Redis database