Redis VS. Memcached is not suitable for data volume higher than 10 million, and ensure data integrity of the Key-value storage

Introduction:

In the large data age, there is always the hope that there is a key-value storage mechanism, like hashmap in memory to handle a large number of (tens of millions of orders) of the key-value pairs, in order to improve data lookup, modify speed.

So, we'll think of the two NoSQL databases memcached and Redis, which are strictly not counted as databases.

1, memcached is a cache mechanism, when the memory is not enough to use the LRU mechanism to replace the stale data, so he can not guarantee that our data as in the HashMap is not lost, and there is no data persistence mechanism;

2. Redis overcomes this shortcoming and adopts disk storage mechanism to realize data persistence. However, when the amount of data reaches about 10 million, because the memory can not store such a large number of data, frequent data exchange with the disk, resulting in data query, storage performance of a sharp decline, will lead to service is not available.

Conclusion: At present, there is no good product to achieve key-value guarantee data integrity, tens order, efficient storage and query support products.

Appendix I: The following is transferred from other users of the test data:

Appendix II: Comparison of memcached and Redis, and their respective uses

Appendix I:

From the diagram can be guessed that there will also be Redis 2.2.1 test, the same test environment, 1K of data, using the Servicestack.redis client to do the following test:

1) Set operation

2) Get operation

3) del operation

Each set of tests is tested using three configurations:

1 Green Line is open dump mode of persistence, 5 minutes lasting once

2 The Blue Line is open aof mode of persistence, write disk per second

3) The red line is to turn off any persistent way

The same other configuration is used for each configuration:

1 open VM maximum memory 10GB (128 byte one page) after start swapping out, VM space 160GB

2 maximum use of memory 15GB, to ensure that there is enough residual memory in the dump

3) Open compression, no configuration Master

Now take a look at the test results:

　　

As you can see from this diagram:

1 for the lack of persistence in the way, read and write in the amount of data reached 8 million times, performance dropped several times, this time is the memory 10g,redis began to swap out to disk. And since then there is no way to perk up again, performance than mongodb a lot worse.

2 for the way of aof persistence, the overall performance will not be much worse than the way without persistence, are in the amount of tens of millions of data, memory is full after the performance of the read only hundreds of.

3 for the Dump persistence mode, read and write performance fluctuations are relatively large, may be in the period of dump also has a relationship, and reached 14 million data volume, read and write performance post. In the dump, will not be swapped out, and all the modified data or create a new page, memory consumption than usual high, more than 15GB. And dump also compresses, takes up a lot of CPU. In other words, at that time memory, disk and CPU pressure are close to the limit, performance is not bad.

To sum up:

1 Redis is actually only suitable as a cache, not a database or storage. Its persistence method is suitable for saving the emergency what, not very suitable as a common function to use. For this version of Redis, it is not recommended to use any persistence method. Otherwise, it might be ugly to die. Plainly, expect Redis is memcached upgrade version, with a variety of data structure, but do not expect Redis to and mongodb/kt.

2 for VM is not recommended to open, although open VM can let Redis save more data than memory, but if the hot and cold data is not very obvious performance will be very poor (my test is random query key, hot and cold is not obvious). Of course, for hot and cold conditions, you can set 200%-400% of memory as a VM space, and it is not recommended to set 10 times times as much memory space as a VM (like my configuration).

3 Servicestack.redis client seems to have several bugs, first redistypedclient Dispose incredibly did not implement, should be to call Client.dispose (), Second Redisnativeclient's info attribute is not to get the latest value every time, the third pooledredisclientmanager of Writepoolindex and Readpoolindex only see Add to see minus the place, Do not know what this is, in fact, every time take the first is not active client can be, Pooledredisclientmanager also did not use the timeout of active client forced collection ( Avoid using when you forget to dispose of too many connections. I will try to contact Servicestack.redis's author about these points.

Appendix II:

The two years Redis fire, and Redis are often referred to as memcached Challenger on the table. The comparison of Redis and memcached is everywhere. However, Redis really goes beyond memcached in functionality, performance, and memory usage efficiencies.

The following is an answer from the Redis author on StackOverflow, which corresponds to the question "is memcached a dinosaur in comparison to Redis?" (compared to redis,memcached really obsolete. ）

There is no need to focus too much on performance because both are already high enough. Since Redis only uses a single core, and memcached can use multi-core, the two compare, on average, on each kernel, Redis is more efficient at storing small data than memcached. And in the data above 100k, memcached performance is higher than Redis. Although Redis has recently been optimized for the performance of storing large data, it is slightly inferior to memcached. Having said so much, the conclusion is that no matter which one you use, the number of requests processed per second will not be a bottleneck. In the efficiency of memory usage, if using simple key-value storage, memcached memory utilization is higher. But if Redis uses the hash structure to do the key-value storage, because of its combined compression, its memory utilization rate will be higher than memcached. Of course, this is related to your application scenarios and data features. If you have data persistence and data synchronization requirements, then recommend you choose Redis. because these two characteristics are not memcached. Even if you just want the cached data to be lost after upgrading or restarting the system, it's also wise to choose Redis. Of course, finally have to talk about your specific application needs. Redis has more data structures than memcached, and supports richer data operations. Usually in memcached, you need to take the data to the client for similar modifications and set back. This greatly increases the number of network IO and the volume of data. In Redis, these complex operations are often as efficient as the general get/set. So, if you need caching to support more complex structures and operations, then Redis would be a good choice.