How to store a large number of key => value values?

There are a large number of key & amp; gt; value pairs, such as: xxxx & amp; gt; aaaaaaa. The value will not be changed once it is created, and the number will reach millions in the future, it is frequently accessed and the concurrency is not high, but data integrity must be ensured. Which solution is better to store? There are a large number of value pairs in the form of key => value, such as: xxxx => aaaaaaa. The value will not be changed once it is set up, and the number will reach millions in the future and will be accessed frequently, the concurrency is not high, but data integrity must be ensured. Which solution is better to store?

Reply content:

There are a large number of value pairs in the form of key => value, such as: xxxx => aaaaaaa. The value will not be changed once it is set up, and the number will reach millions in the future and will be accessed frequently, the concurrency is not high, but data integrity must be ensured. Which solution is better to store?

Actually, the answer is a little more complete.

• In addition to the "frequent access and low concurrency" access method, is the access method a single key query or multiple queries? Is Data ordered? Is there a requirement similar to "Search for keys with a prefix of foo? Is there a need to "list 10 keys? Is there a need to "list all keys? Are access keys random or some hotspot key values concentrated? What is the ratio of hot and cold key values? Currently, the concurrency is not high. Do you need to consider the scheme after the concurrency increases?

• Complete data. Do you need to back up the data? How often does the key increase? Is the newly added key accessible immediately?

• What is the length of key and value? One million records 1 kb is a memory, and one million records 1 MB is a disk.

Generally, if the conditions are not extreme, you can choose redis & backup, single mysql & backup, or mysql + redis, check what you want and what resources you have in your hands to find a balance.

PS: Don't forget that mysql is also divided into mysql on ssd and mysql on mechanical disk.

Redis is easier to store in memory or hard disk.

The concurrency is not high, the database is enough, and a single table of millions is enough. If the concurrency is high, add a layer of memcached.

From your needs, we mainly aim at two points: the value will not be changed once it is created, and the data integrity will be ensured.
Databases cannot be run. Currently, databases are the first choice to ensure data integrity, and they are also the most common and trustworthy. In the future, the number of requests will reach the level of millions, which is frequently accessed. If the concurrency is low, you can consider using the memory database to provide access efficiency, such as redis, use a common k-v structure for the data structure.

Memcached

Hash the key to different files, and decide whether to add a layer of memory cache based on the traffic.

Use mysql for storage, and then add redis for caching.

If it is only Key-Value, you do not need to store it in the database. you can use kv nosql. if persistence exists, you can consider redis, SSDB, and so on. on Nosql, key-based sharding can be easily implemented to solve your data capacity and access volume problems.

Use apc directly. This is relatively simple!