Currently, there are 3 kW data, and each piece of data is very small. If we look at the key-value, the key is a 32-bit MD5 string, and the value is about 100 bytes on average.
Now we need to cache the data so that we can still respond quickly when it is not high.
Because the data is basically not divided into hot and cold data, you need to put all the data in the cache.
1. directly generate static files and use nginx to efficiently cache static files.
- At that time, the hardware resources of the server were limited, so this method was adopted, and the source has been used till now.
- Servers are shared through NFS
- Too many small files, inconvenient to manage
- NFS inconvenient O & M and Expansion
- The file content is very small (about 100 bytes), and about kW is about GB.
- However, the file storage is related to the family size of the hard disk partition. Here, the family size of the disk partition is 8 K, so although the file content is only 100 bytes, however, when a single file is actually stored on a disk, it is actually 8 KB.
- Therefore, the 3KW file world occupies about GB of disk space (A waste of space.)
2. Redis (V2.2.11) [KV database]
- I heard from my colleagues that enabling the VM will cause a sharp reduction in performance, So I basically ignore the VM and put all the data in the memory.
- The key is a 32-bit MD5 string.
- Test data: 10 million data accounts for about 20 mb of memory
- Test data: 350 million data accounts for about 1 GB of memory, and the persistent rdb data file is about MB.
- Estimate 3 kW data: 6 GB memory, and the persistent rdb data file is about 2 GB (compressed ?)
- Because Redis doubles the memory during persistence, and considering the growth of data, it is okay to have a machine with 20 GB memory (Disaster Tolerance ).
- Redis is very fast. If there is no problem with the hardware, this is the best choice.
3. Moosefs (MFS) [Distributed File Storage System]
- Mfs supports NFS mounting to local direct operations (for example, if you use mfs, you do not need to change the current architecture)
- The most basic requirement is a Master Server and a data Server)
- File and directory indexes must all be loaded into the memory of the master server. Therefore, the memory of the master server is required.
- Write 30 W files, and the write will start to drop sharply by 20 W (My 5400 RPM laptop hard drive)
- 30 W files, 4 w directory, the master server occupies about MB of memory
- The file storage method seems to be the same as the common file storage (uncertain). The 30 W file occupies about 2G disk space (8 K files ).
- There are as many small files as it is, which is inconvenient for maintenance and Migration (I wonder if I set it wrong ?).
- The memory and hard disk occupy a lot, and the performance is not very outstanding.
- So this is basically not taken into consideration.
