Cache rebuilds that cannot be ignored in the database

The main content of this paper comes from MongoDB official blog, supplemented by Nosqlfan, this paper analyzes the traditional distributed cache system, and points out that it will bring great pressure to the database in cache reconstruction. and analyzes how MongoDB's mmap scheme avoids this problem.

The following diagram of the architecture, in the database front-end with a distributed cache (such as our commonly used memcached), so that the client to access the first time to find Cache,cache do not hit the read the database and cache the structure. This is a commonly used method of sharing the reading pressure.

But there's a problem with this approach, if the front end of the cache hung, or more extreme the entire machine room power, then in the machine restart, the original cache machine in memory will be all empty, in the client access process, will absolutely not hit, so that the database will be in an instant to accept the huge reading pressure.

Imagine if a 64GB cache fails, in its rebuilding, suppose the gigabit NIC connected to the database, assuming that it takes data from the database at an extreme speed of 100M per second to rebuild the cache, it also takes 10 minutes to build, not to mention the ideal scenario for a client-triggered random cache rebuild. It may take a long time. This is also the case that the database can provide 100M per second of data read requests.

We often see some sites hang up and then recover, and then hang up after the recovery, so many times to really recover, because the first time the cache fell, the database can not withstand the corresponding reading pressure, in the cache rebuilt a small part after the death. The equivalent of each restart of the database can restore a portion of the cache until the cached non-hit rate reaches the database to withstand the pressure to truly restore the service.

This problem can be implemented with some caching that can provide persistent functionality, such as Redis, which automatically recovers most of the data from the RDB files that are periodically released when the AOF is not open, and of course, sometimes this can result in inconsistencies between the cache and the database data. Need to select applications based on application scenarios.

This is the problem with distributed cache, and for many database stores, virtually all of the thermal data is put in memory as much as possible. But many databases in the implementation of their own memory in the implementation of the cache mechanism, in this way, when the database is restarted (non-OS reboot), the cache may also be emptied, and the database needs to be rebuilt, and then all the operations of the database may fall on disk IO. Brings the same problem.

Unlike the MongoDB, MongoDB uses mmap to map data files to memory, so when MongoDB restarts, the mapped memory is not cleared because they are maintained by the operating system (so when the operating system restarts, MongoDB will have the same problem). Compared with other databases that maintain cache, MongoDB does not need to be rebuilt and warmed up after reboot.

In addition, Sina Weibo's Timyang also has proposed one kind of cache reconstruction lock way, also can solve this problem partially. In simple terms, when caching is rebuilt, when more than one client initiates a request for the same cached data, the client uses a lock-and-wait approach, and the rebuild of the same cache needs to acquire the corresponding lock, only one client can get the lock, and only the client that gets the lock can access the database rebuild cache. Other clients will need to wait for the client to get the lock to rebuild the cache and read the cache directly, and the result is a database reconstruction access to the same cached data.

Here are some more practical knowledge:

Regardless of which storage you use, it is best to understand the process of cache rebuilding first, if a reboot can cause the database to crash, or to be careful.

Restarting the MongoDB does not cause the MongoDB cache to fail (unless the server is restarted)

When you reload the disk, the file system's cache is invalidated, as is the case with restarting the machine, and MongoDB cannot avoid

A small trick to use MongoDB, when the MongoDB server is just started, you can copy all of its files to dev, which triggers the operating system to read the files, so that the memory allows Maps as many MongoDB data files as possible to physical memory. Of course, if you're in the process of MongoDB, you can determine which files hold data that is hot, or copy the files to dev to gain more physical memory for them.

(Responsible editor: Lu Guang)