Pay attention to cache expiration issues at the same time

[ArticleAuthor: Sun Li link: http://www.cnblogs.com/sunli/ updated by: 2010-09-06

I remember that last year, an application had a resource-consuming backend query and used memcached for a layer of cache. The cache time was 30 minutes (based on business needs, it does not matter if it is longer ), it is reasonable to say that there should be no problem at all, but there is a strange problem, that is, after memcached is restarted during the peak period, a large number of backend queries will occur, and the backend services will not be able to survive. This is understandable. However, as access increases during normal operation, queries are also a burst of accesses to the database, formingRipple.

I still remember that when I first came to the company in, there was a Java application under tomcat, And the cache was built inside JVM (in fact, the cache of many Java applications is built inside the JVM, because it is simple and efficient.) as the business volume increases, it will not support access traffic and will crash frequently, shutdown and restart Tomcat is a big problem. It didn't start directly at all. After the startup, it will crash again a few seconds, and the traffic can only be blocked on nginx, it takes about 10-20 minutes for Tomcat to start up and wait for the internal cache to be established.

The two problems above are caused by cache expiration at the same time. For example, if the data of a function of an application is set to cache for 30 minutes during caching, a large number of backend queries will be generated after the cache is restarted to recreate the cache, A large amount of data is set to expire in 30 minutes. After 30 minutes, you need to query the backend to reconstruct the cache after expiration, this poses unnecessary risks to the database. For example

Figure 1: cache-expired ripple dB penetration

 

Solution:

In a large application, the cache should be isolated from the Web application as much as possible to avoid the impact of rebuilding the cache on the backend caused by application restart. (In fact, this can improve the cache utilization of multiple web pages)

The cache service should be used as many as possible to avoid the impact of rebuilding all the caches on the backend. Multiple cache services should use consistent hash mode whenever possible.

Cache expiration at the same time. For example, if the cache expires within 30 seconds, you can find a range that can be received by the service. For example, if a random expiration time is set between 25-40 seconds, the probability of simultaneous expiration can be reduced.

You can consider using multi-level cache.

In fact, when designing a system, you need to consider the impact of cache failure on the system and how to avoid these problems. Restart the system to consider a cache push process.