The phenomenon of Redis in the super time

background:

The Redis cache database is one of the most effective ways to mitigate database pressure for quick response to clients, one of which is the failure cache, which has the advantage of releasing a limited amount of memory from the less frequently used business space. But one problem for synchronizing the data between the database and the cache is that when a cached data fails after a large amount of concurrency, it causes multiple concurrent threads to initiate a request to the back-end database to get the same business data (every time the cache fails, we ideally There are 1 threads going to the database to fetch the data, then write the 1 data into the Redis, but in a high concurrency environment, there may be 100 threads to the database to obtain data, and then the 100 data will be written into the Redis, resulting in a large database pressure, a large number of cache failure. So that if a large amount of cache is generated over a period of time, and then there is a large amount of cache invalidation in another period of time, this can lead to a steep rise in the pressure on the back-end database, which is called a "surprise group phenomenon caused by cache expiration".

Processing ideas:

True expiration time in cache time1

stored in cache value for human failure timestamp: time2 (time2 is always less than time1)

The lock lock corresponding to the cache value (that is, another key corresponding to value) is used primarily to determine the number of threads that read Redis value

When the data of the database is written to the cache, there is the first time the client reads the cache, taking the current system time: System_time if System_time >= time2 the default cache is considered expired (if system_time< time1 Then get the lock lock of value, then call the Redis incr function (single line Cheng function) to determine the number of threads that acquire the lock, and return 1 when it is the first thread, and then increment gradually. The first access thread goes to the database to get the latest value to reload the cache and remove the lock lock key and reset the timestamp; All Access value client threads get lock value greater than 1 before the lock is removed, and these threads still read the old values in Redis. The database is not centrally accessed.

Problem Solving python code:

Import JSON import pickle import redis Import time Import math class Redisapi (Redis.
        Redis): Def get_json (self, name): "" If the values set by the old API can be taken in this way:p Aram Name:: return: "" "Value = Self.get (name) if value is None:return None Try:return J
                    Son.loads (value) except Exception:if Value.startswith (b '! '): Try: Return Json.loads (Pickle.loads (value[1:)) except Pickle. Pickleerror:return None Else:return json.loads (Value.decode (). Replace ("'
        ", '" ")" Def get_bylock (Self, Key): "" To avoid the Redis phenomenon, you must cooperate with ' Set_bylock ' using the call method and ' get ' "" "Lock_key = key +". Lock "data = Self.get (key) current = Int (Time.time ()) if no T Data:return None else:real_data = json.loads (data) # if artificially setTimeout timed out if real_data[' expireat '] <= current: # If you get to the lock if Self.set (Lock_key,
                    "X", ex=2, Nx=true): Return None # if not acquired lock else: Return real_data[' data '] else:return real_data[' data ' def set_bylock (self, key, data, E  Xpire_time): "" "to avoid redis the phenomenon of surprise group, please must cooperate with the ' Get_bylock ' using the call method and ' Setex '" "" "" "" "
        Int (Time.time ()) Real_data = {' Data ': Data, ' expireat ': Current + Expire_time-math.ceil (EXPIRE_TIME/2)} Self.setex (Key, Json.dumps (Real_data), expire_time)

The Redisapi class inherits the Redis.redis class, which is the same as the box Redis class, except that when the user uses the method of. Get () and. Setex (), the corresponding replacement is the. Get_bylock () and. Set_bylock () method