How to optimize 30 GB redis

Suddenly we found that our redis has used 30 GB. Well, this is a very embarrassing number, because our cache machine's memory is currently 32 GB, and the memory is exhausted. Fortunately, last week, the company purchased 90 GB of machines, and now it has migrated to one of them at zero. (In this case, the 90 GB memory is too good. I used 90 GB memory for the second time except for koding.com.

Suddenly we found that our redis has used 30 GB. Well, this is a very embarrassing number, because our cache machine's memory is currently 32 GB, and the memory is exhausted. Fortunately, last week, the company purchased 90 GB of machines, and now it has migrated to one of them at zero. (In this case, the 90 GB memory is too good. I used 90 GB memory for the second time except for koding.com.

Suddenly we found that our redis has used 30 GB. Well, this is a very embarrassing number, because our cache machine's memory is currently 32 GB, and the memory is exhausted. Fortunately, last week, the company purchased 90 GB of machines, and now it has migrated to one of them at zero. (In this case, 90 GB of memory is so cool that I used 90 GB of memory for the second time except koding.com. koding is a good website and online programming IDE .) However, as the data volume grows bigger and bigger, a single machine will never be able to afford it, and transformation is imperative. After preliminary thinking, we come up with a very simple solution, which is to summarize it as "both internal and external"

1. Internal strength Cultivation

Let's start with our application layer to check redis usage. Is there any way to recycle some keys? First, go to the redis server and execute info, and delete the keys.

1: redis 127.0.0.1: 6391> info

2: used_memory_human: 35.58G

3: keyspace_hits: 2580207188

4: db0: keys = 2706740, expires = 1440700

Currently, we only use one DB, but there are too many keys with 144 keys and expired keys. The first thought was to let me know how to have so many keys. The second thought was that there may be too many keys.

Can we see if we can optimize large keys? Unfortunately, the database key size is not displayed on the official website. We can only find a solution on our own.

Google, found that foreign friends have already written shell

Transport: https://gist.github.com/epicserve/5699837

The size of each key can be listed. However, this does not apply to us, because our key is too big and we haven't finished running for nine hours, so we can't afford to vomit. Another option is to use another tool.

Transport: https://github.com/sripathikrishnan/redis-rdb-tools

Unfortunately, this is too heavy. If we don't want to bother ops, we can only pick up our sleeves and create wheels.

Let's take a look at the shell code. It's a good thing to issue a debug object. google found the official website.

Simple debugging information is available, and the rest will be processed.

1: # coding = UTF-8 2: import redis 3: 4: COLOR_RED = "\ 033 [31; 49; 1 m % s \ 033 [31; 49; 0 m" 5: 6: COLOR_GREED = "\ 033 [32; 49; 1 m % s \ 033 [39; 49; 0 m" 7: 8: COLOR_YELLOW = "\ 033 [33; 49; 1 m % s \ 033 [33; 49; 0 m "9: 10: COLOR_BLUE =" \ 033 [34; 49; 1 m % s \ 033 [34; 49; 0 m "11: 12: COLOR_PINK =" \ 033 [35; 49; 1 m % s \ 033 [35; 49; 0 m "13: 14: COLOR_GREENBLUE = "\ 033 [36; 49; 1 m % s \ 033 [36; 49; 0 m" 15: 16: 17: def getHumanSize (value): 18: gb = 1024*1024*1024.0 19: mb = 1024*1024.0 20: kb = 1024.0> = gb: 22: return COLOR_RED % (str (round (value/gb, 2 )) + "gb") 23: elif value> = mb: 24: return COLOR_YELLOW % (str (round (value/mb, 2) + "mb") 25: elif value> = kb: 26: return COLOR_BLUE % (str (round (value/kb, 2) + "kb") 27: else: 28: return COLOR_GREED % (str (value) + "B") 29: 30: 31: month = 3600*24*30 32: result = [] 33: client = redis. redis (host = "XXXXX", port = XXXX) 36: client.info () 37: 38: count = 0 39: for key in client. keys ('*'): 40: try: 41: count + = 1 42: idleTime = client. object ('idletime', key) 43: refcount = client. object ('refcount', key) 44: length = client. debug_object (key) ['serializedlength'] 45: value = idleTime * refcount 46: print "% s key: % s, idletime: % s, refcount: % s, length: % s, humSize: % s "% (count, key, idleTime, refcount, length, getHumanSize (length) 47: Failed t Exception: 48: pass

Write a simple python script to output the size of each key, idle time, and refer count. With so much data combined with awk, We can well count the usage of each key. Note that this size is the size of the key in redis, not the actual size, which is compressed by redis. After analysis, it is found that there is no large key, but some keys have not been accessed Orz for half a year.

Next we will handle it well. We will set the expiration time for each key. If the key is hit, this expire time will be updated. In this way, cold data can be gradually eliminated to achieve cold/hot separation.

2. External skill cultivation

We cleared invalid keys internally. We wanted to achieve horizontal expansion and the single-host load was always limited, so we started the legendary distributed transformation.

The Distributed File seems to be more difficult to understand. Fortunately, we have limited CAP constraints on the cache service. The best way for the cache service to be distributed is consistent hash. In fact, after the transformation, we found that the official website is ready to implement this distributed cache system (drool) but now we still have a backup Twemproxy in development. But we have done it. We will use it first. Wait for the official test and let us know.

Portal:

We have achieved smooth data migration and minimal impact on server modifications. Because phpredis is used in the past, it is extended and the code can be smoothly transitioned.

Our own: https://github.com/trigged/redis_con_hash

In fact, if we pull so much, we need to split up redis data. Single-host bearing is always a bottleneck, but redis is not well-developed in this respect, but it will get better and better in the future.

,