Redis memory fragments

Redis memory fragments

Redis uses jemalloc to allocate memory by default. According to the best article in fast '14, when a variable-length key-value load occurs, the fragmentation problem may occur: low memory utilization, the actually allocated memory is more than the required memory. Therefore, I used YCSB to test Redis memory.

1. Stage 1 of the experiment: insert 1st records into Redis, with the key being 20 bytes and the value being 1 million bytes. Both of them are fixed length. Theoretically, the valid data is about 100 MB.
2. Phase 2: 0.9 million update requests are generated. Requests are evenly distributed randomly (each key has the same probability of being selected). The key value remains unchanged, but the value changes to 130 bytes. An update operation is equivalent to deleting the original 100 bytes and then inserting another 130 bytes.
3. Phase 3: 0.9 million update requests are generated, and the value is changed to 170 bytes.
4. Phase 4: 0.9 million update requests are generated, and the value is changed to 210 bytes.
5. In the next 12 phases, each time there are 0.9 million update requests, and the value length increases progressively.
6. In the last phase, redis is restarted without any request.

After each stage ends, run the info command to view the memory of redis. The result is shown in.

Used indicates the memory allocated by redis using jemalloc, and rss indicates the memory occupied by the process (which can be seen as the actual memory allocated by jemalloc ). Rss/used can be used as an indicator to measure fragments. The larger the size, the more serious the fragments are.

As the value increases, the memory allocated by redis increases linearly. Rss is growing faster than used, so the memory fragmentation problem exists. rss/used even exceeds 2.0 at one time (the maximum value reached 440 in 2.06 bytes ), this means that every 1 GB of memory used by redis requires an additional 1 GB. After the peak, rss/used dropped to about 1.6. If we restart redis, fragments will no longer exist.

One noteworthy problem is that redis uses much more memory than the actual data volume. For example, after the first stage, the valid data should be 120 MB, but redis allocates MB of memory. Similar phenomena exist in other stages. The specific reason is unclear, but I feel a little bit more. Therefore, in addition to the fragmentation problem of jemalloc, is redis itself too proud to use the memory?

I also tried to test the experiment with 10 million records. The test method is exactly the same as that of the million records. However, on my 16 GB memory server, the experiment was almost impossible. During the 2nd and 3 phases, redis reported the error of memory allocation failure, resulting in an experiment failure. My guess is that redis's rdb mechanism uses copy-on-write. Under our high update load, redis needs two times of memory to complete rdb.

So I disabled rdb and started the experiment again. This experiment can run to 8th stages (370 bytes), but at 9th stages, the error of memory allocation fails because redis uses more memory than the physical memory. Let's take a look at the valid data volume in the 9th phase. The worst case is 9 million key values of 410 + 20, 1 million key values of 370 + 20, and about 4 GB of data. However, redis cannot complete the task when rdb is disabled! (Considering that rss/used is close to 2, 8 GB memory is required even if there is no overhead)

The actual situation is: The above experiments are extreme, and the actual load may not be so abnormal. Therefore, I tested the load published by facebook (that is, the length of the key is subject to the extreme distribution, and the length of the value is subject to the Multi-Party distribution), and found that jemalloc has no memory fragmentation problem, rss/used is always around 1.03. Therefore, the memory fragmentation problem may exist, but it depends on the load.

Install and test Redis in Ubuntu 14.04

Redis cluster details

Install Redis in Ubuntu 12.10 (graphic explanation) + Jedis to connect to Redis

Redis series-installation, deployment, and maintenance

Install Redis in CentOS 6.3

Learning notes on Redis installation and deployment

Redis. conf

Redis details: click here
Redis: click here

This article permanently updates the link address: