Question and contrast test of memory footprint using redis different data structures in C #

recently in a large number of use of Redis for data statistics before the cleaning and finishing, the daily data volume of over 50 million +, in the development process, the data volume is small, pay attention to business rules processing, on-line basic test found a large number of problems, one of which is the Redis storage data too much, The amount of memory used is greatly increased. In a simple analysis, the storage of very frequent entity class has been improved, the field name is abbreviated processing, all of a sudden reduce the amount of memory usage. In the study of Redis, the following article was found:Some of the pits on Redis-the American Regiment, found one of the sections: "Four, Redis Memory usage optimization", the use of different redis storage structure is compared, it is very interested in this, Also found themselves in the use of the process may be stored errors, so according to their own business situation carried out the same test, to see if there is no room for optimization.

1. Test environment and comparison project

C # 4.0 + Servicestack.redis 3.9 + Windows Redis 2.6.2

Test the number of keys stored under the same data structure 1 million:

1) General K-V structure storage

2) List Structure storage

3) isolated hash structure storage

4) Multiple hash structure storage

Here's a look at the simple code and results, for simplicity, we use the same entity structure and the same data, which is used in the business to obfuscate the field values.

Since the storage structure of the above-mentioned group of articles is relatively simple, I have chosen a physical structure that is closer to actual use. 7 fields, the value type also basically has.

Static Sendscanmsg getentity () {return new sendscanmsg () {SN = "test Test", NN = "Test", SM = "Color height 7", SC = "123445.888", P = "A5911590 1094 ", ST = Convert.todatetime (" 2016-01-18 09:54:53 "), RT = Convert.todatetime (" 2016-01-18 10:59:44 ")};}

2. Separate Key-value storage

Simple code, where the Redis operation has been encapsulated to understand the meaning. K-v storage can be imagined that the storage space is the largest, because to a certain number of levels, key length is very important, also accounted for memory.

public static void Testkeymemorysingle () {String key = "701183714183_8801_6222"; var model = GetEntity (); Int32 N = 1000000;f  or (int i = 0; i < N; i++) {msgredis.redishelper.item_set<sendscanmsg> (key + i.tostring (), model, new TimeSpan (10, 0, 0));}}

The results are as follows:

# Memoryused_memory:317253140used_memory_human:302.56mused_memory_rss:317253140used_memory_peak:317253324used_ memory_peak_human:302.56mused_memory_lua:31744mem_fragmentation_ratio:1.00mem_allocator:libc

3. List structure Storage

Look at the code, list structure only array type, theoretically also the most space-saving, because there are only 1 keys to see the results:

public static void Testkeymemorylist () {String key = "701183714183_8801_6222"; var model = GetEntity (); Int32 N = 1000000;for (int i = 0; i < N; i++) {msgredis.redishelper.list_add<sendscanmsg> (key, model);}}

The results are as follows:

# Memoryused_memory:220861160used_memory_human:210.63mused_memory_rss:220861160used_memory_peak:410351028used_ memory_peak_human:391.34mused_memory_lua:31744mem_fragmentation_ratio:1.00mem_allocator:libc

4. Separate hash structure

The code is as follows, similar in principle to the above. In actual use, the hash structure is used very often, but there are some corresponding inconvenient, such as can not set the expiration time for a single key, can only set the expiration time for the whole hash key, cannot be paged, etc., the specific use according to the circumstances of choice.

public static void Testkeymemoryhash () {String key = "701183714183_8801_6222"; var model = GetEntity (); Int32 N = 1000000;for (int i = 0; i < N; i++) {msgredis.redishelper.hash_set<sendscanmsg> (Key,i.tostring (), model);}}

As a result, you can see that the results are not much more than the list and should be the cause of the key:

# Memoryused_memory:253009020used_memory_human:241.29mused_memory_rss:253009020used_memory_peak:471307216used_ memory_peak_human:449.47mused_memory_lua:31744mem_fragmentation_ratio:1.00mem_allocator:libc

5. Multiple hash structure storage

Based on the information provided in the previous article, multiple hash structure storage is more space-saving than a single hash. So I also deliberately contrast, we will be the ID of the hash is divided into 0-100 queue, take the remainder implementation:

public static void Testkeymemorysplithash () {String key = "701183714183_8801_6222"; var model = GetEntity (); Int32 N = 100000 0;for (int i = 0; i < N; i++) {msgredis.redishelper.hash_set<sendscanmsg> (key+ (i%100). ToString (), i.tostring (), model);}}

The result is, in fact, a small difference from a single hash, and the reason for the analysis may be related to the specific use of the entity class key. Not all cases are several times the difference. This is also the real purpose of my test to see if the real gap is not so much.

# Memoryused_memory:264309588used_memory_human:252.07mused_memory_rss:264309588used_memory_peak:266261980used_ memory_peak_human:253.93mused_memory_lua:31744mem_fragmentation_ratio:1.00mem_allocator:libc

6. Conclusion

The above results of the direct to for example, because the actual use of the entity structure and the above mentioned article is not the same, so the results are not comparative, we can not completely superstitious my results, specific problems, specific analysis, we can only from the test to find the approximate trend, as to the specific gap will vary according to the actual situation :

Redis uses on key storage and specific business to be related, as to whether the list or hash to figure out its characteristics is not difficult, as for the independent hash structure and multi-hash space-saving problems, most similar, also need to be used on the basis of Business division as well, nor alone in order to save memory space loss of business flexibility. Here is a brief talk about the differences between several data structures:

1. Independent K-V structure: The advantage is that a single store can flexibly set the expiration time, while the same data type has an increased memory footprint, and is not structurally significant in redis;

2. List structure: The list structure benefits can be very flexible to obtain a certain range of data, or paging, but also the most memory-saving, but the entity independent lookup is more difficult; only the entire list structure can set the expiration time;

3. Hash structure: The greatest advantage is that the search efficiency of the element is high, flexible, but the disadvantage is not as a list as the scope of access, but also set the expiration time;

After a period of development, there is a deep understanding of the different problems in the data analysis process and the appropriate structure for business sampling. Each of the advantages and disadvantages of the structure is complementary, as long as patience, careful analysis, in fact, these several structures are very powerful. Time progress, as for the use of Redis experience, the individual has a lot of shortcomings, if there is a problem, please correct me.

Question and contrast test of memory footprint using redis different data structures in C #