Ramcloud: Memory allocation mechanism for memory cloud storage

Now the full flash array has become a Xtremio, EMC's, and Vnx-f (Rockies), IBM Flashsystem. Full Flash is truly efficient, redefining the storage speed. With the ultimate performance, high availability, you greatly improve the efficiency of enterprise-class applications. Mention the advantages of flash, then there is no doubt that speed! But behind the speed advantage, SSD faces the limit of price, capacity and life.

Of course, with the development of technology, the cost of decline, SSDs may replace the mechanical hard drive, the next generation of enterprises to store the main media. A mechanical hard drive may turn into a tape role.

But is the flash speed really the limit of the storage system now? There is now a database that needs to be based on memory, such as Redis,timesten. Also have to mention the excellent practice of caching system memcached. Spark also puts all the intermediate data into memory, avoiding the problem of poor real-time and usability of Hadoop, which could have a profound impact on Hadoop's biosphere. Spark has officially become the top project of the Apache Foundation on February 27, 2014.

Ramcloud, a storage system that uses DRAM completely, all of its data is stored in memory. Of course, in order to recover Ramcloud, the log and data backups are persisted to the normal hard drive. In the fast meeting in 2014, Ramcloud published a paper on memory allocation and management mechanisms, and was named best paper, after reading this article to share experiences.

1. Why not use malloc?

Now many memory allocator, such as the Malloc,google Tcmalloc of C. Of course, there is also Java memory management and GC. But they have several problems, one of which is efficiency, and it is very easy to generate memory fragmentation when the access mode is variable. Experiments show that using malloc can only maximize memory utilization to 50%.

Memory allocator can be divided into two categories: non-copying and copying. Non-copying allocator is not going to move it after the memory is allocated. For a single program, this allocation is very natural, because after all, the size of the memory once it is applied will not change. But for a storage system, a file can be modified, such as increasing or decreasing, so if you use malloc, you need to reapply for a new block of memory, so it's very easy to create fragmentation. For example, the system just started a new 100B size of the file, then some of the files increased to 130B, then, the release of 100B space may produce fragments, such as no longer write <=100b size of the file. (Note: Memory fragmentation is indeed very easy to generate in this memory storage system, and the test case designed by the author does make the malloc memory utilization the largest of 50%.) But whether the actual production environment is like this is now unknown. In short Ramcloud memory allocation and management is very good, but malloc whether in the production environment is really so unbearable, and now can not be conclusive. For malloc memory allocation, read the "malloc memory allocation and free memory release principle".

For copying allocator, the problem of fragmentation can indeed be solved by garbage collector. But the downside is that you need to traverse all the data to redistribute it. However, in the performance considerations, a lot of extra memory is required (1.5 times to 5 times times). This also loses the intention of increasing memory usage through fragmentation management. And there is a problem that requires a long pause in service. With a Java GC, it takes 3-4 seconds for Ramcloud to detect a failed node and recover 64GB of data. (Note: Do not know if it is improper to use Java to make this wronged, although it does have a problem)

2. Ramcloud Overview

Because this article is mainly to explain the log-based memory allocation management, this section mainly introduces the storage management, and why the need for journaling based memory allocation management mechanism, and discard the original.

The most appropriate scenario for Ramcloud is the data center that has divided the server into application servers (primarily implementing application logic such as generating web pages and executing business rules) and storage servers (providing long-term shared storage for application servers). These data centers generally support many applications, some small, using only one part of the capacity of a server, some very large, to use thousands of dedicated applications and storage servers.

Also, the information stored in the memory cloud must be as persistent as the hard disk, and the failure of a single storage server cannot result in data loss and even a few seconds of service being unavailable. Ramcloud stores all the data in DRAM, and performance can be as high as 100~1000 times higher than today's most high-performance hard disk storage systems. In terms of access latency, a process running in an application server in the Ramcloud scenario reads hundreds of bytes of data over the network from a storage server in the same data center with only 5~10μs, At present, the actual system usually spends 0.5~10ms, depending on whether the data is in the server memory cache or hard drive. Also, a multi-core storage server can serve at least 1 million small read requests per second. The same machine in the hard drive system can only service 1000~10000 requests per second.

The system architecture diagram is as follows.