LEVELDB reading notes (1) Memory allocation

Memory management is an important piece for any program, and LEVELDB itself implements a simple memory allocator, rather than using some other open source software Tcmalloc to avoid reliance on other software.

What are the benefits of implementing a memory allocator yourself? I think the main points are as follows:

1. The primary role of the memory pool is to reduce the number of calls, such as new, delete, and so on, to reduce the overhead of system calls.

2. Reduce memory fragmentation.

3. Convenient memory usage statistics and monitoring.

Arena request memory by block, each block size is 4k (this value should be related to paging size), and then use vectors to save the first address of these blocks, the model is as follows:

The member variable is called with the Publlic interface:

classArena { Public:   //returns the allocated memory block  Char*Allocate (size_t bytes); //returns the allocated memory block, the first address satisfies the byte alignment  Char*allocatealigned (size_t bytes); //estimated size of memory used (because of vector with STL, exact size is not OK)size_t Memoryusage ()Const {    returnReinterpret_cast<uintptr_t>(Memory_usage_.  Nobarrier_load ()); } Private:  //Allocation State  Char* ALLOC_PTR_;//point to Current 4k block usage Progresssize_t Alloc_bytes_remaining_;//Current 4k block remaining size//Array of new[] allocated memory blocksstd::vector<Char*> Blocks_;//record the first address of each block of memory requested for easy recovery//Total memory usage of the arena.Port::atomicpointer Memory_usage_;//estimates with memory size already in use}

Memory allocation Policy:

1. Use the remaining memory directly when you need the remaining memory size to meet the allocation requirements (a chunk was previously requested, and some of it was useless)

Otherwise you need to reapply a piece to the system.

2. The remaining memory size of the current block does not meet the allocation requirements, and requires a large allocation of memory (>4096/4 = 1k), separate application for a separate piece of memory.

3. If the current block has insufficient memory and the new allocation requirement is less than 1k, another chunk of 4k is requested, and the remainder is returned to the caller, and the rest is for the next use.

The source of the comments also said, the 2nd is to avoid excessive memory waste, why this can be avoided? Consider a situation where:

If the current block is still 1k in size, the allocation demand is 1025 bytes > 1k, do not follow the above practice, it is necessary to new application a 4k block from the 1025 bytes return, but in doing so, the last piece of the remaining 1k will no longer be used, this is a waste. Instead, the remaining 1k memory can continue to be used.

So this avoids a big chunk of waste, but it's still possible to waste 1k of memory, why not set this value to a small amount? That is almost the same as using new directly, lost the original meaning of the memory allocator, set to this value is a result of the pros and cons.

Specific implementation:

InlineChar*arena::allocate (size_t bytes) {//The semantics of what to return is a bit messy if we allow//0-byte Allocations, so we disallow them here (we don ' t need//them for we internal use).Assert (Bytes >0); if(Bytes <=alloc_bytes_remaining_) {    Char* result =alloc_ptr_; Alloc_ptr_+=bytes; Alloc_bytes_remaining_-=bytes; returnresult; }  returnallocatefallback (bytes);}Char*arena::allocatefallback (size_t bytes) {if(Bytes > Kblocksize/4) {    //Object is more than a quarter of our block size. Allocate it separately//To avoid wasting too much space in leftover bytes.    Char* result =allocatenewblock (bytes); returnresult; }  //We Waste The remaining space in the current block.Alloc_ptr_ =Allocatenewblock (kblocksize); Alloc_bytes_remaining_=kblocksize; Char* result =alloc_ptr_; Alloc_ptr_+=bytes; Alloc_bytes_remaining_-=bytes; returnresult;}Char*Arena::allocatenewblock (size_t block_bytes) {Char* result =New Char[Block_bytes];  Blocks_.push_back (result); Memory_usage_. Nobarrier_store (reinterpret_cast<void*> (Memoryusage () + Block_bytes +sizeof(Char*))); returnresult;}

In addition, Arena provides a way to guarantee direct alignment:

Char*arena::allocatealigned (size_t bytes) {Const intalign = (sizeof(void*) >8) ?sizeof(void*) :8; Press a pointer to the size of the byte to which the minimum is 8 assert (align& (align-1)) ==0);//ensure that the power to its large hour 2size_t current_mod = reinterpret_cast<uintptr_t> (alloc_ptr_) & (align-1); The equivalent of the align to take the mold size_t slop= (Current_mod = =0?0: Align-current_mod); Need to fill the size size_t needed= bytes +slop; Really need the sizeChar*result;
The following is the same as the normal distribution process.if(Needed <=alloc_bytes_remaining_) {Result= Alloc_ptr_ +slop; Alloc_ptr_+=needed; Alloc_bytes_remaining_-=needed; } Else {    //Allocatefallback always returned aligned memoryresult =allocatefallback (bytes); } assert ((Reinterpret_cast<uintptr_t> (Result) & (align-1)) ==0); returnresult;}

Summarize:

LEVELDB implementation of the memory allocator is very simple, a bit shabby feeling. Compared to leveldb only with a vector maintenance, C + + STL implementation of the default memory allocator is much more fine, it is 8, 16, the 、...... Byte size to do a multilevel management, the current level of memory can be used for the next level of use, there is little memory waste, but also to maintain the structure of the higher complexity, but also need to save additional redundant information.

LEVELDB reading notes (1) Memory allocation