[Busy with blog writing] Memory Optimization for table configuration files: simple memory distributor

Recently, it was the first time in my life that I spent all my time on coding, and even no energy on blog writing. This feeling is not very good. I hope it can be adjusted as soon as possible. I think it is very important to think deeply after working hard.

At present, Excel configuration files are often used in large-scale game development in China. This is a text file, which can be saved in TXT format after being edited in Excel. Except for some special cases, it can be considered as a CSV file separated by line breaks. The Analysis and reading of such configuration files are not difficult to write. You only need to test whether Excel treats special symbols such as quotation marks and line breaks.

This configuration file has one feature: There are a lot of blank cells, and a large number of repeated strings in super large projects will make people feel uncomfortable. Therefore, there is considerable room for optimization when saving this file in the memory. With limited energy, I recorded my conclusions:

Optimization is divided into two parts:

First, analyze the memory allocation optimization when loading tables. As you can imagine, in a large game, there are many tables, rows, and cells, and the number of times allocated when reading tables to the memory is as high as that of the ox. The advantage is that once these configuration files are read to the memory, they do not need to be destroyed, because they may be used at any time during the game. Therefore, it is appropriate to use a simple memory distributor that cannot separately release some of the memory-that is, to repeatedly apply for memory and release all the memory at the last time.

Second, string sharing. The table contains a large number of repeated short strings. If a string library with reference count can be used, there should be a considerable memory optimization space (actually proved by a project ). The reference counting method of STL is not thorough, but optimized to some extent. I just found out today that the STL library of vs2010 has removed the reference counting function because it may be a bug in multithreading.

(I just suddenly thought that if it is a blank string, it takes up only one byte of memory. At this time, sharing will be greatly reduced (instead, it will increase to 1 Int ?). But for 5 ~ It makes sense to optimize a string of 30 bytes .)

In this blog, record the memory splitter I just created. The interface is simple, as follows:

Struct roheap; typedef struct roheap; # ifdef _ cplusplus // This is for the vs compiler. If a CPP file contains this header file, the extern "C ", to prevent the C ++ function name from modifying extern "C" {# endifint roheapinit (roheap ** heap, int page_size); void * roheapalloc (roheap * heap, int byte_size ); void roheapdeallocall (roheap ** heap); # ifdef _ cplusplus} # endif

Implementation: roheap. c

#include "ROHeap.h"#include <malloc.h>#include <assert.h>typedef struct Page{struct Page* next;char* p;} Page;#define SIZE_PAGE_HEAD ((int)(sizeof(Page)))struct ROHeap{Page* head;Page* cur_page;int page_size;};int ROHeapInit(ROHeap** heap, int page_size){ROHeap* h;assert( page_size==0 || page_size>SIZE_PAGE_HEAD );if ( page_size == 0 ){page_size = 4096;}else{page_size = page_size;}*heap = (ROHeap*)malloc( sizeof(ROHeap) );h = *heap;h->page_size = page_size;h->cur_page = (Page*)malloc( page_size );h->cur_page->next = 0;h->cur_page->p = (char*)h->cur_page + SIZE_PAGE_HEAD;h->head = h->cur_page;return 0;}void* ROHeapAlloc(ROHeap* heap, int byte_size){void* ret;assert( byte_size>0  && byte_size <= heap->page_size - SIZE_PAGE_HEAD );assert( heap->cur_page != 0 );if ( byte_size > heap->page_size - (heap->cur_page->p - (char*)heap->cur_page) ){heap->cur_page->next = (Page*)malloc( heap->page_size );heap->cur_page = heap->cur_page->next;heap->cur_page->next = 0;heap->cur_page->p = (char*)heap->cur_page + SIZE_PAGE_HEAD;}ret = heap->cur_page->p;heap->cur_page->p += byte_size;return ret;}void ROHeapDeallocAll(ROHeap** heap){Page* page;Page* next;assert(heap);page = (*heap)->head;do{next = page->next;free( page );page = next;}while ( page != 0 );free( *heap );*heap = 0;}

Simple Description:

1. I used pure C language for implementation. During the compilation process, struct was very difficult (typedef struct was written many times ). In addition, note that the local variable declaration in the C language must be performed at the beginning of the function, and a strange error will be reported in the middle of the function: Error c2143: syntax error: Missing "; "(before" type)

2. vs2010 will determine the compilation behavior based on whether the file extension is C or CPP. when calling the C language interface, CPP should pay attention to the declaration of extern "C", and use macro to protect it, ensure compatibility when compiling to C files and CPP files (see the previous example ). The underlying principle is related to the name modification of c ++ functions. For details, see related books.

3. How to Use the memory leak check tool provided by vs: First, write the following three Macros in sequence:

# Define crtdbg_map_alloc

# Include <stdlib. h>
# Include <crtdbg. h>

Then, call the function: _ crtdumpmemoryleaks ();

In debug mode, the test is very useful. The release version does not overload malloc, so it is invalid. When writing a memory distributor, you must take this for a good test :)