BOLTDB Source Analysis-Introduction-0

This is a creation in Article, where the information may have evolved or changed.

Boltdb

boltdbis an embedded K-V storage engine implemented by Golang. In the source code of BOLTDB, Doc.go has a brief description of it. It provides transaction, acid language, no lock MVCC support. The data provides 0 copies, B-tree indexes. Its main design stems from Howard Chu's Lmdb.

Persistence of

boltdbA single file is used as a persistent store. It divides different file parts into different page, different page stores different types of data (meta, key, value) and so on. After repeated additions and deletions, the file may appear without data. There boltdb is no intention to move the data and truncate the file to return the extra space to the operating system. Instead, these parts are added to the internal FreeList to maintain, and when new data is written, the space is reused. Therefore, from the user's perspective, boltdb the persistence file will only increase, and will not be reduced because of data deletion.

boltdbWhen writing data, in order to improve efficiency, unsafe the method used directly obtains the related struct system memory layout, but does not use some serialization tools. Therefore, the data format recorded in the persistence file is operating system-related. Therefore, it is not possible to copy persistent files from a big-endian system to a small-end system.

Because of the boltdb B-tree index, there is a random write, so in a scenario where there is a write bottleneck, it can only be improved by using SSDs.

When the user writes a key-value pair, the boltdb associated B-tree index is written according to the specified file offset.

Its 0 copy is characterized primarily by mapping the persisted file in read-only mode to the mmap memory space, and then by indexing the space pointed to by the value of the key in memory, and then returning the memory to the user. Therefore, when the database stores more data, it may show that the memory usage of the program is very high. boltdbin the file mmap mapping strategy is relatively simple, the overall mapping of the file into memory space, the full use of the operating system virtual memory space and physical memory dynamic mapping, swap out mechanism, and not to do dynamic mapping changes, in the address space is relatively small 32bit system may bring some restrictions.

Using a read-only default to map also protects persistent files, preventing user programs from going out of memory and polluting persistent files.

This also poses a problem, boltdb is a read-only memory that the user cannot modify. and the memory life cycle exists only in the relevant transaction duration.