MongoDB Data File internal structure

Someone on the Quora asked: What is the organizational structure inside the MongoDB data file ? Then 10gen engineer Jared Rosoff came out to do a short answer.

Each database has its own separate files. If you turn on the DIRECTORYPERDB option, the files for each library will be placed in a separate folder.

The database file is internally sliced into a single block, with each block holding only one namespace of data. In MongoDB, namespaces are used to differentiate between different storage classes. For example, each collection has a separate namespace, and each index has its own namespace.

In one block, multiple records are saved, each record is in Bson format, and the record is connected to the record through a doubly linked list.

The index data also exists in the data file, but the index is organized into a B tree structure instead of a doubly linked list.

For each database, there is a namespace file that holds the metadata corresponding to each namespace. We can find the storage block location of the corresponding namespace by querying these meta-data.

If you turn on the jorunaling log, there will be files that store all of your operational records.

The following image is excerpted from 10gen engineer Mathias Stearn in the MongoSV2011 conference, a hand-drawn data file structure.

1. Each database has corresponding data files and namespace files

2. data file starting from 16MB, the new data file is one times larger than the previous file, the maximum is 2GB

3. The file uses mmap for memory mapping, which maps all data files into memory, but only virtual memory, which is exchanged to physical memory only when the data is accessed.

4.MongoDB data files are mapped to locations in memory tables

5. Using a 32-bit machine, the memory address can be used to identify the maximum 4GB memory

6. However, on 32-bit machines, 4GB of memory will have 1GB of kernel warfare, about 0.5GB will be used in the stack space of the mongod process, only about 2.5GB can be used to map data files.

7. Can represent up to 128TB of space on 64-bit machine

8. Each data file is partitioned into a single block of data, connected by a doubly linked list between blocks and blocks

9. In the namespace file, a hash table is saved, and the stored information metadata for each namespace is saved, including its size, number of blocks, first position, last position, linked list of deleted blocks, and index information

10. These locations are stored through the DISKLOC data structure, storing the data file number and the location of the block in the file

11. For each block, the head contains metadata about the block, such as its position, the position of the previous and next block, and the position pointer of the first and last record in the block. The remainder is used to store the specific data, and the data is connected through a two-way link.

12. Below is the storage structure and working principle of B tree

MongoDB Data File internal structure