The internal structure of MongoDB from the MongoDB the definitive guide

Today downloaded the "MongoDB The Definitive Guide" electronic version, browsing the content inside, or quite rich. is a supplement to the actual application of official website documents. Similar to the official documentation, the introduction of MongoDB's internal principles is minimal, and only a section of the appendix describes the relevant content.

MongoDB is like a big black box for most mongodb users, but it will help you better understand and use MongoDB if you can learn about some of the constructs inside MongoDB.

BSON

In MongoDB, a document is an abstraction of the data that is used in interaction between the client side and the server side. All clients (driver in various languages) use this abstraction, and its expression is what we often call Bson (Binary JSON).

Bson is a lightweight binary data format. MongoDB is able to use Bson and store Bson as data storage on disk.

When the client wants to write to the document, use the query, and so on, the document needs to be encoded in Bson format and then sent to the server side. Similarly, the return result of the server side is encoded in Bson format and put back to the client side.

Use the Bson format for the following 3 purposes:

Efficiency

Bson is designed for efficiency and requires little space. Even in the worst case scenario, the Bson format is much more efficient than the JSON format, which is best for storage.

Transmission of

In some cases, Bson will sacrifice additional space to make data transfer more convenient. For example, the transmission prefix of a string identifies the length of the string, not the end tag at the end of the string. This form of transmission facilitates mongodb to modify the transmitted data.

Performance

Finally, the encoding and decoding of the Bson format is very fast. It uses a C-style data representation that can be used efficiently in a variety of languages.

For more information about Bson, refer to:http://www.bsonspec.org.

Write protocol

The client side accesses the server side using a lightweight TCP/IP write protocol. This protocol is described in detail in the MongoDB wiki, which actually makes a simple wrapper over the Bson data. For example, the command that writes the data contains a 20-byte message header (consisting of the length of the message and the Write command identifier), the collection name to write, and the data to write.

Data files

In the MongoDB Data folder (the default path is/data/db), all the files that make up the database. Each database contains an. ns file and some data files, where data files become more numerous as the amount of data increases. So if there is a database named Foo, then the file that makes up Foo is composed of foo.ns,foo.0,foo.1,foo.2 and so on.

Each new data file will be twice times the size of the previous data file, with a maximum of 2G per data file. This kind of design is helpful to prevent the database with small amount of data to waste too much space, at the same time can ensure the database with large data volume has corresponding space use.

MongoDB uses a pre-allocation approach to ensure stable write performance (this can be turned off using--noprealloc). The pre-allocation is performed in the background, and each pre-allocated file is populated with 0. This allows MongoDB to always keep extra space and free data files, thus avoiding the congestion caused by the allocation of disk space due to the rapid growth of data.

Namespaces and disk areas

Each database consists of multiple namespaces, each of which stores the corresponding type of data. Each collection in a database has its own namespace, and the index file also has a namespace. The metadata for all namespaces is stored in the. ns file.

The data in the namespace is divided into multiple intervals in the disk, called the disk area. In, foo This database contains 3 data files, and the third data file belongs to an empty pre-allocated file. The first two data files are divided into corresponding extents corresponding to different namespaces.

Shows the relevant characteristics of the namespace and the area. Each namespace can contain a number of different extents, which are not contiguous. As the data files grow, each namespace corresponds to the size of the extents that are increasing as the number of allocations increases. This is done to balance the wasted space in the namespace with the continuity of the data in a given namespace. There is also a namespace to be aware of: $freelist, this namespace is used to record the extents that are no longer used (collection or indexes that are deleted). Whenever a namespace needs to allocate new extents, it will first see if the $freelist has the right size to use.

Memory-mapped storage engine

MongoDB currently supports the memory mapping engine for the storage engine. When MongoDB starts, all data files are mapped to memory, and the operating system hosts all disk operations. This storage engine has the following characteristics:

* The code for memory management in MongoDB is very streamlined, after all the relevant work has been managed by the operating system.
* The virtual memory used by the MongoDB server will be very large and will exceed the size of the entire data file. Don't worry, the OS will handle all this.
* MongoDB cannot control the order in which data is written to disk, which will cause MongoDB to fail to implement the Writeahead log feature. So, if MongoDB wants to provide a durability feature (this feature can be referenced in my writing about Cassandra article: http://www.cnblogs.com/gpcuster/tag/Cassandra/), Another storage engine needs to be implemented.
* 32-bit System MongoDB server each Mongod instance can use only 2G of data files. This is because the address pointer can only support 32-bit.

Other

There are only so many internal MongoDB constructs introduced in MongoDB the definitive guide, and if you really want to make it clear, you may need another book to tell. For example, the internal JS parsing, query optimization, indexing and so on. Interested friends can directly refer to the source code:)

MongoDB's internal structure from MongoDB the definitive guide