MongoDB Hardware and Development standard specification

Big Data Platform Department

Operation and Maintenance Development Group

MongoDB Hardware and Development standard specification

Description

No special circumstances, all of this document as a reference document, if there is a special situation, you need to discuss with the operations development team and development team after the change.

Mongodb

Title hardware standards and research and Development code

Directory

The 1th chapter of MONGODB Configuration Standard specification ....... ..... ................. ....... ..... ....................... 1

1.1 MongoDB Version Select ..... ..... ...................... ....... ..... ....................... 1

1.1.1 Stable version ..... ..... .... ..... ..... ...................... ..... ..... ..... ..... ..... .............. .......... 1

1.2 Research and development of data cluster planning ..... ....... .....?????????.... ....... ..... .................. ......... 1

1.2.1 Standard cluster (REPL set replica set) ... ..... ... ..... ..... ..... ..... ..... ..... ..... ..... ....... ................. 1

The deployment process can be found in the MongoDB high availability cluster C production deployment document V1.0 ...... 1

1.2.2 Standard cluster (REPL set replica set +sharding shard) ....... ..... .................. 1

The deployment process can be found in the MongoDB high availability cluster A production deployment document V1.1 ...... 2

1.2.3 Ultra-large-scale clusters (REPL set replica set +sharding shards) ........ .......... 2

The deployment process can be found in the MongoDB high availability cluster B production deployment documentation V1.0 ...... 2

1.3 Big Data mass data cluster planning ........ ................ ....................... ........ 2

1.2.1 Mass data standard cluster (REPL set replica set) ...... ..... ..... ..... ....... ................. 2

The deployment process can be found in the MongoDB high availability cluster C production deployment document V1.0 ...... 2

1.2.2 Mass Data Standard cluster (REPL set replica set +sharding shard) ........ ....... 2

The deployment process can be found in the MongoDB high availability cluster A production deployment document V1.1 ...... 2

1.2.3 Massive data super-large-scale cluster (REPL set replica set +sharding shard) ... 3

The deployment process can be found in the MongoDB high availability cluster B production deployment documentation V1.0 ...... 3

1.4 Host Configuration Normalization parameters ..... ..... ..................... ....... ..... ....................... 3

1.5 Basic volume allocation principles ..... ..... ..... ...................... ..... ..... ..... ..... .............. ........... 6

1.6 Volume monitoring threshold requirements ..... ....... ..... ..................... ..... ..... ..... ..... .............. ........... 7

The 2nd chapter is about MONGODB research and development code ..... ....... .................. ..... ..... ..... ..... .............. ........... 8

2.1 Library Design ................................................................................................................ 8

2.1.1 MongoDB library design ...... ..... ...................... ....... ..... ................. ......... 8

2.2 Integrated Design ............................................................................................................ 8

2.2.1 MongoDB Collection Design ...... ..... ................... ............................. 8

2.3 Document Design ............................................................................................................ 9

2.3.1 MongoDB Document Design ...... ..... ................... ............................. 9

2.4 Index Design ............................................................................................................ 9

2.4.1 MongoDB Index Design ..... ....... .................. ............................. 9

2.4 Index Design .......................................................................................................... 10

1th. MongoDB Configuration Standard Specification

1.1 MongoDB Version Selection

1.1.1 Stable Version

The default newly installed database uses the MongoDB 3.X Community Edition (see the table below for details), except for systems that are already in use and have a large amount of data and that have not been upgraded.

Version	Suggestions
3.2.10	See Standard installation package folder
3.4.X	Research to be tested

1.2 Cluster planning

1.2.1 Standard cluster (REPL set replica set)

The deployment process can be found in the MongoDB high availability cluster a production deployment documentation V1.0

L Cluster Node: 6 node (1 primary +4 Secondry + 1 quorum)

L Host Configuration: Core64, 256G Memory (request resources on demand see chapter III Computing standards)

L HDD Configuration: Priority SSD/HDD 2T RAID-10

L Capacity and expansion planning: Single node Memory utilization control at 70%, cluster LUN total up to 2T to plan a new large library cluster or to expand the corresponding hardware

1.2.2 Standard cluster (REPL set replica set +sharding shard)

Deployment process See "MongoDB High Availability cluster B production deployment document V1.1"

L Cluster Nodes: 3 nodes

L Host Configuration: Core64, 128G memory

L HDD Configuration: Priority SSD/HDD 5T RAID-10 ( assembled with 1TB MLC SSD )

L Capacity and expansion planning: Single node Memory utilization control at 70%, cluster LUN total up to 5T to plan a new large library cluster or corresponding hardware

1.2.3 Hyper-scale cluster (REPL set replica set +sharding shard)

Deployment process See "MongoDB High Availability cluster C production deployment document V1.0"

L Cluster Nodes: 7 nodes

L Host Configuration: Core64, 128G memory

L HDD Configuration: Priority SSD/HDD 10T RAID-10

L Capacity and expansion planning: Single node Memory utilization control at 70%, cluster LUN total up to 10T to plan a new large library cluster or corresponding hardware

1.3 Basic Volume Allocation principle

Volume type	Volume name	Initial Size G	Storage type	Use	Other Notes
Software volumes	/app/mongodb	30	Local	MongoDB Software	All database shares on this machine
Data volumes	/data/mongodb/*/data where * represents the file name of each instance	On request	Local	Database data	Share to all hosts in the cluster
Log volume	/data/mongodb/*/log where * represents the file name of each instance	On request	Local	Database log data	Share to all hosts in the cluster
Backing up volumes	/mongodb_backup/	500	The site	Receive DP Backup	All database sharing in this machine is currently only required in disaster recovery room

1.4 Volume monitoring threshold requirements

Volume type	Volume name	Warning	Critical
Software volumes	/app/mongodb	90D	95D
Data volumes	/data/mongodb/*/data	90D	95D
Log volume	/data/mongodb/*/log	90D	95D
Backing up volumes	/mongodb_backup/	90D	85%

The 2nd Chapter MongoDB Research and Development code

2.1 Library Design

2.1.1 MongoDB library Design

L MONGODB Database naming specification: DB_XXXX
Prohibit use of any special characters outside of "_" (underscore)

L Library name All lowercase, prohibit use of any special characters other than _, prohibit the use of the name of the library with the beginning of the number, such as: 123_ABC

The library is in the form of a folder, and using special characters or other nonstandard naming methods can cause naming confusion
Database name up to 64 characters

• Before you create a new library, you should try to evaluate the volume, QPS, and so forth of the library, and discuss with the DBA in advance whether a new library should be created, or the library should be built specifically for it.

L cannot be an empty string ("")

L cannot start with $

L cannot contain. (dot) and empty string

Database names are case-sensitive (it is recommended that the database names are all lowercase)

L database name is up to 64 bytes long

L do not keep the same database name as the system, this write database includes: Admin,local,config, etc.

2.2 Integrated Design

2.2.1 MongoDB Collection Design

L MONGODB Collection naming specification: t_xxxx

L Set name all lowercase, prohibit use of any special characters other than _, prohibit the use of the name of the set with the beginning of the number, such as: 123_ABC, prohibit the system begin

The system is a collection prefix
Collection name up to 64 characters
To avoid problems with library-level locks, you should use the "single Library single collection" Structure to write large collections, so you should create new libraries instead of creating new collections in existing libraries for new businesses

L write large collections in one library can affect read and write performance of other collections
If the amount of data in the evaluation single collection is large, you can split a large table into multiple small tables and then store each small table in a separate library, because MongoDB is a library-level lock, so doing so can significantly reduce the lock contention problems associated with concurrent writes

L The collection name cannot be an empty string ("")

L cannot contain a \ n or a null character, which indicates the end of a key

L The collection name cannot be "system." At the beginning, the prefix is reserved by the system itself

The collection name cannot contain the $ character (note: can contain. dot number)

2.3 Document Design

2.3.1 MongoDB Document Design

L The key in the document prohibits the use of any special characters other than _

L try to keep documents of the same type in a collection, and distribute different types of documents in different collections

L The same type of document can greatly improve the index utilization, if the document is mixed storage may appear the query often needs full table scan situation

L prohibit the use of _id, such as: Write custom content to _id

L KEEP the uniform lowercase data as much as possible

L shorten the length of key as much as possible (note is as far as possible!) Will involve performance issues)

2.4 Index Design

2.4.1 MongoDB Index Design

L Use overriding index first

L MongoDB's Combined index usage policy is consistent with MySQL and follows the "leftmost principle"

L index name length should not exceed 128 characters

l should evaluate the query scenario as much as possible, by evaluating the inclusion of a single-column index into the combined index to reduce the number, combined with 1, 2 points

When creating a composite index, you should evaluate the fields contained in the index and try to place the fields with large data bases in front of the combined index

When the amount of data is large, the creation of MongoDB index is a slow process, so should be evaluated before the front line or the data is very large, as needed to create the index used

The index creation of MongoDB is a library-level lock, and the library is not readable and writable when the index is created, so if you need to add an index, contact your DBA

L pay particular attention to the problems that can occur when location-based indexing is established.

2.4 Index Design

The 3rd chapter refer to MONGODB Hardware resource calculation Standard 3.1 CPU Resource computing standard

3.1.1 Replica set

Standard: 64core---------2TB snappy compression algorithm CPU a thread will run full of one core. Suppose that a thread is written to read a thread, and that two threads run out of two CPU cores at close to full load.

----------------------------------------------------------------------------------

Primary Node: Example

CPU core = 2 (System reserved) +12000 (read and write)/1000 core, configure CPU resources as per the actual situation

-----------------------------------------------------------------------------------

3.1.2 Shard Replica Set

------------------------------------------------------------------------------------

Evaluation to be calculated.

-------------------------------------------------------------------------------------

3.2 Computing standards for memory resources

Data Volume: Memory ≈3:1 (minimum standard 5:1)

eg:700g Data Volume:

The minimum amount of 128G memory should be used in replica set mode. Follow the minimum standard or approach principle.

--------------------------------------

700/5=140≈128

---------------------------------------

Each instance machine should use 64GB of memory on the Shard replica set (under 3 shards)

------------------------------------------------------------

700/5=140g=<64g*3

------------------------------------------------------------

3.3 Hard disk resource computing standard

3.3.1 Hard Drive Type

In the case of resource permitting, MongoDB takes priority to SSD-dominated. If the resource is not allowed, an SSD must be used if the following conditions are true.

---------------------------------------------------------------------------------------------

Read more than 100M of data read per second, read queue blocked up, you must use SSD.

Write to read more than 100M of data per second, write queue blocking rise, you must use an SSD.

---------------------------------------------------------------------------------------------

Disk Specifications:

Take 700G data Volume For example: Use disk size ≈ cardinality + Daily growth data volume * 90 days

3.3.2 Hard disk size

By default, the data volume should be kept at least 3 months, depending on the actual data volume growth rate.

3.4 Network Card Resource calculation standard

If the hard disk is an SSD, you can provide a gigabit network card on the master and second standby nodes in the case of superelevation 200m/s writes or reads.

Other cases with Gigabit network card can be.

MongoDB Hardware and Development standard specification