10 things that MongoDB should know on a large scale

Asya Kamsky, chief solution architect at MongoDB, recently published an article outlining 10 things to know about running MongoDB on a large scale.

MongoDB also needs DevOps.

MongoDB is a database. Like any other data store, it also requires capacity planning, tuning, monitoring and maintenance. Do not assume that MongoDB does not require proper care and feeding because it is easy to install, get started, and meets developers' paradigms more naturally than relational databases. Being able to run it ultra-fast on small sample datasets does not mean that you do not need good mode, indexing strategies, and the right hardware resources for your production environment. But if you're well prepared and understand best practices, running a large MongoDB cluster can get boring, not a headache.

2, Successful MongoDB users will monitor everything and be prepared for growth.

Tracking current capacity and capacity planning in any database system is a basic exercise, as is MongoDB. You need to know how much work the cluster can now support and what requirements it will handle at maximum usage. If you fail to notice the increased load on your server, you will end up with errors that do not have enough capacity. Monitoring MongoDB MongoDB Management Services (MMS) allows you to visualize your actions by looking at opscounters charts:

3, you may not want the system with performance growth barriers as the usage grows.

According to a large number of users experience deployment, performance bottlenecks are usually (in order):

It turns out that the biggest impact on performance in true large-scale deployment practices is the extent to which pattern design fits the needs of the application. The lack of indexing, indexing or indexing is the second most important factor affecting performance. Disk IO throughput is the next limiting factor when the pattern is perfect and the indexes are optimal, especially for write throughput. Insufficient RAM can cause a lot of page faults and increase the pressure on disk IO.

Many successful MongoDB users use a single replication set.

Too early sharding may be premature optimization, not every MongoDB deployment needs sharding. Fragmentation of very special needs, can not be considered without thinking that it is the best solution to the "database is very slow." Fragmentation does not solve the problem if your coordination pattern is poor or has a wrong index, but you end up with some poor coordination and poor execution fragmentation. When a particular machine on a single machine or a replication set becomes a bottleneck and is unable to add more such resources based on cost considerations, it is suitable for sharding. You may need more disk IO throughput, or more memory, or more storage, or more concurrency, in which case shards make sense.

5, MongoDB can still achieve very good performance even without the entire database in memory.

One common misconception about MongoDB is that you need to put the entire database in memory for better performance. This is probably the most erroneous thing because it depends on the type of load the cluster is processing. There are some signs and indicators that tell you whether you have enough memory relative to the type of load you put on the database. As you can see, as the size of the database grows, the relevant parts that can fit into memory will be limited by the size of the available physical memory. If the amount of memory does not meet the performance requirements, then you will see the page fault, as the page error rate increases, opcounters eventually will be lower than expected.

6, the data must be rewritten to disk.

If the disk utilization reached 100%, then the speed of processing more writes than the current did not get the slightest improvement. The "Background flush average" graph in MMS can be used to see how long it takes to flush dirty pages in data files to disk. Through this trend you will find that as the write operation grows, refreshing will take more time. This problem can be solved by using a faster disk, splitting the work into more slices, or tweaking the application to reduce the amount of data written. You should remember that everything you write is flushed to the disk twice - immediately flushing to the log and refreshing the data file periodically. Separating these two operations onto different physical devices will eliminate their competition for available disk IO bandwidth.

7, copy! = Backup.

Everyone knows the importance of backup. But why backup so important? Presumably because we can recover data when some catastrophic event that affects all replicate set nodes occurs. The reason why replication is not a backup is that it does not allow you to avoid human error - for example, someone suddenly deleted product data or deployed the wrong version of the application code to mess up some or all of the data. There must be a backup that will allow us to recover data from this scenario. Practice data recovery through file system snapshots, mongodump, or MMS backups. The first operation to restore product data from backup should not happen at the time of a real "data emergency."

8, copy set health is not just delayed replication.

The "replication delay" is just one of the indicators that replicates the health of a person. It is just as important to focus on the replication oplog window as monitoring replication latency. It represents the amount of time it takes to completely "roll" the oplog based on the current write traffic. In other words, it refers to the time it takes to take a replicate node and still be able to rejoin the collection without having to resynchronize all of the data. Over time, the copy operation log window will float as the write load changes. The window will be shortened at the peak of traffic. This is very important in capacity planning, and you need to be prepared for the busiest time to absorb data. Below is a parallel view of the MMS that shows the copy operation log window for the entire copy set.

9, MongoDB does not know what kind of data security level.

Like other databases, you should follow the principle of least privilege. You must configure your own database security. Do not let everyone have access to your data. It is important to turn on MongoDB's own security mechanism, but this also locks in access to the cluster from anywhere unless you really think your client process can run there. Modifying only the default port for MongoDB processes does not guarantee security.

10, no need to modify the engine inside things.

Unless the documentation or MongoDB support tells you to do something very special, you do not need to modify the system collection, local, manage, or configure the database directly. You can do this with the help of administrative commands and the shell, and if the database does not run as expected or if something goes wrong, the key to success is not to try to force it to run by manipulating the internal bits directly. The only "special" you need to be familiar with, the system-generated collection is a collection of analyzers, and analyzing your queries on a regular basis is a great way to make sure everything runs as expected.