Those pitfalls of MongoDB (developer perspective)

Comments: This article describes the pitfalls encountered from the perspective of common developers. Most problems can be avoided, but it still has some reference significance for most developers who are new to them. MongoDB is a popular NoSQL document-based database. It provides some excellent features, such as automatic failover, automatic sharding, and schema-less schemaless,

Comments: This article describes the pitfalls encountered from the perspective of common developers. Most problems can be avoided, but it still has some reference significance for most developers who are new to them. MongoDB is a popular NoSQL document-based database. It provides some excellent features, such as automatic failover, automatic sharding, and schema-less schemaless,

Comments:
This article describes the pitfalls encountered from the perspective of common developers. Most problems can be avoided, but it is of some reference significance for most new developers.

MongoDB is currently a popular NoSQL document-based database. It provides some excellent features, such as the automatic failover mechanism, automatic sharding, and schema-less schemaless. In most cases, the performance is also great. However, mint encountered a lot of problems when using MongoDB in depth. Below we will summarize several pitfalls we have encountered. Note: The MongoDB version we currently use is 2.4.10, which has been upgraded to MongoDB 2.6.0. the problem persists and is rolled back to version 2.4.10.

MongoDB database-Level Lock

Potholes index: 5 stars (up to 5 stars)

MongoDB locks differ greatly from general relational databases such as MySQL (InnoDB) and Oracle. InnoDB and Oracle can provide row-level granularity locks, while MongoDB can only provide database-level granularity locks, this means that when a MongoDB write lock is in use, other read/write operations are required.

It seems that the database-Level Lock has serious problems in a large concurrency environment, but MongoDB can still maintain a large concurrency and high performance, because although the MongoDB lock granularity is very rough, however, there is a big difference between the lock processing mechanism and the relational database lock, mainly manifested in:

MongoDB does not support complete transactions, and the operation atomicity only reaches the single document level. Therefore, the Operation granularity is usually small;
The actual time occupied by MongoDB locks is the memory data computing and change time, usually very fast;
MongoDB locks have a temporary discard mechanism. When you need to wait for a slow IO to read and write data, you can give up temporarily and then obtain the lock again after IO is complete.
Usually no problem is not equal to no problem. If the data operation is not proper, it will still occupy the write lock for a long time. For example, in the frontend index creation operation mentioned below, when this happens, the entire database is completely blocked and cannot perform any read/write operations, which is very serious.

To solve the problem, try to avoid occupying the write Lock operation for a long time. If there are some set operations that are hard to avoid, you can consider putting this set into a separate MongoDB library, because the locks of different MongoDB databases are isolated from each other, separating a set can avoid global blocking caused by a collection operation.

Index creation causes database congestion

Potholes index: 3 stars

As mentioned above, the MongoDB database-Level Lock issue is that index creation is a problem that may easily cause long-time write locks. MongoDB needs to occupy a write lock (and will not temporarily give up) during index creation at the front end ), if the data volume of a set is large, it usually takes a long time to create an index, which is particularly prone to problems.

The solution is simple. MongoDB provides two types of index creation access, one is the background mode, which does not require a long time to occupy the write lock, and the other is the non-background mode, it takes a long time to occupy the lock. You can solve the problem by using the background method.
For example, to create an index for posts,
Never use

db.posts.ensureIndex({user_id: 1})

Instead, use

db.posts.ensureIndex({user_id: 1}, {background: 1})

Improper use of embedded embed document

Potholes index: 5 stars

Embed document is a significant difference between apsaradb for MongoDB and relational databases. You can embed other sub-documents in a document so that the parent and child documents can be kept in a single collection, making it easier to search and modify them.

For example, there is a Group document in the mint application scenario. The user applies to join the Group and creates the GroupRequest document. At first, we used the embed Method to place the GroupRequest in the Group.
The Ruby code is as follows (using Mongoid ORM ):

class Group  include Mongoid::Document  ...  embeds_many :group_requests  ...endclass GroupRequest  include Mongoid::Document  ...  embedded_in :group  ...end

This method has dropped us into the trap, and we almost cannot climb it. It causes system problems in nearly two weeks. during peak hours, the system gets stuck for several minutes, the most serious one may even cause MongoDB downtime.

After careful analysis, it is found that group_requests of some active groups increase frequently (when a new application is made) and changes (when a user application is approved or rejected, these operations often occupy the write lock for a long time, resulting in the entire database being blocked. The reason is that when the group_request operation is added, the pre-allocated space of the Group is insufficient, and the space needs to be re-allocated (both memory and hard disk are required), which takes a long time. In addition, the Group has many indexes, moving the Group location causes a large number of Index Update operations to take a long time.

The solution to the problem is also simple, that is, to change the embed Association to a common foreign key Association, which is similar to the practice of relational databases. In this way, adding or modifying group_request only occurs on GroupRequest, it is simple and fast to avoid the issue of occupying the write lock for a long time. When the data of the associated object is not fixed or changes frequently, you must avoid using embed Association, otherwise it will die badly.

Improper use of Array Fields

Potholes index: 4 stars

The Array field of MongoDB is a unique feature. It can store some simple one-to-many relationships in a single document.

Mint has an application scenario that causes serious performance problems. The Code is as follows:

class User  include Mongoid::Document  ...  field :follower_user_ids, type: Array, default: []  ...end

In the User, the follower_user_ids field of the Array type is used to save the id of the person who follows the User. The number of people who follow the User ranges from 10 to 3000, and the changes are frequent, similar to the problems caused by the above embed, frequent modifications to follower_user_ids lead to a large number of database write locks for a long time, leading to a sharp decline in MongoDB database performance.

Solution: we transferred follower_user_ids to the memory database redis to avoid frequent changes to the users in MongoDB, thus completely solving the problem. If you do not use redis, you can create a UserFollower set and associate it with a foreign key.

First, list the above pitfalls. They are all harmless traps. You must pay more attention when using MongoDB to avoid falling into the trap.

References:
1. MongoDB lock mechanism details
2. MongoDB index creation operation documentation

Source: http://xiewenwei.github.io/blog/2014/06/22/trap-in-mongodb/

Original article address: those pitfalls of MongoDB (developer perspective), thanks to the original author for sharing.