recently, MongoDB has been researched, which can be used to build a flexible no schema storage system simply and quickly. in this paper, we use MongoDB to build a set of storage systems with good performance and availability to meet hundreds of billions of dollars through demonstration and analysis requirements. On the selection of NoSQL databases, it is necessary to combine their own
Data Model、
access ModeAnd
costand other aspects of the consideration make a tradeoff (trade off).
So after studying MongoDB (2.6.4 version) has the following features: usability:
1. Support Highly available and flexible service cluster configuration, with master-slave, replica set, auto-shard mode.
2. Document-based queries, high performance, simple query tens of thousands of QPS.
3. Support full-text search.
Consistency:
1. Support in-document update mode, high efficiency.
2. The latest 2.6 version adopts read-write lock, write lock priority, and the granularity of lock is collection level.
Ease of Use:
1. Approximate SQL usage for traditional relational databases.
2. Rich Management Configuration tool.
3. Support User role-based rights management system.
Storage mechanism:
The mmap file + memory index is used. Memory and cache management is the responsibility of the operating system, which degrades performance when the size of the data set is exceeded. Memory control under Linux can be implemented by configuring user Ulimit-u.
Real needs: 1. Need to store flexible semi-structured data for fields. 2. The storage size of billions of records, average 20k per record. 3. Read demand is much larger than write (in 8:2), write in bulk write, read need to support flexible and complex query mode. 4. Write performance: 5000QPS read performance: 2W QPS
Thanks to MongoDB's flexible storage and access, as well as good query performance and scalability and maintenance costs, it is thought to use MongoDB to store about 200 million of the amount of data. According to demand analysis as follows: 1. At 200 million scale, the total storage volume is approximately 4TB. Now the market server hard disk maximum 2TB specifications, the deployment can be considered in LVM mode, first install the 1~2TB disk, waiting for capacity growth in accordance with the demand for convenient expansion. 2. Insert 1000W records with 100 million data already in existence, and each time 4 indexes are written, the QPS can reach a requirement of 8000 to fulfill the write performance. 3. Single-threaded simple reading in the case of QPS can reach 8W QPS, multi-threaded concurrent read Total performance is also close to 8W QPS to meet the query performance of 2W QPS, follow-up if the read performance has increased demand can be considered from the node open Read permission to share read pressure. In the test, read delay < 1ms.
Based on the results of the demand argument, it is feasible to use MongoDB to store the 200 million records. In actual deployment, MongoDB supports many ways to have master-slave, replica set, and Shard. The replica set has the advantage of automatic failover relative to master-slave mode, but it also brings the disadvantage of complexity and machine cost increase. Therefore, after comprehensive consideration, choose the master-slave mode to deploy, select the server configuration to
16 Physical core + 256G memory + 2TB HDDTwo sets of main and standby nodes for the machine. Where the Read permission is opened from the node, while the application layer can initiate read requests to the slave node when the primary is unreadable, on the other hand the front end can allocate some read requests to the slave nodes according to the load. Because the data write to belong to the offline operation, so only need to monitor the state of the master and slave nodes, can restore the good service status in time.
Through the above practice, we have completed the use of MongoDB rapid construction to meet the requirements of hundreds of millions of levels of storage.
Practice thinking of MongoDB application
