How to configure a high-availability mongodb cluster

In the era of big data, traditional relational databases must solve the problems of high concurrent read/write, efficient storage of massive data, high scalability and high availability to provide higher services. However, Nosql was born because of these problems.

NOSQL has these advantages:

Large data volumes allow you to store large amounts of data on low-cost servers, so you can easily get rid of the limit on the storage capacity of a traditional mysql single table.

With high scalability, Nosql removes the relational characteristics of relational databases and is easy to scale horizontally. This frees you from the criticism of persistent vertical scaling.

High performance. Nosql uses a simple key-value method to , which is very fast. In addition, NoSQL caches are record-level and fine-grained. Therefore, NoSQL requires much higher performance at this level.

Flexible Data models enable NoSQL to store custom data formats at any time without having to create fields for the data to be stored in advance. Adding or deleting fields in relational databases is a very troublesome task. Adding fields to a table with a very large data volume is a nightmare.

High Availability: NoSQL can easily implement a high-availability architecture without affecting performance. For example, mongodb can quickly configure high-availability configurations through mongos and mongo shards.

In nosql databases, most queries use key-value pairs (keys and values. MongoDB is a product between relational databases and non-relational databases. It is the most like a relational database in non-relational databases. Supports similar to the object-oriented query language, and supports almost most of the functions of querying a single table in a relational database. It also supports data indexing. Therefore, this is very convenient. We can use SQL to operate MongoDB and migrate it from a relational database. The learning costs of developers will be greatly reduced. If you encapsulate the underlying SQL API, the development will not feel the difference between mongodb and relational databases. MongoDB also claims to be able to quickly build a highly available and scalable distributed cluster. There are a lot of articles on the Internet, and we still need to find and modify a lot of things when building it, therefore, we should record the actual steps to be forgotten. Let's take a look at how to build this stuff step by step.

1. mongodb single instance. This configuration is only suitable for simple development and cannot be used in production, because the entire data service is suspended on a single node, as shown in the figure below.

 

Although it cannot be used in production, this mode can be quickly set up and started, and the database can be operated using mongodb commands. The steps for installing a single-node mongodb in linux are listed below

1. Create a mongodb test folder.

# Store the entire mongodb file

Mkdir-p/data/mongodbtest/single

# Store mongodb data files

Mkdir-p/data/mongodbtest/single/data

# Go to the mongodb folder

Cd/data/mongodbtest/single

2. Download the mongodb installation package

Wget http://fastdl.mongodb.org/linux/mongodb-linux-x86_64-2.4.6.tgz

# Decompress the downloaded package

Tar xvzf mongodb-linux-x86_64-2.4.6.tgz

# Enter the mongodb program execution folder

Cd mongodb-linux-x86_64-2.4.6/bin/

3. Start a single-instance mongodb instance

Mongod -- dbpath/data/mongodbtest/single/data

The output log is as follows. Success!

[Initandlisten] db version v2.4.6

........

[Initandlisten] waiting for connections on port 27017

[Websvr] admin web console waiting for connections on port 28017

By default, mongodb provides a web access interface, which can be accessed through IP address + Port.

Http: // 192.168.0.1: 28017/

 

II. Master-slave mode. Mysql databases are widely used. After a dual-host backup is used, the master node fails and the Slave node can take over the master node for further service. Therefore, this mode is much better than single-node high availability.

 

Next, let's take a look at how to build a mongodb master-slave replication node step by step:

1. Prepare two machines: 192.168.0.1 and 192.168.0.2. 192.168.0.1 serves as the master node, and 192.168.0.2 serves as the Slave node.

2. Download the mongodb installation package. Create a folder/data/mongodbtest/master on 192.168.0.1 and a folder/data/mongodbtest/slave at 192.168.0.2.

3. Start the mongodb master node program on 192.168.0.1. Pay attention to the following "-master" parameter, which indicates the master node.

Mongod-dbpath/data/mongodbtest/master-master

The output log is as follows. Success!

[Initandlisten] MongoDB starting: pid = 18285 port = 27017 dbpath =/data/mongodbtest/master = 1

# Log display master node parameters

[Initandlisten] options: {dbpath: "/data/mongodbtest/master", master: true}

........

[Initandlisten] waiting for connections on port 27017

4. Start the mongodb Slave node program at 192.168.0.2. Key configuration: specify the master node IP address and port-source 192.168.0.1: 27017, and specify the Slave node-source parameter.

Mongod-dbpath/data/mongodbtest/slave-source 192.168.0.1: 27017

The output log is as follows. Success!

[Initandlisten] MongoDB starting: pid = 17888 port = 27017 dbpath =/data/mongodbtest/slave = 1

........

# Log display Slave node parameters

[Initandlisten] options: {dbpath: "/data/mongodbtest/slave", slave: true, source: "192.168.0.1: 27017 & Prime ;}

........

[Initandlisten] waiting for connections on port 27017

# The log shows that the Slave node replicates data synchronously from the master node.

[Replslave] repl: from host: 192.168.0.1: 27017

5. Test master-slave replication.

Connect to the terminal on the master node:

Mongo 127.0.0.1

# Create a test database.
Use test;

Insert data to the testdb table.
> Db. testdb. insert ({"test1": "testval1 "})

Query the testdb data to see if it is successful.
> Db. testdb. find ();
{"_ Id": ObjectId ("5284e5cb1f4eb215b2ecc463"), "test1": "testval1 "}

You can see the synchronization log of the host.

[Initandlisten] connection accepted from 192.168.0.2: 37285 #3 (2 connections now open)

[SlaveTracking] update local. slaves query: {_ id: ObjectId ('5284e6268ed115d6238bdb39 & prime;), config: {host: "192.168.0.2: 35271 & Prime;, upgradeNeeded: true}, ns:" local. oplog. $ main "} update: {$ set: {syncedTo: Timestamp 1384441570000 | 1} nscanned: 1 nupdated: 1 fastmod: 1 keyUpdates: 0 locks (micros) w: 132015 132 ms

Check the data of the slave host.

Mongo 127.0.0.1

View the current database.

> Show dbs;
Local 0.203125 GB
Test 0.203125 GB

Use test;
Db. testdb. find ();
{"_ Id": ObjectId ("5284e5cb1f4eb215b2ecc463"), "test1": "testval1 "}

After the query, the data has been synchronized. Check the log and find that the slave host indeed synchronizes data from the host.

Thu Nov 14 23:05:13 [replslave] repl: checkpoint applied 15 operations
Thu Nov 14 23:05:13 [replslave] repl: syncedTo: Nov 14 23:08:10 5284e75a: 1

View Service Status

> Db. printReplicationInfo ();
This is a slave, printing slave replication info.
Source: 192.168.0.1: 27017
SyncedTo: Sun Nov 17 2013 16:04:02 GMT + 0800 (CST)
=-54 secs ago (-0.01hrs)

Mongodb with the master-slave structure has been set up.

In the failover test, can the slave server run normally if the master server fails?

A. Test whether the slave server can be used as the master server, that is, can the slave server be synchronized to the master server?

Connect to mongodb on 192.168.0.2.

Mongo 127.0.0.1: 27017

> Db. testdb. insert ({"test3": "testval3 "});

Not master

As you can see, mongodb slave nodes cannot provide write operations, but only read operations.

B. If the slave server fails, the master server can also provide services. If the master server fails, can the slave server automatically become writable.

Test it!

First, kill the original mongodb master server.

Kill-3 'PS-ef | grep mongod | grep-v grep | awk' {print $2 }''

Test whether the slave server is writable. Connect to mongodb on 192.168.0.2 for test.

> Db. testdb. insert ({"test3": "testval3 "});
Not master

It seems that the slave server does not have the function of automatically taking over the master server. It is only handled manually!

Stop the slave server, start the original data file, and add the master server ID.

Mongod -- dbpath/data/mongodbtest/slave -- master

Wait until the startup is successful (a bit long ). Connect to 192.168.0.2

Mongo 192.168.0.2: 27017

.

> Db. testdb. find ();
{"_ Id": ObjectId ("5288629e9b0318be4b20bd4c"), "test1": "testval1 "}
{"_ Id": ObjectId ("528862d69b0318be4b20bd4d"), "test2": "testval2 "}

Successful!

Multiple slave nodes. Currently, it is only a database server that provides writing and reading, and machine load will encounter bottlenecks. Do you still remember the read/write splitting in mysql? Putting 20% of the write data on the master node and 80% of the read data on the Slave node reduces the server load. However, most applications are under the pressure of read operations. If a node cannot load itself, you can change one from node to multiple nodes. Can one-master-Multiple-slave mongodb be supported? The answer is yes.

 

To facilitate the test, create a folder/data/mongodbtest/slave1 on 192.168.0.2 as another slave server.

Start the slave2 service,

Mongod -- dbpath/data/mongodbtest/slave1 -- slave -- port 27017 -- source 192.168.0.1: 27017.

After successful startup, use mongodb connection test:

> Db. testdb. find ();
{"_ Id": ObjectId ("5288629e9b0318be4b20bd4c"), "test1": "testval1 "}
{"_ Id": ObjectId ("528862d69b0318be4b20bd4d"), "test2": "testval2 "}

Is it very stable to set up this master-slave replication system... Let's look at these questions?

Can I automatically switch the connection when the master node fails? Manual switchover is required.

How can I solve the high write pressure on the master node?

Each data above the Slave node is a full copy of the database. Will the pressure on the Slave node be too high?

Can I implement automatic scaling even if I implement a routing access policy for a slave node route?