Mongodb uses rs to implement ha and backup

Mongodb is deployed on a single node. I found an 8-core, 48 GB RAM server before the National Day and ran for a week under the pressure of more than 100 concurrent servers. No data loss or excessive server load was found.

However, there are still some problems with single-node deployment. The first is that HA cannot be performed. If the mongod is down or the deployed server is down, the application cannot work. The second is not conducive to backup, this is because the additional burden on mongod during Backup may affect the business. The third is that read/write splitting cannot be performed. Therefore, in the production environment, cluster deployment is still considered.

Mongod supports three cluster deployment methods:

1. master-slave

2. replica set

3. sharding

Master-slave can solve the backup problem, but it cannot be transparent, so it is not very good. sharding is a highlights feature of mongo and can be automatically sharded. However, according to my test results, sharding began to reflect the performance advantage after a single set reaches the threshold of 20 million data records, while sharding data is distributed, so the backup is complicated, in addition, more servers are required, which is not conducive to the cost. In the end, it is more appropriate to deploy a cluster using the replica set method. It can solve HA, backup, and read/write splitting problems.

Basically, we use the TOPO recommended on the official website:

Ideally, it is best to deploy three mongod instances on a separate server, which is required. For cost consideration, you can also deploy two secondary instances on the same server, primary needs a lot of memory to process read/write requests (read/write is not separated at the moment), and considering the HA factor, it is better to ensure that a server is exclusive.

I. cluster configuration

1. Start mongod in-replset mode.

You can also use the command line to start it, but it is not conducive to management. Therefore, we recommend that you use the -- config parameter to start it. The configuration file is as follows:

Port = 2222
Bind_ip = 127.0.0.1
Dbpath =/home/zhengshengdong/pai_dbs/db1
Fork = true
ReplSet = rs0
Logpath =/home/zhengshengdong/pai_log/log1.txt
Logappend = true
Journal = true

./Mongod -- config/home/zhengshengdong/mongo_confs/mongo1.conf

Then, start the other two mongod instances.

2. initialize the replica set and add secondary

Use./mongo -- port 2222 to connect to the mongod instance of primary, and then execute the following commands in sequence.

rs.initiate()rs.conf()rs.add("host1:port")rs.add("host2:port")

3. Check

Run the following command on the primary instance:

rs.status()

You should be able to see similar results

Ii. Verify HA scenarios

The following code is written in java driver for verification.

public static void main(String[] args) throws UnknownHostException {ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();final MongoClient client = MongoConnectionHelper.getClient();client.setWriteConcern(WriteConcern.REPLICA_ACKNOWLEDGED);Runnable task = new Runnable() {private int i = 0;@Overridepublic void run() {DB db = client.getDB("yilos");DBCollection collection = db.getCollection("test");DBObject o = new BasicDBObject("name", "MongoDB" + i).append("count", i);try {collection.insert(o);} catch (Exception exc) {exc.printStackTrace();}i++;}};executor.scheduleWithFixedDelay(task, 1, 1, TimeUnit.SECONDS);}

Write a piece of data to the cluster every one second, and then manually shut down the primary. The following message is displayed on the jvm console:

Warning Primary switching from zhengshengdong-K43SA/127.0.0.1: 2222 to mongodb.kyfxbl.net/127.0.0.1:4444

This message indicates that mongo automatically handles primary switching and is transparent to applications. Check the records in mongo and find that the write operation continues after the switchover is complete.

We can see that the primary switching is in progress between 2926 and 2931. After the switching, the application can continue to write data.

However, we can see that the four data entries 2927,2928, 2929,2930 in the middle are lost (mongo's primary switching takes about 3-5 seconds). For the business, although the time is not long, however, the loss of business data is unacceptable.

Next, let's take a closer look at the exceptions thrown by the code during this period:

Com. mongodb. MongoException $ Network: Write operation to server mongodb.kyfxbl.net/127.0.0.:4444 failed on database yilos
At com. mongodb. DBTCPConnector. say (dbtcpconne. java: 153)
At com. mongodb. DBTCPConnector. say (dbtcpconne. java: 115)
At com. mongodb. DBApiLayer $ MyCollection. insert (DBApiLayer. java: 248)
At com. mongodb. DBApiLayer $ MyCollection. insert (DBApiLayer. java: 204)
At com. mongodb. DBCollection. insert (DBCollection. java: 76)
At com. mongodb. DBCollection. insert (DBCollection. java: 60)
At com. apsaradb for mongodb. DBCollection. insert (DBCollection. java: 105)
At com. yilos. mongo. HATester $ 1.run( HATester. java: 36)
At java. util. concurrent. Executors $ RunnableAdapter. call (Executors. java: 441)
At java. util. concurrent. FutureTask $ Sync. innerRunAndReset (FutureTask. java: 317)
At java. util. concurrent. FutureTask. runAndReset (FutureTask. java: 150)
At java. util. concurrent. ScheduledThreadPoolExecutor $ ScheduledFutureTask. access $101 (ScheduledThreadPoolExecutor. java: 98)
At java. util. concurrent. ScheduledThreadPoolExecutor $ ScheduledFutureTask. runPeriodic (ScheduledThreadPoolExecutor. java: 180)
At java. util. concurrent. ScheduledThreadPoolExecutor $ ScheduledFutureTask. run (ScheduledThreadPoolExecutor. java: 204)
At java. util. concurrent. ThreadPoolExecutor $ Worker. runTask (ThreadPoolExecutor. java: 886)
At java. util. concurrent. ThreadPoolExecutor $ Worker. run (ThreadPoolExecutor. java: 908)
At java. lang. Thread. run (Thread. java: 662)
Caused by: java.net. SocketException: Broken pipe
At java.net. SocketOutputStream. socketWrite0 (Native Method)
At java.net. SocketOutputStream. socketWrite (SocketOutputStream. java: 92)
At java.net. SocketOutputStream. write (SocketOutputStream. java: 136)
At org. bson. io. PoolOutputBuffer. pipe (PoolOutputBuffer. java: 129)
At com. apsaradb for mongodb. OutMessage. pipe (OutMessage. java: 236)
At com. mongodb. DBPort. go (DBPort. java: 133)
At com. mongodb. DBPort. go (DBPort. java: 106)
At com. mongodb. DBPort. findOne (DBPort. java: 162)
At com. mongodb. DBPort. runCommand (DBPort. java: 170)
At com. mongodb. DBTCPConnector. _ checkWriteError (DBTCPConnector. java: 100)
At com. mongodb. DBTCPConnector. say (dbtcpconne. java: 142)
... 16 more

It can be found that this is not a mongo issue, but caused by a BUG in the test code:

Runnable task = new Runnable() {private int i = 0;@Overridepublic void run() {DB db = client.getDB("yilos");DBCollection collection = db.getCollection("test");DBObject o = new BasicDBObject("name", "MongoDB" + i).append("count", i);try {collection.insert(o);} catch (Exception exc) {exc.printStackTrace();}i++;}};

This is where the problem occurs. The try statement captures a fatal exception, but the code I wrote is not processed, but the exception is simply printed. After I is auto-incremented, enter the next cycle.

It can also be determined that the java driver provided by mongo will not store failed write operations in the queue and try again later. Instead, it will discard the write operation and throw an exception so that the client can handle it by itself. I think this design is okay, but the client must handle it. Since the js driver is used in the future, I will not continue to study this code, and I will study it carefully in the js driver. The key is to handle mongo exception and set write concern at a higher level. The HA mechanism of replica set itself is feasible.

Iii. Backup Solution

2-level backup:

The first layer is the natural backup provided by cluster deployment. Because there are two secondary nodes, they will always be synchronized with primary. Therefore, the cluster has two complete data image copies at any time.

The second layer uses mongodump and fsync tools provided by mongo to obtain the daily copies of key collections when the service load is low (AM) by means of manual execution or script. Backup collection is also executed on secondary, without additional pressure on primary

The collected backups are stored locally or on other servers to avoid storage corruption and loss of data and backup.

At the same time, when mongod is started, open the journal parameter, so that in extreme cases (the above two types of backup failed), you can also perform manual recovery through oplog