Automatic expansion of NoSQL cluster on cloud

The chief technical officer of Schindler Yuncheng (Speedycloud) Dong at the meeting with a speech entitled "Automation extension of NoSQL cluster on the Yunping platform". Demonstrates how to create and initialize a NoSQL cluster on a cloud host, and how to detect a host failure through a monitoring system, and automatically invoke the cloud API for faulty device replacement, and finally show how to destroy a host from the command line.

But because the time is hasty, did not give the small partner to leave too many time for the question solution and the difficult point detailed introduction, so dong after the meeting will be more concerned about some of the problems of integration, and analysis of the construction process encountered some difficult problems and solutions, the following is his collation of this part of the content:

Hello everyone, I am honored to be the keynote speaker and share the meeting. However, due to the hasty time, did not make full communication with everyone, so after the meeting I would like to solve some of the issues of concern, and to introduce some of us in the construction process encountered some of the difficulties and solutions.

Before that, let me introduce you to the architecture of the NoSQL database MongoDB cluster that we used during the build process and some of its features.

The cluster we are building is a MongoDB replica set (Replica set, which can refer to the official document), and the cluster structure is similar to the following figure (1 main, 4 pairs of structures are used in the presentation, but the principle is the same).

In this cluster there is a primary (Primary) node that receives the write operation, and the remainder is the secondary (secondary) node, which is synchronized oplog from the master node and applied to the local data to synchronize the data, and there is a heartbeat between each node in the cluster ( Heartbeat) detection to ensure interoperability between clusters.

When the primary node in the cluster is unavailable, the remaining nodes automatically generate a new master node in the form of an election, a process that is characteristic of the MongoDB cluster, as shown in the following illustration:

I showed you three scenes:

First, MongoDB the initialization of the cluster. We first called the Speedycloud Cloud API to complete the cloud host to create and start a single MongoDB instance, when the state of the database was confirmed, after the completion of a section of initialization code to complete the entire cluster initialization.

Second, MongoDB cluster failed device replacement. In order to simulate the equipment failure in the cluster, we manually shut down a host, and then through monitoring system detection, automatically discover the fault device, and call the cloud host creation process, and then call the Add member process to achieve the cluster of failed devices automatic replacement.

Third, destroy the entire cluster. When the lifecycle of the whole cluster is over, the automatic destruction process of cloud host can be realized by calling the Speedycloud Cloud API and passing in the ID number of the cloud host.

The whole process involved four technical applications: Speedycloud Cloud API invocation, puppet Configuration management system usage, MONGODB cluster maintenance and Zabbix monitoring system use.

We have built the puppet agent in the cloud host, let it interact with puppet Master, implemented the password modification and some configuration file management, etc. through the Zabbix agent to realize the automatic registration of the equipment to the monitoring system, and monitor the data reporting function At the same time, the discovery of the failed host is implemented through the trigger of the Zabbix server, and the processing script call after the fault is discovered. The overall structure of the system is shown below: