The problems and solving techniques in the process of setting up MongoDB replica set

in my previous cognition, once a system is officially on-line, most of the servers will not be migrated easily, especially those involving multiple related applications involving multiple hardware servers, because this migration will be reaching.
However, there is still such a situation, as I am mainly responsible for the past few days, is a system of all the server migration part of the machine migration, and a part of others are responsible.
This system involves flume data acquisition, storm data analysis, RABBITMQ message distribution, Ehcache caching to improve system performance, MongoDB replica set storage data, Tomcat management system applications, and so on, the architecture is basically as follows:

And here I am mainly responsible for is RABBITMQ, Tomcat, Ehcache, MongoDB, here side tomcat, Ehcache installation and configuration are relatively simple, but rabbitmq need to rely on Erlang. And Erlang had to rely on something else, which required root to execute yum, and we didn't have root privileges, and it took a little bit of effort.
In addition, MongoDB replica set also slightly encountered a little bit of a problem, but fortunately all in accordance with the expected development, has not encountered before the problem also through the experience of the perfect solution.

The reason why the MongoDB replica set will encounter some problems, mostly because this is not done in person, but by the new people with the operation.
First of all, according to the document that I gave him step by step, the result on the port, do not know because of the habit or because of what, he set the port is not the port we require.
So at this point when I asked him to change to the required port, he was a bit dazed, did not know whether to remove all the configuration of the re-match, or what.
Because of the time, I offered him a solution to get rid of the port directly using the configuration priority. I have written a copy of the set of documents should have said the priority of how to change, basically the following three steps:

config=rs.conf()config.members[0].priority=2rs.reconfig(config)

So according to this, we envision the port should be the following (the bottom IP and port is just a casual hypothesis, the production environment can not be casually disclosed):

config=rs.conf()config.members[0].host="192.168.117.88:37017"rs.reconfig(config)

But the result, in the third step when the exception, unfortunately, only to solve the problem and not, forget the specific is what is abnormal, but the general meaning is that the port member does not exist.
So, I gave him a second plan, that is, three members of the non-primary service of any one from the member to remove:

rs.remove("ip:port")

Then change the port of this machine to the 37017 we need, and then add it with the added member command:

rs.add("ip:port")

Then after doing this three times, the ports of the three servers are successfully modified to the required ports.
This process, when modified to the main service, because the first set the highest priority, it is necessary to set the other first to a higher priority operation.
The problem is solved, but I thought about it afterwards, it seems that this scheme is not the best of the most concise, because at that time did not think about the first solution that the cause of the problem, and later thought, mostly because the machine or the original port has not been restarted.
If we first change the port of the non-primary service machine to reboot, then it should be feasible to use the first solution again, and it will be simpler than the second one and have the opportunity to try it.

I thought it would be okay, and then, when we were all migrated, we were told that the machines were in the network segment of the test service, to be converted into a production network segment.
As a whole, all the machine IPs have changed so that our MongoDB replica set is reconfigured again.
But this time the trouble is, the previous change port because I can at least guarantee that there are two machines or normal operation, you can operate the RS command, but this time the IP changes, I three machines can not normally become the main service, so that the RS command failed.
A few times toss, always did not come up with a good plan, so had to the data directory content deleted, and then really reconfigure again.
However, in this novice operation, the configuration process, the command should be the following:

config={_id:”reptest”,members:[{_id:0,host:”192.168.0.160:57017”},{_id:1,host:”192.168.0.211:57017”},{_id:,host:”192.168.0.213:57017”}]}rs.initiate(config)

Made it this way:

config={_id : "Reptest ", Members:[{_id : 0 , Host:"  192.168  .0  .160 : 57017  "},{_id : 1 , Host: "192.168  .0  .211 : 57017  "},{_id :, Host: "192.168  .0  .213 : 57017  "}]}rs . initiate  ()  

In other words, although he was assigned to config, but it was not used when the load, it is also strange to forget to tell him before the discovery of a problem.
usually we operate on the Windows system on the Linux application, will use the CRT or putty these tools, the two tools each have advantages and disadvantages, and I found that when we go into the MONGO shell operation, the two tools are different, using putty can be rolled back, The CRT is no longer able to fall back in the MongoDB shell.
So when he finishes Rs.initiate () and wants to return to the parentheses with config, there is no turning back, just bite the bullet and enter.
At this time, rs.initiate () can only execute once, and then the action in my document is not the same, what should I do?
After the last problem, this problem seems to be a good solution, how to do it, I think it can be used Rs.add and modify the configuration of the way to solve, and then the idea told him, he took this operation, sure enough to do it once!
OK, this time the two problems basically solved, I do not know other friends, whether there is a better solution to this situation? Welcome to leave a message.

The problems and solving techniques in the process of setting up MongoDB replica set