Also talk about the virtual machine ha in OpenStack

OpenStack is an open source project designed to provide software for the building and management of public and private clouds. Its community is home to more than a few companies and 1350 developers, both of which use OpenStack as a common front-end for infrastructure as a service (IaaS) resources. The first task of the OpenStack project is to streamline the deployment of the cloud and bring it to a good scale. As a cloud computing IaaS layer de facto standard, OpenStack has a wide range of applications and industries. So far, the OpenStack community does not have a complete solution for virtual machine ha. At first, the community considered that the ha of the virtual machine was not the cloud platform level, should not be implemented at the cloud platform level, and the ha of the virtual machine should be implemented by application level ha. However, many applications do not default to the application-level ha,openstack and lack of such an important feature. So many companies offer their own solutions for the ha of virtual machines. Recently the community has also begun to focus on the implementation of HA for virtual machines. This article describes the progress of virtual machine ha in OpenStack and the virtual machine ha implementations of several companies. Finally, based on the advantages of various schemes, the implementation of HA for a virtual machine is introduced. I. History and status of Virtual machine ha in OpenStack

The initial discussion of HA for virtual machines in OpenStack can be seen in this article, as an important contributor to the Nova project, and his article has a broad impact on the implementation of HA for virtual machines. This article also gives a basic idea of the implementation of the virtual machine ha, which requires some key components to solve the problem:

Monitoring (monitoring)-the system detects a failure of the virtualization layer

Isolation (or fence, Fencing)-system isolation fault compute node

Recovery (Recovery)-Recovering a virtual machine from a failed virtualization

Below are some solutions for virtual machine ha in OpenStack.

1. Support in Nova

Nova already has some HA capabilities.

1) The evacuate command is provided in Nova to enable the VM to rebuild from the failed compute node on the target node. The implementation of this feature requires shared storage between the source node and the target node.

2) in the ha of the VM, it is very fine to judge whether the compute node is faulty, and at present, every Nova-compute service in OpenStack starts a timer, which periodically writes the heartbeat to the database. This makes it easy to know the state of the compute node from the control node.

But OpenStack only has these features that are not enough to complete the perfect support for VM ha functionality.

1) only through the Nova-compute service to determine the state of the compute node is unreliable, such as only the Nova-compute service failure, or the network flash, it will also cause a heartbeat expiration, so that the HA can not be accurately judged. It is therefore necessary to ensure that the state of the node is accurately obtained by other means. The most important is the best practice deployment of OpenStack, usually the management, business, and storage segments are separate segments, when the service state of the Nova service can only reflect the state of the Management network segment, and cannot reflect the state of the nova-compute nodes of the storage and business segments.

2) OpenStack does not lock the VM, so there is a split in the evacuate command (the same disk that launches multiple VMs).

3) for virtual machines that need to be protected, you need to provide a list of which VMs are protected. The current evacuate command will award the failure of all VMS on the host to rebuild such implementations are also unreasonable.

2. NEUTRON+VRRP

To prevent ARP spoofing, neutron is not allowed to bind multiple IP addresses on top of a port. Neutron in Havana release adds a "Allowed-address-pairs" feature that allows a port of a virtual machine to bind an additional IP as a floating IP. This can be installed in the virtual machine VRRP implementation software, more than keepalived, floating IP configuration For additional IP, multiple virtual machine binding port is bound to this additional floating IP, two virtual machines through VRRP can elect a master to provide services.

This solution is first configured in a complex configuration, requiring the neutron to participate in the port-bound IP for HA and configuring VRRP support software in the virtual machine. This scheme is not a complete solution, it is equivalent to the HA implementation of the application layer to do a neutron in the support function. Refer to this article for specific reference.

3. Heat Restarter

The HA feature of Heat is implemented with OpenStack multi-module, which involves Nova,ceilometer,heat-cfn-api,heat-cloudwatch,heat-cfntools and so on.

Nova-> providing virtual machines

ceilometer-> Send Alarm

Heat-cfntools-> Monitoring Virtual machine status

When the application process on the virtual machine is down, first try to resolve it by restarting the application process, and if it doesn't work, restart or rebuild the virtual machine, if it doesn't work, rebuild the entire stack. From this point of view, the function of heat ha is much more powerful than that of pure virtual machine ha.

But for normal web stateless applications, deleting the original virtual machine via os::heat::harestarter and recreating it might be appropriate, but what if it's a stateful application like a database? How to ensure that the data in the original database is not lost, back-end volume virtual machine? So how do you guarantee to use the original Fixed-ip?

Officially because Harestarter is the way to remove legacy virtual machines and some of the dependencies of virtual machines, the OpenStack community has deprecated harestarter in the kilo version.

4. Scenarios for virtual machine ha in OpenStack Two, the implementation of major companies 1. Masakari

Masakari is a set of virtual HA solutions provided by NTT Japan. Masakari supports the monitoring of virtual machine processes, virtualization processes, and compute node processes. Monitor virtual machine processes through shell scripts, nova-compute services and compute node status.

The virtual machine process hangs and shuts down and starts the virtual machine through the virtual machine's API.

The virtualization process hangs by setting the Nova-compute service to the down state through the Nova-compute API.

The Nova-compute process hangs up the virtual machine on the evacuation compute node.

Architecture of the Masakari:

Masakari-controller: The controller for this HA service.

Masakari-instancemonitor: Detects if the virtual machine process has been hung out.

Masakari-processmonitor: Detects if the Nova-compute is hung.

Masakari-hostmonitor: Detects if the compute node is hung.

This scenario is desirable in that there are three different levels of failures considered in the solution for virtual machine ha. However, without considering the isolation of the virtual machine's brain and compute nodes, there are three network segments of management, business and storage for the usual OpenStack deployment, and it is not enough to simply monitor the state of the compute nodes through a network segment.

2. Redhat programme

The deployment method is as follows:

Using the Pacemaker cluster as the control plane

The compute nodes are added to the Pacemaker cluster as Partial members and are managed and monitored by them. At this time, its number is not limited by the total number of nodes within the Corosync cluster.

HA Implementation Details:

Pacemaker starts various services on the compute nodes by Pacemaker_remote in order (Neutron-ovs-agent-Ceilometer-compute->nova-compute). The previous service failed to start, and subsequent services are not started.

Pacemaker monitors the connection to the pacemaker_remote on each compute node to check whether the node is active. Start the recovery (recovery) process by discovering that it cannot be connected.

The Pacemaker monitors the status of each service and is restarted if the status fails. A reboot failure triggers a defensive behavior (fencing action), and when all services are started, the VM's network is restored, so the network will only be affected for a short time.

When a node fails, the recovery (recovery) process is triggered and the Pacemaker is followed by:

1) Running Nova service-disable

2) Shut down the node

3) wait for Nova to discover that the node is dead.

4) Boot the node

5) If the node starts successfully, execute novaservice-enable

6) If the node fails to start, the novaevacuate is executed to move the virtual machine on the node to the other available compute nodes.

which

L steps (1) and (5) are optional, and the main purpose is to prevent Nova-scheduler from assigning a new VM to the node.

L Step (2) Ensure the machine will be shut down.

In step (3), Nova needs to wait for a longer timeout before judging the node down. There's a Blueprint in Liberty to add a Nova API to set the node state directly to down.

L Some other prerequisites:

The VM must be deployed on cinder-volume or shared temporary storage such as RBD or NFS, so the VM evaculation will not cause data loss.

L If the virtual machine does not use shared storage, you must periodically create a snapshot of the VM and save it to Glance. After the virtual machine is damaged, it can be recovered from the Glance snapshot. However, this can result in state or data loss.

L Control and compute nodes need to be installed rhel7.1+

• The compute nodes need to have a protective mechanism, such as IPMI, hardware dog, etc.

For specific reference here.

3. Sea-Yun Jiexun programme

The Distributed Health Check scheme of sea Yun Jiexun is a scheme that I agree to monitor whether the compute nodes are hung off, and consider the monitoring of three network segments of management, business and storage. It also supports the application layer's custom monitoring mode. For specific reference here.

The scheme introduces the Consul monitoring tool to monitor the state of compute nodes through the management, business and storage of the consul cluster in three segments, and to make different processing methods according to different combination situations. I think is the virtual machine ha scheme in the monitoring part of the deep and fine control, can achieve the precise recovery of virtual machine, effectively prevent the virtual machine brain crack situation.

Iv. Summary

Given that companies are doing the HA for OpenStack virtual machines, communities are beginning to consider the HA scenario for virtual machines, which can be consulted here.

Integration of the advantages of the various scenarios, as far as possible to deal with the various virtual machine anomalies, to ensure high availability of applications on the cloud. The following virtual machine ha scenarios are conceived,

Service Framework: Drawing on the idea of Https://github.com/gryf/mistral-evacuate , the service framework for virtual machine ha should be a relatively generic framework for dealing with different scenarios for various users. A universal framework, to be able to handle different user scenarios, the service framework must be a certain abstraction and commonality. Here is the service processing flow for the HA service.

Due to the fact that there is not much choice in the isolation and Recovery section, the basic differences between the virtual machine ha schemes of each company lie in the Monitoring section, and how to monitor the state of the compute nodes in detail. In view of the OpenStack environment is basically the management, business and storage network three separate deployment mode, so I think the above sea Yun Jiexun distributed Health Check method is a more practical way to monitor, plus ovirt in the virtual machine disk locking mechanism. I think it would be better to solve the problem of HA for virtual machines. Reference:

Virtual machine ha seems to be a simple requirement, but from the top of the various implementation methods, have their own little and shortcomings. So the problem is actually quite complicated. You are welcome to communicate with me on this issue.

Third, reference documents

Https://review.openstack.org/#/c/257809/2

Https://etherpad.openstack.org/p/automatic-evacuation

https://www.youtube.com/watch?v=yq5nYPKxBCo&feature=youtu.be&t=23m22s

Https://review.openstack.org/#/c/270015/7/user-stories/draft/ha_vm.rst

Https://github.com/gryf/mistral-evacuate

http://blog.clusterlabs.org/blog/2015/openstack-ha-compute/

Https://blueprints.launchpad.net/nova/+spec/vm-auto-ha-when-host-broken

OpenStack Related Technology Exchange please Dabigatran: 314889201

Also talk about the virtual machine ha in OpenStack