How to use heartbeat with floating IP to create high availability settings on Ubuntu 14.04

Provide Zstack community

Content Introduction

Heartbeat is an open source program that delivers cluster infrastructure capacity-including cluster members and messaging-to client servers. Hearbeat plays a key role in the High Availability server infrastructure. We often need to combine heartbeat with cluster resource managers such as pacemaker (CRM) to achieve full high availability settings. In today's tutorial, however, we will demonstrate how to easily create a set of two-node high-availability server settings using heartbeat and Digitalocean floating IP.

If you want to further improve usability levels, try Corosync and pacemaker as a combination or keepalived.

Target setting

Upon completion, this set of high availability settings will consist of an active/passive configuration established by two Ubuntu 14.04 servers. To achieve this goal, we need to use floating ip--to be responsible for specifying the actual way users access the service or the site-to point to the primary server or to the active server, unless it detects that the server has failed. Once the Hearbeat service discovers that the primary server is unavailable, the second server will automatically run a set of scripts to reassign the floating IP to itself through the Digitalocean API. As a result, subsequent network traffic to the floating IP will be directed to our secondary server, which will assume the role of the active server before the primary server returns to normal (after the primary server returns to normal, it also assigns the floating IP to itself for switchover).

Note: Today's tutorial only involves setting up an active/passive high availability system at the gateway level, which includes the floating IP and the Load Balancer server-the primary and secondary servers. In addition, for demonstration purposes, we do not configure a reverse proxy load balancer for each server, but simply configure it to respond to the corresponding host name and public IP address, respectively.

To achieve this goal, we will take the following steps:

• Create 2 droplet to receive traffic
• Create a floating IP and assign it to 1 of them droplet
• Create a DNS A record and point to the floating IP (optional)
• Installation of heartbeat on 2 droplet
• Configure heartbeat to run floating IP reassignment service
• Create floating IP Reassignment service
• Test failover effects

Prerequisite

In order to achieve automatic redistribution of floating IP, we must use the Digitalocean API. This means that you need to generate a personal access token (PAT) that can be used to authenticate our Digitalocean account and read and write access based on the instructions in the API Guide section, "How to generate a personal access token" section. Everyone's Pat will be used by script, which is added to two servers in the cluster to ensure that it securely accesses the Digitalocean account for reference.

In addition to the API, this tutorial will also take the following Digitalocean elements:

• Floating IPs
• Metadata
• User data (Cloud configuration script)

You can click on the link to learn more about the details.

Create droplet

The first step is to create 2 Ubuntu Droplet within the same datacenter, which will be used as the primary and secondary server as previously mentioned. In our example settings, the two will be named "Primary" and "secondary" respectively for easy reference. We will install Nginx on each of the 2 droplet and replace its index page with its unique identifying information. In this way, we can easily prove whether this set of high-availability settings is actually effective. In real settings, the server should run the Web server or load balancer according to the actual needs.

Create 2 ubuntu 14.04 Droplet, respectively primary and secondary, while the following Bash Scripts act as user data:

Examples of user data:

#!/bin/bashapt-get -y updateapt-get -y install nginxexport HOSTNAME=$(curl -s http://169.254.169.254/metadata/v1/hostname)export PUBLIC_IPV4=$(curl -s http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/address)echo Droplet: $HOSTNAME, IP Address: $PUBLIC_IPV4 > /usr/share/nginx/html/index.html

This script installs Nginx and replaces the droplet's host name with the IP address (by referencing the metadata service) index.html . When accessed through a public IP address, the corresponding droplet will display a set of basic pages containing the droplet host name and IP address to help the user test which droplet the floating IP points to in any period of time.

Create floating IP

In the Digitalocean Control Panel, click "Networking" in the top menu, then select "Floating IPs" in the subsequent menu.

Assign a floating IP to our primary droplet and click on the "Assign floating IP" button.

After the floating IP assignment is complete, the IP is accessed through a Web browser to confirm that it is properly assigned to the established droplet.

http://your_floating_ip

At this point you should be able to see your primary droplet index page.

To configure DNS (optional)

If you want to access your primary availability settings through a domain name, you can create a record in DNS that will be responsible for pointing the domain name to the corresponding floating IP address. If your domain name uses Digitalocean's domain name server, you can follow the third step in the "How to use Digitalocean to set a host name" tutorial. Once setup is complete, you can access your Active server through the domain name.

Here we use the example domain name for example.com . If you do not have a domain name available for the time being, you should use the floating IP address.

Installing Heartbeat

The next step is to install heartbeat on both servers. The simplest way to install heartbeat is to use Apt-get:

sudo apt-get updatesudo apt-get install heartbeat

Heartbeat is now installed, but we need to configure it to make it work.

Configure Heartbeat

In order to ensure that the cluster starts and runs properly, we must configure the heartbeat in the files in the/ETC/HA.D directory-two servers are configured in the same way:

1. The global configuration of the Ha.cf:Heartbeat cluster, including its member nodes.
2. Authkeys: Add a security key to ensure that the cluster authenticates the node.
3. Haresources: Specifies the service that is managed by the cluster and the corresponding node that is the owner of the service. It is important to note that we do not need to use this file when setting up with CRM scenarios such as pacemaker.

We also need to provide a set of scripts to perform floating IP reassignment when the availability of primary droplet is changed.

Collect node Information

Before you configure HA.CF, we should first look at each node name. Heartbeat requires that the name of each node match its corresponding uname -n output.

Run the following command on both servers to find the corresponding node name:

* uname -n

Note the output of the command. The node name in the example is "primary" and "secondary" and exactly matches our droplet name.

We also need to look at the network interfaces and IP addresses that each node uses to communicate with other nodes in the cluster to see which nodes are currently available. You can use any network interface, as long as the nodes within the cluster to ensure the ability to communicate with each other. Here we use droplet's public interface, which happens to be eth0.

On both servers, use the following command to find the IP address of the Eth0 interface (or view it through the Digitalocean Control Panel):

* ip addr show eth0ip addr show eth0 output:2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000    link/ether 04:01:76:a5:45:01 brd ff:ff:ff:ff:ff:ff    inet 104.236.6.11/18 brd 104.236.63.255 scope global eth0       valid_lft forever preferred_lft forever    inet 10.17.0.28/16 scope global eth0       valid_lft forever preferred_lft forever    inet6 fe80::601:76ff:fea5:4501/64 scope link       valid_lft forever preferred_lft forever

Note the IP address of the network interface (highlighted in this example). Make sure that you get the corresponding IP address for both servers.

Create a HA.CF file

Open the/etc/ha.d/ha.cf file with a familiar editor on both servers. We use VI here:

* sudo vi /etc/ha.d/ha.cf

The contents of this file should be empty this time. We need to add the network interface and name of each node within the cluster.

Copy and paste the configuration into this file, and then replace the specific values with the corresponding host name and IP address you found earlier. In this example, the IP address of the primary is 104.236.6.11, and the IP address of secondary is 104.236.6.22:

node primaryucast eth0 104.236.6.11node secondaryucast eth0 104.236.6.22

Save and exit. Next, we'll set up the authentication key for this cluster.

Create a Authkeys file

The purpose of the validation key is to allow the cluster members to join the cluster. For this purpose, we can easily create an accompanying secret key.

Run the following command in the primary node to generate an appropriate authentication key in the environment variable named Authkey:

if [ -z "${AUTH_KEY}" ]; then  export AUTH_KEY="$(command dd if=‘/dev/urandom‘ bs=512 count=1 2>‘/dev/null‘ | command openssl sha1 | command cut --delimiter=‘ ‘ --fields=2)"fi

Later, write the/etc/ha.d/authkeys file using the following command:

sudo bash -c "{ echo auth1  echo 1 sha1 $AUTH_KEY  

Check the contents of the Authkeys file with the following command:

* sudo cat /etc/ha.d/authkeys   

The execution results should look like this (only the specific verification key differs):

/etc/ha.d/authkeys example:auth11 sha1 d1e6557e2fcb30ff8d4d3ae65b50345fa46a2faa

Make sure that the file is only available for root read:

sudo chmod 600 /etc/ha.d/authkeys

Now copy the/etc/ha.d/authkeys file from the primary node to our secondary node. You can do this manually or by using the SCP.

On the secondary server, ensure that the permissions for the Authkeys file are set:

sudo chmod 600 /etc/ha.d/authkeys

Now the two servers should already have the same/etc/ha.d/authkeys file.

Create a Haresources file

The Haresources file here is responsible for specifying the preferred host corresponding to the services managed by the cluster. The so-called preferred host is preferred, which refers to the operating options that the service should prioritize when the corresponding node is in the available state. If the preferred host is unavailable, i.e. it cannot be accessed by the cluster, the other node will take its responsibility. In other words, the secondary server will replace its role in the event of a primary server failure.

Open the Haresources file on both servers through a familiar editor. Here we use VI:

sudo vi /etc/ha.d/haresources

Now add the following line to the file and replace our own primary node name in:

/etc/ha.d/haresources

primary floatip

Save and exit. The primary server is now configured as the preferred host for the Floatip service, but the Floatip service itself has not yet been defined. So the next task is to set up the Floatip service.

Create floating IP Reassignment service

The purpose of our heartbeat cluster is to ensure that the Floatip service is always working, and that one node is used to assign floating IP to itself. However, the service itself also needs to be created separately. Before setting up the service ontology, we first create a set of scripts to assign the floating IP to the node where it is running through the Digitalocean API. Next, we will create the Floatip service that is responsible for running the floating IP reassignment script.

Create a ASSIGN-IP script

In the example, we will download a set of basic Python scripts to use the Digitalocean API to assign a floating IP to a given droplet ID.

Download the Assign-ip Python script on both servers:

sudo curl -L -o /usr/local/bin/assign-ip http://do.co/assign-ip

To provide execution permissions on both servers:

sudo chmod +x /usr/local/bin/assign-ip

The following details also need to be noted with the Assign-ip script:

• Floating IP: The first parameter in the script that specifies the floating IP that needs to be assigned.
• Droplet ID: The second parameter in the script that specifies the Droplet ID to which the floating IP is assigned.
• Digitalocean Pat (API Token): passed as an environment variable dotoken, that is, our read/write Digitalocean Pat.

Please review this part of the script carefully before proceeding to the next step.

Now we are ready to create the Floatip service.

Create a Floatip service

To create the Floatip service, we need to first create an INIT script to invoke the previously written Assign-ip script, while responding to the start and stop subcommands. The init script will be responsible for locating the server's droplet ID through the droplet metadata service. In addition, the script needs to obtain the assigned floating IP as well as the Digitalocean API token (that is, the personal access token we mentioned in the Prerequisites section).

On both servers, use the editor to open/etc/init.d/floatip:

sudo vi /etc/init.d/floatip

Then copy and paste the following into the Init script, replacing the highlights with your own Digitalocean API key and the floating IP you need to assign:

/etc/init.d/floatip

#!/bin/bashparam=$1export DO_TOKEN=‘b7d03a6947b217efb6f3ec3bd3504582‘IP=‘45.55.96.8‘ID=$(curl -s http://169.254.169.254/metadata/v1/id)if [ "start" == "$param" ] ; then    python /usr/local/bin/assign-ip $IP $ID     exit 0elif [ "stop" == "$param" ] ; then    exit 0;elif [ "status" == "$param" ] ; then    exit 0;else    echo "no such command $param"    exit 1;fi

Save and exit.

Provides execution permissions for the script.

sudo chmod u+x /etc/init.d/floatip

When the Floatip service starts, it invokes the Assign-ip Python script directly and assigns the specified floating IP to the droplet that executes the script. In normal operation, the script is used by the secondary server to assign the floating IP to itself, thus enabling the failover capability of the primary server. Similarly, the script can be used to reassign the floating IP to itself after the primary is re-added to the cluster.

Start Heartbeat

Now that the heartbeat is configured and all the scripts needed to run it are ready-we can start the heartbeat cluster right away!

Run the following command on both servers to start heartbeat:

sudo service heartbeat start

You should see the following output:

Heartbeat output:Starting High-Availability services: Done.

Our high availability settings are complete! The next step is to test the state of its work.

Testing for high Availability

It's important to test the working state of the high availability settings, so let's get started.

Currently, the floating IP is assigned to the primary node. We can access the floating IP via the IP address or the domain name that points to it, and view the index page of the primary server. If you use the user data script provided by the sample, the results should look like this:
Floating IP is pointing to primary server
Droplet:primary, IP address:104.236.6.11

This means that the floating IP is actually assigned to primary droplet.
Now let's open the terminal and use curl to iterate through the floating IP for 1 seconds. You can do this with the following command, but be careful to replace the URL section with your actual domain name or floating IP address:

while true; do curl http://example.com; sleep 1; done

Currently it will output the same droplet name as the primary server IP address. If we leave the primary server in an unusable state-such as shutting it down or stopping the heartbeat service-you will see that the floating IP is reassigned to the secondary server.

Now let's shut down the primary server. You can turn it off by Digitalocean the control Panel or by running the following command on the primary server:

sudo poweroff

After a few moments, the primary server will no longer be available. You need to be aware of the output of the curl loop running inside the terminal. You should see the following output:

curl loop output:Droplet: primary, IP Address: 104.236.6.11...curl: (7) Failed to connect to example.com port 80: Connection refusedDroplet: secondary, IP Address: 104.236.6.22Droplet: secondary, IP Address: 104.236.6.22...

At this point, the floating IP address should be reassigned to the IP address of the secondary server, which means that our high availability settings are working properly, that is, the automatic failover function has been successfully implemented.

You may find a connection refused error--or it may not, because everyone's floating IP access attempts occur between the primary server failure and the floating IP reallocation complete.

Now you can restart your primary Droplet via the Digitalocean Control Panel. Since heartbeat is configured to run the floating IP reassignment script as preferred host in the configuration, the floating IP will automatically re-point to the primary server after availability recovery.

Summarize

Congratulations, everyone! Now we have a set of basic high Availability server settings that leverage heartbeat and Digitalocean floating IP implementations.

If you want to further improve usability levels, consider using corosync and pacemaker combinations or keepalived.

If you want to expand your heartbeat settings, the next step is to replace the Nginx settings in the example with a reverse proxy load balancer. You can also use Nginx or haproxy to achieve the same effect. It is important to note that you need to bind the load balancer to another IP address to ensure that users only have access to the server through the floating IP address (not through the public IP address of each server).

This article originates from Digitalocean Community. The original English: How to Create a high availability Setup with Heartbeat and floating IPs on Ubuntu 14.04 by Mitchell Anicas

Translation: DIRADW

How to use heartbeat with floating IP to create high availability settings on Ubuntu 14.04