FortiOS 6.0 Expert Recipe: FGCP Virtual Clustering with four FortiGates

In this recipe you set up a FortiGate Clustering Protocol (FGCP) virtual clustering configuration with four FortiGates to provide redundancy and failover protection for two networks. The FortiGate configuration includes two VDOMs. The root VDOM handles internal network traffic and the Engineering VDOM handles Engineering network traffic. This recipe describes a very simple two-VDOM configuration.  However, the same principles described in this example apply to a virtual cluster with more VDOMs.

In this virtual cluster configuration the primary FortiGate processes all internal network traffic and the backup FortiGate processes all Engineering network traffic. Virtual clustering enables override and uses device priorities to distribute traffic between the primary and backup FortiGates in the virtual cluster. For details about how this configuration works, see Configuring virtual clustering.

The third FortiGate (the recipe names it Backup-2) acts as a backup to the primary FortiGate; if the primary FortiGate fails, all primary FortiGate network traffic transfers to the Backup-2 FortiGate, which becomes the new primary FortiGate.

The fourth FortiGate (Backup-3) acts as a backup to the backup FortiGate; if the backup FortiGate fails, all backup FortiGate network traffic transfers to the Backup-3 FortiGate, which becomes the new backup FortiGate.

This recipe describes the recommended steps for setting up a virtual cluster of four FortiGates. You can follow the procedure described in High Availability with FGCP (expert) to configure virtual clustering by converting a FortiGate with VDOMs to HA mode and then adding another FortiGate to form a cluster. However, taking this approach with virtual clustering is not as foolproof as a normal HA configuration. If you accidentally add the management VDOM to virtual cluster 2 before adding the backup FortiGate, the configuration of the primary FortiGate can be overwritten by the backup FortiGate. If want to experiment with this approach, make sure you don’t add the management VDOM to virtual cluster 2 until all of the FortiGates have joined the cluster.

Before you start, the FortiGates should be running the same FortiOS firmware version and their interfaces should not be configured to get addresses from DHCP or PPPoE.

This recipe features four FortiGate-51Es. FortiGate-51Es have a 5-port switch lan interface. Before configuring HA, the lan interface was converted to 5 separate interfaces (lan1 to lan5).

1. Preparing the FortiGates

If required, upgrade the firmware running on the FortiGates. All of the FortiGates should be running the same version of FortiOS.

On each FortiGate, enter the following command to reset them factory default settings.

execute factoryreset

You can skip this step if the FortiGates are fresh from the factory. But if their configurations have changed at all, it’s a best practice to reset them to factory defaults to reduce the chance of synchronization problems.

In some cases, after resetting to factory defaults you may want to make some initial configuration changes to connect the FortiGates to the network or for other reasons. To write this recipe, the lan switch on the FortiGate-51Es was converted to separate lan1 to lan5 interfaces.

Change the primary FortiGate Host name to identify it as the primary FortiGate by going to System > Settings.

Change the backup FortiGate Host name to identify it as Backup-1 by going to System > Settings.

Change the third FortiGate Host name to identify it as Backup-2 by going to System > Settings.

Change the fourth FortiGate Host name to identify it as Backup-3 by going to System > Settings.

You can also use the CLI to change the host name. From the Primary FortiGate:

config system global
  set hostname Primary
end

From the Backup-1 FortiGate:

config system global
  set hostname Backup-1
end

From the Backup-2 FortiGate:

config system global
  set hostname Backup-2
end

From the Backup-3 FortiGate:

config system global
  set hostname Backup-3
end

Register and apply licenses to the FortiGates before forming the cluster. This includes licensing for FortiCare Support, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, Security Rating, Outbreak Prevention, and additional virtual domains (VDOMs).

All of the FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs.

You can add FortiToken licenses and install third-party certificates at any time because they’re synchronized to all cluster members.

2. Configuring clustering

On the primary FortiGate, enter the following CLI command to set the HA mode to active-passive, set a group-id, group name, and password, increase the device priority to 200, enable override, and configure the heartbeat interfaces (lan4 and lan5 in this example).

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 200
  set override enable
  set hbdev lan4 200 lan5 100
end

You can also configure most of these settings from the GUI (go to Global > System > HA). The group-id and override can only be configured from the CLI.

On the Backup-1 FortiGate, duplicate the primary FortiGate HA mode, group-id, group-name, password, override, and heartbeat device settings. Set the device priority to 50. Setting the device priority to a relatively low value means the Backup-1 FortiGate will most likely always become the backup FortiGate.

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 50
  set override enable
  set hbdev lan4 200 lan5 100
end

On the Backup-2 FortiGate, duplicate the primary FortiGate HA mode, group-id, group-name, password, override, and heartbeat device settings. Set the device priority to 150. A device priority of 150 is almost as high as the device priority of the primary FortiGate. So if the primary FortiGate fails, the Backup-2 FortiGate should become the new primary FortiGate.

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 150
  set override enable
  set hbdev lan4 200 lan5 100
end

On the Backup-3 FortiGate, duplicate the primary FortiGate HA mode, group-id, group-name, password, override, and heartbeat device settings. Set the device priority to 100. A device priority of 100 means that if the backup FortiGate fails, the Backup-3 FortiGate will have the lowest device priority so will become the new backup FortiGate.

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 100
  set override enable
  set hbdev lan4 200 lan5 100
end

After you enable HA, each FortiGate negotiates to establish an HA cluster. You may temporarily lose connectivity as FGCP negotiation takes place and the MAC addresses of the FortiGate interfaces change to HA virtual MAC addresses.

To reconnect sooner, you can update the ARP table of your management PC by deleting the ARP table entry for the FortiGate (or just deleting all ARP table entries). You can usually delete the ARP table from a command prompt using a command similar to arp -d.

The FGCP uses virtual MAC addresses for failover. The virtual MAC address assigned to each FortiGate interface depends on the HA group ID. A group ID of 88 sets FortiGate interfaces to the following MAC addresses: 00:09:0f:09:58:00, 00:09:0f:09:58:01, 00:09:0f:09:58:02 and so on. For details, see Cluster virtual MAC addresses.

You can verify that the FGCP has set the virtual MAC addresses by viewing the configuration of each FortiGate interface from the GUI (go to Network > Interfaces) or by entering the following CLI command (shown below for lan2 on a FortiGate-51E):

get hardware nic lan2
... 
Current_HWaddr 00:09:0f:09:58:01 
Permanent_HWaddr 70:4c:a5:98:11:54
...

You can also use the diagnose hardware deviceinfo nic lan2 command to display this information.

The output shows the current hardware (MAC) address (the virtual MAC set by the FGCP) and the permanent hardware (MAC) address for the interface.

3. Connecting and verifying cluster operation

Connect the FortiGates together and to your networks as shown in the network diagram at the start of the recipe. Making these connections disrupts network traffic as you disconnect and re-connect cables.

Switches must be used between the cluster and the Internet, between the cluster and the internal network, and between the cluster and the Engineering network as shown in the diagram. You can use any good quality switches to make these connections.

To make HA heartbeat connections, connect all of the lan4 interfaces to the same switch and all of the lan5 interfaces to another switch.

You can also use fewer switches for all of these connections as long as you configure the switches to separate traffic from the different networks.

When you connect the heartbeat interfaces and power on the FortiGates, they find each other and negotiate to form a cluster. The cluster will have the same IP addresses as the primary FortiGate. You can log into the cluster by logging into the primary FortiGate GUI or CLI using one of the original IP addresses of the primary FortiGate.

Check the cluster synchronization status to make sure the primary and backup FortiGates both have the same configuration. Log into the primary FortiGate CLI and enter this command:

diagnose sys ha checksum cluster

The command output lists all cluster members’ configuration checksums. If both cluster members have identical checksums you can be sure that their configurations are synchronized. If the checksums are different, wait a short while and enter the command again. Repeat until the checksums are identical. It may take a while for some parts of the configuration to be synchronized. If the checksums never become identical you can use the information in Synchronizing the configuration to troubleshoot the problem or visit the Fortinet Support website for assistance.

You can also use the get system ha status command to display detailed information about the cluster. For information about this command, see Viewing cluster status from the CLI for details.

The HA Status dashboard widget widget also shows synchronization status. Hover over the host names of each FortiGate in the widget to verify that they are synchronized and both have the same checksum.

4. Adding VDOMs and setting up virtual clustering

Enable VDOMs by going to System > Settings > System Operation Settings and enabling Virtual Domains. Or enter the following CLI command.

config system global
  set vdom-admin enable
end

Add VDOMs as required. Go to Global > System > VDOM and select Create New. Or enter the following CLI command to add the Engineering VDOM.

config global
  edit Engineering
end

Configure virtual clustering and VDOM partitioning on the primary FortiGate. The following command enables virtual cluster 2, adds the Engineering VDOM to virtual cluster 2, and sets the virtual cluster 2 device priority of the primary FortiGate to 50.

config global
  config system ha
    set vcluster2 enable
      config secondary-vcluster
        set vdom Engineering
        set priority 50
      end

You can also configure virtual clustering and VDOM partitioning from the GUI by going to Global > System > HA.

Set the virtual cluster 2 priority of the Backup-1 FortiGate to a relatively high value (in this example, 200) so that this FortiGate processes traffic for the VDOMs in virtual cluster 2. The FGCP synchronizes all other HA settings from the primary FortiGate. 

You can only configure the virtual cluster 2 priority of the backup FortiGate from the CLI. Use execute ha manage to access the backup FortiGate CLI.

config global
  config system ha
    config secondary-vcluster
      set priority 200
  end

Set the virtual cluster 2 priority of the Backup-2 FortiGate to 100 so that if the primary FortiGate fails, Backup-2 will become the primary FortiGate but will have the lowest virtual cluster 2 priority. The FGCP synchronizes all other HA settings from the primary FortiGate. 

You can only configure the virtual cluster 2 priority of the Backup-2 FortiGate from the CLI. Use execute ha manage to access the backup FortiGate CLI.

config global
  config system ha
    config secondary-vcluster
      set priority 100
  end

Set the virtual cluster 2 priority of the Backup-3 FortiGate to 150 so that if the backup FortiGate fails, Backup-3 will have the highest virtual cluster 2 device priority. The FGCP synchronizes all other HA settings from the primary FortiGate. 

You can only configure the virtual cluster 2 priority of the backup FortiGate from the CLI. Use execute ha manage to access the backup FortiGate CLI.

config global
  config system ha
    config secondary-vcluster
      set priority 150
  end

5. Checking virtual cluster operation

Once again use the diagnose sys ha checksum cluster command and the get system ha status command to check the cluster synchronization status to make sure the primary and backup FortiGates both have the same configuration.

The HA Status dashboard widget shows the VDOMs in the virtual clusters. You can hover over the VDOM names to see status information for the VDOMs. You can hover over the host names of each FortiGate to verify that they are synchronized and both have the same checksum.

To view more information about the cluster status, click on the HA Status widget and select Configure Settings in System > HA (or go to System > HA).

The HA status page shows all four FortiGates in the cluster. It also shows that Primary is the primary (master) FortiGate for the root VDOM (so the primary FortiGate processes all root VDOM traffic). The page also shows that Backup-1 is the primary (master) FortiGate for the Engineering VDOM (so the backup FortiGate processes all Engineering VDOM traffic).

6. Results

All root VDOM traffic should now be flowing through the primary FortiGate and Engineering VDOM traffic should be flowing through the backup FortiGate. If the primary FortiGate becomes unavailable, the cluster negotiates and traffic fails over and all traffic would be processed by the backup FortiGate.

To test failover, ping a reliable IP address from a PC on the internal or Engineering network. After a moment, power off the primary FortiGate. You will see a momentary pause in the ping results while the cluster negotiates and then ping traffic can continue.

64 bytes from 184.25.76.114: icmp_seq=69 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=70 ttl=52 time=8.822 ms\
64 bytes from 184.25.76.114: icmp_seq=71 ttl=52 time=9.034 ms\
64 bytes from 184.25.76.114: icmp_seq=72 ttl=52 time=9.536 ms\
64 bytes from 184.25.76.114: icmp_seq=73 ttl=52 time=8.877 ms\
64 bytes from 184.25.76.114: icmp_seq=74 ttl=52 time=8.901 ms\
Request timeout for icmp_seq 75\
64 bytes from 184.25.76.114: icmp_seq=76 ttl=52 time=8.860 ms\
64 bytes from 184.25.76.114: icmp_seq=77 ttl=52 time=9.174 ms\
64 bytes from 184.25.76.114: icmp_seq=78 ttl=52 time=10.108 ms\
64 bytes from 184.25.76.114: icmp_seq=79 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=80 ttl=52 time=10.861 ms\
64 bytes from 184.25.76.114: icmp_seq=81 ttl=52 time=10.757 ms\
64 bytes from 184.25.76.114: icmp_seq=82 ttl=52 time=8.158 ms\
64 bytes from 184.25.76.114: icmp_seq=83 ttl=52 time=8.639 ms}

You can log into the cluster GUI or CLI using the same IP address as you had been using to the log into the primary FortiGate. If the primary FortiGate is powered off you will be logging into the Backup-1 FortiGate. Check the host name to verify the FortiGate that you have logged into.

After the primary FortiGate fails the HA Status dashboard widget shows that the Backup-2 has become the primary (master) FortiGate.

The System > HA page shows that the Backup-2 FortiGate has become the primary FortiGate for virtual cluster 1. This page also shows that the Backup-1 FortiGate continues to process virtual cluster 2 traffic.

If you restart the primary FortiGate, after a few minutes it should rejoin the cluster and because override is enabled, the original virtual cluster configuration should be re-established. Traffic may be temporarily disrupted when the restarted primary FortiGate rejoins the cluster.

You can also try powering off other FortiGates in the virtual cluster to see how the cluster adapts to the failover. Because of the device priority configuration, if two FortiGates are operating, virtual cluster 1 and virtual cluster 2 traffic will be distributed between them.