FortiOS 6.0 Expert Recipe: FGCP Virtual Clustering with two FortiGates

In this recipe you set up a FortiGate Clustering Protocol (FGCP) virtual clustering configuration with two FortiGates to provide redundancy and failover protection for two networks. The FortiGate configuration includes two VDOMs. The root VDOM handles internal network traffic and the Engineering VDOM handles  Engineering network traffic. This recipe describes a very simple two-VDOM configuration.  However, the same principles described in this example apply to a virtual cluster with more VDOMs.

In this virtual cluster configuration the primary FortiGate processes all internal network traffic and the backup FortiGate processes all Engineering network traffic. Virtual clustering enables override and uses device priorities to distribute traffic between the primary and backup FortiGates in the virtual cluster. For details about how this configuration works, see Configuring virtual clustering.

This recipe describes the recommended steps for setting up a virtual cluster of two FortiGates. You can follow the procedure described in High Availability with FGCP (expert) to configure virtual clustering by converting a FortiGate with VDOMs to HA mode and then adding another FortiGate to form a cluster. However, taking this approach with virtual clustering is not as foolproof as a normal HA configuration. If you accidentally add the management VDOM to virtual cluster 2 before adding the backup FortiGate, the configuration of the primary FortiGate can be overwritten by the backup FortiGate. If want to experiment with this approach, make sure you don’t add the management VDOM to virtual cluster 2 until all of the FortiGates have joined the cluster.

Before you start, the FortiGates should be running the same FortiOS firmware version and their interfaces should not be configured to get addresses from DHCP or PPPoE.

This recipe features two FortiGate-51Es. FortiGate-51Es have a 5-port switch lan interface. Before configuring HA, the lan interface was converted to 5 separate interfaces (lan1 to lan5).

1. Configuring clustering

If required, upgrade the firmware running on both FortiGates. Both FortiGates should be running the same version of FortiOS.

On each FortiGate, enter the following command to reset it to factory default settings.

execute factoryreset

You can skip this step if the FortiGates are fresh from the factory. But if their configurations have changed at all, it’s a best practice to reset them to factory defaults to reduce the chance of synchronization problems.*

Change the primary FortiGate Host name to identify it as the primary FortiGate by going to System > Settings.

Change the backup FortiGate Host name to identify it as the backup FortiGate by going to System > Settings.

You can also use the CLI to change the host name. From the Primary FortiGate:

config system global
  set hostname Primary
end

From the Backup FortiGate:

config system global
  set hostname Backup
end

Register and apply licenses to both FortiGates before forming the cluster. This includes licensing for FortiCare Support, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, Security Rating, Outbreak Prevention, and additional virtual domains (VDOMs).

Both FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs.

You can add FortiToken licenses and install third-party certificates at any time because they’re synchronized to all cluster members.

On the primary FortiGate, enter the following CLI command to set the HA mode to active-passive, set a group-id, group name, and password, increase the device priority to 200, enable override, and configure the heartbeat interfaces (lan4 and lan5 in this example).

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 200
  set override enable
  set hbdev lan4 200 lan5 100
end

You can also configure most of these settings from the GUI (go to Global > System > HA). The group-id and override can only be configured from the CLI.

On the backup FortiGate, duplicate the primary FortiGate HA mode, group-id, group-name, password, override, and heartbeat device settings. Set the device priority to 50.

config system ha
  set mode a-p
  set group-id 88
  set group-name My-vcluster
  set password <password>
  set priority 50
  set override enable
  set hbdev lan4 200 lan5 100
end

After you enabe HA, the FortiGates negotiate to establish an HA cluster. You may temporarily lose connectivity as FGCP negotiation takes place and the MAC addresses of the FortiGate interfaces change to HA virtual MAC addresses.

To reconnect sooner, you can update the ARP table of your management PC by deleting the ARP table entry for the FortiGate (or just deleting all ARP table entries). You can usually delete the ARP table from a command prompt using a command similar to arp -d.

The FGCP uses virtual MAC addresses for failover. The virtual MAC address assigned to each FortiGate interface depends on the HA group ID. A group ID of 88 sets FortiGate interfaces to the following MAC addresses: 00:09:0f:09:58:00, 00:09:0f:09:58:01, 00:09:0f:09:58:02 and so on. For details, see Cluster virtual MAC addresses.

You can verify that the FGCP has set the virtual MAC addresses by viewing the configuration of each FortiGate interface from the GUI (go to Network > Interfaces) or by entering the following CLI command (shown below for lan2 on a FortiGate-51E):

get hardware nic lan2
... 
Current_HWaddr 00:09:0f:09:58:01 
Permanent_HWaddr 70:4c:a5:98:11:54
...

You can also use the diagnose hardware deviceinfo nic lan2 command to display this information.

The output shows the current hardware (MAC) address (the virtual MAC set by the FGCP) and the permanent hardware (MAC) address for the interface.

2. Connecting and verifying cluster operation

Connect the primary and backup FortiGates together and to your networks as shown in the network diagram at the start of the recipe. Making these connections disrupts network traffic as you disconnect and re-connect cables.

Switches must be used between the cluster and the Internet, between the cluster and the internal network, and between the cluster and the Engineering network as shown in the diagram. You can use any good quality switches to make these connections. You can also use one switch for all of these connections as long as you configure the switch to separate traffic from the different networks.

You can connect the heartbeat interfaces (lan4 and lan5) with direct cable connections as shown.

When you connect the heartbeat interfaces and power on the FortiGates, they find each other and negotiate to form a cluster. The cluster will have the same IP addresses as the primary FortiGate. You can log into the cluster by logging into the primary FortiGate GUI or CLI using one of the original IP addresses of the primary FortiGate.

Check the cluster synchronization status to make sure the primary and backup FortiGates both have the same configuration. Log into the primary FortiGate CLI and enter this command:

diagnose sys ha checksum cluster

The command output lists all cluster members’ configuration checksums. If both cluster members have identical checksums you can be sure that their configurations are synchronized. If the checksums are different, wait a short while and enter the command again. Repeat until the checksums are identical. It may take a while for some parts of the configuration to be synchronized. If the checksums never become identical you can use the information in Synchronizing the configuration to troubleshoot the problem or visit the Fortinet Support website for assistance.

You can also use the get system ha status command to display detailed information about the cluster. For information about this command, see Viewing cluster status from the CLI for details.

The HA Status dashboard widget widget also shows synchronization status. Hover over the host names of each FortiGate in the widget to verify that they are synchronized and both have the same checksum.

3. Adding VDOMs and setting up virtual clustering

Enable VDOMs by going to System > Settings > System Operation Settings and enabling Virtual Domains. Or enter the following CLI command.

config system global
  set vdom-admin enable
end

Add VDOMs as required. Go to Global > System > VDOM and select Create New. Or enter the following CLI command to add the Engineering VDOM.

config global
  edit Engineering
end

Configure virtual clustering and VDOM partitioning on the primary FortiGate. The following command enables virtual cluster 2, adds the Engineering VDOM to virtual cluster 2, and sets the virtual cluster 2 device priority of the primary FortiGate to 50.

config global
  config system ha
    set vcluster2 enable
      config secondary-vcluster
        set vdom Engineering
        set priority 50
      end

You can also configure virtual clustering and VDOM partitioning from the GUI by going to Global > System > HA.

Set the virtual cluster 2 priority of the backup FortiGate to a relatively high value (in this example, 200) so that the backup FortiGate processes traffic for the VDOMs in virtual cluster 2. The FGCP synchronizes all other HA settings from the primary FortiGate. 

You can only configure the virtual cluster 2 priority of the backup FortiGate from the CLI. Use execute ha manage to access the backup FortiGate CLI.

config global
  config system ha
    config secondary-vcluster
      set priority 200
  end

4. Checking virtual cluster operation

Once again use the diagnose sys ha checksum cluster command and the get system ha status command to check the cluster synchronization status to make sure the primary and backup FortiGates both have the same configuration.

The HA Status dashboard widget shows the VDOMs in the virtual clusters. You can hover over the VDOM names to see status information for the VDOMs. You can hover over the host names of each FortiGate to verify that they are synchronized and both have the same checksum.

To view more information about the cluster status, click on the HA Status widget and select Configure Settings in System > HA (or go to System > HA).

The HA status page shows both FortiGates in the cluster. It also shows that Primary is the primary (master) FortiGate for the root VDOM (so the primary FortiGate processes all root VDOM traffic). The page also shows that Backup is the primary (Master) FortiGate for the Engineering VDOM (so the backup FortiGate processes all Engineering VDOM traffic).

7. Results

All root VDOM traffic should now be flowing through the primary FortiGate and Engineering VDOM traffic should be flowing through the backup FortiGate. If the primary FortiGate becomes unavailable, the cluster negotiates and traffic fails over and all traffic would be processed by the backup FortiGate.

To test failover, ping a reliable IP address from a PC on the internal or Engineering network. After a moment, power off the primary FortiGate.* You will see a momentary pause in the ping results while the cluster negotiates and then ping traffic can continue.

64 bytes from 184.25.76.114: icmp_seq=69 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=70 ttl=52 time=8.822 ms\
64 bytes from 184.25.76.114: icmp_seq=71 ttl=52 time=9.034 ms\
64 bytes from 184.25.76.114: icmp_seq=72 ttl=52 time=9.536 ms\
64 bytes from 184.25.76.114: icmp_seq=73 ttl=52 time=8.877 ms\
64 bytes from 184.25.76.114: icmp_seq=74 ttl=52 time=8.901 ms\
Request timeout for icmp_seq 75\
64 bytes from 184.25.76.114: icmp_seq=76 ttl=52 time=8.860 ms\
64 bytes from 184.25.76.114: icmp_seq=77 ttl=52 time=9.174 ms\
64 bytes from 184.25.76.114: icmp_seq=78 ttl=52 time=10.108 ms\
64 bytes from 184.25.76.114: icmp_seq=79 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=80 ttl=52 time=10.861 ms\
64 bytes from 184.25.76.114: icmp_seq=81 ttl=52 time=10.757 ms\
64 bytes from 184.25.76.114: icmp_seq=82 ttl=52 time=8.158 ms\
64 bytes from 184.25.76.114: icmp_seq=83 ttl=52 time=8.639 ms}

You can log into the cluster GUI or CLI using the same IP address as you had been using to the log into the primary FortiGate. If the primary FortiGate is powered off you will be logging into the backup FortiGate. Check the host name to verify the FortiGate that you have logged into.

When you restart the primary FortiGate, after a few minutes it should rejoin the cluster and because override is enabled, the original virtual cluster configuration should be re-established. Traffic may be temporarily disrupted when the restarted primary FortiGate rejoins the cluster.