FortiOS 6.0 Expert Recipe: Adding a third FortiGate to an FGCP cluster

You prepare the new FortiGate by:

• Setting it to factory defaults to wipe any configuration changes.
• Licensing it (if required).
• Enabling HA without changing the device priority and without enabling override.
• Connecting it to the FGCP cluster already on the network.
• The new FortiGate becomes a second backup FortiGate; its configuration synchronized to match the configuration of the cluster.

Before you start, the new FortiGate should be running the same FortiOS firmware version as the cluster and its interfaces should not be configured to get addresses from DHCP or PPPoE.

After the third FortiGate joins the cluster, this recipe also describes how to switch the cluster to operate in active-active (or a-a) mode. Active-active HA enables proxy-based NGFW/UTM load-balancing to allow the three FortiGates to share proxy-based NGFW/UTM processing. If the cluster handles a large amount of NGFW/UTM traffic, active-active HA with three FortiGates may enhance performance.

This recipe features three FortiGate-51Es. These FortiGate models include a 5-port switch lan interface. Before configuring HA, the lan interface was converted to five separate interfaces (lan1 to lan5). The lan1 interface connects to the internal network and the wan1 interface connects to the Internet. The lan4 and lan5 interfaces become the HA heartbeat interfaces.

1. Enabling override on the primary FortiGate (optional)

Before adding the third FortiGate to the cluster, enable override on the primary FortiGate. In most cases this step would not be necessary but it is a best practice because enabling override makes sure the configuration of the primary FortiGate is not overwritten by the configuration of the new backup FortiGate.

To enable override, log into the primary FortiGate CLI and enter this command:

config system ha
  set override enable
end

2. Configuring the new FortiGate

Enter this command to reset the new FortiGate to factory default settings:

execute factoryreset

You can skip this step if the new FortiGate is fresh from the factory. But if its configuration has been changed at all it’s recommended to set it back to factory defaults to reduce the chance of synchronization problems.

If required, change the firmware running on the new FortiGate to match the cluster firmware version.

Register and apply licenses to the new FortiGate before adding it to the cluster. This includes licensing for FortiCare Support, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, and additional virtual domains (VDOMs). All FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs. FortiToken licenses can be added at any time because they are synchronized to all cluster members.

Change the host name of the new FortiGate to identify it as Backup-2 by clicking on the System Information dashboard widget and selecting Configure settings in System > Settings and changing the Host name.

You can also enter this CLI command:

config system global 
   set hostname Backup-2
end

Duplicate the primary FortiGate HA settings, except set the Device Priority to a lower value (for example, 50) and do not enable override.

config system ha
  set mode a-p
  set group-id 100
  set group-name My-cluster
  set password <password>
  set priority 50
  set hbdev lan4 200 lan5 100
end

Once you enter the CLI command the new FortiGate negotiates to establish an HA cluster. You may temporarily lose connectivity with the FortiGate while FGCP negotiation takes place and the FortiGate interface MAC addresses change to HA virtual MAC addresses.

If the group ID is the same, the backup FortiGate interfaces get the same virtual MAC addresses as the primary FortiGate. You can check Network > Interfaces on the GUI or use the get hardware nic command.

3. Connecting the new FortiGate to the cluster

Connect the new FortiGate to the cluster and your network as shown in the network diagram above. Making these connections disrupts network traffic as you disconnect and re-connect the heartbeat interfaces. If you have already added switches to connect the heartbeat interfaces, you can connect the new FortiGate without disrupting network traffic.

When you add a third FortiGate to a cluster you need to connect the heartbeat interfaces together using switches. You can use separate switches for each heartbeat interface (recommended for redundancy) or you can use the same switch for both heartbeat interfaces as long as you separate the traffic from each heartbeat interface.

When you connect the heartbeat interfaces of the new FortiGate, the cluster re-negotiates. If you have enabled override on the primary FortiGate and set its priority highest, the primary FortiGate synchronizes its configuration to the new FortiGate. The cluster automatically forms with minimal or no additional disruption to network traffic.

The new cluster will have the same IP addresses as the primary FortiGate. You can log into the cluster by logging into the primary FortiGate CLI or GUI.

4. Checking cluster operation

Check the cluster synchronization status to make sure all FortiGates have the same configuration. Log into the primary FortiGate CLI and enter this command:

diagnose sys ha checksum cluster

The command output lists all cluster members’ configuration checksums. If they all have identical checksums you can be sure that the configurations are synchronized. If the checksums are different, wait a short while and enter the command again. Repeat until the checksums are identical. It may take a while for some parts of the configuration to be synchronized. If the checksums never become identical you can use the information in Synchronizing the configuration to troubleshoot the problem or visit the Fortinet Support website.

The HA Status dashboard widget also shows the synchronization status of the cluster members. Mouse over each FortiGate in the widget to verify their synchronization status and that all have the same checksum.

For more detailed cluster status information, from the HA Status widget, select Configure Settings in System > HA (or go to System > HA) to view more detailed cluster status information.

5. Disabling override (recommended)

When the checksums are identical, disable override on the primary FortiGate by entering the following command:

config system ha
   set override disable
end

6. Converting to an active-active cluster

Log into the primary FortiGate CLI and enter this command to convert the cluster from an active-passive to an active-active cluster. The cluster changes modes without any traffic interruption.

config system ha
  set mode a-a
end

7. Results

Most traffic should now be flowing through the primary FortiGate with proxy-based NGFW/UTM sessions distributed to all three FortiGates in the cluster. If the primary FortiGate becomes unavailable, the cluster negotiates and one of the backup FortiGates becomes the new primary FortiGate. If you restore the primary FortiGate, when it rejoins the cluster, the same backup FortiGate should continue operating as the primary FortiGate. Enable override if you want to make sure the same FortiGate always becomes the primary FortiGate.

To test failover, ping a reliable IP address from a PC on the internal network. After a moment, power off the primary FortiGate.* You will see a momentary pause in the ping results as the cluster negotiates and chooses a new primary FortiGate. Once the cluster selects a new primary FortiGate, the ping traffic continues:

64 bytes from 184.25.76.114: icmp_seq=69 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=70 ttl=52 time=8.822 ms\
64 bytes from 184.25.76.114: icmp_seq=71 ttl=52 time=9.034 ms\
64 bytes from 184.25.76.114: icmp_seq=72 ttl=52 time=9.536 ms\
64 bytes from 184.25.76.114: icmp_seq=73 ttl=52 time=8.877 ms\
64 bytes from 184.25.76.114: icmp_seq=74 ttl=52 time=8.901 ms\
Request timeout for icmp_seq 75\
64 bytes from 184.25.76.114: icmp_seq=76 ttl=52 time=8.860 ms\
64 bytes from 184.25.76.114: icmp_seq=77 ttl=52 time=9.174 ms\
64 bytes from 184.25.76.114: icmp_seq=78 ttl=52 time=10.108 ms\
64 bytes from 184.25.76.114: icmp_seq=79 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=80 ttl=52 time=10.861 ms\
64 bytes from 184.25.76.114: icmp_seq=81 ttl=52 time=10.757 ms\
64 bytes from 184.25.76.114: icmp_seq=82 ttl=52 time=8.158 ms\
64 bytes from 184.25.76.114: icmp_seq=83 ttl=52 time=8.639 ms}