FortiOS 6.0 Expert Recipe: SD-WAN with FGCP HA

This SD-WAN HA configuration allows you to load balance your Internet traffic between multiple ISP links and provides redundancy for your network’s Internet connection if your primary ISP is unavailable or if one of the FortiGates in the HA cluster fails.

This recipe features two FortiGate-51Es, which have a 5-port switch lan interface. Before starting the steps in this recipe, we converted the lan interface to 5 separate interfaces (lan1 to lan5). The lan1 interface connects to the internal network, the wan1 interface connects to one Internet service provider (ISP) and the wan2 to a second ISP.  For the FGCP HA configuration, the lan4 and lan5 interfaces become HA heartbeat interfaces.

1. Connecting the FortiGate to your ISP devices

Connect the Internet-facing ports (WAN ports) on the FortiGate to your ISP devices. Connect WAN1 to the ISP that you want to use for most traffic. Connect WAN2 to the other ISP.

2. Removing existing configuration references to interfaces

Before you can configure FortiGate interfaces as SD-WAN members, you must remove or redirect existing configuration references to those interfaces in routes and security policies. This includes the default Internet access policy that’s included with many FortiGate models. Note that after you remove the routes and security policies, traffic can’t reach the WAN ports through the FortiGate.

Redirecting the routes and policies to reference other interfaces avoids your having to create them again later. After you configure SD-WAN, you can reconfigure the routes and policies to reference the SD-WAN interface.

Go to Network > Static Routes and delete any routes that use WAN1 or WAN2.

Go to Policy & Objects > IPv4 Policy and delete any policies that use WAN1 or WAN2.

3. Creating the SD-WAN interface

Go to Network > SD-WAN and set Status to Enable.

Under SD-WAN Interface Members, select + and select wan1. Set the Gateway to the default gateway for this interface. This is usually the default gateway IP address of the ISP that this interface is connected to. Repeat these steps to add wan2.

Go to Network > Interfaces and verify that the virtual interface for SD-WAN appears in the interface list. You can expand SD-WAN to view the ports that are included in the SD-WAN interface.

4. Configuring SD-WAN load balancing

Go to Network > SD-WAN Rules and edit the rule named sd-wan.

In the Load Balancing Algorithm field, select Volume, and prioritize WAN1 to serve more traffic.

In the example, the ISP connected to WAN1 is a 40Mb link, and the ISP connected to WAN2 is a 10Mb link, so we balance the weight 75% to 25% in favor of WAN1.

5. Creating a static route for the SD-WAN interface

Go to Network > Static Routes and create a route.

In the Destination field, select Subnet, and leave the destination IP address and subnet mask as 0.0.0.0/0.0.0.0.

In the Interface field, select the SD-WAN interface from the drop-down menu.

Ensure that Status is set to Enable.

If you previously removed or redirected existing references in routes to interfaces that you wanted to add as SD-WAN interface members, you can now reconfigure those routes to reference the SD-WAN interface.

6. Configuring a security policy for SD-WAN

Configure a security policy that allows traffic from your organization’s internal network to the SD-WAN interface.

Go to Policy & Objects > IPv4 Policy and create a policy.

Set Incoming Interface to the interface that connects to your organization’s internal network, and set Outgoing Interface to the SD-WAN interface.

Enable NAT and apply Security Profiles as required.

Configure other policy options as required.

7. Configuring the FortiGate for HA

Change the Host name to identify this FortiGate as the primary FortiGate. From the System Information dashboard widget, select Configure settings in System > Settings.

You can also enter this CLI command:

config system global 
   set hostname Primary
end

Register and apply licenses to the primary FortiGate before configuring it for HA operation.

Enter this CLI command to set the HA mode to active-passive; set a group ID, group name and password; increase the device priority to a higher value (for example, 250); and enable override.

config system ha 
  set mode a-p 
  set group-id 100 
  set group-name My-cluster
  set password <password> 
  set priority 250 
  set override enable
  set hbdev lan4 200 lan5 100 
end

Enabling override and increasing the device priority means this FortiGate always becomes the primary unit.

This configuration also selects lan4 and lan5 to be the heartbeat interfaces and sets their priorities to 200 and 100 respectively. It’s a best practice to set different priorities for the heartbeat interfaces (but not a requirement).

If you have more than one cluster on the same network, each cluster should have a different group ID. Changing the group id changes the cluster interface virtual MAC addresses. If your group ID causes a MAC address conflict on your network, you can select a different group ID.

Override and the group ID can only be configured from the CLI.

config system ha
  set group-id 100
  set override enable
end

You can also configure most of these settings from the GUI (go to System > HA).

After you enter the CLI command or make changes from the GUI, the FortiGate negotiates to establish an HA cluster. You may temporarily lose connectivity with the FortiGate as FGCP negotiation takes place and the MAC addresses of the FortiGate interfaces are changed to HA virtual MAC addresses.

To reconnect sooner, you can update the ARP table of your management PC by deleting the ARP table entry for the FortiGate unit (or just deleting all ARP table entries). You can usually delete the ARP table from a command prompt using a command similar to arp -d.

8. Configuring the backup FortiGate

If required, change the firmware running on the new FortiGate to the same version as is running on the primary FortiGate.

Enter the following command to reset the new backup FortiGate to factory default settings.

execute factoryreset

You can skip this step if the new FortiGate is fresh from the factory. But if its configuration has been changed at all, it’s a best practice to reset your FortiGate to factory defaults to reduce the chance of synchronization problems.

Register and apply licenses to the backup FortiGate before configuring it for HA operation. This includes licensing for FortiCare Support, IPS, AntiVirus, Web Filtering, Mobile Malware, FortiClient, FortiCloud, Security Rating, Outbreak Prevention, and additional virtual domains (VDOMs). All FortiGates in the cluster must have the same level of licensing for FortiGuard, FortiCloud, FortiClient, and VDOMs. FortiToken licenses can be added at any time because they are synchronized to all cluster members.

Click on the System Information dashboard widget, and select Configure settings in System > Settings. Change the FortiGate’s Host name to identify it as the backup FortiGate.

You can also enter this CLI command:

config system global 
   set hostname Backup 
end

Duplicate the primary FortiGate HA settings, except set the Device Priority to a lower value (for example, 50) and do not enable override.

config system ha 
  set mode a-p 
  set group-id 100 
  set group-name My-cluster
  set password <password> 
  set priority 50 
  set hbdev lan4 200 lan5 100 
end

Similar to when configuring the primary FortiGate, when you enter the CLI command, the backup FortiGate negotiates to establish an HA cluster. You may temporarily lose connectivity with the FortiGate as FGCP negotiation takes place and the MAC addresses of the FortiGate interfaces are changed to HA virtual MAC addresses.

9. Connecting the primary and backup FortiGates

Connect the primary and backup FortiGates to each other and to your network as shown. Making these connections disrupts network traffic as you disconnect and re-connect cables.

Switches must be used between the cluster and the ISPs and between the cluster and the internal network as shown in the network diagram. You can use any good quality switches to make these connections. You can also use one switch for all of these connections as long as you configure the switch to separate traffic from the different networks.

The example shows the recommended configuration of direct connections between the lan4 heartbeat interfaces and between the lan5 heartbeat interfaces.

When the heartbeat interfaces are connected, the FortiGates find each other and negotiate to form a cluster. The primary FortiGate synchronizes its configuration to the backup FortiGate. The cluster forms automatically with minimal or no additional disruption to network traffic.

The cluster will have the same IP addresses as the primary FortiGate had. You can log into the cluster by logging into the primary FortiGate CLI or GUI using one of the original IP addresses of the primary FortiGate.

10. Checking cluster operation

Check the cluster synchronization status to make sure the primary and backup FortiGates have the same configuration. Log into the primary FortiGate CLI and enter this command:

diagnose sys ha checksum cluster

The command output lists all cluster members’ configuration checksums. If both cluster members have identical checksums, you can be sure that their configurations are synchronized. If the checksums are different, wait a short while and enter the command again. Repeat until the checksums are identical. It may take a while for some parts of the configuration to be synchronized. If the checksums never become identical, you can use the information in Synchronizing the configuration to troubleshoot the problem or visit the Fortinet Support website for assistance.

The HA Status dashboard widget also shows synchronization status. Mouse over the host names of each FortiGate in the widget to verify that they are synchronized and both have the same checksum.

To view more information about the cluster status, click on the HA Status widget and select Configure Settings in System > HA (or go to System > HA).

11. Disabling override (recommended)

When the checksums are identical, disable override on the primary FortiGate by entering the following command:

config system ha
   set override disable
end

FGCP clusters dynamically respond to network conditions. If you keep override enabled, the same FortiGate always becomes the primary FortiGate. With override enabled, however, the cluster may negotiate more often to keep the same FortiGate as the primary FortiGate, potentially increasing traffic disruptions.

If you disable override, it is more likely that the backup FortiGate could become the primary FortiGate. Disabling override is recommended unless it’s important that the same FortiGate remains the primary FortiGate.

12. Results

Browse the Internet using a computer on your internal network and then go to Network > SD-WAN.

In the SD-WAN Usage section, you can see the bandwidth, volume, and sessions for traffic on the SD-WAN interfaces.

Go to Monitor > SD-WAN Monitor to view the number of sessions, bit rate, and more information for each interface.

13. Testing HA failover

All traffic should now be flowing through the primary FortiGate. If the primary FortiGate becomes unavailable, traffic fails over to the backup FortiGate. When the primary FortiGate rejoins the cluster, the backup FortiGate should continue operating as the primary FortiGate.

To test this, ping a reliable IP address from a PC on the internal network. After a moment, power off the primary FortiGate. You will see a momentary pause in the ping results, until traffic diverts to the backup FortiGate, allowing the ping traffic to continue.

64 bytes from 184.25.76.114: icmp_seq=69 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=70 ttl=52 time=8.822 ms\
64 bytes from 184.25.76.114: icmp_seq=71 ttl=52 time=9.034 ms\
64 bytes from 184.25.76.114: icmp_seq=72 ttl=52 time=9.536 ms\
64 bytes from 184.25.76.114: icmp_seq=73 ttl=52 time=8.877 ms\
64 bytes from 184.25.76.114: icmp_seq=74 ttl=52 time=8.901 ms\
Request timeout for icmp_seq 75\
64 bytes from 184.25.76.114: icmp_seq=76 ttl=52 time=8.860 ms\
64 bytes from 184.25.76.114: icmp_seq=77 ttl=52 time=9.174 ms\
64 bytes from 184.25.76.114: icmp_seq=78 ttl=52 time=10.108 ms\
64 bytes from 184.25.76.114: icmp_seq=79 ttl=52 time=8.719 ms\
64 bytes from 184.25.76.114: icmp_seq=80 ttl=52 time=10.861 ms\
64 bytes from 184.25.76.114: icmp_seq=81 ttl=52 time=10.757 ms\
64 bytes from 184.25.76.114: icmp_seq=82 ttl=52 time=8.158 ms\
64 bytes from 184.25.76.114: icmp_seq=83 ttl=52 time=8.639 ms}

You can log in to the cluster GUI or CLI using the same IP address as you had been using to the log into the primary FortiGate. If the primary FortiGate is powered off, you will be logging into the backup FortiGate. Check the host name to verify the FortiGate that you have logged into. The FortiGate continues to operate in HA mode and if you restart the primary FortiGate, after a few minutes it should rejoin the cluster and operate as the backup FortiGate. Traffic should not be disrupted when the restarted primary unit rejoins the cluster.

14. Testing ISP failover

To test failover of the redundant Internet configuration, you must simulate a failed Internet connection to one of the ports. You can do so by disconnecting power from the wan1 switch or otherwise disconnecting the wan1 interfaces of both FortiGates from ISP 1.

Verify that users still have Internet access by navigating to Monitor > SD-WAN Monitor. The Upload and Download values for WAN1 show that traffic isn’t going through that interface.

Go to Network > SD-WAN. In the SD-WAN Usage section, you can see that bandwidth, volume, and sessions have diverted entirely through WAN2.

Users on the internal network shouldn’t notice the WAN1 failure. Likewise, if you’re using the WAN1 gateway IP address to connect to the admin dashboard, nothing should change from your perspective. It appears as though you’re still connecting through WAN1.

After you verify successful failover, re-establish the connection to ISP 1.