Tips for troubleshooting proxy firewall connection faults

With more proxy firewalls installed between the client and the server, troubleshooting of connection failures is more complicated than before. However, a little understanding of the Transmission Control Protocol (TCP) "handshake" process helps diagnose connection problems, including proxy firewalls.
First, for some of us who have grown white hair since our last online course launch, let's review how TCP handshake works. TCP is a connection-oriented protocol. Before any communication between a pair of devices, the two-way channel between these devices must be enabled. This is achieved by using special packets containing tags. A syn (synchronous) flag is used to indicate a connection request, and the ACK mark confirms that a connection has been opened.

Shows the complete three-way handshake:

First, the client sends a SYN packet to the server. The server will edit an answer packet after receiving this packet. This server opens the ACK mark to confirm the Client Communication Request. Then, the server opens a channel from the server to the client that marks the Relative Direction of the request. After the client receives the SYN/ACK packet, it must reply an ACK packet to confirm the server-Client Connection Request. At this time, the client and server may be able to communicate in two directions.

However, the proxy firewall injects itself into these processes, thus affecting the normal TCP handshake. See:

The client connects to the proxy firewall and thinks they have established a connection with the server. This proxy firewall then opens a separate connection with the server on behalf of this client. This unique method provides an additional isolation layer between the client and the server, and allows the firewall to adjust this session, and may check potential malicious content in application-layer communication.

The existence of the proxy firewall makes it more complicated to troubleshoot the connection between the client and the server. To diagnose a fault, both sides of the firewall must be monitored. We need an external monitoring device to view the external communication of the firewall, and an internal monitoring device to view each internal interface of the firewall. Let's take a look at some common connection problems and how they occur in the tcpdump output. All of our examples use the connection information described above. It should be noted that the web server has two IP addresses: 192.168.3.150 is the dedicated IP address of the server inside the firewall LAN. 10.0.0.120 is a public address exposed to the world.

First case: server problems

The server may have connection problems due to denial-of-service attacks, setup errors, successful hacker attacks, or other reasons. In these cases, a complete three-way handshake occurs outside the firewall because the firewall works normally. The output data looks like the following:

19:24:36. 959777 IP 10.1.1.16.1450> 10.0.0.120: 80: S 735906036: 735906036 (0) win 16384
19:24:36. 987938 IP 10.0.0.120: 80> 10.1.1.16.1450: S 3607622985: 3607622985 (0) ack 735906037 win 8190
19:24:36. 987961 IP 10.1.1.16.1450> 10.0.0.120: 80:. ack 1 win 17640
Note: For your convenience, both SYN and ACK use italics. This is a suitable three-way handshake. The problem will occur between the firewall and the Web server. If the server breaks down or does not respond, the indicator signal is a series of SYN packets sent from the firewall to the server, as shown below:

19:22:40. 214672 IP 192.168.3.124.1439> 10.0.0.120.80: S 4182454: 4182454 (0) win 16384
19:22:43. 154580 IP 192.168.3.124.1439> 10.0.0.120.80: S 4182454: 4182454 (0) win 16384
19:22:43. 154580 IP 192.168.3.124.1439> 10.0.0.120.80: S 4182454: 4182454 (0) win 16384
19:22:43. 154580 IP 192.168.3.124.1439> 10.0.0.120.80: S 4182454: 4182454 (0) win 16384
19:22:43. 154580 IP 192.168.3.124.1439> 10.0.0.120.80: S 4182454: 4182454 (0) win 16384

Case 2: Firewall Problems

Depending on the specific problem, firewall problems may be displayed in different ways. If the firewall is completely broken, the public address from the firewall to the server will have a series of SYN packets that do not respond. These packets will appear on the external monitor, and related communication will not be displayed on the internal monitor.

On the other hand, if the firewall rules are set incorrectly, the external monitor may display a suitable three-way handshake. The internal monitor may not display any communication.

Case 3: Application Problems

If an appropriate three-way handshake is displayed on both the internal and external monitors, firewall problems can be ruled out. In this case, the most likely cause is the application itself. For example, an attacker may break the application and change its behavior. The service may be configured incorrectly or the client may make an illegal request.

These situations cannot cover everything. It cannot involve all possible troubleshooting situations. These examples provide an excellent starting point for diagnosing connection problems with proxy firewalls.