Windows 2000 TCP Performance Tuning Tips

Windows 2000 TCP Performance Tuning Tips

Carl Harris

Communications Network ServicesVirginia Tech

 

I was dismayed by the rather dismal TCP performance in Win2000, especially for FTP up/downloads to distant sites, where the round-trip time (RTT) is nearly triple-digit. Being a UNIX bigot of sorts, I also found it amusing to watch my Win2000 machine dribble a download along at 1/6th the speed of my UNIX box for the same file, on network connections and host systems that are virtually identical. I would have simply filed this away in the "Microsoft Will Never Get It Right" folder, but I've heard from reliable sources that the Win2K TCP/IP stack is largely an adaptation of the TCP/IP implementation in NetBSD -- assuming that's true, given such venerable roots it ought to be able to get much better TCP throughput than I've observed.

 

I did a little digging around and found the following TechNet document that describes the TCP/IP implementation in an astonishing level of detail.

 

            http://www.microsoft.com/technet/itsolutions/network/deploy/depovg/tcpip2k.asp

 

(Because Microsoft seems to want to remedy world unemployment by employing armies of people charged with the single task of relocating every document on the Microsoft web site on an ongoing basis, you can download a copy of a PDF version of this article here.)

 

After reading this document, two things became abundantly clear.

 

1. Microsoft has really done a remarkable job. The TCP implementation includes virtually all of the recent extensions to improve performance on long fat networks -- in which there is plenty of bandwidth but significant round-trip time (RTT). TCP window scaling and timestamps, selective acknowledgement, delayed acknowledgement, as well as other enhancements are all implemented.

 

2. The default parameters for the TCP window size and scaling parameters options are suitable for small 10 Mbps LANs or Internet access where the smallest bottleneck is no more than 2 Mbps. However, for Internet access at 10 Mbps or more, or for 100 Mbps LANs, the default parameters are too conservative, and throughput may suffer.

 

Bottom line, I installed registry parameters to change the TCP window size and to enable the TCP scaling windows and timestamps options (RFC 1323). With these new parameters in place (and after the obligatory reboot), my Win2000 box is able to achieve 7 Mbps throughput, downloading large files from well-connected distant FTP sites. Previously, it could do no better than about 1.8 Mbps throughput.

 

The parameters I installed go into

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]

 

The names/values I installed are:

GlobalMaxTcpWindowSize	REG_DWORD	131400 (decimal)
TcpWindowSize	REG_DWORD	131400 (decimal)
Tcp1323Opts	REG_DWORD	3

 

These apply only to Windows 2000, as far as I can tell. There are analogous parameters for the window size for WinNT/98/95, but the RFC 1323 options (for window scaling, in particular) are not implemented, which means that the maximum feasible window size is 64 KB.

 

The window size parameters represent a nearly eight-fold increase over the default, and should be adequate for most 10/100 connected stations communicating over just about any path reachable from the campus. For server systems communicating with lots of downstream clients, you may want to be more conservative; with this configuration, every TCP socket will consume at least 128 KB in memory. The RFC-1323 opts parameter must be either 1 or 3 to allow the window size to scale beyond 64 KB. The setting of 3 also enables TCP timestamp headers; this TCP option improves the round-trip delay estimate (improving overall window size dynamics), at the cost of some marginal amount of throughput.

 

So what is the TCP window size thing?

 

The TCP window is the amount of unacknowledged data in flight between the sender and the receiver. Data is sent by TCP in segments that are typically 1460 bytes in length. If the sender is permitted a window size of only 1 segment, the sender transmits a single segment, and waits for an acknowledgement from the receiver. If the transmission delay between sender and receiver is long, this means very low throughput (very few segments transferred per unit time). Both sender and receiver spend most of their time waiting for messages to be transmitted from one end of the connection to the other.

 

In order to improve throughput, the sender transmits multiple segments without waiting for the next acknowledgement from the receiver. The TCP window is an estimate of the upper bound on the number of segments that will fit in the length of the pipe between sender and receiver. The window size is increased during a TCP transfer until the end-to-end path becomes too full (indicated by a segment being dropped somewhere in the network), then the size is backed off and then increased slowly again until the limit is reached. This cycle of shrinking and slowly expanding window size continues throughout the TCP connection. In this way, TCP tries to optimize the transmit window to maximize throughput over the lifetime of the connection. The receiver advertises his maximum window size to give the sender an idea of how much buffer space the receiver has available. This puts a hard limit on size of the window, even if more bandwidth is available in the network.

 

If the pipe is pretty big, and the round-trip delay is long, a lot of segments will fit in the network between the sender and receiver.

The window size needs to be pretty big in a long, fat network. How big?

 

            window size = bandwidth * delay

 

For 10 Mbps bandwidth and a round-trip delay of 0.010 sec, this gives a window size of about 12 KB (or nine 1460-byte segments). This should yield maximal throughput on a 10 Mbps LAN, even if the delay is as high as 10 ms -- most LANs have round-trip delay of less than a few milliseconds.   When bandwidth is lower, more delay can be tolerated for the same fixed window size, so a window size of 12 KB works well at lower speeds, too.

 

For a fixed window size, large increases in bandwidth mean that small increases in round-trip delay will have a significant effect on throughput. If the end-to-end bandwidth is 100 Mbps, a 12 KB window size implies that no more than 1.05 ms of delay is tolerable if TCP throughput near 100 Mbps is desired -- that's a pretty tight bound, even for a LAN.

 

The default maximum TCP window size in Win2000 is 17520 bytes (12 segments). This should work nicely in LANs -- even at 100 Mbps -- assuming that end-to-end delay is not much more than 2 ms. The default window size should yield TCP throughput of around 70 Mbps on a 100 Mbps LAN, assuming congestion is minimal. This default max TCP window size is also large enough for round trip delays of 70 ms and end-to-end bandwidth of up to 2 Mbps. That's no coincidence; most entities on the Internet have access pipes that are T1 (1.5 Mbps) or E1 (2 Mbps) and the nominal round-trip delay coast-to-coast in the U.S. is about 70 ms. So, as a first cut at a reasonable default setting for the TCP max window size, this ain't too bad.

 

But what if you have a WAN connection with nominal 10 Mbps bandwidth available, and you're downloading from a distant, well-connected server. Perhaps you’re on the US East coast, and you’re downloading from a West coast server. The typical delay in this case is about 85 ms.   In order to get 10 Mbps throughput over a connection with 85 ms round trip delay, the TCP window will need to be at least 100 KB. If the delay varies by as much as 20 ms, you'll need about 128 KB in the window.

 

The parameters I chose provide a maximum TCP window of 90 1460-byte segments (about 128 KB). This window size should be able to compensate for as much as 105 ms delay, and still deliver 10 Mbps throughput, assuming that packet loss due to congestion is low.

 

Is this a bad thing to do? If everyone sets their TCP window size at 128 KB and thus has the potential to get 10 Mbps throughput on downloads across a substantial chunk of the Internet, you might be thinking that Bad Things might happen. TCP is smarter than that. The window size advertised by the receiver is just an upper bound. The sender increases the size of the transmit window slowly, and uses packet loss as the indication that the transmit window is too big. Thus, when there are lots of senders, each will see proportionally less bandwidth available for transmit window expansion, and throughput for each will be proportionally lower. This happens with virtually no adverse affect on the network.

 

Is this going to "hurt" performance for connections with much lower delay (e.g. in the LAN)? The round trip time (RTT) plays a crucial role in determining the rate of growth of the window. Even though a receiver might advertise a large maximum window size, the TCP slow start ensures that the sender won't try to put that much data in flight before received acknowledgements start providing a good estimate of the RTT, hence the length of the pipe, hence the number of segments that should be kept in flight.

 

The only remaining issue is whether the allocation of 128 KB buffers for each TCP socket poses a significant threat to system performance. Given a 128 MB system that is behaving mostly as a client, it's unlikely that memory resources are tight enough to warrant much concern there. You’ll want more memory for socket buffers if you plan to do this on a server that has lots of concurrent connections on the network.

 

Copyright 2001, Carl Harris, Jr.

ceharris@vt.edu