File descriptor and squid in UNIX

A file descriptor is a simple integer used to indicate the files and sockets opened by each process. The first open file is 0, the second is 1, and so on. A Unix operating system usually imposes a limit on the number of files that can be opened by each process. What's more, Unix usually has a system-level limit.

Because of squid's working method, file descriptor restrictions may greatly affect performance. After squid runs out of all file descriptors, it cannot receive new connections from users. That is to say, a denial of service is caused by the use of file descriptors. After a part of the current request is completed, the corresponding file and socket are closed, and squid cannot receive new requests. When squid detects a file descriptor shortage, it will issue a warning.

Before running./configure, check whether the file descriptor restrictions of your system are appropriate to avoid some trouble. In most cases, 1024 file descriptors are sufficient. A very busy cache may require 4096 or more. When setting the file descriptor limit, we recommend that you set the system-level limit to twice the limit for each process.

Generally, you can find the system's file descriptor limits in your UNIX shell. All C shells and similar shells have built-in limit commands. The updated Bourne shell and similar shell have a command called ulimit. To discover the file descriptor limitations of your system, run the following command:

csh% limit descriptors unlimited        csh% limit descriptors        descriptors    4096

Or

sh$ ulimit -n unlimitedsh$ ulimit -n4096

On FreeBSD, you can use the sysctl command:

% sysctl -a | grep maxfileskern.maxfiles: 8192kern.maxfilesperproc: 4096

If you cannot confirm the file descriptor limit, Squid's./configure script can do it for you. When you run./configure, see section 3.4 to observe the output at the end:

checking Maximum number of file descriptors we can open... 4096

If other limit, ulimit, or./configure reports that the value is less than 1024, you have to spend time to increase the size of this limit before compiling squid. Otherwise, squid performs poorly at high loads.

The methods for increasing the file descriptor limit vary with systems. The following sections provide methods to help you start.

3.3.1.1 FreeBSD, NetBSD, and OpenBSD

Edit your Kernel configuration file and add the following line:

options       MAXFILES=8192

On OpenBSD, use option instead of options. Then, configure, compile, and install the new kernel. Finally, restart the system to make the kernel take effect.

3.3.1.2 Linux

Configuration file descriptors on Linux are a bit complicated. Before compiling squid, you must edit one of the system include files and execute some shell commands. Edit the/usr/include/bits/types. h file to change the value of _ fd_setsize:

#define _ _FD_SETSIZE    8192

Next, use this command to add the restrictions on Kernel file descriptors:

# echo 8192 > /proc/sys/fs/file-max 

Finally, add the process file descriptor restriction and execute it in the same shell where you are about to compile squid:

sh# ulimit -Hn 8192

This command must be run as root and only run in bash shell. You do not have to restart the machine. To use this technology, you must execute the echo and ulimit commands after each system startup, or at least before the squid is started. If you use an rc. d script to start squid, it is a good place to place these commands.

3.3.1.3 Solaris

Add the row to your/etc/system file:

set rlim_fd_max = 4096

Then, restart the machine for the change to take effect.