Varnish cache 503 error, 400 Error cause and Solution

The main reason why varnish is used to replace squid in the production environment:

1. squid does not support multi-core CPUs. Most of the production environments use the Dell r610 series. This type of machine is configured with two 4-core dual-thread CPUs and the operating system recognizes 16. For squid, only one logical CPU can be used. The other 15 logical CPUs are a waste of resources.

2. Squid cannot clear cache in batches

3. squid3.0 currently does not support HTTP 1.1, which is unnecessary for some management adjustments (such as gzip compression and cache-control ).

4. Currently, I have not found the squid Grace mode (that is, when the origin site cannot arrive, it will continue to use the expired cache to provide services for users), and varnish is fully supported.

5. after several months of repeated tests in the production environment, varnish consumes less and has never experienced a crash. Squid does not support multiple cores, as a result, the Dell r610 server load often exceeds 4, but after varnish is used, the server load never exceeds 1.5.

After repeated tests, we have started to replace squid with varnish in millions of independent daily IP addresses.

However, this process is full of many unknown situations. The key problems we have currently encountered are:

1. iptables packet loss in high traffic situations
Ip_conntrack: Table full, dropping packet.
According to the conventional method on the internet, modifying only/etc/sysctl. conf is invalid.

Follow these steps:
Add parameters to/boot/GRUB/grub. conf kernel.
Ip_conntrack.hashsize = 524288
Added in/etc/sysctl. conf
Net. ipv4.netfilter. ip_conntrack_max = 2097152

Note: The above operation requires you to restart the machine.

2. default value for logging on to shell ulimit
/Etc/security/limits. conf
* Soft nofile 655360
* Hard nofile 655360
The default file descriptor is limited to 1024, which is too small. Even if it is adjusted to 65536, it may not be enough for High-load machines. Therefore, it is set to 655360. Note that this value cannot be too large, otherwise, the shell cannot log on.

After modifying/etc/security/limits. conf, it is only valid for Shell logon. Pay attention to this problem for programs running at startup (set ulimit before startup)
 
Shell-level resource restrictions come from/etc/security/limits. conf. Where can the system-level resource restrictions be modified?

3. Squid is a solution for increasing traffic after Varnish:
/Etc/security/limits. conf
* Soft memlock 1048576
* Soft memlock 1048576
Or run ulimit-HSL 1048576
The default memlock is only 32 KB, which is too small for varnish. Varnish uses memory blocks to save logs, so this value needs to be increased.
The above value is adjusted to 1 GB to solve the problem that varnish traffic is higher than squid.

4. Errors 503 and 400 frequently occur during varnish operation

The error 503 is returned because varnish has a limit on the backend server response header. The default length is 2048, which can be increased.
-P http_resp_hdr_len = 8192

Then restart to solve the 503 error.

The error 400 occurs because varnish considers that the number of header lines and length of the client request are too large. By default, the maximum number of accepted request header lines is 64, and the maximum length (the sum of the length of all request header lines) is 2048, it is relatively simple to solve this problem. Add the following to the varnish startup parameters:
-P http_max_hdr = 256
-P http_req_hdr_len = 8192

Restart varnish to solve the 400 error.