Analysis on network packet loss caused by full tracking of ip_conntrack connections

Analysis on network packet loss caused by full tracking of ip_conntrack connections

When the access volume of our online web server is very large, packet loss occurs during network connection. You can view the log by running the dmesg command and find the following information:

kernel:ip_conntrack:tablefull,droppingpacket.kernel:printk:1messagessuppressed.kernel:ip_conntrack:tablefull,droppingpacket.kernel:printk:2messagessuppressed.kernel:ip_conntrack:tablefull,droppingpacket.

The key information here is "ip_conntrack: table full, dropping packet". It can be determined from this that this is related to iptables, because iptables Firewall uses the ip_conntrack kernel module to implement the connection tracking function, all inbound and outbound data packets are recorded in the connection trace table, including tcp, udp, and icmp. Once the connection trace table is filled up, packet loss occurs, leading to network instability.

Our server does open the iptables firewall, and this problem often occurs when the website traffic is very high. This problem occurs because the web server receives a large number of connections. When iptables is enabled, iptables will track all connections, in this way, iptables will have a link tracking table. When the table is full, the above error will occur.

The maximum capacity configuration file of the Link Tracking table of iptables is as follows:

Centos5 netfilter parameter configuration file:

/Proc/sys/net/ipv4/netfilter/ip_conntrack_max or/proc/sys/net/ipv4/ip_conntrack_max

Centos6 netfilter parameter configuration file:

/proc/sys/net/netfilter/nf_conntrack_max

Because nf_conntrack works at Layer 3 and supports IPv4 and IPv6 while ip_conntrack only supports IPv4, The nf_conntrack module is introduced in Linux kernel 2.6.15, the ip_conntrack kernel in Linux 2.6.22 is removed (centos6.x). Therefore, the configuration files are different for different versions of the system. Currently, most ip_conntrack _ * has been replaced by nf_conntrack _ *. Many ip_conntrack _ * are only soft links, the original ip_conntrack configuration directory/proc/sys/net/ipv4/netfilter/still exists, but the new nf_conntrack is in/proc/sys/net/netfilter, this is done to achieve backward compatibility.

After learning about the changes in the configuration file, let's take a look at how to solve this problem. There are generally two solutions:

1. Adjust the parameters below/proc/

You can increase the entries of the appropriate conntrack in CentOS5/RHEL 5:

(1) Run

sysctl-wnet.ipv4.netfilter.ip_conntrack_max=655360

(2) Add the following to/etc/sysctl. conf:

net.ipv4.netfilter.ip_conntrack_max=655360

(3). Make it take effect

sysctl-p

Under CentOS 6/RHEL6:

(1) Run

sysctl-wnet.nf_conntrack_max=100000

(2) Add the following to/etc/sysctl. conf:

net.nf_conntrack_max=100000

(3) make it take effect

sysctl-p

2. Do not use the ip_conntrack Module

Under CentOS5/RHEL 5:

If ip_conntrack is not used, the state module needs to be removed because ip_conntrack needs to be loaded when this module is used. Make sure that the iptables rule does not contain a rule similar to the state module. If yes, remove it:

Then, comment out the following in/etc/sysconfig/iptables-config:

IPTABLES_MODULES="ip_conntrack_netbios_ns"

Finally, remove the ip_conntrack module:

[root@waiweiipv4]#modprobe-rip_conntrack_netbios_nsxt_state

Under CentOS6/RHEL6:

[root@waiweiipv4]#modprobe-rnf_conntrack_ipv4xt_state[root@waiweiipv4]#modprobe-rnf_conntrack

Now there should be no nf_conntrack in/proc/net.

Among the two methods, the first method is simple, but the solution is not permanent, and the second method is a little troublesome. However, after all, you can choose based on your situation.