Analysis on the problem of network packet loss caused by Ip_conntrack tracking connection full

Our online Web server when the traffic is very large, there will be a network connection drops problem, through the DMESG command to view the log, the following information is found:

Kernel:ip_conntrack:table full, dropping packet.kernel:printk:1 messages suppressed.kernel:ip_conntrack:table full, Dropping packet.kernel:printk:2 messages suppressed.kernel:ip_conntrack:table full, dropping packet.

The key message is "ip_conntrack:table full, dropping packet", from here can be judged that this is related to iptables, because Iptables firewall uses IP_ Conntrack Kernel module realizes the connection tracking function, all incoming and outgoing packets are recorded in the connection tracking table, including TCP,UDP,ICMP, and once the connection tracking table is filled, the packet loss will occur, causing the network instability.

And our servers do open the Iptables firewall, and this is a problem that often occurs when the site is very high-traffic. The cause of this problem is because the Web server received a large number of connections, in the case of iptables enabled, Iptables will all the connections are linked tracking processing, so that iptables will have a link tracking table, when the table full, the above error will occur.

The maximum capacity profile for the Iptables Link Tracking table is as follows:

CENTOS5 netfilter parameter configuration file:

/proc/sys/net/ipv4/netfilter/ip_conntrack_max or/proc/sys/net/ipv4/ip_conntrack_max

CENTOS6 netfilter parameter configuration file:

/proc/sys/net/netfilter/nf_conntrack_max

Since Nf_conntrack works on Layer 3, supports IPV4 and IPV6, and Ip_conntrack only supports IPV4, nf_conntrack modules are introduced in the 2.6.15 kernel of Linux, and IP_ Conntrack is removed from the Linux 2.6.22 kernel (centos6.x version), so different versions of the system, the configuration files are not the same. Most of the current ip_conntrack_* has been replaced by nf_conntrack_*, many ip_conntrack_* is just a soft link, the original ip_conntrack configuration directory/proc/sys/net/ipv4/netfilter/ still exists, but the new nf_conntrack in/proc/sys/net/netfilter/, this is done in order to be able to be downward compatible.

After understanding the configuration file changes, we see how to solve this problem, the solution is generally two:

1, adjust the/proc/the following parameters

You can increase the entry for the appropriate conntrack, under Centos5/rhel 5:

(1) Operation

Sysctl-w net.ipv4.netfilter.ip_conntrack_max=655360

(2). Add in/etc/sysctl.conf:

Net.ipv4.netfilter.ip_conntrack_max = 655360

(3). Make it effective

Sysctl-p

Under the CentOS 6/rhel6:

(1) Operation

Sysctl-w net.nf_conntrack_max=100000

(2) Add in/etc/sysctl.conf:

Net.nf_conntrack_max = 100000

(3) Make it effective

Sysctl-p

2. Do not use Ip_conntrack module

Under Centos5/rhel 5:

If you do not use ip_conntrack, you need to remove the state module because the module needs to be loaded with ip_conntrack. Make sure that no rules like the state module appear in the Iptables rule and remove it if there is one:

Then comment in the/etc/sysconfig/iptables-config:

Iptables_modules= "Ip_conntrack_netbios_ns"

Finally, remove the Ip_conntrack module:

[Email protected] ipv4]# modprobe-r Ip_conntrack_netbios_ns xt_state

Under the CENTOS6/RHEL6:

[Email protected] ipv4]# modprobe-r nf_conntrack_ipv4 xt_state[[email protected] ipv4]# modprobe-r nf_conntrack

Now there should be no nf_conntrack under the/proc/net/.

Of the two methods, the first one is simple, but the symptoms do not cure the problem, the second kind of a little trouble, but after all, we can choose according to the situation.

This article is from the "Technical Achievement Dream" blog, please be sure to keep this source http://ixdba.blog.51cto.com/2895551/1737642

Analysis on the problem of network packet loss caused by Ip_conntrack tracking connection full