Network Packet Troubleshooting Guide-class Linux Platform

Network Packet Troubleshooting Guide-class Linux platform background information

I've been testing k8s recently, and if you know or get in touch with Docker, you know that most of the networks associated with Docker are on bridging, routing, Iptables. If you happen to have contact with k8s, and understand the principle behind it, you must know that Kube-proxy iptables play to fly up. Of course, you may think of some of the wrong tools, such as I used to grab the package tool, or route tracking tools, but these tools in the present complex environment, is not too much to take advantage of, especially the package in the machine's multiple network cards or virtual network card, there are a lot of iptables strategy, routing and so the package in In the kernel space. The Grab kit tool does not capture this information, and traceroute tracks the route when you find that you need to track the routing information for a SRC,DST and port packet is not reached by law.

Here are some new debugging tools:

• IPTables Tracking Troubleshooting
• Local routing troubleshooting
• Some network-related kernel parameter settings.

Iptables Tracking Troubleshooting

Speaking of iptables wrong, I have to come up with this logic very clear figure out, it is suggested that iptables often control this diagram, see the delivery path of the packet. In my previous iptables knowledge category, I thought it was not a route to choose between multiple tables, and the actual error was added to the graph. It turns out that there may be routing decision between different table.

Please refer to the troubleshooting process in my k8s Issue.

Then I had to say Iptables's trace target, without knowing the target, I used the log target, and found that to write more than one iptables you would not necessarily be able to track the entire strategy of each package, and how the policy was handled.

I am currently demonstrating on Ubuntu above:

######### check is if the trace related mod is loaded modprobe nf_log_ipv4########## TRACE Target can only be applied to Raw Tablesudo iptables-t raw-i prerouting -P tcp-m TCP--dport 8081-j tracesudo iptables-t raw-i output-p tcp-m tcp--dport 8081-j trace########### grep TRACE in/var/log/kern.loggrep trace/var/log/kern.log[email protected]:~$ grep trace/var/log/kern.log|grep 2213090174May 8 16:30:29 Ceph3 kernel: [324781.838361] trace:raw:output:policy:2 in= out=enp3s0 src=192.168.235.13 DST=1 0.43.206.251 len=60 tos=0x00 prec=0x00 ttl=64 id=57266 DF proto=tcp spt=18130 dpt=8081 seq=2213090174 ACK=0 WINDOW=29200 R es=0x00 SYN urgp=0 OPT (020405b40402080a04d5cccc0000000001030307) uid=1000 gid=1000may 8 16:30:29 ceph3 kernel: [324781.8 38389] trace:nat:output:rule:1 in= out=enp3s0 src=192.168.235.13 dst=10.43.206.251 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=5 7266 DF proto=tcp spt=18130 dpt=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN urgp=0 OPT (020405B40402080A04D5CCCC00 00000001030307) uid=1000 gid=1000May 8 16:30:29 Ceph3 kernel: [324781.838417] trace:nat:kube-services:rule:9 in= out=enp3s0 src=192.168.235.13 DST=10.43. 206.251 len=60 tos=0x00 prec=0x00 ttl=64 id=57266 DF proto=tcp spt=18130 dpt=8081 seq=2213090174 ACK=0 WINDOW=29200 RES=0x The SYN urgp=0 OPT (020405b40402080a04d5cccc0000000001030307) uid=1000 gid=1000may 8 16:30:29 ceph3 kernel: [324781.838439 ] trace:nat:kube-svc-zp4vkujytbcrozyy:rule:1 in= out=enp3s0 src=192.168.235.13 dst=10.43.206.251 LEN=60 TOS=0x00 PREC= 0x00 ttl=64 id=57266 DF proto=tcp spt=18130 dpt=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN URGP=0 OPT (020405B404 02080a04d5cccc0000000001030307) uid=1000 Gid=1000may 8 16:30:29 ceph3 kernel: [324781.838454] Trace:nat: Kube-sep-or6jeccppingggrc:rule:2 in= out=enp3s0 src=192.168.235.13 dst=10.43.206.251 LEN=60 TOS=0x00 PREC=0x00 TTL=64 id=57266 DF proto=tcp spt=18130 dpt=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN urgp=0 OPT (020405B40402080A04D5CC CC0000000001030307) uid=1000 Gid=1000may 8 16:30:29 Ceph3 kernel: [324781.838479] trace:filter:output:rule:1 in= out=enp3s0 src=192.168.235.13 DST=10.0.1.12 LEN=60 to  s=0x00 prec=0x00 ttl=64 id=57266 DF proto=tcp spt=18130 dpt=8081 seq=2213090174 ack=0 window=29200 RES=0x00 SYN URGP=0 OPT (020405b40402080a04d5cccc0000000001030307) uid=1000 gid=1000may 8 16:30:29 ceph3 kernel: [324781.838493] Trace:filter: Kube-services:return:2 in= out=enp3s0 src=192.168.235.13 dst=10.0.1.12 len=60 tos=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO =tcp spt=18130 dpt=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN urgp=0 OPT ( 020405b40402080a04d5cccc0000000001030307) uid=1000 Gid=1000may 8 16:30:29 ceph3 kernel: [324781.838505] Trace:filter:o Utput:rule:2 in= out=enp3s0 src=192.168.235.13 dst=10.0.1.12 len=60 tos=0x00 prec=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18 dpt=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN urgp=0 OPT (020405b40402080a04d5cccc0000000001030307) UID= Gid=1000may 8 16:30:29 Ceph3 kernel: [324781.838518] Trace:filter: Kube-firewall:return:2 in= out=enp3s0 src=192.168.235.13 dst=10.0.1.12 len=60 tos=0x00 PREC=0x00 TTL=64 ID=57266 DF PROT O=tcp spt=18130 dpt=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN urgp=0 OPT ( 020405b40402080a04d5cccc0000000001030307) uid=1000 Gid=1000may 8 16:30:29 ceph3 kernel: [324781.838531] Trace:filter:o Utput:rule:4 in= out=enp3s0 src=192.168.235.13 dst=10.0.1.12 len=60 tos=0x00 prec=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18 dpt=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN urgp=0 OPT (020405b40402080a04d5cccc0000000001030307) UID=  Gid=1000may 8 16:30:29 Ceph3 kernel: [324781.838551] trace:filter:output:policy:6 in= out=enp3s0 src=192.168.235.13  dst=10.0.1.12 len=60 tos=0x00 prec=0x00 ttl=64 id=57266 DF proto=tcp spt=18130 dpt=8081 SEQ=2213090174 ACK=0 WINDOW=29200 res=0x00 SYN urgp=0 OPT (020405b40402080a04d5cccc0000000001030307) uid=1000 gid=1000may 8 16:30:29 ceph3 kernel: [324781 .838564] trace:nat:postrouting:rule:1 in= out=enp5s0 src=192.168.235. dst=10.0.1.12 len=60 tos=0x00 prec=0x00 ttl=64 id=57266 DF proto=tcp spt=18130 dpt=8081 SEQ=2213090174 ACK=0 WINDOW=29 res=0x00 SYN urgp=0 OPT (020405b40402080a04d5cccc0000000001030307) uid=1000 gid=1000may 8 16:30:29 ceph3 kernel: [324 781.838577] Trace:nat:kube-postrouting:return:2 in= out=enp5s0 src=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC= 0x00 ttl=64 id=57266 DF proto=tcp spt=18130 dpt=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN URGP=0 OPT (020405B404 02080a04d5cccc0000000001030307) uid=1000 Gid=1000may 8 16:30:29 ceph3 kernel: [324781.838589] TRACE:nat:POSTROUTING: Rule:8 in= out=enp5s0 src=192.168.235.13 dst=10.0.1.12 len=60 tos=0x00 prec=0x00 ttl=64 ID=57266 DF PROTO=TCP SPT=18130 DP t=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN urgp=0 OPT (020405b40402080a04d5cccc0000000001030307) UID=1000 GID =1000may 8 16:30:29 Ceph3 kernel: [324781.838609] trace:nat:postrouting:policy:10 in= out=enp5s0 src=192.168.235.13 DST= 10.0.1.12 len=60 tos=0x00 prec=0x00 ttl=64 id=57266 DF proto=tcp spt=18130 dpt=8081 seq=2213090174 ack=0 window=29200 res=0x00 SYN URGP=0 OPT (020405B404020 80a04d5cccc0000000001030307) uid=1000 gid=1000

To explain, each trace will record the name of the table, such as raw:OUTPUT:policy:2 or nat:OUTPUT:rule:1 , indicated as Table:chain: the explicit policy is rule,table the default policy is policy:rule number. I usually use grep id=57266 this method to filter the same package.

Above all crawl iptables log, if the middle encounter routing problems, such as I this problem packet log as follows, package to Nat:prerouting:policy:3 there is no below, should continue into mangle:input or filter:input, None of the results, refer to the above packet diagram, you can find there is a route decision process. So next I'll look at the problem of excluding local routes.

[email protected]:~$ grep trace/var/log/kern.log|grep 1726587944May 8 15:51:07 ceph2 kernel: [309854.514762] TRACE : Raw:prerouting:policy:2 in=enp5s0 out= mac=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 DST= 10.0.1.12 len=60 tos=0x00 prec=0x00 ttl=64 id=28265 DF proto=tcp spt=14024 dpt=8081 seq=1726587944 ACK=0 WINDOW=29200 RES= 0x00 SYN urgp=0 OPT (020405b40402080a04ccca410000000001030307) May 8 15:51:07 ceph2 kernel: [309854.514799] TRACE:nat:PRE Routing:rule:1 in=enp5s0 out= mac=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 src=192.168.235.13 DST=10.0.1.12 LEN=60 tos=0x00 prec=0x00 ttl=64 id=28265 DF proto=tcp spt=14024 dpt=8081 seq=1726587944 ack=0 WINDOW=29200 RES=0x00 SYN URGP=0 O PT (020405b40402080a04ccca410000000001030307) May 8 15:51:07 ceph2 kernel: [309854.514841] Trace:nat:kube-services: Rule:13 in=enp5s0 out= mac=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 src=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 prec=0x00 ttl=64 id=28265 DF proto=tcp spt=14024 dpt=8081 seq=1726587944 ack=0 window=29200 res=0x00 SYN urgp=0 OPT (020405b40402080a04ccca410000000001030307) May 8 15:51:07 C EPH2 kernel: [309854.514861] trace:nat:kube-nodeports:return:1 in=enp5s0 out= mac=00:23:7d:5b:96:ec:00:21:5a:ef:39: fe:08:00 src=192.168.235.13 dst=10.0.1.12 len=60 tos=0x00 prec=0x00 ttl=64 id=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=17 26587944 ack=0 window=29200 res=0x00 SYN urgp=0 OPT (020405b40402080a04ccca410000000001030307) May 8 15:51:07 CEPH2 kernel : [309854.514881] trace:nat:kube-services:return:14 in=enp5s0 out= mac=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC =192.168.235.13 dst=10.0.1.12 len=60 tos=0x00 prec=0x00 ttl=64 id=28265 DF proto=tcp spt=14024 DPT=8081 SEQ=1726587944 ACK =0 window=29200 res=0x00 SYN urgp=0 OPT (020405b40402080a04ccca410000000001030307) May 8 15:51:07 ceph2 kernel: [309854.51 4897] trace:nat:prerouting:rule:2 in=enp5s0 out= mac=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 dst=10.0.1.12 len=60 tos=0x00 prec=0x00 TTl=64 id=28265 DF proto=tcp spt=14024 dpt=8081 seq=1726587944 ack=0 window=29200 res=0x00 SYN URGP=0 OPT (020405B40402080A0 4ccca410000000001030307) May 8 15:51:07 ceph2 kernel: [309854.514914] Trace:nat:docker:return:2 in=enp5s0 OUT= MAC=00:23 : 7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 src=192.168.235.13 dst=10.0.1.12 len=60 tos=0x00 PREC=0x00 TTL=64 ID=28265 DF Proto=tcp spt=14024 dpt=8081 seq=1726587944 ack=0 window=29200 res=0x00 SYN urgp=0 OPT ( 020405b40402080a04ccca410000000001030307) May 8 15:51:07 ceph2 kernel: [309854.514930] Trace:nat:prerouting:policy:3 In=enp5s0 out= mac=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 src=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC= 0x00 ttl=64 id=28265 DF proto=tcp spt=14024 dpt=8081 seq=1726587944 ack=0 window=29200 res=0x00 SYN URGP=0 OPT (020405B404 02080a04ccca410000000001030307)

Local routing troubleshooting

In general, there are not many local routes, so the general traditional method is one-to-one routing entries, and then the artificial judgment will eventually be thrown there, if not found that the route can be processed, it will be drop. The new version of Linux with IP rule and IP route display and operation of the routing table, IP belongs to the suite in the Iproute2 package, after a general look at the following document, only to find 还有这种操作的 a feeling.

######## IP rule to list IP route tables[email protected]:~$ IP rule0:from all lookup local32766:from all loo Kup Main32767:from All lookup default######## Here are three table, first check 0 number of local, then check 32766 Main, and then check 32767 of default########   List the rule[email protected]:~$ IP Route list table localbroadcast 10.0.1.0 dev enp5s0 proto kernel scope link in each table src 10.0.1.12local 10.0.1.12 dev enp5s0 proto kernel scope host src 10.0.1.12broadcast 10.0.1.255 dev enp5s0 Proto K Ernel scope link src 10.0.1.12local 10.42.2.0 dev flannel.1 proto kernel scope host src 10.42.2.0broadcast 10.42.2.0 Dev cni0 proto kernel scope link src 10.42.2.1local 10.42.2.1 dev cni0 proto kernel scope host src 10.42.2.1broadcas T 10.42.2.255 Dev cni0 proto kernel scope link src 10.42.2.1broadcast 127.0.0.0 dev lo proto kernel scope link src 1  27.0.0.1local 127.0.0.0/8 Dev lo proto kernel scope host src 127.0.0.1local 127.0.0.1 dev lo proto kernel scope host SRC 127.0.0.1broadcast 127.255.255.255 Dev Lo proto kernel scope link src 127.0.0.1broadcast 172.17.0.0 dev docker0 proto kernel scope link src 172.17 .0.1 linkdownlocal 172.17.0.1 Dev Docker0 proto kernel scope host src 172.17.0.1broadcast 172.17.255.255 dev Docker0 p Roto kernel scope link src 172.17.0.1 linkdownbroadcast 192.168.235.0 dev enp3s0 proto kernel scope link src 192.168. 235.12local 192.168.235.12 Dev enp3s0 proto kernel scope host src 192.168.235.12broadcast 192.168.235.255 dev enp3s0 p  Roto kernel scope link src 192.168.235.12[email protected]:~$ IP route list table Maindefault via 192.168.235.2 Dev Enp3s0 onlink10.0.1.0/24 Dev enp5s0 proto kernel scope link src 10.0.1.1210.42.0.0/24 via 10.42.0.0 Dev flannel.1 Onli nk10.42.1.0/24 via 10.42.1.0 Dev flannel.1 onlink10.42.2.0/24 dev cni0 proto kernel scope link src 10.42.2.110.42.3.0/2   4 via 10.42.3.0 Dev flannel.1 onlink10.42.4.0/24 via 10.42.4.0 dev flannel.1 onlink172.17.0.0/16 dev Docker0 Proto kernel Scope link src 172.17.0.1 linkdown192.168.235.0/24 Dev enp3s0 proto kernel scope link src 192.168.235.12[email protected]:~$ IP route l IST table default[email protected]:~$

In front of the general method is to look at the route to see whether the routing entry to a particular package has a regular correspondence, but this method requires you to be familiar with the routing rules, and manual easy to determine the leak. So here's a command to test the routing rulesip route get

###### 偷懒摘抄下man 8 ip 里的说明ip route get - get a single routethis command gets a single route to a destination and prints its contents exactly as the kernel sees it.to ADDRESS (default) #the destination address.from ADDRESS #the source address.tos TOSdsfield TOS # TOS=the Type Of Service.iif NAME #the device from which this packet is expected to arrive.oif NAME #force the output device on which this packet will be routed.

Take this log of the Iptables track section above as an example

May  8 15:51:07 ceph2 kernel: [309854.514930] TRACE: nat:PREROUTING:policy:3 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307)

Suppose we want to determine the routing of this package will match which route, we can look at the following command, the first day to see the results will be the initial response is not my command grammar has a problem error? Let's see what I did when I referenced the rp_filter_kernel_setting.

[email protected]:~$ ip route get from 192.168.235.13 to 10.0.1.12 iif enp5s0 tos 0x00RTNETLINK answers: Invalid cross-device link######### change rp_filter kernel setting[email protected]:~$ sudo bash[email protected]:~# echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter[email protected]:~# echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter######### Ok 在更改rp_filter 内核参数后，我们的同样的命令有匹配结果了。[email protected]:~# ip route get from 192.168.235.13 to 10.0.1.12 iif enp5s0 tos 0x00local 10.0.1.12 from 192.168.235.13 dev lo    cache <local>  iif enp5s0[email protected]:~#

Here I think you can try some other IP route get command again and again to see the output, such as

[email protected]:~$ ip route get from 172.18.0.3 to 10.0.1.12RTNETLINK answers: Invalid argument # 本机根本没法从172.18.0.3 路由到10.0.1.12[email protected]:~$ ip route get from 192.168.235.3 to 10.0.1.12RTNETLINK answers: Invalid argument # 本机也没法从192.168.235.3 路由到10.0.1.12[email protected]:~$ ip route get from 192.168.235.12 to 10.0.1.12 #从本地的一个卡为192.168.235.12 可以从lo 上路由到10.0.1.12local 10.0.1.12 from 192.168.235.12 dev lo    cache <local>[email protected]:~$ ip route get from 192.168.235.12 to 10.0.1.13 #从本地的一个卡为192.168.235.12 可以从enp5s0 上路由到10.0.1.13(另外一个主机）10.0.1.13 from 192.168.235.12 dev enp5s0    cache

Finally, we will explain in detail the display meaning of the routing rules, the specific reference is "Iproute2 Doc"

Take this longer broadcast 10.0.1.0 dev enp5s0 proto kernel scope link src 10.0.1.12 as an example.

• broadcast 10.0.1.0The first is the route type, can be broadcast,unicast,local, and so on, and if not, the unicast,10.0.1.0 is the destination network.
• dev enp5s0That means when you go out. Nic Enp5s0
• via 10.42.3.0You may see this in some rules, representing the next hop gateway is 10.42.3.0
• proto kernelThe routing protocol is kernel, generated by kernel.
• scope linkThis address is valid only on the link
• src 10.0.1.12The source IP is 10.0.1.12, where the 10.0.1.12 must be found on the local NIC address
• onlinkPretend that the next hop gateway is on this link.

Some network-related kernel parameter settings

OK, fast to the end have to mention the Linux kernel parameter settings, these parameters can be set in the kernel, is often refined and refined the essence of the part. So that's the problem?

• Q: How do I know which parameters I need?

  These parameters are described in detail in A:linux's kernel documentation, so we can read the kernel documents, such as IP-related parameters, to find the parameters we might need, and my idea is to search the Internet by the name of the kernel parameter that I think is possible. Then look at the results of other people use this parameter to solve what the problem specifically.

• Q: Where do I find the Linux kernel documentation?

  A: In Ubuntu, for example, Linux-doc is the current kernel document package, and after the installation of the file in the /usr/share/doc/linux-doc/ home directory, you can look dpkg -L linux-doc for the required documents. For example zcat /usr/share/doc/linux-doc/networking/ip-sysctl.txt.gz , you can read the network-related kernel parameters of the document.

A few important kernel parameters

• When the Rp_filter is set to 2, the package's SRC is matched for all network cards, and if the matching route is set to 1 o'clock, the packet is discarded if the packet-passing NIC discovers that the return path is not optimal.

  Net.ipv4.conf.default.rp_filter = 2

  net.ipv4.conf.all.rp_filter=2

• When the Log_martians Boolean is set to enable, it is possible to log information in the kernel log when it is considered an impossible address by the kernel.

Other supplements

• When cross-host, the grab kit is replenished.
• Knowledge needs to spread, dig deep, welcome to add.

Network Packet Troubleshooting Guide-class Linux Platform