The problem of network error in Kubernetes

The problem of network error in Kubernetes

2. System environment

   #系统版本cat /etc/redhat-releaseCentOS Linux release 7.4.1708 (Core)

   #kubelet版本kubelet --versionKubernetes v1.10.0

   #selinux状态getenforceDisabled

   #系统防火墙状态systemctl status firewalld● firewalld.service - firewalld - dynamic firewall daemonLoaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)Active: inactive (dead) Docs: man:firewalld(1)

4. Pod anomaly Problem

   #dns的Pod 一直处于 Waiting 或 ContainerCreating 状态kubectl get po -n kube-systemNAME                                    READY     STATUS             RESTARTS   AGEkube-dns-86f4d74b45-ffwjf        0/3       ContainerCreating   0          6m

   #查看Pod详细情况kubectl  describe pod kube-dns-86f4d74b45-ffwjf  -n kube-system##我们看到如下信息：Error syncing podPod sandbox changed, it will be killed and re-created.##可以发现，该 Pod 的 Sandbox 容器无法正常启动，具体原因需要查看 Kubelet 日志。

   #查看Pod的logjournalctl -u kubelet##看到如下报错内容：RunPodSandbox from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod "kube-dns-86f4d74b45-ffwjf" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.4.1/24

   # #说明
   There are several containers in one pod here, so we use the KUBECTL Logs command to view the log is very limited, regarding the use of Kubectl logs, refer to the pod Introduction and practice in Kubernetes and kubernetes Chinese documents.

5. Process steps

   #在master节点之外的节点进行操作kubeadm resetsystemctl stop kubeletsystemctl stop dockerrm -rf /var/lib/cni/rm -rf /var/lib/kubelet/*rm -rf /etc/cni/ifconfig cni0 downifconfig flannel.1 downifconfig docker0 downip link delete cni0ip link delete flannel.1##重启kubeletsystemctl restart kubelet##重启dockersystemctl restart docker

   #说明##如果上面操作之后还是报相同的错误或是如下错误："CreatePodSandbox for pod \" kube-dns-86f4d74b45-ffwjf _default(78e796f5-eb7c-11e7-b903-b827ebd42d30)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \" kube-dns-86f4d74b45-ffwjf _default\"network: failed to allocate for range 0: no IP addresses available in range set:10.244.1.1-10.244.1.254"

   #执行如下操作步骤：##在master主机上kubeadm resetsystemctl stop kubeletsystemctl stop dockerrm -rf /var/lib/cni/rm -rf /var/lib/kubelet/*rm -rf /etc/cni/ifconfig cni0 downifconfig flannel.1 downifconfig docker0 downip link delete cni0ip link delete flannel.1##重启kubeletsystemctl restart kubelet##重启dockersystemctl restart docker

   ##初始化kubeadm init --kubernetes-version=v1.10.1 --pod-network-cidr=10.244.0.0/16--apiserver-advertise-address=10.0.0.39##说明：最后给出了将节点加入集群的命令：kubeadm join 10.0.0.39:6443 --token 4g0p8w.w5p29ukwvitim2ti --discovery-token-ca-cert-hash sha256:21d0adbfcb409dca97e655641573b2ee51c77a212f194e20a307cb459e5f77c8这条命令一定保存好，因为后期没法重现的！！

   ##建立.kuberm -rf /root/.kube/mkdir -p /root/.kube/cp -i /etc/kubernetes/admin.conf /root/.kube/configchown root:root /root/.kube/config

   #在node（非master）节点上kubeadm resetsystemctl stop kubeletsystemctl stop dockerrm -rf /var/lib/cni/rm -rf /var/lib/kubelet/*rm -rf /etc/cni/ifconfig cni0 downifconfig flannel.1 downifconfig docker0 downip link delete cni0ip link delete flannel.1##重启kubeletsystemctl restart kubelet##重启dockersystemctl restart docker

   ## kubeadm joinkubeadm join 10.0.0.39:6443 --token 4g0p8w.w5p29ukwvitim2ti --discovery-token-ca-cert-hash sha256:21d0adbfcb409dca97e655641573b2ee51c77a212f194e20a307cb459e5f77c8

7. Summarize
   In addition to the above errors, other possible causes are:
   Mirror pull failed, for example:
   (1) The wrong image is configured
   (2) Kubelet cannot access the mirror (domestic environment access Gcr.io requires special handling
   (3) secret key configuration error for private image
   (4) Mirror too large, pull timeout (can adjust the Kubelet--image-pull-progress-deadline and--runtime-request-timeout options appropriately)
   The distribution network error, generally need to check the configuration of the network plug-ins, such as:
   (1) Unable to configure Pod network
   (2) Unable to assign IP address
   The container does not start, and you need to check whether the correct image is packaged or configured with the correct container parameters.
9. Reference articles
   https://github.com/kubernetes/kubernetes/issues/57280

Problem with network error in kubernetes