Openvswitch GRE implementation of Kubernetes network model

1. Preface

Kubernetes designed a special network model that deviates from the native Docker network model. In this design, Kubernetes defines an abstract concept: pods, each pod is a collection of containers, and there is a shared IP, and all containers share the same network namespace. Pods can communicate not only with physical machines, but also between containers across the network. Kubernetes's ip-per-pod design idea has many benefits, such as: From port assignment, network, naming, service discovery, load balancing, application configuration, and migration, this model allows developers, operators to treat pods as a virtual machine or as a physical machine. Have a good back-to-compatibility capability. Google Now implements this ip-per-pod model on its cloud platform GCE, but if you use kubernetes locally, you have to implement this model yourself, and this article focuses on using Openvswitch GRE to implement this model.

2. Implementation scenarios

This article uses 2 units to run CentOS 7, the implementation can be described as: 650) this.width=650; "src=" Http://i.imgur.com/w7djVFg.png "alt=" W7djvfg.png "/>

2.1 Installing Docker

The installation steps refer to the following script:

 #Docker默认配置文件DOCKER_CONFIG =/etc/sysconfig/docker# Download the latest Docker installation files wget https://get.docker.com/builds/ linux/x86_64/docker-latest -o /usr/bin/dockerchmod +x /usr/bin/docker# Configuration docker  Unit file Cat <<eof >/usr/lib/systemd/system/docker.socket[unit]description=docker socket  for the api[socket]listenstream=/var/run/docker.socksocketmode=0660socketuser=rootsocketgroup= docker[install]wantedby=sockets.targeteofsource  $DOCKER _configcat <<eof >/usr/lib/ Systemd/system/docker.service[unit]description=docker application container enginedocumentation =http://docs.docker.comafter=network.target docker.socketrequires=docker.socket[service]type= notifyenvironmentfile=-$DOCKER _configexecstart=/usr/bin/docker -d  $OPTIONSLimitNOFILE = 1048576limitnproc=1048576[install]also=docker.socketeofsystemctl daemon-reloadsystemctl enable  Dockersystemctl start docker 

2.2 Installing Openvswitch

Download Openvswitch and install:

wget http://cbs.centos.org/kojifiles/packages/openvswitch/2.3.1/2.el7/x86_64/openvswitch-2.3.1-2.el7.x86_64.rpm RPM-IVH openvswitch-2.3.1-2.el7.x86_64.rpmsystemctl start Openvswitchsystemctl enable Openvswitch

3.3 Setting up the Docker default bridge

Here, we follow the steps below to create a new Linux bridge localbr0 and replace the default bridge Docker0,

#停止Docker Daemon Process Systemctl stop Docker.socketsystemctl stop docker# set default bridge Docker0 to down and remove IP link set Dev Docker0 Downbrctl delbr docker0# new Linux bridge localbr0brctl ADDBR localbr0# change 10.244.x.0/24 on each host, set LOCALBR0 address as below: # 10.224.106.127ip addr Add 10.244.1.1/24 dev localbr0#10.224.106.128ip addr add 10.244.2.1/24 dev localbr0ip link set dev l Ocalbr0 Upecho ' options= "--bridge localbr0--iptables=false" ' >>/etc/sysconfig/dockersystemctl start Docker

3.4 Setting up Openvswitch Bridge and GRE

#新建Openvswitch网桥ovs-vsctl add-br ovsbr# enable SPT protocol to prevent Bridge Loop ovs-vsctl set bridge ovsbr  stp_enable=true# add ovsbr to local localbr0, allowing container traffic to flow through OvS Tunnelbrctl addif localbr0 ovsbrip link  set dev ovsbr up# Creating greovs-vsctl add-port ovsbr tep0 -- set  interface tep0 type=internal# the TEP0 IP address needs to be modified on each host ip addr add 192.168.1.1/24  dev tep0ip addr add 192.168.1.2/24 dev tep0ip link set dev  tep0 up# Use the GRE tunnel to connect the Openvswitch bridge on each host #10.224.106.127ovs-vsctl add-port ovsbr gre0 --  set interface gre0 type=gre options:remote_ip=10.224.106.128#10.224.106.128ovs-vsctl  add-port ovsbr gre0 -- set interface gre0 type=gre options: remote_ip=10.224.106.127# Configure routing to enable communication between containers across hosts ip route add 10.244.0.0/16 dev tep0# To enable the container to access the Internet, configure the Natipta on both hostsbles -t nat -a postrouting -s 10.244.0.0/16 -o ens192 -j  Masquerade

3.5 Verification

Complete the above operation, the following should be able to work properly:

2. Can ping Tep0 's address to each other

   [Email protected] ~]# ping 192.168.1.2PING 192.168.1.2 (192.168.1.2) (+) bytes of data.64 bytes from 192.168.1.2:icmp  _seq=1 ttl=64 time=0.929 ms64 bytes from 192.168.1.2:icmp_seq=2 ttl=64 time=0.642 ms64 bytes from 192.168.1.2:icmp_seq=3 ttl=64 time=0.322 ms64 bytes from 192.168.1.2:icmp_seq=4 ttl=64 time=0.366 ms^c---192.168.1.2 ping statistics---4 pack ETS transmitted, 4 received, 0% packet loss, time 3000msrtt Min/avg/max/mdev = 0.322/0.564/0.929/0.245 ms

4. Can ping Localbr0 's address to each other

   [Email protected] ~]# ping 10.244.2.1PING 10.244.2.1 (10.244.2.1) (+) bytes of data.64 bytes from 10.244.2.1:icmp_seq =1 ttl=64 time=0.927 ms64 bytes from 10.244.2.1:icmp_seq=2 ttl=64 time=0.337 ms64 bytes from 10.244.2.1:icmp_seq=3 ttl=6 4 time=0.409 ms^c---10.244.2.1 ping statistics---3 packets transmitted, 3 received, 0% packet loss, time 2000msrtt min/a Vg/max/mdev = 0.337/0.557/0.927/0.264 ms

5. The containers on both hosts can ping each other

   Run a new container on the host 10.224.106.127 and 10.224.106.128 by executing the following command:

   Docker Run-ti Ubuntu/bin/bash

   Then the 10.224.106.127 container pings the 10.224.106.128 container,

   [Email protected]:/# ping 10.244.2.3PING 10.244.2.3 (10.244.2.3) (+) bytes of data.64 bytes from 10.244.2.3:icmp_seq=1 ttl=63 time=0.781 ms64 bytes from 10.244.2.3:icmp_seq=2 ttl=63 time=0.404 ms^c---10.244.2.3 ping statistics---2 packet s transmitted, 2 received, 0% packet loss, time 1000msrtt Min/avg/max/mdev = 0.404/0.592/0.781/0.190 ms

   Ping the 10.224.106.127 container from the 10.224.106.128 container.

   [Email protected]:/# ping 10.244.1.3PING 10.244.1.3 (10.244.1.3) (+) bytes of data.64 bytes from 10.244.1.3:icmp_seq=1  ttl=63 time=1.70 ms64 bytes from 10.244.1.3:icmp_seq=2 ttl=63 time=0.400 ms^c---10.244.1.3 ping statistics---2 packets Transmitted, 2 received, 0% packet loss, time 1001msrtt Min/avg/max/mdev = 0.400/1.054/1.708/0.654 ms

3. Conclusion

This paper implements the Kubernetes network model by Openvswitch GRE, but this approach is a bit clumsy in large-scale systems. For example, if the system has n hosts and they need to communicate between them, an n (n-1)/2 GRE tunnel is required, although it is possible to prevent the bridge loop by enabling the SPT protocol, but maintaining N (n-1)/2 tunnels is still a lot of work. So next consider how you can automate the implementation.

4. References

1. Https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/design/networking.md

2. Https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/ovs-networking.md

3. https://docs.docker.com/installation/centos/

4. https://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/

5. Introduction of the author

Yang Zhangxin, currently working for Cisco, is mainly engaged in WebEx SaaS service operations, system performance analysis and so on. Special attention is paid to technologies such as cloud computing, automated operations and deployment, especially go, Openvswitch, Docker and its ecosystem technologies, such as Docker-related open source projects such as Kubernetes and Flocker. Email: [Email protected]

This article is from the "brain Original not Reliable" blog, please be sure to keep this source http://aresy.blog.51cto.com/5100031/1600956

Openvswitch GRE implementation of Kubernetes network model