Use Kubernetes to manage containers on centos 7
1. Preface
The previous section describes the Kubernetes system architecture, which gives you a preliminary understanding of Kubernetes. However, you may not know how to use Kubernetes. This article describes how to deploy and configure the network environment of the Kubernetes cluster locally and demonstrate cross-machine service communication through instances. The main content of this article is as follows:
- Deployment Environment
- Logic architecture of Kubernetes Clusters
- Deploy Open vSwitch, Kubernetes, and Etcd Components
- Demonstrate Kubernetes Container Management
2. Deployment Environment
3. logic architecture of Kubernetes Clusters
Before you deploy a Kubernetes cluster in detail, we will first show you the logical architecture of the cluster. It can be seen that the entire system is divided into two parts: the first part is Kubernetes APIServer, which is the core of the entire system and manages all containers in the cluster; the second part is minion, which runs Container Daemon, it is the place where all containers reside. At the same time, the Open vSwitch program is run on minion. Through GRE Tunnel, it is responsible for the network communication between pods of minion.
4. Deploy Open vSwitch, Kubernetes, and Etcd component 4.1 to install Open vSwitch and configure GRE
To solve the problem of Pod communication across minion, we install Open vSwtich on each minion and use GRE or VxLAN to enable inter-machine Pod communication. This article uses GRE, vxLAN is usually used in large-scale networks that require isolation. For detailed installation steps of Open vSwitch, refer to this blog. We will not detail the installation steps here. After the Open vSwitch is installed, a tunnel between minion1 and minion2. First, create an OVS Bridge on minion1 and minion2,
[root@minion1 ~]# ovs-vsctl add-br obr0
Next, create gre, add the new gre0 to obr0, and execute the following command on minion1,
[Root @ minion1 ~] # Ovs-vsctl add-port obr0 gre0 -- set Interface gre0 type = gre options: remote_ip = 192.168.230.5
Run on minion2,
[root@minion2 ~]# ovs-vsctl add-port obr0 gre0 -- set Interface gre0 type=gre options:remote_ip=192.168.230.4
At this point, the tunnel between minion1 and minion2 has been established. Then we create the Linux bridge kbr0 on minion1 and minion2 to replace Docker's default docker0 (we assume that Docker is installed on both minion1 and minion2), and set the kbr0 address of minion1 to 172.17.1.1/24, the kbr0 address of minion2 is 172.17.2.1/24, and the obr0 interface is added as kbr0. The following command is executed on minion1 and minion2.
[Root @ minion1 ~] # Brctl addbr kbr0 // create a linux bridge [root @ minion1 ~] # Brctl addif kbr0 obr0 // Add an interface whose obr0 is kbr0 [root @ minion1 ~] # Ip link set dev docker0 down // set docker0 to down [root @ minion1 ~] # Ip link del dev docker0 // Delete docker0
To make the newly created kbr0 valid after each system restart, we create a ifcfg-kbr0 for minion1 in the/etc/sysconfig/network-scripts/directory as follows:
DEVICE=kbr0ONBOOT=yesBOOTPROTO=staticIPADDR=172.17.1.1NETMASK=255.255.255.0GATEWAY=172.17.1.0USERCTL=noTYPE=BridgeIPV6INIT=no
Also create a ifcfg-kbr0 on minion2, you just need to modify ipaddr to 172.17.2.1 and gateway to 172.17.2.0, then execute systemctl restart network to restart the system network service, on minion1 and minion2, you can find that kbr0 has a corresponding IP address. In order to verify whether the tunnel we created can communicate with each other, we ping the IP address of kbr0 of the other Party on minion1 and minion2. The following results show that the communication fails, this is because the routes to access 172.17.1.1 and 172.17.2.1 are missing on minion1 and minion2. Therefore, we need to add routes to ensure communication between them.
[root@minion1 network-scripts]# ping 172.17.2.1PING 172.17.2.1 (172.17.2.1) 56(84) bytes of data.^C--- 172.17.2.1 ping statistics ---2 packets transmitted, 0 received, 100% packet loss, time 1000ms[root@minion2 ~]# ping 172.17.1.1PING 172.17.1.1 (172.17.1.1) 56(84) bytes of data.^C--- 172.17.1.1 ping statistics ---2 packets transmitted, 0 received, 100% packet loss, time 1000ms
Because the route added through ip route add will expire after the next system restart, So we create a new file route-eth0 storage route under the/etc/sysconfig/network-scripts directory, it should be noted that the route-eth0 and the ifcfg-eth0 of the simhei part must be consistent, otherwise it cannot work, so that the added route will not expire after the next restart. To ensure that the two minion kbr0 can communicate with each other, we add route 172.17.2.0/24 via 192.168.230.5 dev eno16777736 in the route-eth0 of minion1, eno16777736 is the NIC of minion1, also add route 172.17.1.0/24 via 192.168.230.4 dev eno16777736 in the route-eth0 of minion2. Verify again after restarting the network service. ping each other's kbr0 addresses, for example:
[root@minion2 network-scripts]# ping 172.17.1.1PING 172.17.1.1 (172.17.1.1) 56(84) bytes of data.64 bytes from 172.17.1.1: icmp_seq=1 ttl=64 time=2.49 ms64 bytes from 172.17.1.1: icmp_seq=2 ttl=64 time=0.512 ms^C--- 172.17.1.1 ping statistics ---2 packets transmitted, 2 received, 0% packet loss, time 1002msrtt min/avg/max/mdev = 0.512/1.505/2.498/0.993 ms
Now we have established a tunnel between two minion and can work correctly. The following describes how to install Kubernetes APIServer, kubelet, proxy, and other services.
4.2 install Kubernetes APIServer
Before installing APIServer, we first download Kubernetes and Etcd for some preparation. The specific operations on kubernetes are as follows:
[root@kubernetes ~]# mkdir /tmp/kubernetes[root@kubernetes ~]# cd /tmp/kubernetes/[root@kubernetes kubernetes]# wget https://github.com/GoogleCloudPlatform/kubernetes/releases/download/v0.5.2/kubernetes.tar.gz[root@kubernetes kubernetes]# wget https://github.com/coreos/etcd/releases/download/v0.4.6/etcd-v0.4.6-linux-amd64.tar.gz
Decompress the downloaded kubernetes and etcd packages, and create the/opt/kubernetes/bin directory on kubernetes, minion1, and minion2,
[root@kubernetes kubernetes]# mkdir -p /opt/kubernetes/bin[root@kubernetes kubernetes]# tar xf kubernetes.tar.gz[root@kubernetes kubernetes]# tar xf etcd-v0.4.6-linux-amd64.tar.gz[root@kubernetes kubernetes]# cd ~/kubernetes/server[root@kubernetes server]# tar xf kubernetes-server-linux-amd64.tar.gz[root@kubernetes kubernetes]# /tmp/kubernetes/kubernetes/server/kubernetes/server/bin
Copy kube-apiserver, kube-controller-manager, kube-schedfg, and kubecfg to the/opt/kubernetes/bin directory of kubernetes, while kubelet, kube-proxy is copied to/opt/kubernetes/bin of minion1 and minion2, And it is executable.
[root@kubernetes amd64]# cp kube-apiserver kube-controller-manager kubecfg kube-scheduler /opt/kubernetes/bin[root@kubernetes amd64]# scp kube-proxy kubelet root@192.168.230.4:/opt/kubernetes/bin[root@kubernetes amd64]# scp kube-proxy kubelet root@192.168.230.5:/opt/kubernetes/bin
To deploy only one etcd server, if you need to deploy the etcd cluster, refer to the official documentation and deploy it on the same machine as the Kubernetes APIServer in this article, in addition, place etcd in/opt/kubernetes/bin. etcdctl is in the same directory as ectd.
[root@kubernetes kubernetes]# cd /tmp/kubernetes/etcd-v0.4.6-linux-amd64[root@kubernetes etcd-v0.4.6-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin
Note that files in the/opt/kubernetes/bin directory of kubernetes and minion must be executable. At present, we have prepared almost the same work. Now we start to configure the unit file for apiserver, controller-manager, scheduler, and etcd. First, we use the following script etcd. sh to configure the etcd unit file,
#!/bin/shETCD_PEER_ADDR=192.168.230.3:7001ETCD_ADDR=192.168.230.3:4001ETCD_DATA_DIR=/var/lib/etcdETCD_NAME=kubernetes! test -d $ETCD_DATA_DIR && mkdir -p $ETCD_DATA_DIRcat <<EOF >/usr/lib/systemd/system/etcd.service[Unit]Description=Etcd Server[Service]ExecStart=/opt/kubernetes/bin/etcd \\ -peer-addr=$ETCD_PEER_ADDR \\ -addr=$ETCD_ADDR \\ -data-dir=$ETCD_DATA_DIR \\ -name=$ETCD_NAME \\ -bind-addr=0.0.0.0[Install]WantedBy=multi-user.targetEOFsystemctl daemon-reloadsystemctl enable etcdsystemctl start etcd
The scripts configured for the unit files of the remaining apiserver, controller-manager, and scheduler can be found on GetStartingKubernetes on github, and will not be listed here. After running the script, the etcd, APIServer, controller-manager, and scheduler services on the apiserver can run normally.
4.3 install Kubernetes Kubelet and Proxy
According to the Kubernetes design architecture, docker, kubelet, and kube-proxy must be deployed on minion. When APIServer is deployed in section 4.2, we have already distributed kubelet and kube-proxy to two minion, so we only need to configure the unit file of docker, kubelet, and proxy, and then start the service. For detailed configuration, see GetStartingKubernetes.
5. Demonstration of Kubernetes Container Management
For convenience, we use the example Guestbook provided by Kubernetes to demonstrate how Kubernetes manages containers running across machines. Next we create containers and services according to the steps of Guestbook. In the following process, if this is the first operation, there may be a certain wait time, And the status is pending, because it takes some time to download images for the first time.
5.1 create redis-master Pod and redis-master service
[root@kubernetes ~]# cd /tmp/kubernetes/kubernetes/examples/guestbook[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 -c redis-master.json create pods[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 -c redis-master-service.json create services
After completing the above operations, we can see that the following redis-master Pod is scheduled to 192.168.230.4.
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 list podsName Image(s) Host Labels Status---------- ---------- ---------- ---------- ----------redis-master dockerfile/redis 192.168.230.4/ name=redis-master Running
However, in addition to the redis-master service, there are two default Kubernetes-ro and kubernetes services in the kubernetes system. In addition, we can see that each service has a service IP address and corresponding port. The service IP address is a virtual address, which is selected based on the cidr ip address segment set by the portal_net option of the apiserver, set it to 10.10.10.0/24 in our cluster. For this reason, each time a new service is created, the apiserver randomly selects an IP address in this address segment as the IP address of the service, and the port is determined in advance. For the redis-master service, the service address is 10.10.206 and the port is 6379.
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 list servicesName Labels Selector IP Port---------- ---------- ---------- ---------- ----------kubernetes-ro component=apiserver,provider=kubernetes 10.10.10.207 80redis-master name=redis-master name=redis-master 10.10.10.206 6379kubernetes component=apiserver,provider=kubernetes 10.10.10.161 443
5.2 create redis-slave Pod and redis-slave services
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 -c redis-slave-controller.json create replicationControllers[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 -c redis-slave-service.json create services
THE list command shows that the newly created redis-slave Pod is scheduled to two minion Servers Based on the scheduling algorithm. The service IP address is 10.10.10.92 and the port is 6379.
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 list podsName Image(s) Host Labels Status---------- ---------- ---------- ---------- ----------redis-master dockerfile/redis 192.168.230.4/ name=redis-master Running8c0ddbda-728c-11e4-8233-000c297db206 brendanburns/redis-slave 192.168.230.5/ name=redisslave,uses=redis-master Running8c0e1430-728c-11e4-8233-000c297db206 brendanburns/redis-slave 192.168.230.4/ name=redisslave,uses=redis-master Running[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 list servicesName Labels Selector IP Port---------- ---------- ---------- ---------- ----------redisslave name=redisslave name=redisslave 10.10.10.92 6379kubernetes component=apiserver,provider=kubernetes 10.10.10.161 443kubernetes-ro component=apiserver,provider=kubernetes 10.10.10.207 80redis-master name=redis-master name=redis-master 10.10.10.206 6379
5.3 create Frontend Pod and Frontend services
Modify the number of Replicas for the frontend-controller.json to 2 before creating, because our cluster only has 2 minion, if the default value of Replicas for the frontend-controller.json is 3, this will cause two pods to be scheduled to the same minion, resulting in port conflict. One Pod will remain in the pending state and cannot be scheduled.
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 -c frontend-controller.json create replicationControllers[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 -c frontend-service.json create services
The Frontend Pod is also scheduled to two minion servers. The service IP address is 10.10.10.220 and the port is 80.
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 list podsName Image(s) Host Labels Status---------- ---------- ---------- ---------- ----------redis-master dockerfile/redis 192.168.230.4/ name=redis-master Running8c0ddbda-728c-11e4-8233-000c297db206 brendanburns/redis-slave 192.168.230.5/ name=redisslave,uses=redis-master Running8c0e1430-728c-11e4-8233-000c297db206 brendanburns/redis-slave 192.168.230.4/ name=redisslave,uses=redis-master Runninga880b119-7295-11e4-8233-000c297db206 brendanburns/php-redis 192.168.230.4/ name=frontend,uses=redisslave,redis-master Runninga881674d-7295-11e4-8233-000c297db206 brendanburns/php-redis 192.168.230.5/ name=frontend,uses=redisslave,redis-master Running[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 list servicesName Labels Selector IP Port---------- ---------- ---------- ---------- ----------kubernetes-ro component=apiserver,provider=kubernetes 10.10.10.207 80redis-master name=redis-master name=redis-master 10.10.10.206 6379redisslave name=redisslave name=redisslave 10.10.10.92 6379frontend name=frontend name=frontend 10.10.10.220 80kubernetes component=apiserver,provider=kubernetes 10.10.10.161 443
In addition, you can delete the Pod, Service, and Replicas of the ReplicationController, such as deleting the Frontend Service:
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 delete services/frontendStatus----------Success
You can also update the number of Replicas of ReplicationController. The following is the information of ReplicationController before Replicas.
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 list replicationControllersName Image(s) Selector Replicas---------- ---------- ---------- ----------redisSlaveController brendanburns/redis-slave name=redisslave 2frontendController brendanburns/php-redis name=frontend 2
Now we want to update the frontendController's Replicas to 1, and then use the following command to view the frontendController information. It is found that Replicas has changed to 1.
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 resize frontendController 1[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 list replicationControllersName Image(s) Selector Replicas---------- ---------- ---------- ----------redisSlaveController brendanburns/redis-slave name=redisslave 2frontendController brendanburns/php-redis name=frontend 1
5.4 demonstrate cross-machine service communication
After completing the preceding operations, we can see the Pod information running in the current Kubernetes cluster.
[root@kubernetes guestbook]# kubecfg -h http://192.168.230.3:8080 list podsName Image(s) Host Labels Status---------- ---------- ---------- ---------- ----------a881674d-7295-11e4-8233-000c297db206 brendanburns/php-redis 192.168.230.5/ name=frontend,uses=redisslave,redis-master Runningredis-master dockerfile/redis 192.168.230.4/ name=redis-master Running8c0ddbda-728c-11e4-8233-000c297db206 brendanburns/redis-slave 192.168.230.5/ name=redisslave,uses=redis-master Running8c0e1430-728c-11e4-8233-000c297db206 brendanburns/redis-slave 192.168.230.4/ name=redisslave,uses=redis-master Running
Through the above results, we can see that the pods of the current PHP and Redis master, which provide data storage services, run on 192.168.230.5 and 192.168.230.4 respectively, that is, containers run on different hosts, redis slave also runs on two different hosts, which will write data to the Redis master from the Redis master synchronization front-end. Next we will verify from two aspects that Kubernetes can provide cross-machine container communication:
Open http: // $ {IP in the browserAddress}: 8000, IPAddress is the IP Address of the minion running on the PHP container, and the exposed port is 8000. Here IP_Address is 192.168.230.5. The following information is displayed when you open your browser:
You can enter and Submit information, such as "Hello Kubernetes" and "Container", and the information you entered will be displayed below the Submit button.
Because the frontend PHP container and the backend Redis master container are on two minion instances, PHP must communicate with each other when accessing the Redis master service, it can be seen that the implementation of Kubernetes avoids the defect that link can only implement inter-container communication on the same host. The implementation of Kubernetes cross-machine communication will be described in detail later.
From the above results, we can see that cross-machine communication has been implemented. Now we verify the communication between containers on different machines from the back-end data layer. According to the above output, we found that Redis slave and Redis master are scheduled to two different minion instances, and run docker exec-ti c000011cc8971/bin/sh on host 192.168.230.4, c201711cc8971 is the container ID of the Redis master. after entering the container, run the redis-cli command to view the information entered from the browser as follows:
If we find the same information in the Redis slave container running on 192.168.230.5 as in the Redis master container, the data synchronization between the Redis master and Redis slave works normally, the following information is queried from the Redis slave container running on 192.168.230.5:
It can be seen that the data synchronization between the Redis master and Redis slave is normal, and the ovs gre tunnel technology enables normal communication between containers across machines.
6. Conclusion
This document describes how to deploy a Kubernetes cluster in a local environment and how to use Kubernetes to manage containers running in a cluster, and how to use OVS to manage network communication between pods of different minion clusters. Next, we will analyze the source code of each Kubernetes component and describe how Kubernetes works.
7. Personal Profile
Yang zhangxian, now working in Cisco, is mainly engaged in WebEx SaaS service O & M and system performance analysis. Pay special attention to cloud computing, automated O & M, deployment, and other technologies, especially Go, OpenvSwitch, Docker and its ecosystem technologies, such as Kubernetes, Flocker, and other open-source Docker projects. Email: yangzhangxian@gmail.com
8. References
- Https://n40lab.wordpress.com/2014/09/04/openvswitch-2-3-0-lts-and-centos-7/
- Https://github.com/GoogleCloudPlatform/kubernetes/tree/master/examples/guestbook
OpenStack, Kubernetes, and Mesos
Problems encountered during Kubernetes cluster construction and Solutions
For details about Kubernetes, click here
Kubernetes: click here
This article permanently updates the link address: