k8s1.9.2 based on Kubeadm high-availability installation tutorials, including offline installation packages, support for simple and quick installation

This is a creation in Article, where the information may have evolved or changed.

Installing a secure high-availability kubernetes cluster using Kubeadm

Install package address such as non-highly available installation Please ignore this tutorial and look directly at the three-step installation of the product page.

Single master process:

1. Unzip the CD shell && sh init.sh on master and then SH master.sh (note because the script uses a relative path so no more files will be found in the current directory)
2. On node CD shell && sh init.sh. Then execute the join command for the master output on node

Prepare in advance

Assuming that a 3master+2node k8s cluster is built, 5 nodes are required to share the following conditions:

1. ( yum install -y docker是1.12.6版本需要改cg )
   17.06 Installation Tutorials:

   #0.删除老旧的$ yum remove -y docker*  #如果默认之前yum安装的1.12版本,可以这样删没装可以跳过此步#1.安装需要的包$ yum install -y yum-utils \  device-mapper-persistent-data \  lvm2  #2.添加源,不然默认的找不到$ yum-config-manager \    --add-repo \    https://download.docker.com/linux/centos/docker-ce.repo    #3.根据实际查找当前版本 (可选)$ yum list docker-ce --showduplicates | sort -r#4.如果确定了版本,直接安装,如果要装17。03直接修改下面数字即可$ yum install  docker-ce-17.06.1.ce  # 主意版本填写包名的格式.#5.开启docker服务,和开机启动$ systemctl start docker && systemctl enable docker

2. It is suggested that the binary method should be deployed well in advance docker-compose, step reference
3. It is recommended to permanently turn off selinux and swap to avoid subsequent problems
4. Recommend stopping and shutting down firewalls such as Firewalld/iptables
5. Remember to change the network name when the new node startshostnamectl set-hostname masterX
6. Between nodes to be able to communicate with the intranet environment stable
7. There is a problem in the installation to see log journalctl-n 10, running log viewtail -f 10 /var/log/messages

System Architecture Diagram

          kubectl dashboard                 |                 V      +------------------------+ join     | LB  10.1.245.94        | <--- Nodes     +------------------------+     |                                                        |--master1 manager1 schedule1   10.1.245.93                                                     |--master2 manager2 schedule2   10.1.245.95    =============>  etcd cluster  http://10.1.245.93:2379,http://10.1.245.94:2379,http://10.1.245.95:2379     |--master3 manager3 schedule3   10.1.245.94                                              

Installation package Introduction

After the decompression, see the following directory:

├── bin          所需要的k8s相关的bin文件│   ├── kubeadm│   ├── kubectl│   └── kubelet├── image        依赖的所有镜像包│   └── images.tar├── out          所有的配置文件│   ├── dashboard  dashboard相关配置│   │   ├── dashboard-admin.yaml│   │   └── kubernetes-dashboard.yaml│   ├── etcd  etcd相关配置│   │   ├── etcd-docker-compose-0.yml│   │   ├── etcd-docker-compose-1.yml│   │   └── etcd-docker-compose-2.yml│   ├── haproxy  haproxy相关配置│   │   └── haproxy.cfg│   ├── heapster   heapster相关yaml配置│   │   ├── influxdb│   │   │   ├── grafana.yaml│   │   │   ├── heapster.yaml│   │   │   └── influxdb.yaml│   │   └── rbac│   │       └── heapster-rbac.yaml│   ├── kube    k8s自身配置│   │   ├── 10-kubeadm.conf│   │   ├── config    kubeadm配置│   │   └── kubelet.service│   ├── kubeinit.json  忽略│   └── net  网络相关配置│       ├── calico.yaml│       └── calicoctl.yaml└── shell    初始化脚本    ├── init.sh   初始化节点,安装bin文件，systemd配置等    └── master.sh  执行kubeadm init和其它组件

Initializing nodes

After unpacking the package and then scp -r xxx root@ip:/root distributing the unpacked package to the other nodes

all nodes in the cluster need to be executed cd shell && sh init.sh (if you run a single master then sh master.sh you need to execute , multi-master do not run)

Here are some things to keep in mind:

1. Modify the init.sh script to add later if the binaries do not have permission to executechmod +x /usr/bin/kube*
2. Cgroups driver needs to choose the docker17.0x version, there is no need to adjust, if the 1.1x version of the Docker need to manually modify the Kubelet boot file inside the cgroups configuration to systemd (modify location /etc/systemd/system/kubelet.service.d ) and Docker Info|grep CG Consistent
3. Modify the default init or manual execution in advance or the sysctl -w net.ipv4.ip_forward=1 seventh line error

After the completion of execution through kubectl get pod -n kube-system the command view, the state is all running normal

Starter ETCD Cluster

ETCD cluster installation using Docker-compose mode deployment

A. Use Docker-compose to start, if not installed:

$ pip install docker-compose

B. Starting docker-compose with a binary package (offline optional)

$ wget https://github.com/docker/compose/releases/download/1.18.0/docker-compose-Linux-x86_64  #官方推荐是用curl,不建议$ mv docker-compose-Linux-x86_64 /usr/local/bin/docker-compose && chmod a+x /usr/local/bin/docker-compose  #也有写+x的.#这样就完成了,测试$ docker-compose version  #下面是正常输出docker-compose version 1.18.0, build 8dd22a9docker-py version: 2.6.1CPython version: 2.7.13OpenSSL version: OpenSSL 1.0.1t  3 May 2016

In the OUT/ETCD directory has the relevant template etcd-docker-compose-x.yam , the start of multiple nodes when modified to their own IP address two other nodes copied, modify the IP, image that line should be replaced gcr.io/google_containers/etcd-amd64:3.1.11 , the actual version number can be changed.

#需要修改所有含有ip的地方，下面的9，10，11，12行改为当前节点ip，15行三个ip顺序改为etcd集群部署的三台节点ipversion: '2.1'services:    etcd0:        container_name: etcd_infra0        image: gcr.io/google_containers/etcd-amd64:3.0.17  #这里最后改为3.1.11        command: |                etcd --name infra0                --initial-advertisie-peer-urls http://10.230.204.160:2380                --listen-peer-urls http://10.230.204.160:2380                --listen-client-urls http://10.230.204.160:2379,http://127.0.0.1:2379                --advertise-client-urls http://10.230.204.160:2379                --data-dir /etcd-data.etcd                --initial-cluster-token etcd-cluster-1                --initial-cluster infra0=http://10.230.204.160:2380,infra1=http://10.230.204.165:2380,infra2=http://10.230.204.151:2380                --initial-cluster-state new        restart: always        volumes:           - /data/etcd-data.etcd:/etcd-data.etcd        network_mode: "host"

Three nodes are started separately:

$ docker-compose -f out/etcd/etcd-docker-compose-x.yml up -d  #正常输出Creating etcd_infrax ... done  x为每个etcd编号

Check whether the installation was successful:

$ docker exec etcd_infra0 etcdctl member list  #master1上可能运行报错容易提示容器正在重启。。原因暂时未知，其他master上可以#成功应该是类似显示5ded6dd284b89d31: name=infra1 peerURLs=http://10.230.204.153:2380 clientURLs=http://10.230.204.153:2379 isLeader=true6d4b5eee32c1497a: name=infra0 peerURLs=http://10.230.204.150:2380 clientURLs=http://10.230.204.150:2379 isLeader=false729d9cd56debb1a1: name=infra2 peerURLs=http://10.230.204.154:2380 clientURLs=http://10.230.204.154:2379 isLeader=false#如果出现有peerURL不显示说明没有成功，尝试remove重新创建$ docker-compose -f  out/etcd/etcd-docker-compose-x.yml down -v

KUBEADM Configuration

Modifying a configuration out/kube/config file

apiVersion: kubeadm.k8s.io/v1alpha1kind: MasterConfigurationapiServerCertSANs:    #此处填所有的masterip和lbip和其它你可能需要通过它访问apiserver的地址和域名或者主机名等，如阿里fip，证书中会允许这些ip- 172.31.244.231- 172.31.244.232- 172.31.244.233- 172.31.244.234- master1- master2- master3- node1- 47.75.1.72etcd:  endpoints:   #这里填之前安装的etcd集群地址列表，修改IP地址  - http://172.31.244.232:2379  - http://172.31.244.233:2379  - http://172.31.244.234:2379apiServerExtraArgs:  endpoint-reconciler-type: leasenetworking:  podSubnet: 192.168.0.0/16  #不用改kubernetesVersion: v1.9.2  #不用改featureGates:  #不用改   CoreDNS: true

Then execute:

$ kubeadm init --config out/kube/config

After the successful KUBEADM join command exists in the file, that thing cannot be lost.

Start Calico, etc.

mkdir ~/.kube && cp /etc/kubernetes/admin.conf ~/.kube/config (if it already exists please check if it is the same, not sure recommend deleting the re CP past)

Modify the Calico configuration and replace the ETCD address with your installed cluster address:
Out/net/calico.yaml:

kind: ConfigMapapiVersion: v1metadata:  name: calico-config  namespace: kube-systemdata:  # The location of your etcd cluster.  This uses the Service clusterIP  # defined below.  etcd_endpoints: "http://10.96.232.136:6666" # 这里改成etcd集群地址如 "http://172.31.244.232:2379,http://172.31.244.233:2379,http://172.31.244.234:2379"

$ kubectl apply -f out/net/calico.yaml$ kubectl apply -f out/heapster/influxdb$ kubectl apply -f out/heapster/rbac$ kubectl apply -f out/dashboard#上面命令可整合为$ kubectl apply -f out/net/calico.yaml -f out/heapster/influxdb -f out/heapster/rbac -f out/dashboard

1. Then access the https://master1IP:32000 port, under Chrome can not enter the prompt certificate is wrong, the replacement of Firefox, prompted that the certificate date is not correct (to be fixed)

Start multiple Master

The first master we call MASTER0 (assuming the other master has init.sh), now copy the first master's/etc/kubernetes/pki directory to the other master node

$ mkdir -p /etc/kubernetes$ scp -r /etc/kubernetes/pki root@10.1.245.93:/etc/kubernetes/pki

Remove the APISERVER.CRT and Apiserver.key files under the PKI directory rm -rf apiserver.crt apiserver.key , and note that if you do not delete, you will see only one master, which is not normal.

Also use out/kube/config files on Master0, copy content, copy to Master1, scp out/kube/config root@10.230.204.151:/root/ executekubeadm init --config ~/config

MASTER2/3 node with Master1

Start LoadBalance

I recommend using the four-tier proxy haproxy configuration Out/haproxy directory:

vi out/haproxy/haproxy.cfg

global  daemon  log 127.0.0.1 local0  log 127.0.0.1 local1 notice  maxconn 4096defaults  log               global  retries           3  maxconn           2000  timeout connect   5s  timeout client    50s  timeout server    50sfrontend k8s  bind *:6444  mode tcp  default_backend k8s-backendbackend k8s-backend  balance roundrobin  mode tcp  #下面三个ip替换成三个你自己master的地址  server k8s-1 10.1.245.93:6443 check   server k8s-1 10.1.245.94:6443 check  server k8s-2 10.1.245.95:6443 check

mkdir /etc/haproxyand then copy this file incp out/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg

$ docker run --net=host -v /etc/haproxy:/usr/local/etc/haproxy --name ha -d haproxy:1.7

Modifying the Kubeproxy configuration

$ kubectl -n kube-system edit configmap kube-proxy

Find the master address, modify the LB address, 6444 port (the key here is how to know the address of LB is Which?) After configuring the above three Masterip polling does not know which is the LB address)

#找到文件的这一块，第七行server 有个ip地址apiVersion: v1    kind: Config    clusters:    - cluster:        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt        server: https://10.230.204.151:6443 #修改为 LoadBalanceIP:6444      name: default    contexts:    - context:        cluster: default        namespace: default        user: default      name: default    current-context: default    users:    - name: default      user:        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

Join node nodes

Or a command that executes the first master output on the node nodes

$ kubeadm join --token <token> 10.1.245.94:6443 --discovery-token-ca-cert-hash sha256:

Modify node Node Kubelet configuration

vi /etc/kubernetes/kubelet.conf Also change the address to a LB address, such as: 10.1.245.94:6444 , modify the following line fifth (show example has been modified)

apiVersion: v1clusters:- cluster:    certificate-authority-data: xxxxxx #此处省略几百字符    server: https://10.230.204.160:6444 #修改这里为LB:6444，原本是另外的ip:6443  name: default-clustercontexts:- context:    cluster: default-cluster    namespace: default    user: default-auth  name: default-contextcurrent-context: default-context

KUBECTL Configuration

Modify ~/.kube/config The file, the IP of the server is changed to lb IP 10.1.245.94:6444

or modify by command:

$ kubectl config set-cluster kubernetes --server=https://47.52.227.242:6444 --kubeconfig=$HOME/.kube/config

Start a multi-DNS replica

$ kubectl edit deploy coredns -n kube-system

Replicas:3

[root@master1 ~]$ kubectl get pod -n kube-system -o wide|grep corecoredns-65dcdb4cf-4j5s8                  1/1       Running   0          39m       192.168.137.65    master1coredns-65dcdb4cf-ngx4h                  1/1       Running   0          38s       192.168.180.1     master2coredns-65dcdb4cf-qbsr6                  1/1       Running   0          38s       192.168.166.132   node1

In this way, three DNS was started

Validation and testing

$ kubectl run test --image busybox sleep 10000$ kubectl exec your-busybox-pod-name nslookup kubernetes

Kill non-LB master, multiple tests to see if the creation of pod and DNS is normal, you can also telnet 10.96.0.1 443 to verify that Clusterip is normal