Kubernetes1.9 production environment high available practice--003-node installation and configuration in Docker

Apiserver installs the deployment in a highly available manner. This article is attached to a "Kubernetes1.9 production environment high available practice –002-apiserver high Availability Installation Deployment".

In this article, we focus on how to install Docker in node nodes and the configuration of Docker.
The configuration of the Kubernetes node is discussed in the next section.

This article in a lot of articles are relatively simple to write. We write here in a relatively troublesome way, but if you are configuring in a production environment, use this configuration.

We are installing three nodes here:
192.168.3.56 yds-dev-svc02-node01
192.168.3.57 YDS-DEV-SVC02-NODE02
192.168.3.58 yds-dev-svc02-node03

The three-node installation process is the same, so we only configure YDS-DEV-SVC02-NODE01 here, and the other nodes use the same configuration. 01 Server Configuration 01.01 Configuring the server name and IP

[Root@localhost ~]# hostnamectl set-hostname yds-dev-svc02-node01
[root@localhost ~]# cat/etc/sysconfig/ Network-scripts/ifcfg-ens32 
type=ethernet
proxy_method=none
browser_only=no
bootproto=static
Defroute=yes
Ipv4_failure_fatal=no
ipv6init=yes
ipv6_autoconf=yes
ipv6_defroute=yes
ipv6_failure_fatal=no
ipv6_addr_gen_mode=stable-privacy
Name=ens32
uuid=7d6fb2ed-364c-415f-9b02-0e54436ff1ec
device=ens32
onboot=yes
ipaddr= 192.168.3.56
netmask=255.255.255.0
gateway=192.168.3.1
dns1=192.168.3.10
dns2=61.139.2.69

When the configuration is complete, exit the login again. Docker Installation 02.01 Installation Epel-release

Yum update-y 
Yum install-y epel-release wget

02.02 Installation Docker

Yum install-y Docker

[root@yds-dev-svc02-node01 ~]# Docker--version
version docker, build 1.13.1 1.13.1

02.03 Start Docker

Systemctl Enable Docker
systemctl start Docker
systemctl status Docker

02.04 Check Docker

[Root@yds-dev-svc02-node01 ~]# Docker info containers:0 running:0 paused:0 stopped:0 images:0 Server version:1.13 .1 Storage driver:overlay2 backing filesystem:xfs Supports d_type:true Native Overlay diff:true Logging Driver:jou  Rnald cgroup driver:systemd Plugins:Volume:local Network:bridge host Macvlan null overlay swarm:inactive runtimes: Docker-runc runc Default runtime:docker-runc Init binary:docker-init containerd version: (Expected:aa8187dbd3b7ad67d8 e5e3a15115d3eef43a7ed1) Runc version:n/a (expected:9df8b306d01f59d3a8029be411de015b7304dd8f) init version:n/a ( expected:949e6facb77383876aeff8a6944dde66b3089574) Security Options:seccomp warning:you ' re not using the ' default sec Comp Profile profile:/etc/docker/seccomp.json selinux Kernel version:3.10.0-693.21.1.el7.x86_64 operating System:cen TOS Linux 7 (Core) ostype:linux architecture:x86_64 number of Docker hooks:3 cpus:2 total memory:1.78 GiB name:yds-d EV-SVC02-NODE01 Id:ykwt:7y6m:o3FB:C7BC:KU3Q:ZI5I:KM7E:QGTW:7TZV:2WF4:S5LD:ROKB Docker Root Dir:/var/lib/docker debug Mode (client): False debug Mode (s erver): false Registry:https://index.docker.io/v1/experimental:false insecure registries:127.0.0.0/8 Live Restore Ena Bled:false Registries:docker.io (Secure)

The default storage driver is now overlay2 and is docker for test environments only. But we are in the production environment, we need to change this to Devicemapper DIRECT-LVM mode, do not use LOOP-LVM mode in production. For the Docker storage drive, there are a lot of explanations on the Internet, if sufficient time, will translate a few comparative documents on this.

Next, we change the Docker storage to DIRECT-LVM. 03 Configuration Docker DIRECT-LVM Mode 03.01 HDD Check

Let's look at the hard drive information first.

[Root@yds-dev-svc02-node01 ~]# fdisk-l disk/dev/sda:128.8 gigabyte, 128849018880 bytes, 251658240 sectors Units = sectors of 1 * bytes = sector size (logical/physical): bytes/512 bytes I/o size (minimum/optimal): bytes/512 byt ES Disk label type:dos disk IDENTIFIER:0X000A9FCF Device Boot Start end Blocks Id system/dev/s

DA1 * 2048 2099199 1048576-linux/dev/sda2 2099200 251658239 124779520 8e Linux LVM disk/dev/sdb:257.7 GB, 257698037760 bytes, 503316480 sectors Units = sectors of 1 * The bytes of sector size (Logica l/physical): bytes/512 bytes I/o size (minimum/optimal): bytes/512 bytes GB, 53687091200 bytes, 104857600 sectors Units = sectors of 1 *-the bytes of sector size (logical/physical): bytes /bytes I/O size (minimum/optimal): bytes/512 bytes disk/dev/mapper/centos-swap:2147 MB, 2147483648 bytes, 4 194304 Sectors Units = SEctors of 1 * bytes = sector size (logical/physical): bytes/512 bytes I/o size (minimum/optimal): bytes  /bytes disk/dev/mapper/centos-home:71.9 GB, 71932313600 bytes, 140492800 sectors Units = sectors of 1 * 512 = 512 Bytes sector size (logical/physical): bytes/512 bytes I/o size (minimum/optimal): bytes/512 bytes

You can see/DEV/SDB as an unused hard drive.
Now, let's deal with/DEV/SDB to store Docker data. 03.02 Stop Docker

Systemctl Stop Docker

03.03 Installing software

Yum install-y lvm2 Device-mapper-persistent-data

03.04 Creating a physical volume

[Root@yds-dev-svc02-node01 ~]# pvcreate/dev/sdb
  Physical Volume "/dev/sdb" successfully created.

03.05 Creating Docker volume groups

[Root@yds-dev-svc02-node01 ~]# vgcreate docker/dev/sdb
  Volume Group "Docker" successfully created

03.06 Creating logical volumes

Here you need to create two logical volumes, named Thinpool and Thinpoolmeta.

[Root@yds-dev-svc02-node01 ~]# lvcreate--wipesignatures y-n thinpool docker-l 95%vg Logical volume
  ' thinpool ' creat Ed.

[Root@yds-dev-svc02-node01 ~]# lvcreate--wipesignatures y-n thinpoolmeta docker-l 1%VG Logical volume
  "thinpoolmet A "created.

03.07 Volume Conversion

Convert the volumes to a thin pool and a storage location for metadata for the thin pool, using the Lvconvert command.
Convert the volume you just created to the thin pool and use the Thinpoolmeta volume that you just created.

[root@yds-dev-svc02-node01 ~]# lvconvert-y \
--zero n \
c 512K \
--thinpool docker/thinpool \
-- Poolmetadata Docker/thinpoolmeta
  Thin Pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
  warning:converting Logical Volume Docker/thinpool and Docker/thinpoolmeta to thin pool ' s data and metadata volumes H metadata wiping.
  This would DESTROY CONTENT of LOGICAL VOLUME (filesystem etc.)
  Converted Docker/thinpool_tdata to thin pool.

03.08 Automatic expansion configuration

The options you need to configure are Thin_pool_autoextend_threshold and thin_pool_autoextend_percent.

Thin_pool_autoextend_threshold: When the amount of usage is reached, it will attempt to expand automatically, using existing space. The configuration is 100 for no expansion (Disable).
Thin_pool_autoextend_percent: In the expansion is the increase in the amount of space.
The example below adds 20% more capacity when the disk usage reaches 80%.
The following configuration is to increase the capacity by 20% when volume usage reaches 80%.

Activation {
  thin_pool_autoextend_threshold=80
  thin_pool_autoextend_percent=20
}

Now we write this configuration to the configuration file/etc/lvm/profile/docker-thinpool.profile.

[root@yds-dev-svc02-node01 ~]# cat/etc/lvm/profile/docker-thinpool.profile
activation {
  Thin_pool_ autoextend_threshold=80
  thin_pool_autoextend_percent=20
}

03.09 applying LVM configuration Files

[Root@yds-dev-svc02-node01 ~]# lvchange--metadataprofile docker-thinpool docker/thinpool Logical volume docker/
  Thinpool changed.

03.10 enabling LVM monitoring

If LVM monitoring is not enabled, the automatic expansion we have just configured is not effective.

[Root@yds-dev-svc02-node01 ~]# lvs-o+seg_monitor
  LV       VG     Attr       lsize    Pool Origin data%  meta% Move  Log cpy%sync Convert Monitor  
  home     centos-wi-ao----   66.99g                                                              
  root     centos-wi-ao----   50.00g                                                              
  swap     centos-wi-ao----    2.00g                                                              
  thinpool Docker twi-a-t---<228.00g             0.00   0.01                             Monitored

03.11 eliminate Docker data

If there is a/var/lib/docker, back up or empty the files inside.

MKDIR/VAR/LIB/DOCKER.BK
mv/var/lib/docker/*/VAR/LIB/DOCKER.BK

When the configuration is complete, if it is good, you can delete the directory/VAR/LIB/DOCKER.BK 03.12 configuration Docker Storage driver

If/etc/docker/daemon.json is empty before the configuration. Now we're going to modify this file to read the following.

[Root@yds-dev-svc02-node01 docker]# Cat/etc/docker/daemon.json
{"
    storage-driver": "Devicemapper",
    " Storage-opts ": [
    " Dm.thinpooldev=/dev/mapper/docker-thinpool ",
    " Dm.use_deferred_removal=true "," "
    Dm.use_deferred_deletion=true "
    ]
}

If the/etc/sysconfig/docker-storage is in the following configuration, remove it.
docker_storage_options= "–storage-driver Devicemapper"

[Root@yds-dev-svc02-node01 docker]# Cat/etc/sysconfig/docker-storage

View information in/etc/sysconfig/docker-storage-setup.

[Root@yds-dev-svc02-node01 docker]# cat/etc/sysconfig/docker-storage-setup 
storage_driver=devicemapper

03.13 start Docker

Systemctl start Docker
systemctl status Docker

03.13 Authentication Configuration

[Root@yds-dev-svc02-node01 docker]# Docker info containers:0 running:0 paused:0 stopped:0 Server Version: 1.13.1 Storage driver:devicemapper pool name:docker-thinpool pool blocksize:524.3 kB Base Device size:10.74 GB Ba cking FILESYSTEM:XFS data file:metadata file:data space used:20.45 MB data spaces total:244.8 GB data space Ava ilable:244.8 GB Metadata space used:311.3 KB Metadata spaces total:2.575 GB Metadata space available:2.575 GB Thin Pool Minimum free space:24.48 GB Udev Sync supported:true Deferred removal enabled:true Deferred deletion Rue Deferred Deleted Device count:0 Library version:1.02.140-rhel7 (2017-05-03), Logging Driver:journald cgroup Driver : Systemd Plugins:Volume:local Network:bridge host Macvlan null overlay swarm:inactive runtimes:docker-runc Runc D Efault runtime:docker-runc Init binary:docker-init containerd version: (Expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1) Runc version:n/a (expected:9df8b306d01f59d3a8029be411de015b7304dd8f) Init version:n/a (expected: 949e6facb77383876aeff8a6944dde66b3089574) Security Options:seccomp warning:you ' re not using the ' default Seccomp ProFi Le profile:/etc/docker/seccomp.json selinux Kernel version:3.10.0-693.21.1.el7.x86_64 operating System:centos Linux 7 (Core) ostype:linux architecture:x86_64 number of Docker hooks:3 cpus:2 total memory:1.78 GiB name:yds-dev-svc02-n ODE01 id:ykwt:7y6m:o3fb:c7bc:ku3q:zi5i:km7e:qgtw:7tzv:2wf4:s5ld:rokb Docker Root Dir:/var/lib/docker Debug Mode (
 Client): false Debug Mode (server): false Registry:https://index.docker.io/v1/experimental:false Insecure registries: 127.0.0.0/8 Live Restore enabled:false Registries:docker.io (Secure)

If the configuration is correct, Data file and metadata file two are empty and pool name is Docker-thinpool. 03.13 Cleaning up

After verifying our configuration, we can delete the directory that we backed up before.

Rm-rf/var/lib/docker.bk

some configurations of Docker

Because the default base Device size is 10G and the size of the regular Docker exceeds 10G, you need to modify the size of this value.
We modify the base Device size to 30G here.
Only need to add parameters in/etc/docker/daemon.json: dm.basesize=20g

[Root@yds-dev-svc02-node01 docker]# Cat/etc/docker/daemon.json 
{"
    storage-driver": "Devicemapper",
    " Storage-opts ": [
    " Dm.thinpooldev=/dev/mapper/docker-thinpool ",
    " Dm.use_deferred_removal=true "," "
    Dm.use_deferred_deletion=true ",
    " dm.basesize=20g "
    ]
}

Restart the Docker and verify the following:

[Root@yds-dev-svc02-node01 docker]# Docker info containers:0 running:0 paused:0 stopped:0 Server Version: 1.13.1 Storage driver:devicemapper pool name:docker-thinpool pool blocksize:524.3 kB Base Device size:21.47 GB Ba cking FILESYSTEM:XFS data file:metadata file:data space used:29.36 MB data spaces total:244.8 GB data space Ava ilable:244.8 GB Metadata space used:311.3 KB Metadata spaces total:2.575 GB Metadata space available:2.575 GB Thin Pool Minimum free space:24.48 GB Udev Sync supported:true Deferred removal enabled:true Deferred deletion Rue Deferred Deleted Device count:0 Library version:1.02.140-rhel7 (2017-05-03), Logging Driver:journald cgroup Driver : Systemd Plugins:Volume:local Network:bridge host Macvlan null overlay swarm:inactive runtimes:docker-runc Runc D Efault runtime:docker-runc Init binary:docker-init containerd version: (Expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1) Runc version:n/a (expected:9df8b306d01f59d3a8029be411de015b7304dd8f) Init version:n/a (expected: 949e6facb77383876aeff8a6944dde66b3089574) Security Options:seccomp warning:you ' re not using the ' default Seccomp ProFi Le profile:/etc/docker/seccomp.json selinux Kernel version:3.10.0-693.21.1.el7.x86_64 operating System:centos Linux 7 (Core) ostype:linux architecture:x86_64 number of Docker hooks:3 cpus:2 total memory:1.78 GiB name:yds-dev-svc02-n ODE01 id:ykwt:7y6m:o3fb:c7bc:ku3q:zi5i:km7e:qgtw:7tzv:2wf4:s5ld:rokb Docker Root Dir:/var/lib/docker Debug Mode (
 Client): false Debug Mode (server): false Registry:https://index.docker.io/v1/experimental:false Insecure registries: 127.0.0.0/8 Live Restore enabled:false Registries:docker.io (Secure)

51 Custom configurations for Docker 05.01/etc/sysconfig/docker configuration file

[Root@yds-dev-svc02-node01 ~]# Cat/etc/sysconfig/docker #/etc/sysconfig/docker # Modify These options if you are want to CH Ange the way the Docker daemon runs ' if [-Z ' options=} ']; Then Docker_cert_path=/etc/docker Fi # does not add registries to this file anymore.
Use/etc/containers/registries.conf # from the Atomic-registries package. # on a SELinux system, if you remove the--selinux-enabled option, your # also need to turn on the Docker_transition_un
Confined Boolean.  # setsebool-p docker_transition_unconfined 1 # Location used for temporary files, such as those created by # Docker load and build operations.
Default is/var/lib/docker/tmp # Can is overriden by setting the following environment variable.
# docker_tmpdir=/var/tmp # Controls the/etc/cron.daily/docker-logrotate cron job status.
# To disable, uncomment the line below.
# logrotate=false # docker-latest daemon can be used by starting the Docker-latest Unitfile. # To use Docker-latest client, uncomment below lines #DOCKERBINARY =/usr/bin/docker-latest #DOCKERDBINARY =/usr/bin/dockerd-latest #DOCKER_ Containerd_binary=/usr/bin/docker-containerd-latest #DOCKER_CONTAINERD_SHIM_BINARY =/usr/bin/ Docker-containerd-shim-latest

05.02 docker-storage-setup configuration file

[Root@yds-dev-svc02-node01 ~]# cat/etc/sysconfig/docker-storage-setup 
storage_driver=devicemapper

05.03/etc/docker/daemon.json configuration file

[Root@yds-dev-svc02-node01 ~]# Cat/etc/docker/daemon.json 
{"
    storage-driver": "Devicemapper",
    " Storage-opts ": [
        " Dm.thinpooldev=/dev/mapper/docker-thinpool ",
        " Dm.use_deferred_removal=true "," "
        Dm.use_deferred_deletion=true ",
        " dm.basesize=20g "
    ],
    " log-driver ":" Json-file ",
    " log-opts ": {
        "Max-size": "200m", "
        max-file": "5",
        "labels": "Prod"
    },
    "Insecure-registries": [
        " 192.168.0.0/16 "
    ],
    " DNS ": [
        " 10.254.0.2 ",
        " 61.139.2.69 "
    ],
        " selinux-enabled ": False,
        "Dns-search": [
                "default.svc.cluster.local",
                "svc.cluster.local"
         ],
        "dns-opt": [
                "Ndots:2",
                "Timeout:2",
                "Attempts:2"
         ]
}

Your support is the author's greatest motivation: