Docker - 用Flannel跨主機

這是一個建立於 的文章，其中的資訊可能已經有所發展或是發生改變。

試了下比較流行的幾種SDN，感覺flannel還是比較好用，這裡簡單記錄一下。

用的是virtualbox，3個機器，分別為:

genesis: inet 192.168.99.103/24 brd 192.168.99.255 scope global dynamic enp0s3

exodus: inet 192.168.99.105/24 brd 192.168.99.255 scope global dynamic enp0s3

leviticus: inet 192.168.99.106/24 brd 192.168.99.255 scope global dynamic enp0s3

虛機資訊如下

[root@localhostyum.repos.d]# uname -mars

Linuxlocalhost.localdomain3.10.0-229.el7.x86_64#1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@localhostyum.repos.d]# cat /etc/*-release

CentOSLinuxrelease7.3.1611(Core)

NAME="CentOS Linux"VERSION="7 "

(Core)"ID="centos"ID_LIKE="rhel fedora"VERSION_ID="7"

RETTY_NAME="CentOS Linux 7 (Core)"ANSI_COLOR="0;31"

CPE_NAME="cpe:/o:centos:centos:7"

HOME_URL="https://www.centos.org/"BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"CENTOS_MANTISBT_PROJECT_VERSION="7"REDHAT_SUPPORT_PRODUCT="centos"REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOSLinuxrelease7.3.1611(Core)

CentOSLinuxrelease7.3.1611(Core)

[root@localhostyum.repos.d]# docker version

Client:Version:1.12.5APIversion:1.24Goversion:go1.6.4Gitcommit:7392c3bBuilt:FriDec1602:23:592016OS/Arch:linux/amd64

隨便選擇兩台機器run一下，在容器中ifconfig:

[root@localhost~]# docker run -it busybox

/#ifconfig

eth0

Linkencap:EthernetHWaddr02:42:AC:11:00:02inetaddr:172.17.0.2Bcast:0.0.0.0Mask:255.255.0.0inet6addr:fe80::42:acff:fe11:2/64Scope:LinkUPBROADCASTRUNNINGMULTICASTMTU:1500Metric:1RXpackets:12errors:0dropped:0overruns:0frame:0TXpackets:6errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RXbytes:1016(1016.0B)TXbytes:508(508.0B)

lo

Linkencap:LocalLoopbackinetaddr:127.0.0.1Mask:255.0.0.0inet6addr:::1/128Scope:HostUPLOOPBACKRUNNINGMTU:65536Metric:1RXpackets:0errors:0dropped:0overruns:0frame:0TXpackets:0errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RXbytes:0(0.0B)TXbytes:0(0.0B)

發現參數完全相同，單純bridge模式下並沒有跨主機互連，而host模式是並不建議使用的。

install

先yum install -y etcd flannel，如果沒問題是再好不過了。

etcd 3.x支援--config-file參數，需要的話可以從原始碼install(需要golang 1.6+)。

先從etcd開始，簡單說就是"distributed key value store"。

etcd叢集的3種方式:

static

etcd discovery

DNS discovery

DNS discovery主要是用srv record，這裡先不搞DNS服務，下面對static和etcd discovery兩種方式簡單說明一下。

static

參數可以在啟動時置頂，或者寫到設定檔中，預設設定檔為/etc/etcd/etcd.conf。

genesis的配置如下:

ETCD_NAME=genesis

ETCD_DATA_DIR="/var/lib/etcd/genesis"

ETCD_LISTEN_PEER_URLS="http://192.168.99.103:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.99.103:2379,http://127.0.0.1:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.99.103:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.99.103:2379"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="etct-fantasy"ETCD_INITIAL_CLUSTER="exodus=http://192.168.99.105:2380,genesis=http://192.168.99.103:2380"

exodus的配置如下:

ETCD_NAME=exodusETCD_DATA_DIR="/var/lib/etcd/exodus"ETCD_LISTEN_PEER_URLS="http://192.168.99.105:2380"ETCD_LISTEN_CLIENT_URLS="http://192.168.99.105:2379,http://127.0.0.1:2379"ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.99.105:2380"ETCD_ADVERTISE_CLIENT_URLS="http://192.168.99.105:2379"ETCD_INITIAL_CLUSTER_STATE="new"ETCD_INITIAL_CLUSTER_TOKEN="etctfantasy"

ETCD_INITIAL_CLUSTER="exodus=http://192.168.99.105:2380,genesis=http://192.168.99.103:2380"

啟動方式看自己喜好，如果打算用systemctl啟動的話，注意/usr/lib/systemd/system/etcd.service中的內容可能不會如你所願。

啟動後，檢查一下叢集狀態，順便也看看有哪些成員:

[root@localhost etcd]# etcdctl cluster-health

member7a4f27f78a05e755ishealthy: g

ot healthy resultfromhttp://192.168.99.103:2379

failed to check the health of member8e8718b335c6c9a2 on http://192.168.99.105:2379: 

Get http://192.168.99.105:2379/health: dial tcp 192.168.99.105:2379: i/o timeoutmember8e8718b335c6c9a2isunreachable: [http://192.168.99.105:2379] are all unreachableclusterishealthy

提示"member unreachable"，看來是被exodus的防火牆攔住了，我們先粗暴一點。

[root@localhost etcd]# systemctl stop firewalld

[root@localhost etcd]# etcdctl cluster-health

member7a4f27f78a05e755ishealthy: got healthy resultfromhttp://192.168.99.103:2379 member8e8718b335c6c9a2ishealthy: got healthy resultfromhttp://192.168.99.105:2379 

cluster is healthy etcd discovery

當然，這樣配置的前提是已經知道各個節點的資訊。

但實際情境中可能無法預知各個member，所以我們需要讓etcd自己去發現(discovery)。

首先，etcd提供了一個public discovery service - discovery.etcd.io，我們用它來產生一個discovery token，並在genesis建立目錄:

[root@localhostetcd]# curl https://discovery.etcd.io/newsize=3

https://discovery.etcd.io/6321c0706046c91f2b2598206ffa3272

[root@localhostetcd]# etcdctl set /discovery/6321c0706046c91f2b2598206ffa3272/_config/size 3

修改exodus的配置，用discovery代替之前的cluster:

ETCD_NAME=exodus

ETCD_DATA_DIR="/var/lib/etcd/exodus"

ETCD_LISTEN_PEER_URLS="http://192.168.99.105:2380"

ETCD_LISTEN_CLIENT_URLS="http://192.168.99.105:2379,http://127.0.0.1:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.99.105:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://192.168.99.105:2379"

ETCD_DISCOVERY=http://192.168.99.103:2379/v2/keys/discovery/98a976dac265a218f1a1959eb8dde57f

如果啟動後一直顯示如下錯誤(參考:raft election):

rafthttp:the clock difference against peer ?????? is too high [??????s > 1s]

簡單的解決方案是通過ntp:

[root@localhostetcd]yum install ntp -y

[root@localhostetcd]# systemctl enable ntpd

Createdsymlinkfrom /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.

[root@localhostetcd]# systemctl start ntpd

flannel

set一個路徑給flannel用:

etcdctlset/coreos.com/network/config'{ "Network": "10.1.0.0/16" }'

以systemctl start flanneld方式啟動時，如果出現以下錯誤

network.go:53]Failedto retrieve networkconfig:100:Keynotfound (/coreos.net) [9]

注意/etc/sysconfig/flanneld中的內容，FLANNEL_ETCD_PREFIX很可能是/atomic.io/network，將其改為/coreos.com/network。

或者也可以通過-etcd-prefix指定。

啟動成功後，查看subnet:

[root@localhostetcd]# etcdctl ls /coreos.com/network/subnets

coreos.com/network/subnets/10.1.90.0-24

coreos.com/network/subnets/10.1.30.0-24

coreos.com/network/subnets/10.1.18.0-24

flannel啟動成功後會產生/run/flannel/docker，內容如下:

DOCKER_OPT_BIP="--bip=10.1.30.1/24"

DOCKER_OPT_IPMASQ="--ip-masq=true"

DOCKER_OPT_MTU="--mtu=1450"

DOCKER_NETWORK_OPTIONS=" --bip=10.1.30.1/24 --ip-masq=true --mtu=1450 "

用以下方式啟動docker:

[root@localhostetcd]# source /run/flannel/docker

[root@localhostetcd]# docker daemon ${DOCKER_NETWORK_OPTIONS} >> /dev/null 2>&1 &

／run/flannel/docker是怎麼來的？

參考flanneld的兩個啟動參數，-subnet-dir和-subnet-file。

在genesis進入容器看看效果:

[root@localhostetcd]# docker run -it busybox

#ifconfig

eth0 Linkencap:EthernetHWaddr02:42:0A:01:5A:02inetaddr:10.1.90.2Bcast:0.0.0.0Mask:255.255.255.0inet6addr:fe80::42:aff:fe01:5a02/64Scope:LinkUPBROADCASTRUNNINGMULTICASTMTU:1450Metric:1RXpackets:6errors:0dropped:0overruns:0frame:0TXpackets:6errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RXbytes:508(508.0B)TXbytes:508(508.0B)

lo

Linkencap:LocalLoopbackinetaddr:127.0.0.1Mask:255.0.0.0inet6addr:::1/128Scope:HostUPLOOPBACKRUNNINGMTU:65536Metric:1RXpackets:0errors:0dropped:0overruns:0frame:0TXpackets:0errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:1RXbytes:0(0.0B)TXbytes:0(0.0B)

在exodus也做類似操作，在exodus中ping一下10.1.90.2，發現是通的。

並在各自容器中也ping一下，檢查是否跨主機互連。