Keepalived實現高可用，keepalived實現

Keepalived實現高可用，keepalived實現

原文發表於cu：2017-03-27

參考文檔：

 本文涉及keepalived的安裝，簡單配置，為haproxy做高可用。

一．環境準備1. 作業系統

CentOS-7-x86_64-Everything-1511

2. Keepalived版本

截至2017-03-22，keepalived版本是1.3.5:

http://www.keepalived.org/software/keepalived-1.3.5.tar.gz

3. 拓撲圖

二．Keepalived安裝配置

以下流程均在node1節點完成，node2節點請參考node1做適當修改。 

1. 依賴軟體

#升級或者安裝相關軟體，不是必需都安裝一次;#一般libnl3-devel ipset-devel iptables-devel libnfnetlink-devel popt popt-static popt-devel等並沒有預先安裝到系統中;#net-snmp-devel是需要開啟相關功能才需要[root@elk-node1 ~]# yum install openssl-devel libnl3-devel ipset-devel iptables-devel libnfnetlink-devel popt popt-static popt-devel gcc kernel-headers kernel-devel net-snmp-devel -y 

2. 下載

[root@elk-node1 ~]# cd /usr/local/src/[root@elk-node1 src]#wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz 

3. 編譯安裝

#編譯前可通過”./configure --help”查看相關編譯參數;#此編譯未帶“--with-kernel-dir”參數，一般認為採用”--with-kernel-dir=/usr/src/kernels/(version)”指定到核心效果更好，這裡環境比較簡單，實際使用後並沒有明顯的問題；#這裡未指定是因為centos7在編譯使用參數之後找不到”linux/netlink.h”標頭檔，即使在相應目錄下能找到相應標頭檔，搜了一下也沒有找到對應的解決方案[root@elk-node1 src]# tar -zxvf keepalived-1.3.5.tar.gz[root@elk-node1 src]# cd keepalived-1.3.5[root@elk-node1 keepalived-1.3.5]# ./configure --prefix=/usr/local/keepalived [root@elk-node1 keepalived-1.3.5]# make[root@elk-node1 keepalived-1.3.5]# make install 

4. 配置開機啟動1）啟動相關命令

#軟連結[root@elk-node1 ~]# cd /usr/local/keepalived/[root@elk-node1 keepalived]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/[root@elk-node1 keepalived]# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/

2）設定檔

#軟連結[root@elk-node1 keepalived]# mkdir -p /etc/keepalived[root@elk-node1 keepalived]# ln -s /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/ 

3）開機啟動

#centos7編譯安裝目錄下，預設沒有”/etc/rc.d/init.d/keepalived”檔案，即自啟指令碼，需要手工配置，前提是將啟動相關命令，設定檔等按指令碼定義的目錄放置；#啟動時，可能需要運行：systemctl daemon-reload再重啟keepalived[root@elk-node1 keepalived]# touch /etc/rc.d/init.d/keepalived[root@elk-node1 keepalived]# chmod +x /etc/rc.d/init.d/keepalived [root@elk-node1 keepalived]# vim /etc/rc.d/init.d/keepalived#!/bin/sh## keepalived   High Availability monitor built upon LVS and VRRP## chkconfig:   - 86 14# description: Robust keepalive facility to the Linux Virtual Server project \#              with multilayer TCP/IP stack checks.### BEGIN INIT INFO# Provides: keepalived# Required-Start: $local_fs $network $named $syslog# Required-Stop: $local_fs $network $named $syslog# Should-Start: smtpdaemon httpd# Should-Stop: smtpdaemon httpd# Default-Start: # Default-Stop: 0 1 2 3 4 5 6# Short-Description: High Availability monitor built upon LVS and VRRP# Description:       Robust keepalive facility to the Linux Virtual Server#                    project with multilayer TCP/IP stack checks.### END INIT INFO# Source function library.. /etc/rc.d/init.d/functionsexec="/usr/sbin/keepalived"prog="keepalived"config="/etc/keepalived/keepalived.conf"[ -e /etc/sysconfig/$prog ] && . /etc/sysconfig/$proglockfile=/var/lock/subsys/keepalivedstart() {    [ -x $exec ] || exit 5    [ -e $config ] || exit 6    echo -n $"Starting $prog: "    daemon $exec $KEEPALIVED_OPTIONS    retval=$?    echo    [ $retval -eq 0 ] && touch $lockfile    return $retval}stop() {    echo -n $"Stopping $prog: "    killproc $prog    retval=$?    echo    [ $retval -eq 0 ] && rm -f $lockfile    return $retval}restart() {    stop    start}reload() {    echo -n $"Reloading $prog: "    killproc $prog -1    retval=$?    echo    return $retval}force_reload() {    restart}rh_status() {    status $prog}rh_status_q() {    rh_status &>/dev/null}case "$1" in    start)        rh_status_q && exit 0        $1        ;;    stop)        rh_status_q || exit 0        $1        ;;    restart)        $1        ;;    reload)        rh_status_q || exit 7        $1        ;;    force-reload)        force_reload        ;;    status)        rh_status        ;;    condrestart|try-restart)        rh_status_q || exit 0        restart        ;;    *)        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"        exit 2esacexit $?#設定開機啟動[root@elk-node1 keepalived]# chkconfig --add keepalived[root@elk-node1 keepalived]# chkconfig --level 35 keepalived on[root@elk-node1 keepalived]# vim /usr/lib/systemd/system/keepalived.service#修改PIDFile，如下：PIDFile=/var/run/keepalived.pid 

5. Keepalived設定檔

[root@elk-node1 ~]# vim /usr/local/keepalived/etc/keepalived/keepalived.conf#=====================================================# keepalived.conf 配置#------------------------------------------------------------# 1、Keepalived 設定檔以block形式組織，每個塊內容都包含在{}# 2、“#”,“!”開頭行為注釋# 3、keepalived 配置為三類：#    (1)全域配置:對整個keepalived都生效的配置#    (2)VRRPD 配置:核心配置，主要實現keepalived高可用功能#    (3)LVS配置#=====================================================! Configuration File for keepalived#########################  全域配置######################### global_defs 全域組態識別;global_defs {# notification_email用於設定警示郵件地址; 可以設定多個,每行一個; 設定郵件警示需開啟本機Sendmail服務   notification_email {     root@localhost.local   }# 設定郵件發送地址, smtp server地址, 串連smtp sever逾時時間   notification_email_from root@localhost.local   smtp_server 10.11.4.151   smtp_connect_timeout 30# 表示運行keepalived伺服器標識，郵件發送時在主題中顯示的資訊   router_id Haproxy_DEVEL}#######################  服務檢測配置####################### 服務探測，chk_haproxy為服務名返回0說明服務是正常的    vrrp_script chk_haproxy {        script "/usr/local/keepalived/etc/chk_haproxy.sh"#每隔1秒探測一次        interval 1#haproxy線上,權重加2#        weight 2}#######################  VRRPD配置####################### VRRPD組態識別，VI_1是執行個體名稱vrrp_instance VI_1 {# 指定Keepalvied角色,MASTER(必須大寫)表示此主機為主伺服器,BACKUP則是表示為待命伺服器;# 這裡因為配置非搶佔模式,nopreempt只作用於BACKUP,將2台主機均配置為BACKUP    state BACKUP# 指定HA監測網路的介面    interface eth0# 虛擬路由辨別碼,標識為數字,1-255可選；# 同1個VRRP執行個體使用唯一的標識,MASTER_ID = BACKUP_ID    virtual_router_id 51# 定義節點優先順序,數字越大表示節點的優先順序越高;# 同1個VRRP_instance下，MASTE_PRIORITY > BACKUP_PRIORITY    priority 100# MASTER與BACKUP主機之間同步檢查的時間間隔,單位為秒    advert_int 1# 從實際應用角度,建議配置非搶佔模式,防止網路頻繁切換震蕩    nopreempt# 設定節點間通訊驗證類型與密碼，驗證類型主要有PASS和AH兩種；# 同1個vrrp_instance，MASTER驗證密碼和BACKUP保持一致    authentication {        auth_type PASS        auth_pass 987654    }# 設定虛擬IP地址(VIP),又叫做漂移IP地址;# 可設定多個，1行1個;# keepalived通過“ip address add”命令的形式將VIP添加到系統    virtual_ipaddress {        10.11.4.150    }# 指令碼追蹤,對應服務檢測    track_script {        chk_haproxy    }}############################################### LVS配置,這裡keepalived只做高可用，並不做lvs############################################### virtual_server LVS組態識別# 格式: virtual_server VIP port [IP 和 port 之間空格隔開] # virtual_server 10.11.4.150 443 {# 設定健全狀態檢查時間間隔，單位為秒#    delay_loop 6# 設定負載調度演算法，常用調度演算法是: rr、wlc，另有:lc、lblc、sh、dh等#    lb_algo rr# 設定LVS實現負載平衡的機制，有NAT、TUN和DR三種模式可選#    lb_kind NAT# 會話保持時間,其對動態網頁非常有用，為叢集系統中的seesion共用提供了一個很好的解決方案;# 使用者的請求會一直分發到某個服務節點，直至超過這個會話的保持時間（指最大無響應逾時時間），# 即使用者操作動態網頁面如果在50s沒有執行任何操作則被分發到另外的節點#    persistence_timeout 50# 轉寄協議類型#    protocol TCP# 設定real server段開始的標識 [ IP為真實IP地址]# 格式:real_server realIP port [IP 和 port 之間空格隔開]#    real_server 192.168.201.100 443 {# real server節點的權值,權值大小用數字表示,數字越大,權值越高#        weight 1# 健全狀態檢查 SSL_GET#        SSL_GET {# 指定SSL檢查的URL資訊,可以指定多個#            url { # 詳細的URL路徑#              path /index.html# SSL檢查後的摘要資訊,可以通過genhash命令工具擷取,命令如下:# [root@elk-node1 bin]# /usr/local/keepalived/bin/genhash -s 192.168.4.171 -p 80 -u /index.html#              digest ff20ad2481f97b1754ef3e12ecd3a9cc#            }#            url {#              path /mrtg/#              digest 9b3a0c85a887a256d6939da88aabd8cd#            }# 無響應逾時時間，單位為秒#            connect_timeout 3# 重試次數#            nb_get_retry 3# 稍候再試#            delay_before_retry 3#        }#    }#} 

6. Keepalived檢測指令碼

#檢測haproxy服務是否正常運行，如果沒有則嘗試拉起來，如果嘗試失敗則重啟keepalived服務，切換keepalived的vip[root@elk-node1 ~]# touch /usr/local/keepalived/etc/chk_haproxy.sh[root@elk-node1 ~]# chmod 755 /usr/local/keepalived/etc/chk_haproxy.sh[root@elk-node1 ~]# vim /usr/local/keepalived/etc/chk_haproxy.sh#!/bin/bash# check haproxy process, if there isn't any process, try to start the process once,# check it again after 3s, if there isn't any process still, restart keepalived process, change state.# 2017-03-22 v0.1 if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ]; then     /etc/rc.d/init.d/haproxy start     sleep 3     if [ $(ps -C haproxy --no-header | wc -l) -eq 0 ]; then          /etc/rc.d/init.d/keepalived restart     fifi# another method to check haproxy process#killall -0 haproxy #if [[ $? -ne 0 ]];then #  /etc/rc.d/init.d/keepalived restart #fi

三．驗證 1. 啟動

[root@elk-node1 ~]# service keepalived start[root@elk-node2 ~]# service keepalived start 

2. 查看日誌1）Node1

[root@elk-node1 ~]# tailf /var/log/messages

2）Node2 

[root@elk-node2 ~]# tailf /var/log/messages

3. VIP

#使用的是"ip address add"添加的vip到系統中，因"ifconfig"命令看不到效果[root@elk-node1 ~]# ip address show eth0

Node1的網卡eth0已經獲得vip 10.11.4.150。 

4. 故障切換1）Haproxy故障拉起

[root@elk-node1 ~]# date ; service haproxy stop[root@elk-node1 ~]# date ; service haproxy status

2）Node1日誌 

 

3）Node2日誌 

4）Node2 VIP

[root@elk-node2 ~]# ip address show eth0

Node2的網卡eth0已經獲得vip 10.11.4.150。