Linux LVS Introduction

Lvs:linux Virtual server,4 layer router, compared to 7 layer agent, 4 layer agent process is not limited by the number of ports, because not to the application layer, Ipvs on the input link to the strong forwarding message to the route, there is a website test can reach 400w concurrency. Similar to the iptables configuration, the LVS framework is Ipvs, the Rule tool ipvsadm.

Scheduling

• static method: Scheduling only according to the algorithm itself;

  1. Rr:roundrobin, polling;
  2. Wrr:weighted RR, weighted polling;
  3. Sh:source Hashing, the realization session sticky, the source IP address hash; the request from the same IP address is always sent to the first pick of the RS, thus realizing the conversation binding;
  4. Dh:destination Hashing; the target address hash, the request to the same destination address is always forwarded to the first pick of the RS, the typical usage scenario is the load balancing in the forward proxy cache scenario;
• Dynamic common methods: mainly based on the current load status and scheduling algorithm for each RS;
  1. Lc:least Connections:overhead=activeconns*256+inactiveconns algorithm: An active process overhead is 256 times times the cost of the inactive process (this is just a ballpark and rationale value)
  2. wlc:weighted LC #默认算法, overhead= (Activeconns*256+inactiveconns)/weight
  3. Sed:shortest expection Delay #不考虑非活动资源 arithmetic Understanding, overhead= (activeconns+1) *256/weight
  4. Nq:never Queue #SED升级, weight 1, 102 units, the first 10 will take the weight 10 of that one, and the other one does not have to deal with, NQ is to deal with the situation.

Nat Mode

Work Flow chart

Preparation conditions:

vs:node1rs:node2, node3, node4node1:      172.16.86.249   #作为私网192.168.1.200   # 作为公网node2:172.16.86.250    网关 172.16.86.249node3:172.16.86.248    网关 172.16.86.249node4 172.16.86.251    网关 172.16.86.249

1. VS Installation Ipvsadm

[[email protected] ~]# yum install ipvsadm

2. Service Management

#添加#ipvsadm -A|E -t|u|f service-address [-s scheduler] [-p [timeout]] #-t: TCP协议的端口，VIP:TCP_PORT #-u: UDP协议的端口，VIP:UDP_PORT #-f：firewall MARK，是一个数字；#______________________________________________________________________________[[email protected] ~]# ipvsadm -A -t 192.168.1.200:80 -s rr#修改[[email protected] ~]# ipvsadm -E -t 192.168.1.200:80 -s wrr#删除[[email protected] ~]# ipvsadm -D -t 192.168.1.200:80

3. Node Management

#增改#ipvsadm -a|e -t|u|f service-address -r server-address [-g|i|m] [-w weight]#lvs类型：    #-g: gateway, dr类型    #-i: ipip, tun类型    #-m: masquerade, nat类型# -w weight：权重；#______________________________________________________________________________[[email protected] ~]# ipvsadm -a -t 192.168.1.200:80 -r 172.16.86.250 -m     #可以在rs的ip后面加端口  默认是把前面的端口映射到后面的端口[[email protected] ~]# ipvsadm -a -t 192.168.1.200:80 -r 172.16.86.248 -m[[email protected] ~]# ipvsadm -a -t 192.168.1.200:80 -r 172.16.86.251 -m#删：#ipvsadm -d -t|u|f service-address -r server-address[[email protected] ~]# ipvsadm -d -t 172.16.86.249:80 -r 172.16.86.251

3. View

[[email protected] ~]# ipvsadm-lnip Virtual Server version 1.2.1 (size=4096) Prot localaddress:port Scheduler Flags   -Remoteaddress:port Forward Weight weight activeconn The number of Inactconn inactive connections being connected TCP 192.168.1.200:80 WLC #默认是wlc调度    -172.16.86.248:80 MASQ 1 0 0-172.16.86.250:80 masq 1 0 0-172.16.86.251:80 masq 1 0 0[[em               Ail protected] ~]# ipvsadm-ln--statsip Virtual Server version 1.2.1 (size=4096) Prot Localaddress:port               Conns connections inpkts Number of packets Outpkts stack messages inbytes into stack bytes outbytes out of stack bytes remoteaddress:porttcp 192.168.1.200:80 282989 1708574 1419365 115244K 141336K-172.16.86.248:80 169787 1026019 852945 69195245 847                05145, 172.16.86.250:80 56599 341318 281848 23003003 28178841-172.16.86.251:80 56603 341237 284572 23046538 28452109[[email protected] ~]# watch-n.1 ' ipvsadm-ln--rate ' IP Virtual Server version 1.2.1 (size=4096 ) Prot Localaddress:port CPS establishes connections per second Inpps number of messages per second Outpps inbps bytes per second outbps remoteaddress:p                 Orttcp 192.168.1.200:80 1699 10176 8482 686755 850122-172.16.86.248:80  1019 6106 5089 412064 508979-172.16.86.250:80 340 2035 1696 137359 170579 -172.16.86.251:80 340 2035 1696 137332 170564

4, first to test the RR polling algorithm

[[email protected] ~]# ipvsadm -E -t 192.168.1.200:80 -s rr[[email protected] ~]# curl http://192.168.1.200/node4[[email protected] ~]# curl http://192.168.1.200/node3[[email protected] ~]# curl http://192.168.1.200/node2

Wrr

[[email protected] ~]# ipvsadm -E -t 192.168.1.200:80 -s wrr[[email protected] ~]# ipvsadm -e -t 192.168.1.200:80 -r 172.16.86.248 -m -w 3[[email protected] ~]# ipvsadm -LnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight  ActiveConn InActConnTCP  192.168.1.200:80 wrr  -> 172.16.86.248:80             Masq    3      0          1           #node3  -> 172.16.86.250:80             Masq    1      0          1           #node2  -> 172.16.86.251:80             Masq    1      0          3           #node4[[email protected] ~]# curl http://192.168.1.200/node4[[email protected] ~]# curl http://192.168.1.200/node3[[email protected] ~]# curl http://192.168.1.200/node2[[email protected] ~]# curl http://192.168.1.200/node3[[email protected] ~]# curl http://192.168.1.200/node3

5. Save the Rules

#查看rpm包中的脚本 [[email protected] ~]# cat/usr/lib/systemd/system/ipvsadm.service[unit]description=initialise the Linux Virtual serverafter=syslog.target network.target[service]type=oneshotexecstart=/bin/bash-c "exec/sbin/ Ipvsadm-restore </etc/sysconfig/ipvsadm "Execstop=/bin/bash-c" Exec/sbin/ipvsadm-save-n >/etc/sysconfig/ Ipvsadm "execstop=/sbin/ipvsadm-cremainafterexit=yes[install]wantedby=multi-user.target# Save the rules to the configuration file [email  protected] ~]# ipvsadm-s-n >/etc/sysconfig/ipvsadm empty [[email protected] ~]# ipvsadm-c[[email  Protected] ~]# Ipvsadm-lnip Virtual Server version 1.2.1 (size=4096) Prot localaddress:port Scheduler Flags-Remotea Ddress:port Forward Weight activeconn inactconn# Reload [[email protected] ~]# Ipvsadm-r </ETC/SYSCONFIG/IPV Sadm[[email protected] ~]# ipvsadm-lnip Virtual Server version 1.2.1 (size=4096) Prot localaddress:port Scheduler Fl AGS-Remoteaddress:port Forward Weight activeconn inactconnTCP 192.168.1.200:80 WRR-172.16.86.248:80 MASQ 3 0 0-172.16.86.250:80 MASQ 1 0 0-172.16.86.251:80 masq 1 0 0

Dr Model

Work Flow chart

Limit response level: Arp_ignore
0: Default value, indicating that any address configured on the local interface can be used to respond;
1: The response is only given when the requested destination IP is configured on the receiving request message interface of the local host;
Limit notification Level: Arp_announce
0: The default value, all the interface on this computer all the information to the network on each interface to advertise;
1: As far as possible to avoid the non-local connection network to advertise;
2: It is necessary to avoid advertising to the non-network;

One of these two parameters is the notification setting, which prevents external connections, and one is the response setting that prevents outgoing

Experiment Preparation:

node1: dip192.168.1.200   vip:192.168.1.205  vsnode2: 192.168.1.201  vip:192.168.1.205   rsnode3: 192.168.1.202   vip:192.168.1.205  rs

Node1 Configure VIP? If the VIP equals dip, the broadcast domain below is not your own

[[email protected] ~]# ip addr add 192.168.1.205/32 broadcast 192.168.1.205 dev ens34:0[[email protected] ~]# ip addr delete 192.168.1.205/32 broadcast 192.168.1.205 dev ens34:0

Rs:node2 NODE3 Configuration

#!/bin/bash#vip=192.168.1.205mask=‘255.255.255.255‘case $1 instart)    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce    ip addr add  $vip/32  broadcast $vip  dev lo:0    #发往vip的报文，必须要从lo：0 出去   所以这个lo：0网卡是用来发报文的，相当于 修改 源ip    ip route add  $vip dev lo:0    ;;stop)    ip addr del $vip/32 dev lo:0    ip route delete $vip dev lo:0    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce    ;;*)     echo "Usage $(basename $0) start|stop"    exit 1    

Vs:

[[email protected] ~]# ipvsadm -A -t 192.168.1.205:80 -s rr#这里添加节点还有另外一个意义：就是通知本机 rs节点会有vip，你只要把包发送回去就行[[email protected] ~]# ipvsadm -a -t 192.168.1.205:80 -r 192.168.1.201 -g[[email protected] ~]# ipvsadm -a -t 192.168.1.205:80 -r 192.168.1.202 -g

View

[[email protected] ~]# ipvsadm -lnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight ActiveConn InActConnTCP  192.168.1.205:80 rr  -> 192.168.1.201:80             Route  路由类型   1      0          0  -> 192.168.1.202:80             Route   1      0          0

Test

marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/node3marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/node2marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/node3marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/node2

Packet Capture analysis

[[email protected] ~]#  tcpdump -i any -nn    port 80#响应在rs节点  [[email protected] ~]#  tcpdump -i any -nn    port 8010:18:54.367851 IP 192.168.1.205.80 > 192.168.1.104.55631: Flags [P.], seq 1:235, ack 78, win 227, options [nop,nop,TS val 352696164 ecr 945230025], length 234: HTTP: HTTP/1.1 200 OK

VS Configuration Script

#!/bin/bashvip=‘192.168.1.205‘iface=‘ens34:0‘mask=‘255.255.255.255‘port=‘80‘rs1=‘192.168.1.201‘rs2=‘192.168.1.202‘scheduler=‘wrr‘type=‘-g‘case $1 instart)    ip addr add $vip/32 broadcast $vip dev $iface    iptables -F    ipvsadm -A -t ${vip}:${port} -s $scheduler    ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1    ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1    ;;stop)    ipvsadm -C    ip addr delete $vip/32  broadcast $vip dev $iface    ;;*)    echo "Usage $(basename $0) start|stop"    exit 1    ;;esac

Tun model

Work Flow chart

Fullnat model

Work Flow chart

Marking multiple services as a service

iptables -t mangle -A PREROUTING -d 192.168.1.200 -p tcp -m multoport --dports 80,443 -j MARK --set-mark 3   #标记3 随意iptables -t mangle -vnLipvsadm -A -f 3 s sh   #对标记3的报文定义一个服务ipvsadm -a -f 3 -r 192.168.1.201 -gipvsadm -a -f 3 -r 192.168.1.202 -g

Persistent connections

-P for persistent connections
Per port persistence: Each port is defined as a Cluster service, and each cluster service is scheduled separately;
Ipvsadm-a 192.168.1.200:80-s Rr-p
Each firewall tag is persistent: Define the Cluster service based on the firewall tag, can realize the unified dispatch of the application on multiple ports, namely the so-called Port Affinity;
Ipvsadm-a-F 3-s rr-p
Per-Client persistence: Define the Cluster service based on 0 port, and the client will dispatch all application requests to the backend host, which must be defined as persistent mode;
Ipvsadm-a-T 192.168.1.200:0-S RR-P
Ipvsadm-a-T 192.168.1.200:0-R 192.168.1.201-G
Ipvsadm-a-T 192.168.1.200:0-R 192.168.1.202-G

Monitoring (starting) Ldirectord

Ldirectord function: Check whether the RS server is broken, if broken, ipvsadm do delete, if bad repair will do ipvsadm add
Configuration Script Description

checktimeout=3                  超时时常checkinterval=1                 每个一秒检查一次，可以设置长点，减少压力fallback=127.0.0.1:80           如果real server全跪了，本机提供autoreload=yes                   配置文件发生修改，自动加载logfile="/var/log/ldirectord.log"quiescent=novirtual=5                       #防火墙标记  ip:port 的话是正常标记    real=172.16.0.7:80 gate 2    real=172.16.0.8:80 gate 1    fallback=127.0.0.1:80 gate    #如果rs都坏了，启用这项    service=http           #用http协议方式监测，关闭这个选项，会用四层方式探测    scheduler=wrr    checktype=negotiate  #谈判方式，不是一次击毙    checkport=80    request="index.html"        receive="CentOS"    #index.html中带有CentOS

Installation configuration

[[email protected] packages]# wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network :/ha-clustering:/stable/centos_centos-6/x86_64/ldirectord-3.9.5-3.1.x86_64.rpm[[email protected] Packages] # yum Install ldirectord-3.9.5-3.1.x86_64.rpm[[email protected] packages]# rpm-ql ldirectord/etc/ha.d/etc/ha.d/ resource.d/etc/ha.d/resource.d/ldirectord/etc/init.d/ldirectord/etc/logrotate.d/ldirectord/usr/lib/ocf/ resource.d/heartbeat/ldirectord/usr/sbin/ldirectord/usr/share/doc/ldirectord-3.9.5/usr/share/doc/ ldirectord-3.9.5/copying/usr/share/doc/ldirectord-3.9.5/ldirectord.cf/usr/share/man/man8/ldirectord.8.gz[[ Email protected] packages]# cp/usr/share/doc/ldirectord-3.9.5/ldirectord.cf/etc/ha.d/[[email protected ] packages]# vim/etc/ha.d/ldirectord.cf# Global directiveschecktimeout=3checkinterval=1#fallback=127.0.0.1:80# Fallback6=[::1]:80autoreload=yeslogfile= "/var/log/ldirectord.log" #logfile = "local0" #emailalert = "[email  Protected] "#emailalertfreq=3600#emailalertstatus=allquiescent=novirtual=192.168.1.205:80 real=192.168.1.201:80 Gate real=192.168. 1.202:80 Gate fallback=127.0.0.1:80 gate# service=http scheduler=rr #persistent =600 #net       mask=255.255.255.255 protocol=tcp checktype=negotiate checkport=80# request= "index.html" # receive= "Test page" # virtualhost=www.x.y.z

Start the service

[[email protected] packages]# /etc/init.d/ldirectord startStarting ldirectord (via systemctl):                       [  OK  ][[email protected] packages]# ipvsadm -lnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight ActiveConn InActConnTCP  192.168.1.205:80 rr  -> 192.168.1.201:80             Route   1      0          0  -> 192.168.1.202:80             Route   1      0          0marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/node3marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/node2

Stop a Node

[[email protected] ~]# systemctl stop mynginx[[email protected] packages]# ipvsadm -lnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight ActiveConn InActConnTCP  192.168.1.205:80 rr  -> 192.168.1.201:80             Route   1      0          6marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/node2marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/node2

Stop all, the local service will start automatically

[[email protected] packages]# ipvsadm -lnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight ActiveConn InActConnTCP  192.168.1.205:80 rr  -> 127.0.0.1:80                 Route   1      0          0marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/sorrymarvindeMacBook-Pro:~ marvin$ curl http://192.168.1.205/sorry

Enable one, return to normal

[[email protected] ~]# systemctl start mynginx[[email protected] packages]# ipvsadm -lnIP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags  -> RemoteAddress:Port           Forward Weight ActiveConn InActConnTCP  192.168.1.205:80 rr  -> 192.168.1.201:80             Route   1      0          0

Linux LVS Introduction