Linux load Balancing concept and practice (i)

According to the online article collation.

Load balancer software LVs (Linux Virtual Server) concept paper

LVS is based on the Linux operating system to establish a virtual server, to achieve the load balance between service nodes. It is implemented based on the Linux kernel. The 2.6.X kernel is integrated with the LVS module by default.

The implementation of common load balancing for LVS is based on IP protocol, so it is generally called Ipvs.

Ipvs is essentially an efficient Layer-4 switch that provides load balancing capabilities. When the initial SYN message arrives for a TCP connection, Ipvs selects a server and forwards the message to it. Thereafter, by checking the IP and TCP header addresses of the text, the subsequent messages of this connection are forwarded to the same server. In this way, Ipvs cannot check the requested content and then select the server, which requires the backend server group to provide the same service, regardless of the request is sent to which server, the return result should be the same. However, in some applications, the backend servers may be functionally different, either a Web server that provides HTML documents, a Web server that provides images, or a Web server that provides CGI. At this point, you need to base content request distribution (content-based request distribution), while content-based request distribution can improve the locality of access on the back-end server.

Ipvs is the core software of the LVS cluster system, its main function is to install on the director server, while > virtual an IP address on the director server, the user must access the service through this virtual IP address. This virtual IP is generally called the LVS VIP, namely virtual IP. The requests that are accessed first go through the VIP to the load scheduler, and then the load Scheduler picks a service node from the real server list to respond to the user's request.

The goal of using LVS technology is to achieve a high-performance, highly available server cluster with the load balancing technology provided by LVS and the Linux operating system, which has good reliability, scalability and operability. Thus, the optimal service performance can be achieved at low cost.

Architecture:

The server cluster system with LVS is composed of three parts: the most front-end load balancer layer, represented by load balancer, the middle server group layer, with server array, the bottom of the data sharing storage layer, with shared storage, in the view of users, All internal applications are transparent and users are using only the high-performance services provided by a virtual server.

The LVS components are described in detail:

? Load Balancer layer: At the forefront of the entire cluster system, there is one or more load scheduler (Director server), the LVS module is installed on the director server, and director's main role is similar to a router, It contains the routing tables set up to complete the LVS function, which distribute the user's requests to the application server (Real server) at the server array level through these routing tables. Also, on the director server, you install the Monitoring module Ldirectord for the real Server service, which is used to monitor the health status of each real Server service. When real server is unavailable, remove it from the LVS routing table and rejoin it upon recovery.

? Server Array layer: Consists of a set of machines that actually run the app service, one or more of the Web server, mail server, FTP server, DNS server, video server, and each real Servers are connected to each other over a high-speed LAN or across a WAN. In a real-world application, Director server can also be the role of real server concurrently.

? Shared storage layer: is a storage area that provides shared storage space and content consistency for all real servers, physically consisting of disk array devices and, in order to provide consistency of content, can generally share data via NFS Network file systems. But NFS in a busy business system, performance is not very good, at this time can use the cluster file system, such as Red Hat GFs file system, Oracle provides the OCFS2 file system and so on.

As can be seen from the entire LVS structure, director server is the core of the entire LVS, currently, the operating system for director server can only be Linux and FreeBSD, The linux2.6 kernel can support LVS without any setup, and FreeBSD as a director server is not a lot of applications, performance is not very good. For real Server, almost all system platforms, Linux, Windows, Solaris, AIX, BSD series can be very well supported.

Load balancing and load scheduling algorithm

1. There are many implementations of IP load balancing technology, such as the method of DNS domain name rotation resolution, the method based on client scheduling access, the scheduling method based on application layer system load, and the scheduling method based on IP address, in which the highest execution efficiency is the IP load balancing technology.

The IP load balancing technology of LVS is realized by Ipvs module, Ipvs is the core software of LVS cluster system, its main function is: Install on Director server, and virtual an IP address on Director server. The user must access the service through this virtual IP address. This virtual IP is generally called the LVS VIP, namely virtual IP. The requests that are accessed first go through the VIP to the load scheduler, and then the load Scheduler picks a service node from the real server list to respond to the user's request.

When a user's request arrives at the load scheduler, how the scheduler sends the request to the real server node that provides the service, and how the real server node returns the data to the user, is the key technology implemented by Ipvs.

There are three types of load balancing mechanisms Ipvs, NAT, Tun, and Dr, which are detailed as follows:

? Vs/nat: That is (virtual server via network address translation), which is the translation technology of Web addresses to implement a virtualized server, when the user requests to reach the scheduler, The scheduler overwrites the destination address of the request message (that is, the virtual IP address) to the selected real server address, while the destination port of the message is also changed to the corresponding port of the selected real server, and finally the message request is sent to the selected real server. After the data is obtained on the server side, when Real server returns the data to the user, it needs to go through the load scheduler again to change the source address and source port of the message to the virtual IP address and the corresponding port, then send the data to the user to complete the load scheduling process. As can be seen, in the NAT mode, the user request and response messages must be rewritten by the Director server address, when the user requests more and more time, the scheduler's processing power will be called bottlenecks.

? Vs/tun: That is, (virtual server via IP tunneling) is the IP tunneling technology to implement a virtualized server. Its connection scheduling and management is the same as the Vs/nat way, but its message forwarding method is different, Vs/tun mode, the Scheduler uses IP tunneling technology to forward user requests to a real server, and this real server will directly respond to the user's request, no longer through the front-end scheduler, In addition, there is no requirement for the GEO location of the real server, either in the same network segment as the director server or as a standalone network. Therefore, in the Tun mode, the scheduler will only process the user's message request, the throughput of the cluster system is greatly improved.

? VS/DR: That is (virtual server via direct Routing), that is, the use of direct routing technology to implement virtualized servers. Its connection scheduling and management is the same as in Vs/nat and Vs/tun, but its message forwarding method is different, vs/dr by overwriting the request message's MAC address, send the request to real server, and real server to return the response directly to the customer, eliminating the vs/ The IP tunneling overhead in the Tun. This is the best performance in three load scheduling mechanisms, but it must be required that both the Director server and the real server have a NIC attached to the same physical network segment.

Load Balancing scheduling algorithm

Ipvs implements the following eight load scheduling algorithms:

? Call scheduling: RR (Round Robin) "Round call" dispatch is also called 1:1 scheduling, the scheduler through the "round call" scheduling algorithm to the external user request in order 1:1 to each real server in the cluster, this algorithm treats each real server equally, Regardless of the actual load condition and connection status on the server.

? Weighted round call scheduling: WRR (Weighted Round Robin) The "Weighted round call" scheduling algorithm dispatches access requests based on the different processing capabilities of real server. You can set different scheduling weights for each real server, and for a relatively good real server, you can set a higher weight, and for a less powerful real server, you can set a lower weight value, which ensures that the processing power of the server handles more traffic. The server resources are utilized fully and rationally. At the same time, the scheduler can automatically query the real server load situation, and dynamically adjust its weight value.

? Minimum link Dispatch: LC (Least Connections) "Least-connection" scheduling algorithm dynamically dispatches network requests to servers with the fewest number of established links. If the real server of the cluster system has similar system performance, the "Minimum connection" scheduling algorithm can be used to balance the load better. Weighted least-Link scheduling: WLC (Weighted Least Connections) "Weighted least-link scheduling" is a superset of "minimum connection scheduling", each service node can use corresponding weights to represent its processing power, and the system administrator can dynamically set the corresponding weights, the default value is 1, The weighted minimum connection schedule is proportional to the number of established connections to the service node and its weights as much as possible when assigning new connection requests.

? Minimum links based on locality: LBLC (locality-based Least Connections) "Least-link based on locality" scheduling algorithm is a load balancing target IP address, and is mainly used in cache cluster system. According to the target IP address of the request, the algorithm finds the most recently used server, if the server is available and not overloaded, sends the request to the server, if the server does not exist, or if the server is overloaded and has half of the workload of the server, the principle of "least link" is used to select an available server. , the request is sent to the server.

? Local least-Link with replication: LBLCR (locality-based Least Connections with Replication) the "Local least-link with replication" Scheduling algorithm is also load balanced against the destination IP address, It is mainly used in cache cluster system. It differs from the LBLC algorithm in that it maintains a mapping from a destination IP address to a set of servers, while the LBLC algorithm maintains a mapping from a destination IP address to a server. According to the target IP address of the request, the algorithm finds the corresponding server group of the target IP address, selects a server from the server group according to the principle of "minimum connection", if the server is not overloaded, sends the request to the server, if the server is overloaded, select a server from this cluster according to the "minimum connection" principle. Join the server to the server group and send the request to the server. Also, when the server group has not been modified for some time, the busiest server is removed from the server group to reduce the degree of replication.

The target address hash: DH (Destination Hashing) "Destination address hash" Scheduling algorithm based on the requested destination IP address, as a hash key (hash key) from the static distribution of the hash list to find the corresponding server, if the server is available and not overloaded, send the request to the server , otherwise the return is empty.

The source address hash: sh (source Hashing) "Source address hash" Scheduling algorithm based on the requested source IP address, as a hash key (hash keys) from the static distribution of the hash list to find the corresponding server, if the server is available and not overloaded, send the request to the server, otherwise return empty.

Ubuntu LVS Dr Mode Practice Deployment

1. Environmental description

System version: ubuntu14.04 LTS

LVS Server: 192.168.1.107

Real server: 192.168.1.106,192.168.1.108

vip:192.168.1.70

Deployment Purpose: Users request 192.168.1.70 messages to be forwarded to 192.168.1.106 and 192.168.1.108 for load balancing.

2. Installing LVS

Since LVS work like iptables in the kernel layer, so only need to install the module ip_vs on it, and no background process in running

#aptitude Install Ipvsadm

# ipvsadm-v Ipvsadm v1.26

# lsmod |grep Ip_vs #查看内核模块, there is a representative ipvsadm loaded into the kernel, when the LVS installation is complete

Ip_vs 136701 0

Nf_conntrack 96976 1 Ip_vs

LIBCRC32C 12644 3 Xfs,btrfs,ip_vs

# ipvsadm-l-N #查看lvs规则

IP Virtual Server version 1.2.1 (size=4096)

Prot Localaddress:port Scheduler Flags

Remoteaddress:port Forward Weight activeconn inactconn

# ipvsadm-l-C #查看连接情况

IPVS Connection Entries

Pro Expire State source virtual destination

3. Configure LVS VIP Service

-A is to add a virtual server record, which is the VIP. Here you configure the algorithm to select.

#ipvsadm-a-t 192.168.1.70-s wrr-p #代理转发所有去往该IP TCP Message,-s specifies the algorithm WRR for the weighted polling algorithm,-p session hold Time (default value is 300s)

#ipvsadm-a-t 192.168.1.70:8888-s wrr-p #代理转发去往该IP TCP8888 Port messages

VIP Maintenance:

Use-E to modify VIP settings

Example: Ipvsadm-e-t 192.168.1.70-s rr-p #可修改会话保持时间与算法.

Use-D to remove VIP settings

Example: ipvsadm-d-T 192.168.1.70 #删除该条VIP

4. Configuring LVS RIP Rules

-A is to add a VIP to which real server IP (that is, RIP) is loaded (by default Dr mode is-g).

Ipvsadm-a-T 192.168.1.70-r 192.168.1.106-g-W 1 #-r Specify real server ip,-g to select Dr Mode (-M for Nat mode,-T for tunnel mode),-W to select weights. Ipvsadm-a-T 192.168.1.70-r 192.168.1.108-g-W 1

.........

#如果只对tcp 80 port for load, use the following command

#ipvsadm-a-t 192.168.1.70:8888-r 192.168.1.106:8888-g-W 1

RIP maintenance:

modifying RIP configurations with-E

Example: Ipvsadm-e-T 192.168.1.70-r 192.168.1.106-g-W 4 #修改这条RIP规则的权重. Delete RIP configuration using-D

Example: ipvsadm-d-T 192.168.1.70-r 192.168.1.106-g-W 1 #删除该条RIP配置

or use a script (executed on LVS)

vim/root/lvs.sh

#!/bin/bash

vip=192.168.1.70

rs1=192.168.1.106

rs2=192.168.1.108

#clear Ipvs Tables Ipvsadm-c

#set LVS Server

Ipvsadm-a-T $VIP: 8888-s RR

Ipvsadm-a-T $VIP: 8888-r $rs 1:8888-g-W 1 #-w for weights, defaults to 1

Ipvsadm-a-T $VIP: 8888-r $rs 2:8888-g #不加-W 1 effect same sentence

# update/etc/sysctl.conf, the following configuration will be restored after reboot

echo "1" >/proc/sys/net/ipv4/ip_forward #启用ip转发

echo "1" >/proc/sys/net/ipv4/conf/all/send_redirects

echo "1" >/proc/sys/net/ipv4/conf/default/send_redirects

echo "1" >/proc/sys/net/ipv4/conf/eth0/send_redirects

#sysctl-P #查看sysctl的配置变化, can not be executed

5, configure the VIP on the real server (real server) is configured on the VIP, that is, the 192.168.1.70 address.

#ifconfig lo:0 192.168.1.70 netmask 255.255.255.255 broadcast 192.168.1.70 up

6. Turn off the ARP response on the real server

Turn off the ARP response from the loop on the real server without shutting down the other ports.

The default Arp_ignore and Arp_announce parameters are 0, that is, the ARP request that responds to all destination IPs is native.

On all real servers, do the following:

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore #只回答目标IP地址是来访网络接口本地地址的ARP查询请求.

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce #对查询目标使用最适当的本地地址.

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

or with script vim/root/rs.sh (performed on Real server)

#!/bin/bash

vip=192.168.1.70

Ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP up #网关要和虚ip一样, the mask cannot be the same as the eth0

Route add-host $vip Dev lo:0 #添加永久路由

#echo "0" >/proc/sys/net/ipv4/ip_forward #关闭real server IP Forwarding, 2.6.* kernel is off by default

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

#sysctl-P #查看sysctl的配置变化, can not be executed

7, turn on the kernel forwarding function (if you have performed lvs.sh this step can be omitted)

Turn on the kernel forwarding function on the LVS.

#echo ' net.ipv4.ip_forward=1 ' >>/etc/sysctl.conf

#sysctl-P

8. LVS Test

Because the HTTP service is already set up on the real server, the LVS that just sets up is forwarding all TCP connections,

192.168.1.70 can be accessed directly.

http://192.168.1.70:8888

Run on LVs

#ipvsadm-L-C #查看lvs转发记录 to determine if the request was forwarded successfully.

Pro Expire State source virtual destination

TCP 00:05 NONE 192.168.1.98:0 192.168.1.70:8888 192.168.1.108:8888

TCP 01:20 fin_wait 192.168.1.98:55936 192.168.1.70:8888 192.168.1.108:8888

The LVS was built successfully.

9, LVs Supplementary thinking

1, you can not live on the real server, if the server down situation, will also forward the request to down the server. The workaround is to keepalived or self-write scripts to monitor connectivity and then purge the corresponding rules. 2, one LVS server will be a single point of failure, need to have a backup LVS server solution is keepalived to carry out LVS fault takeover.

Ipvsadm Command Parameters

-A (--add-service) adds a new virtual IP record to the list of virtual servers in the kernel. That is, add a new virtual server. Virtual IP is the IP address of the virtual server.

-E (--edit-service) edit a virtual server record in the list of kernel virtual servers

-D (--delete-service) Delete a virtual server record from the list of kernel virtual servers

-C (--clear) clears all records in the kernel virtual server list

-R (--restore) restore virtual Server rules

-S (--save) Save virtual Server rule, output to-r option readable format

-A (--add-server) adds a new real server record to a record in the kernel virtual server list. That is, adding a new real server to a virtual server

-E (--edit-server) edit a real server record in a virtual server record

-D (--delete-server) deletes a real server record in a virtual server record

-l|-l–list displaying a list of virtual servers in the kernel

-Z (--zero) Virtual Server list counter clear 0 (empty the current number of connections, etc.)-

-set TCP tcpfin UDP setting Connection timeout value

-T describes the TCP service provided by the virtual server, which is followed by the following format: [Virtual-service-address:port] or [Real-server-ip:port]

-U indicates that the virtual server provides the UDP service, which is followed by the following format: [Virtual-service-address:port] or [Real-server-ip:port]

-F Fwmark Description is a service type that has been iptables marked

-s The scheduling algorithm used with LVS after this option has several options: Rr|wrr|lc|wlc|lblc|lblcr|dh|sh the default scheduling algorithm is: WLC

The continuous service time on a real server for-p [timeout]. This means that multiple requests from the same user will be processed by the same real server. This parameter is typically used for operations with dynamic requests, and the default value for timeout is 300 seconds. For example:-P 600, which indicates a continuous service time of 600 seconds.

-r Specifies the IP address of the real server, which is followed by the following format: [Real-server-ip:port]

-G (--gatewaying) specifies that the LVS work mode is direct route mode (this mode is the LVS default operating mode)

-I (-IPIP) specifies the operating mode of the LVS for tunnel mode

-M (--masquerading) specifies the operating mode of LVS for NAT mode

-W (--weight) weight Specifies the weight of the real server

-C (--connection) displays LVS current connection information such as: Ipvsadm-l-C

-L--timeout Displays the timeout value for "TCP Tcpfin UDP" such as: Ipvsadm-l--timeout

-L--daemon Displays the status of the synchronization daemon, for example: Ipvsadm-l–daemon

-L--stats Displays statistics, for example: Ipvsadm-l–stats

-L--rate Display rate information, for example: Ipvsadm-l--rate

-L--sort sort the output on the virtual server and the real server, for example: Ipvsadm-l–sort

Linux load Balancing concept and practice (i)