Layer-4 load balancing-LVS

LVS   Reference: http://zh.linuxvirtualserver.org/  Several terms:Director: Also known as scheduler, LVS front-end device;

RealServer: a real internal server is actually providing services;

VIP: The public IP address, that is, the IP Address requested by the customer;

Dip: the address for communication between the scheduler and the RealServer;

LVS has three working modes: LVS implements Server Cluster load balancing in three modes: Nat, Dr, And Tun. The differences between these three methods are briefly described below:   LVS-NAT:
The idea of this method is to implement network layer (IP layer) Data spoofing, which replaces the destination address sent from the client to the redirector data IP package.
1. In the network environment, one ctor + N realservers. Director and RealServer are in the same private network segment. Director is the default gateway of RealServer. Only Director has public IP addresses and can be exposed on the wide area network. 2. The client requests a public IP (Director) first. the IP address of the IP packet in the request packet is replaced with the IP address of a RealServer selected by director according to the load balancing policy.

3. RealServer response
After processing the request, the RealServer generates a returned data packet. The source address of the returned data packet is the IP address of the RealServer, And the destination address is the IP address of the client server. Because the default gateway of RealServer is redirector, although the destination address of the returned IP packet is the IP address of the client, the returned data packet is first sent back to redirector. Redire implements spoofing again and changes the source address of the returned IP packet to its own IP address. Then, it is forwarded to the vswitch and returned to the client. The redirector task is to implement two IP layer spoofing modifications, one is to modify the target address of the request data packet, the purpose of this modification is to achieve Load Balancing distribution of data. The other is to modify the source address of the response packet to hide the RealServer, so that the user does not feel the existence of the RealServer.
3. Restrictions: the throughput of the entire cluster is limited by the bandwidth of redirector (mainly the egress bandwidth ).

LVS-DR:
The idea of this method is to implement data spoofing at the data link layer and modify the MAC address of the network frame data.
1. Network Environment
One ctor + N realservers, Director and RealServer both have public IP addresses and are exposed on the wide area network. In addition, RealServer also has an IP alias that is the same as the Director IP address. In other words, the RealServer has two IP addresses, a real IP address, and an IP alias (that is, a public IP address) that is the same as the direcotr address. A public IP address is the IP address opened to the client.
RealServer also needs to make a configuration so that they ignore all ARP broadcasts targeting public IP addresses. When the system ARP broadcast asks which MAC address has a public IP address, only the scheduling server will respond, data sent from outside will not be sent to the actual server.

2. Client Requests
Client requests are first sent to the Public IP address (director). Because the network environment has been configured to have only Director responded to ARP broadcasts, the MAC address of the request packet will be changed to the MAC address of the RealServer. That is to say, redireimplements link layer spoofing to replace the target MAC address of the frame data with the MAC address of a RealServer determined based on the load balancing policy.

3. RealServer response
The RealServer receives the MAC frame and then assembles the MAC frame into an IP package. It finds that the destination address in the IP package is the same as its own IP address alias and there is no problem. Continue with the process, (This is why the IP alias must be the same. If the IP alias is different, the operating system may directly ignore it.) generate a response data and send it back. At this time, because redirector is not the default gateway, the data will be directly sent to the wide area network, and the WAN will send the data to the client.

4. Advantage: The returned data packet does not need to pass through redirector, and there is no redirector Bandwidth bottleneck. In principle, the bandwidth of the cluster is the sum of the bandwidth of all servers. Of course, they cannot exceed the bandwidth of the connected Wan switch.

5. Restrictions: You need to purchase multiple public IP addresses. Director and RealServer must be in the same Wan CIDR block, that is, on the same vswitch. Why must I be in the same CIDR Block? If the RealServer is in another CIDR Block, redirechanges the entire packet and MAC frame before sending it to the switch. The switch finds that the MAC address cannot be found in its Wan and cannot forward it. LVS-TUN:
This method is proposed to break through the restrictions of the same network segment of the LVS-DR. It does not cheat, but is a bright and bright communication, and is encapsulated twice at the network layer.

1. In the network environment, one ctor + N realservers, Director and RealServer both have public IP addresses and are exposed on the wide area network. Public IP addresses are different from each other, and there is no alias restriction or need not be in the same network segment.
2. Client Requests
The client sends data to redirector, and redireputs the IP package into a new IP package as the load, and determines the IP address of a specific RealServer as the destination address of the new IP package according to the scheduling policy. These new IP packets are completely compliant with network protocols, and there is no deception. Therefore, these IP packets must pass through the wan cidr block to reach the specified RealServer.

3. RealServer response
After the RealServer obtains the data, it needs to do one thing, extract the payload of the IP packet, and then use these loads as the IP packet to form TCP, and then form the final request data up. Based on the request data, after the RealServer generates the returned data, it returns the result to the client.
4. Advantage: Like LVS-DR, there is no redirector outbound Bandwidth bottleneck.
5. Disadvantages: Additional packaging and package settlement are required.


LVS ConfigurationIpvsadm Installation

yum -y install ipvsadm

 

Configuration of LVS-NAT 1. RealServer Configuration Requirements:

Configure the Intranet address. The default gateway points to the scheduling server.

 

2. Basic Directer Configuration Requirements:

The scheduling server requires two NICs (one for external network and one for internal network. One module can also be used to configure Sub-interfaces. The external VIP and dip are both configured on the same Nic, but this will further reduce the performance of the scheduler. We recommend that you use a dual Nic)

Disable SELinux and iptables, and enable the packet forwarding function:

setenforce 0service iptables stop

echo "1" > /proc/sys/net/ipv4/ip_forward

(To avoid unnecessary troubles, disable these two services on each server)

 

ipvsadm -A –t $VIP：$Port -s rr

Explanation:-A indicates adding a cluster service (you can add multiple services, such as adding 80 web servers and 443 HTTPS servers );

-T indicates the TCP protocol;

-S indicates that the scheduling algorithm is round robin (a total of 10 scheduling algorithms can be selected as needed)

 

ipvsadm -a –t $VIP：$Port -r $DIP:$Port -m

Explanation:-A indicates adding a RealServer, followed by the address port of the cluster service defined earlier.-R indicates adding the address of the specific RealServer, and-M indicates that the mode is Nat.

LVS-DR Configuration

1. Basic configuration of RealServer Configuration Requirements

First, configure the ARP restriction. Otherwise, address conflict occurs after the address is configured. This is achieved by modifying kernel parameters.

In Linux, ARP broadcasts of IP addresses on all interfaces are advertised by default, and ARP requests from IP addresses on all interfaces are responded.

Arp_announce restrictions ARP announcement

Limit level

0: advertise ARP broadcasts of IP addresses on all interfaces.

1: For ARP requests from other devices, try to restrict broadcast announcement responses on the interface (not strict enough)

2: only advertise ARP broadcasts of IP addresses on this interface

Arp_ignore limits ARP responses

Limit level

0: for ARP requests from other devices, respond to ARP requests from IP addresses on all other interfaces.

1: For ARP requests from other devices, only ARP requests from IP addresses on this interface are responded.

echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignoreecho "2" > /proc/sys/net/ipv4/conf//lo/arp_announceecho "1" > /proc/sys/net/ipv4/conf/all/arp_ignoreecho "2" > /proc/sys/net/ipv4/conf/all/arp_announce

 

Configure the VIP address on the lo port of the RealServer. This limits that the VIP address will not generate a MAC address table on the physical switch to avoid IP conflict.

ifconfig lo:1 $VIP broadcast $VIP netmask 255.255.255.255ifconfig eth0 $DIP up

 

Note that the broadcast address of this VIP interface is still VIP and its broadcast is restricted. The subnet mask is 32 bits. Pay attention to this option for the configurations on the scheduler below.

Configure a special route so that the lo port with the source address as the VIP address of the packet whose destination is vip goes out.

route add –host $VIP dev lo:1

 

Directer Configuration Requirements

Configure VIP and dip. VIP is configured on the sub-interface of the physical Nic.

ifconfig eth0 $DIP broadcast $VIP netmask 255.255.255.0 upifconfig eth0:1 $VIP broadcast $VIP netmask 255.255.255.255 up

 

Configure a special route. The destination is the VIP package.

route add –host $VIP dev eth0:1

 

Cluster configuration

ipvsadm -A –t $VIP：$Port-s rr

Explanation:-A indicates adding a cluster service (which is the same as the configuration in NAT)

 

ipvsadm -a –t $VIP：$Port -r $DIP:$Port-g

Explanation: The rest is similar to Nat, And the last mode is changed to-G, that is, the Dr mode.

LVS Scheduling Algorithm

-S indicates the algorithm used by the Service. Common algorithm parameters are as follows:
Rr round robin)
The scheduler uses the "Wheel call" scheduling algorithm to distribute external requests in turn to the real servers in the cluster. It treats each server equally, regardless of the actual number of connections and system load on the server.

WRR Weighted Round call (Weighted Round Robin)
The scheduler uses the Weighted Round call scheduling algorithm to schedule access requests based on different processing capabilities of the Real Server. This ensures that servers with high processing capabilities can process more access traffic. The scheduler can automatically query the actual server load and dynamically adjust its weight.

Least LC connections (least connections)
The scheduler uses the "least connected" scheduling algorithm to dynamically schedule network requests to servers with the least established connections. If the real server of the cluster system has similar system performance, the "minimum connection" scheduling algorithm can be used to better balance the load.

Wlc weighted least Link (weighted least connections)
When the server performance in the cluster system is significantly different, the scheduler uses the weighted least Link (wlc) scheduling algorithm to optimize the Server Load balancer performance, servers with higher weights will bear a large proportion of the active connection load. The scheduler can automatically query the actual server load and dynamically adjust its weight.

Lblc locality-based least connections)
The local-based least-link scheduling algorithm is a load balancing algorithm for the target IP address. It is mainly used in the cache cluster system. This algorithm finds the Server recently used by the target IP address of the request. If the server is available and is not overloaded, the request is sent to the server. If the server does not exist, alternatively, if the server is overloaded and has half of the server's workload, use the "least link" principle to select an available server and send the request to the server.

Local-based least connections with replication)
The "local least link-based replication" scheduling algorithm is also used for load balancing of the target IP address. It is mainly used in the cache cluster system. It differs from the lblc Algorithm in that it maintains a ing from a target IP address to a group of servers, while the lblc algorithm maintains a ing from a target IP address to a server. This algorithm finds the server group corresponding to the target IP address based on the requested target IP address, and selects a server from the server group based on the "minimum connection" principle. If the server is not overloaded, send the request to the server. If the server is overloaded, select a server from the cluster based on the "minimal connection" principle, add the server to the server group, and send the request to the server. At the same time, when the server group has not been modified for a period of time, delete the busiest server from the server group to reduce the degree of replication.

DH Destination Address (destination hashing)
The "target address hash" scheduling algorithm uses the target IP address of the request as the hash key to find the corresponding server from the static allocation hash list, if the server is available and is not overloaded, send the request to the server. Otherwise, null is returned.

Source hashing)
The "Source Address hash" scheduling algorithm uses the requested source IP address as the hash key to find the corresponding server from the static allocation hash list, if the server is available and is not overloaded, send the request to the server. Otherwise, null is returned.

 

 

Layer-4 load balancing-LVS