Cluster scalability and its distributed architecture (4)

LVS Architecture Analysis

Lin Fan (iamafan@21cn.com)

Chen Xun software studio R & D department manager
March 2002

In the previous three articles, we introduced the distributed architecture of clusters, major elements, and key points of consideration and Design of clusters. In fact, only these basic theoretical knowledge is far from enough. Any point in cluster technology is enough to write a few books. It is important that, through a comprehensive introduction, we can lay a solid foundation for the subsequent case discussion. This article mainly introduces the Linux-based cluster technology-ipvs (Based IP Virtual Server) is also called LVS (Linux virtual server ). Through a detailed analysis of LVS, we can understand the detailed analysis and discussion of LVS architecture design, technical implementation details, and some problems that may be encountered in the application, gradually master the basic knowledge of implementing the cluster system in Linux.

Problem
Today, both enterprise networks, campus networks, and wide area networks, such as the Internet, have seen business growth go beyond the most optimistic estimates of the past. At the same time, users are constantly pursuing higher machine performance, however, upgrading a single server system often results in a high investment and maintenance cost, which is much lower than expected. All of this not only puts forward higher requirements on the hardware but also the software platform:

• Scalability:A computer system with good scalability can linearly increase performance with the increase in costs. And it is easy to streamline or expand it.
• Around-the-clock availability support:Strict commercial environments require transparent and automatic availability support from software and hardware systems to ensure the normal operation of 24x7 systems.
• Manageability:The system can be huge, but it must be easy to manage.
• Relative Cost/performance advantages:Constructing such a system is economical in terms of investment costs, and it is easy to customize a target system with a proper price according to specific needs.

Solution
Through the previous analysis, we know that the high scalability of the cluster system in terms of cost and benefit is the effective way to solve this problem. A computer cluster with a relatively low overall cost can achieve the powerful performance that a single system cannot provide. Here we propose a system with the following features for Internet applications:

• High scalability
• High Availability
• High Performance

It may be called the "Three highs" system.

LVS Architecture
The Linux virtual server project is composedDr. Zhang Wenyu(Developers of open source code and Linux kernel. Founder and main developer of LVS (Linux virtual server), a famous Linux cluster project. He is currently working in the National Key Laboratory of parallel and distributed processing and is mainly engaged in the research of cluster technology, operating systems, Object Storage and database. He also spent a lot of time developing Free Software and enjoyed it.Is hosted by the famous open source project, a solution to achieve the "Three high" system. LVS aims to solve the increasingly prominent problem in the rapid development of Web commerce: how to maximize the potential service performance of Web sites with limited investment.

LVS is a software tool on Linux. With LVS, you can quickly and conveniently build a cluster system with the layer-4 load balancing function. In addition, with the help of third-party toolkit, you can also extend the functions that support the LVS cluster availability. First, let's take a look at the LVS architecture:

Figure 1: LVS three-layer architecture

We can see that the abstract architecture of LVS is divided into three layers.Level 1Is the Server Load balancer, which is the only entry to the cluster. From the client perspective, the cluster uses this layer of service as a single IP address-based system image (SSI). The whole cluster shares this virtual address, the client can regard the entire cluster as an independent host system with a valid IP address. All client access is sent to this virtual IP address.

However, we also found that if there is only one Server Load balancer, it is easy to cause single point of failure of the cluster, making it the most vulnerable link in the cluster. Therefore, it is necessary to provide a fault tolerance mechanism to automatically detect and smoothly Replace the Load balancer when it fails, that is, the HA technology. In the structure, a node that runs as a backup balancing node monitors the running status of the Server Load balancer in real time and responds to the detected status: alarm, take over, and restore. Details will be discussed in the HA section.

Layer 2Is the server group that provides the actual service. After the service request sent by the client is processed by the balancer, it is transferred to the Service pool and the server responds to the request and returns data. We usually provide Web Services, FTP services, or VOD services on the service node pool. Since a single system cannot cope with peak traffic worthy of data access, it is more economical to share these loads with multiple servers.

The server node may also be temporarily invalid, especially when the node provides multiple services, random system faults or sudden changes in the external environment may temporarily render a service of the node unavailable. Therefore, the fault tolerance mechanism extended by Server Load balancer should be able to identify such errors and handle them in a timely manner. Similarly, after troubleshooting, the cluster can automatically identify the recovery event and re-incorporate the node into the cluster to continue running.

Layer 3Is a storage service system that provides stable and consistent file access services for the entire cluster. As an extension of the LVS cluster, this layer provides a single file system portal for the cluster node pool, that is, each service node shares the same root (/); it also automatically completes the underlying functions such as file locking, load balancing, fault tolerance, content consistency, and read/write transactions caused by access to the file system at different nodes, and provides a transparent file access service to the application layer.

The LVS cluster is a loosely coupled cluster system. Because the LVS implements SSI on the IP layer, it does not need to deploy a special middleware layer or OS extension in the cluster, which is more compatible with the server node OS. For the internal nodes where LVS is deployed, it is basically compatible with most IP applications and does not require complicated transplantation and installation. Each internal node can be considered as a relatively independent server system. Even on the server Load balancer, ipvs's core functions are transparently provided to the user space without affecting the normal network applications of the local machine.

In reality, many technologies can implement such a system. With Server Load balancer at a certain level, network requests are reduced to zero and shared by a large number of cluster service nodes to maximize performance.

Server Load balancer Technology
In fact, Server Load balancer is not a traditional "Server Load balancer". Generally, Server Load balancer only distributes loads that may be congested in one place to multiple places for sharing. If you call it "Load Balancing", you may better understand it. To put it bluntly, the role of Server Load balancer in the network is like the rotation duty system, which assigns tasks to everyone to complete, so as to avoid exhausting a person. However, in this sense, the balance is generally static, that is, the pre-determined "rotation" strategy.

Different from the rotation duty system, Dynamic Load Balancing uses some tools to analyze data packets in real time, master the data traffic conditions in the network, and allocate tasks reasonably. The structure is divided into local Server Load balancer and regional Server Load balancer (Global Server Load balancer, the latter type refers to load balancing between different geographic locations and different networks and Server clusters.

In the Server Load balancer system, each service node runs an independent copy of a required server program, such as a web, FTP, telnet, or e-mail server program. For some services (such as those running on Web servers), a copy of the program runs on all hosts in the cluster, network Load Balancing distributes workloads among these hosts. For other services (such as e-mail), only one host processes the workload. For these services, network load balancing allows the network traffic to flow to one host, when the host fails, the communication volume is moved to another host.

Load Balancing Implementation Structure
In general, based on the existing network structure, Server Load balancer provides a cheap and effective method to expand server bandwidth and increase throughput, enhance network data processing capabilities, and improve network flexibility and availability. The main tasks are as follows:

• Solves network congestion and provides nearby services to achieve location independence.
• Providing users with better access quality
• Increase Server Response Speed
• Improves utilization efficiency of servers and other resources
• Prevents single point of failure in key parts of the network

For such a network load balancing technology, we start from different implementation layers of the network and analyze the specific performance bottlenecks. From the perspective of client applications, Server Load balancer can be divided into Client Server Load balancer, application server, high-level protocol exchange, and network access protocol exchange:

Layer of Server Load balancer

At present, there are a large number of technologies at each layer to implement the main functions of Server Load balancer, and their advantages and disadvantages are also different. For our understanding of LVS, you only need to care about the Server Load balancer technology of the network access protocol. The technical features of this layer of Server Load balancer are:

• High execution efficiency, because the underlying protocol can be deployed through the hardware system or at the OS core layer.
• Strong compatibility. The access protocol is often compatible with most existing mainstream network applications, such as the IP layer in the IPv4 system.
• The system implementation is relatively simple. Compared with content-based high-level exchange, it does not need a complex pattern matching mechanism, mainly through port ing for data exchange, and the rules are simple.

Next we will analyze the LVS framework and implementation methods based on the server Load balancer technology.

IP Server Load balancer Technology of LVS
Basically, the implementation of LVS is based on IP exchange, that is, the access protocol exchange technology mentioned above. However, the LVS architecture has certain scalability, which can achieve high performance, high scalability, and ease of management among other features, and become a truly meaningful Cluster System with load balancing as the core.

First, let's take a look at the LVS load balancing model. There are three types: address translation (NAT), IP tunneling (IP tunneling), and direct routing (DR) models.

◆ Address translation mode Nat

Nat structure diagram and Nat Packet Handling Process

We can see that the NAT network structure is a private network structure similar to the firewall, and the dotted line in the middle shows the network isolation band. The service node pool is isolated from the Internet through an internal IP address. The service node cannot communicate directly with the client. Both request data and response data must be processed by an IP packet through the Server Load balancer.

The main task in Nat is to rewrite the Source and Destination Address information of the IP packet, so that the request data sent to the VIP is rewritten and then directed to the internal host again. After the internal response data is changed by the Load balancer, use VIP as the source address to send to the requester. This mode is also called Network Address Translation (also called IP address disguise). We use this model in proxy servers, iptables, transparent gateways, and other applications, it can be said that this is a relatively mature technology.

Due to the use of Nat, it is necessary to rewrite the packet header address for the network packets entering and exiting the cluster. When the load is heavy, the performance of the entire cluster will be affected, and the Load balancer is easy to become a bottleneck.

◆IP tunneling

Ipip structure diagram and ipip Packet Handling Process

The IP address mode adopts an open network structure, and the service node has a valid Internet IP address. You can directly return the response packet to the client through the routing path. Therefore, the Server Load balancer only processes the request data packets entering the cluster, but the returned packets do not pass through the router. Therefore, this mode is called the ticket connection mode (single-direction connection mode ). The connection between the Server Load balancer and the service node can be LAN or on different networks. You only need to ensure that the Server Load balancer can send the IP packet to the service node.

After receiving the request packet from the client, the Server Load balancer reprocesses the IP packet through the ipip protocol to form a new IP packet with the selected service node as the destination IP address, the original IP package data is encapsulated in the new IP package. After receiving the ipip data from the balancer, the service node unpacks the packet and returns the processing result to the client based on the client address (source address, the source address of the response packet becomes the virtual address VIP of the cluster.

The technology of the ipip mode is also embodied in other fields. Because the IP address is re-encapsulated, the entire process is still transparent to the application layer. PPTP is an application of IP tunneling protocol. However, currently, ipip is only implemented in Linux. This protocol must enable the device option support in the kernel and bind the VIP through the network guard device. When the service node returns the response data, it can use the VIP as the source address to construct the response packet.

◆Direct routing mode Dr

Dr structure diagram and Dr Package Processing Process

Like the ipip mode, the Dr mode uses a single-device connection mode, and the response data is directly returned to the client without going through the balancer. A service node must also have a valid IP address that can reach the client. In Dr mode, the Server Load balancer and service nodes must be in the same network segment.

After receiving the client request, the Server Load balancer selects the appropriate service node, and then changes the MAC address of the request packet to make it the MAC address of the target service node, then broadcast the package to the network segment of the balancer. Since each service node has a virtual network-side device (which can be dummy0 or lo: 0), these devices are bound with the same VIP address as the balancer, however, this device does not respond to the rap resolution of the VIP and does not conflict with the VIP address of the balancer. After the Server Load balancer receives an IP packet that matches its MAC, it directly returns the response data to the customer after processing, and the source address is still VIP. In this way, the Client Always accesses and receives the response from the cluster's VIP address.

Comprehensive comparison
Although LVS supports three load balancing modes, from the above analysis, we found that according to the Load balancer's inbound and outbound IP Packet Handling Methods, LVS actually contains two models: work order processing and duplex (two-way connection mode) processing. Obviously, the NAT address translation mode is dual-connection processing. In this mode, the Server Load balancer not only needs to process the IP packets entering the cluster, but also processes the response IP packets returned by the nodes inside the cluster, from sending an access request to receiving a response, a user must be processed by the core Server Load balancer of the cluster. Therefore, a user is called a duplex connection. In the other two modes, the Server Load balancer only processes the IP request packets that enter the cluster, and the response data of the nodes in the cluster is no longer returned to the client through the Server Load balancer, instead, it is directly sent to the destination through the routing channel from the node to the client. Since the Server Load balancer only processes the IP request for one full connection, but does not process the response data for the IP address, it is called the single-work connection mode.

What are the disadvantages of the comparison between the two. You know, in today's web world, most network requests are relatively small. They are nothing more than URL page requests, get or post forms, and some commands, the data is basically several hundred to several K bytes. It is easy to process such IP data packets. On the contrary, the response data in the Web is usually very large, and a common web page requires dozens of K. What's more, if the returned data is a video or audio stream, coupled with increasingly crazy network downloads, even a powerful processor cannot handle such a large number of IP packet processing tasks.

Therefore, in IP Server Load balancer, if you use the duplex mode (NAT), you must not only process the requests that enter the cluster (rewrite the Source and Destination addresses of the IP packet ), we also need to do the same for a large amount of data returned by the service node. As the cluster service node pool grows, the processing capacity of the Server Load balancer will soon reach saturation, which also greatly affects the scalability of the LVS cluster. In the ipip or Dr mode, the Server Load balancer only needs to process a relatively small number of IP data packets. For a large amount of returned data, the service node directly returns the data to the client through a vro, vswitch, and other devices. Therefore, in terms of scalability, the ticket mode has the scalability advantage.

There is always a truth about the existence of things. The three load models designed by the author have their own shortcomings. Nat is not useless either. Although Nat is inferior to the other two models in terms of performance, cluster nodes support more operating systems and have higher network security. The following is a comparison of the three in various aspects:

	Nat Mode	Ipip Mode	Dr Mode
Requirements on service nodes	The service node can be any operating system	The service node must support the IP tunneling protocol. Currently, only Linux	The service node supports the virtual Nic device, which can disable the ARP response function of the device.
Network requirements	Local Network with private IP addresses	LAN or WAN with valid IP addresses	For a LAN with a valid IP address, the service node and the balancer must be in the same network segment
Number of nodes supported	10 ~ 20, depending on the processing capability of the test balancer	High, supporting up to 100 service nodes	High, supporting up to 100 service nodes
Gateway	The balancer is the gateway of the service node.	The service node is connected to its own gateway or vro without going through the balancer.	The service node is connected to its own gateway or vro without going through the balancer.
Service Node Security	Better. Use an internal IP address to conceal the service node.	Poor. Public IP addresses are used, and nodes are completely exposed.	Poor. Public IP addresses are used, and nodes are completely exposed.
IP requirements	Only one valid IP address is required as the VIP address.	Except VIP, each service node must have a valid IP address and can be directly routed to the client.	Except VIP, each service node must have a valid IP address and can be directly routed to the client.

Comparison of Three Modes

To sum up, when selecting LVS as the Cluster Server Load balancer solution, we first determine which IP Server Load balancer structure to use based on the previous application environment. If you only have a valid IP address, or you need to construct a secure cluster without worrying about performance issues, you can use the NAT mode. If you have high performance requirements, if the application is based on Linux, using the ipip or Dr mode will surely surprise you.

Author profile: Lin Fan is currently engaged in Linux-related scientific research at Xiamen University. With great interest in cluster technology, I hope to communicate with like-minded friends. You can contact him by email iamafan@21cn.com.