Ucloud the next generation VPC Architecture smooth evolution solution and technical explanation

This is a creation in Article, where the information may have evolved or changed.

Objective
February 27, Ucloud launched the Next Generation VPC Network (hereinafter referred to as VPCNG). VPCNG is designed to address pain points in customer network usage scenarios, such as autonomous planning of IP segments, disaster tolerance across availability zones, VIP Cross-zone high availability scenarios, seamless connectivity between hybrid and public clouds, and more.
Ucloud believes that VPC (virtual Private Cloud) is a tenant-specific network. In a virtual network, tenants can fully control the network environment, manage all cloud service resources autonomously, and flexibly expand the hybrid cloud capabilities, the ultimate goal of VPC is to provide users with the full consistency of the traditional network functions.
New features such as flexible address space management, VIP cross-availability zones are the industry firsts. But what the user can't see is that we do a lot of work in the background service, and many of the logic of the control plane and forwarding surface is reconstructed. These new features are not only deployed in the new room, ucloud in the stock room through a smooth upgrade, so that the user a large number of deployed businesses can also enjoy these new features.
The in-situ upgrade of the stock room and the deployment of the new engine room are completely different jobs. The former to consider the complex existing network environment, the stock room often contains a number of historical versions, as well as custom-made logic for some customers, the implementation of new features to take into account the compatibility of various scenarios. The current network control data will also be converted into a new format. As a system-level underlying service, the network is used by almost all cloud services, and its in-place upgrade is more complex than other products and is a counter switch in cloud computing. In addition, the network in-place upgrade must ensure that the user's business is not subject to 1 seconds of interruption. In order to control the risk, the whole network one-time upgrade is absolutely impossible, only through the gradual, ant moving gray scale upgrade to achieve a smooth operation, to achieve step-by-step for the win.
Next, take a look at the architectural scenarios and technical details of the Ucloud vpcng smooth upgrade.
VPCNG Architecture
VPCNG network architecture, as shown, is mainly composed of VPC, Subnet, routetable three core modules:
Subnet is the smallest unit of VPC resource management, used to manage the resources of cloud host, physical cloud host, cloud database, container, and so on, Vpcng is a highlight of the subnet can cross availability zone, to provide strong protection for cross-availability zone disaster preparedness.

RouteTable is the core of VPCNG design and is responsible for the management of distributed virtual Routing (DVR). Each subnet needs to be associated with a single routing table.

Figure 1 VPCNG Overall architecture
Distributed virtual Routing (DVR)
In an existing VPC network, some products need to create a virtual router (VRouter) before using custom subnets. Routers have three features: multi-host shared out-of-network, designated port forwarding, inter-network interoperability between subnets. Both east-west and north-south traffic go through a virtual router. Two subnets need to pass through the Vrouter, subnet access to the extranet also need to cross the vrouter, so that not only the physical path of a jump, and vrouter itself will become a performance bottleneck.

Figure 2 Set Chinese router architecture
VPCNG reconstructs the route definition and abstracts it into a distributed virtual route DVR. The carrier of DVR is the virtual switch scattered on each compute node, and the routing table is its core. For example, the traffic is distributed through a virtual switch for point-to-point communication. And NATGW is just as gateway device for extranet access, which provides multi-subnet sharing capability of outbound network. In the routing table design, support for multiple routing types, including direct-attached routing, default routing, hybrid cloud routing, host routing, and so on, in addition to support policy Routing and define route priority.

Figure 3 Distributed Virtual Routing (DVR) architecture
As the core of vpcng design, the smooth upgrade is especially important. At the beginning of the VPCNG design, considering that the on-line process should be transparent to the user, without affecting the existing business situation, the smooth upgrade to the new architecture, so in the upgrade scheme design is also more complex.
The entire upgrade process is split into four phases.
Stage1 Routing Table Data Migration
The most difficult thing to do in a smooth upgrade of a large distributed system is the seamless migration of control data. VPCNG The routing table structure changes very large, in this we learn from the availability Zone Upgrade classic "Double Write scheme", first the stock data into a new table structure, and then transform the existing Management Service module, so that it has new cousin dual write ability. This phase of data consistency is the most important and must be reconciled by reconciliation.

STAGE2 SDN Forwarding Surface service upgrade
Forward polygon Service directly affects the current network services of the stock users, so the upgrade process must be cautious. routing table and Controller module is the focus of the SDN forwarding surface service upgrade, we design a layer of forwarding proxy to achieve the ability of grayscale, only gray users can go to the new forwarding logic, effective control of gray influence range.

STAGE3 SDN Control Surface service upgrade
In order to meet the functional and performance requirements of vpcng, we have done a lot of refactoring at this stage, the original control surface architecture adopts the layered structure, the expansibility is not good. The new plus feature involves multiple changes and can easily cause dirty data. Uvpcfe is a set of micro-service system based on Golang language, which is stable, fast and extensible. UVPCFE also achieves user-level grayscale through the APIGW grayscale system. Since the front end is not open at this stage, it is still necessary to guarantee the "double write" of the data.

Stage4 Front-End open upgrade
After the first three phases are completed, the inventory user background is already running under the new network architecture. So the Stage4 stage to do things become simple, as long as the VPCNG console open to the user can. The green part of the figure represents the Vpcng business logic, the blue part represents the original logic, the two sets of logic in the gray process of the long-term existence, once the new logic is found to be defective, can immediately rollback to the original logic, the existing business is not affected.

Figure 4 VPCNG Grayscale Scheme
The detailed upgrade scheme makes the smooth upgrade process feasible, controllable, rollback-faced, and can also be problematic in practical operations. The upgrading of the underlying network architecture will have an impact on the management services of each business (cloud host, cloud data, etc.), there is inevitably a coupling between the business, even the internal modules of the network system will have gray-scale timing problems. How do I decouple? How to control the grayscale publishing rhythm? These problems can not be separated from the systematic integration and refinement of the operation, which is to test whether a team has a large-scale distributed system of the continuous operation of the ability.

Public service
The Ucloud Cloud platform provides many public services such as intranet DNS, NTP, and software sources, which are open to all tenants. The IP of different tenants may be the same, they may go to the public service back-end server, so the backend server cannot decide which tenant to return through the routing table. For example, tenant A and Tenant B cloud hosts have IP addresses of 172.16.0.16, when accessing intranet DNS, the DNS server cannot decide whether to send to tenant a or B when it replies.

Figure 5 Public service centralized NAT architecture
The existing VPC network is implemented through NAT translation, which is the industry's leading solution, which needs to address two issues:
First of all, to solve the problem of routing conflict caused by overlap between the user subnet and the public service subnet network segment. In view of the small size of the public service subnet, the method of reserving network segment is usually adopted to ensure that the public service network segment is not used by the users.
Second, the problem of NAT translation should be resolved.
The Ucloud team demonstrated a variety of possible NAT scenarios, two of which are described here:
The NAT gateway uses the namespace scheme, creating a separate namespace for each tenant, Snat conversion within the namespace, and access to the public services via the Snat IP. The message of the public service response is Dnat converted within the namespace and sent back to the tenant to achieve the purpose of accessing the public service.
The NAT gateway saves the tenant ID information to SKB's mark, makes Snat transitions through the kernel NetFilter module, and maintains the mark correspondence in connection track. The NAT Gateway receives a public service response message after matching connection track, gets mark, and then returns the mark correspondence message to the tenant via NetFilter Dnat.

These NAT schemes have two problems, one is that the source address is not visible, if the public service depends on the source address, then this scheme is not feasible, the second is the Nat gateway is a performance bottleneck, especially for large traffic public services (such as ufile), need to build multiple servers to support.
Vpcng absorbed the concept of distributed NAT, dispersed the centralized NAT gateway to the virtual switch of each compute node, solved the problem of gateway performance, and developed a set of NAT plugin to solve the source address problem, and the different tenants can access the public service even if the IP is the same. The distributed NAT frame is composed as follows:

Figure 6 Public service distributed NAT architecture
Hybrid Cloud
Hybrid cloud connecting traditional it and public cloud, the director of the audience, is now one of the mainstream trend of cloud computing development. In the traditional hybrid cloud scenario, the hybrid cloud is just as a number of network segments into the public cloud, the user's many daily needs such as routing management, bandwidth capacity management, hybrid cloud daily operations need a lot of manpower and communication costs, needs optimization.
In the VPCNG design scenario, the hybrid cloud is abstracted into a separate VPC for the tenant, which contains multiple subnets, either a managed device segment or a self-built IDC segment that connects to a public cloud pop point via fiber optics, digital links, and VPN connections. The architecture diagram for a hybrid cloud VPC is as follows:

Figure 7 Hybrid Cloud VPC architecture
In order to be fully compatible with VPCNG, the hybrid cloud needs to support user-defined hybrid cloud segment, user-defined hybrid cloud routing strategy, user Autonomy control of hybrid cloud connectivity, and so on, so the control surface of hybrid cloud system needs to be completely reconstructed, and for the hybrid Cloud Gateway which is the core of forwarding surface, the forwarding capability is proposed. A full range of new requirements, such as routing control capability and authentication capability. In addition, we must ensure that users are not aware of upgrading from a traditional hybrid cloud to a hybrid cloud vpc.
Control surface
We have designed a new system completely decoupled from the old hybrid cloud control system, and the database and control program are independent of the old version. When the user adds the route, modifies the network segment and so on, the API layer uses the message queue to write the DB double, while the DB reconciliation guarantees the data consistency. The old and new systems have separate forwarding surfaces, respectively, to pull back the background configuration, to ensure that the old and new systems in addition to the dual-write operation are decoupled.

Forwarding polygons
To support the hybrid cloud VPC, we developed a new version of the hybrid Cloud gateway. By pulling the background configuration, the gateway performs the authentication, forwarding, encapsulation, and encapsulation of related messages. At the same time, the gateway cluster has the capacity of scale out to seamlessly expand and support up to hundreds of g of traffic forwarding.
Seamless switching of hybrid cloud gateways is the core step of upgrading. We have developed several tools to assist this critical process:
NETCONF-based switch configuration API for switching and fallback switch-side routing configurations.
VPCNG-based routing switch API,VPCNG control background combined with push and pull, you can update the whole network three-tier routing flow table within 10s. This API is used to toggle and fallback the routing configuration on the public cloud side.
The OvS Packetout-based Connectivity checker, which is injected into the public cloud host's OvS Virtual interface by splicing ICMP packets, can simulate the user's business interoperability and basically cover the user's connectivity black box inspection scenario.
Traffic statistics tools based on switch statistics and hybrid Cloud gateway statistics can be used to confirm the traffic condition before and after the user's switchover from a statistical perspective.

The handoff process for the forwarding polygon is as follows:

Figure 8 The Hybrid cloud VPC forwarding surface switching process
Step one: Call the switch side routing switch API, switch the route of the PE, at this time, the hybrid cloud side to the public cloud side of the traffic has switched to the new hybrid cloud Gateway, and the public cloud side to the hybrid cloud side of the traffic is still to the old hybrid Cloud gateway. One of our basic concepts in Gateway-class product design is that the routing switch gateway must be stateless, so the user's service will not be affected. After switching, use the connectivity checker and the traffic statistics tool to check for exceptions that are immediately returned.
Step Two: Call the public cloud-side routing switch API to switch public cloud-side routing. When the switchover is complete, the new hybrid Cloud Gateway will host all traffic. The tool is also used to check, and the exception is found to be rolled back.
We have successfully upgraded multiple users ' hybrid cloud systems according to the above strategy, and in practice, the user is completely unaware.

Closing
The Cloud network is the cornerstone of a large public cloud platform, and network architecture tuning is the test of the comprehensive capabilities of cloud service providers, such as development, delivery, and operations. From system design, grayscale scheme, to each stage implementation, change release, final delivery, Ucloud vpcng to maximize the smooth evolution.
In addition, in this technology sharing, we also introduced in detail the VPCNG in the upgrade process of several key issues and countermeasures, of course, VPCNG upgrade complexity more than these, welcome to the relevant technical details to discuss.