The transformation of Google architecture from failover system to multi-homed architecture

The transformation of Google architecture from failover system to multi-homed architecture

Running a single data center is difficult, so switch to a dual datacenter if you need to support multiple data centers in different geographies. Google has a thoughtful, thought-provoking paper that describes the process-"large-scale high availability: Building Google's advertising data infrastructure."

The main idea is that a typical failover architecture does not work well in practice when switching a single data center to multiple datacenters. The ability to work with less resources to provide high availability and consistency is a native multi-homed/multi-Connection architecture (natively multihomed architecture):

Our current approach is to build a native multi-homed architecture. In such a system, multiple data centers will continue to operate and, depending on the situation, automatically balance the load of different data centers while simultaneously resolving data center outages of any size in a fully transparent manner. In addition, planned data center outages and maintenance events are fully transparent, minimizing the impact on operational systems. In the past, to complete downtime and maintenance events, a lot of manpower was needed to move the operating system from one data center to another.

The term "multihomed" may be confusing, because the word multihomed generally refers to a computer connected to multiple networks. But according to Google's size, it's natural to use this to describe connecting multiple data centers.

Google has built multiple multihomed systems to ensure high availability (99.99% to 99.999%) and consistency in the event of datacenter-level outages, which are described in detail in the following articles: Spanner: relational database ; Photon: Deployment of joins for multiple consecutive streams; Mesa: Data warehousing. These papers discuss the ways in which systems are used, as well as the many challenges encountered in building a multihomed system: Global state synchronization, how to set checkpoints, and repeatable input and output that executes only once.

In order to maintain usability and consistency, the system has been greatly constrained. This underscores Google's ongoing efforts to streamline these complex systems and improve its ease of use:

"The simplicity of a multihomed system is extremely valuable to the user, and without a multihomed system, it is a problem that the application needs to solve regardless of failover, system recovery or consistency." With multi-homed systems, the infrastructure handles these troublesome problems, and the application developer can focus on its own application, with the availability and consistency of the system guaranteed. ”

The biggest surprise in this paper is that, in practice, multihomed systems consume less resources than failover systems: a multihomed system deployed on 3 data centers, with a total synchronization performance of 20% in stable state and a total resource occupancy of 170%, much smaller than the 300% required for a failover system.

What's wrong with the failover system?

"Failover-based systems do not really achieve high availability, and because of the need to deploy alternative resources, cost is wasted." We used to have a lot of bad experiences when we were using a failover system. Due to unplanned downtime is rarely seen, the failover system is more than the late addition of functions, can not be automated execution, and has not been well tested. Most of the time, teams take days to recover from downtime, a component is on-line, and a temporary tool such as a custom mapreduces is used to perform state recovery and to step through the tuning as the system processes the backlog of downtime. These conditions not only make the system no longer scalable, but also put the team under great strain due to the complexity of the mission-critical system. ”

How does a multihomed system work?

In contrast, multi-homed systems run multiple datacenters as core design attributes at the outset of design, so there is no need to add additional failover systems. Multi-homed Systems keep multiple data centers running. Each data center is continuously processing work, while the load between data centers is automatically balanced. Once the processing speed of a data center slows down, the system automatically adjusts part of the work to a faster data center. Once a datacenter failure is completely unavailable, all work is automatically distributed to other data centers. Only continuous dynamic load balancing, no longer has a failover process. Multi-homed systems coordinate the work between data centers through a global shared state of real-time synchronization. All critical system states are backed up to ensure that they are ready to work again from a point in another datacenter, while ensuring the semantics (exactly once semantics). In the event of a datacenter-level failure, a multihomed system is the only system that provides high availability and complete consistency. In a typical streaming system, event handling is based on user interaction, and multiple datacenters around the world provide traffic services and log storage services to users. The Log collection service collects these logs on a global scale and replicates them to two or more specific log data centers. Each log datacenter saves full log records and ensures that all events replicated to a datacenter are (gradually) replicated to other log datacenters. The streaming system runs on top of one or more log data centers and handles all events. The content that the stream processing system outputs is usually stored in some replication systems around the world, so that the output can be reliably consumed from anywhere. In a multihomed system, all data centers are continuously running and handling events, typically deploying three data centers. In a stable state, each of these three data centers can handle 33% of the traffic. If a data center fails and is down, the remaining two data centers will be responsible for 50% of traffic processing.

The transformation of Google architecture from failover system to multi-homed architecture