Micro-Blog Config Service practice

A. The things that are configured

1. Business background

As a large and complex distributed system, there are a lot of configuration information in the microblogging platform, which defines the addresses of RPC services and resources (memcached, Redis, MySQL, etc.) in the platform, as well as the metadata information of these services and resources.

In the early days of Weibo, configuration information was scattered in engineering code, which was simple and convenient, but the scale of the microblogging platform system and the increase in the complexity of the business deployment posed some problems:

Services are deployed in multiple pools according to business and importance, with different configuration information for different pools.

Service multiple room deployment, different computer room configuration information is also different.

Configuration information for development, testing, and deployment is also different.

To solve the problem, the microblogging platform has made a large-scale adjustment to the configuration, separating the code and configuration, and configuring it in accordance with the computer room, business pool and environment, and storing the agreed location in the local file system. After the separation, each business pool, the computer room configuration more flexible, but also for the configuration to the Config service migration provided a condition.

As the size of the boping server continues to grow, the failure and change of RPC services and resources has become a norm, and the following scenarios are often encountered during routine maintenance:

Sudden surge in a group of MEMCACHEQ resources requires rapid expansion

An Redis resource access overload requires immediate demotion to call its service

And the existing solution can not solve these problems, we urgently need a distributed environment of the persistent configuration system, providing efficient configuration registration and timely change Notification Services, so Vintage (config service in the micro-blog code) emerged.

2.vintage

Vintage is a typical pub-sub based communication framework, with many similar systems in the industry, such as Taobao's diamond and soft-load centers, and zookeeper play a similar role in some systems, such as Kafka.

Vintage manages the information centrally in the cloud and provides real-time change notification. Vintage is mainly used in two main scenarios: Configuration management and naming management, configuration management mainly maintains static configuration data (such as service timeout, downgrade switch state), and naming management primarily maintains RPC service address information. The two scenarios look very similar, but there are also the following differences:

The lifecycle is different: The naming service needs to manage the lifecycle of the service, to detect the existence of the service in real time, and to configure the service to manage static information.

Data is produced in different ways: The naming service data originates from the registration of the service, and the configuration service data comes from the OP manual addition.

Data accuracy is different: Naming service data can be misjudged in a window of time (as mentioned later); The data for the configuration service is very accurate.

Vintage in the microblogging platform has been greatly promoted and used, RPC service, Cache service, distributed trace system, queue service to some extent rely on vintage, its importance is self-evident. So how is such an important service designed?

Two. Vintage Design and architecture

1. Architecture Design

At the beginning of the design, vintage the following goals:

High Availability: Vintage manages the address and metadata information of the service, and its high availability determines the overall success or failure of the system.

Low latency: Every call in the microblogging platform will theoretically interact with vintage one or more times, so vintage access latency affects the overall performance of the system.

Timeliness: timely response and notification of changes

Vintage Select Redis as configuration information of the landing storage, simple structure, no single point, as shown in the following figure: