Phased release of Internet products

1. Why phased release?

1. Internet services change frequently and the release cycle is short. Speed and quality are always difficult to achieve.
2. Gray release can reduce release risks and scope of impact.
3. Reduces dependencies on tests and reduces offline self-testing data construction costs.
4. This facilitates centralized monitoring of logs and enables full release of load balancing at all layers, making it difficult to track a complete call link.
5. You can perform a gray-scale test on the account. After the account passes the test, the real user account is used to further reduce the risk and impact of the release.
6. To facilitate rollback.

Problems that cannot be solved by phased release

It should be emphasized that the "tolerable impact" mentioned above must be recoverable. For example, an API cannot be called for a period of time, but can be called successfully after the repair. Permanent loss or destruction of user data (such as product information and order information) is intolerable. Therefore, architects of Internet enterprises have the responsibility to design well-developed backup measures (such as regular backup of user data and business flow logs for write operations ), when user data is lost due to production system disorder, manual intervention can still be performed based on historical records (backup data, flow logs, etc ), restores lost user data to a State not long ago (for example, one hour ago to one week ago.

TipsFirst, grayscale testing accounts can reduce the risk of damage or loss of real user data.

2. Expected results

Regardless of the change, we want specific requests to be routed to our change version (gray version) for observation and verification.

3. grayscale strategy

In fact, all requests should be routed to our grayscale version (grayscale machine. This is often highly relevant to the business. For example, Apis generally have the following requirements:

1. Specific users (such as test accounts)
2. Specific apps (such as test apps or partner Apps)
3. Specific modules and interfaces (only some interfaces require grayscale. This is generally the modification of API container. Some APIs are not very important for Grayscale testing .)
4. Specific machine (some request IP addresses are forwarded to grayscale machines)

4. grayscale scheme Scheme 1. Code-level dynamic switching between old and new based on the agreed flag-Amazon's Practice

Implementation:

When the switch is embedded in the code, make if-else judgment. For machines that require grayscale, set the switch to on; otherwise, set the switch to off. Each release has two versions.

Advantages

• Fast rollback without re-publishing or restarting the system.

Disadvantages

• Tends to code.
• Branch logic brings complexity.

I have applied this method, that is, when Alibaba switched the product database from Oracle to MySQL, a state variable was used for control. To achieve smooth migration.

Solution 2: Pre-release machine-Alibaba

In fact, this is not a real gray scale. Because the pre-published machine has an internal IP address and does not provide external services. You need to bind a domain name for verification. However, the data is completely online. Therefore, it is essentially a simple practice for certain users of gray scale (users who can access gray scale machines and internal test users. In fact, there is a similar practice on the API side, that is, our Gamma environment, and we also provide the Domain Name of the Gamma machine to facilitate external cooperation with users for testing.

Advantages

• Simple

Disadvantages

• Waste a machine (this can be put into the official environment after pre-release is completed, and removed from nginx during pre-release, but requires O & M support .)
• Not flexible enough
• It can only be used for access-layer machines. You need to consider the gray scale of the IDL service.

Solution 3. Set deployment 1. Separate deployment by service

For example, in the current API container practice, the deployment granularity can reach the API level, and the frontend forwards data based on nginx. For example:

• Micro-shopping API container: api.weigou.qq.com
• Pat API container: api.paipai.com
• Yi Xun API container: api.yixun.com
• Online shopping API container: api.buy.qq.com

The preceding is highly isolated deployment at the business level. It can also be further refined to the module level. For example, the API of virtual service e-commerce is a sub-business module under the pat. However, after their access, their access traffic increases greatly, to avoid affecting other services, and to avoid being affected by other services, the API deploys two machines separately for them, after configuring nginx, you can direct access to virtual APIs:

Virtual API container: http://api.paipai.com/v2/virbiz

In this way, when we release a version, we can first select Yi Xun, with the smallest business volume, for release, and observe that there is no problem and other platforms will be fully involved.

2. User-isolated deployment

This is not suitable for open platforms, but it is suitable for SNS. For example, in the QQ system, users are divided into several sets based on their number segments. Each set contains 0.1 billion consecutive numbers. Assume that the latest QQ number is close to 1 billion, there are 10 sets (Set 1 to set 10) in total ). In this way, you can select one of the sets to publish each time, and high QQ is often not very important to users, so set10 will be released first.

Advantages

• Isolated deployment with minimal impact on each business line. Automatic gray release is supported.

Disadvantages

• The gray granularity depends on the isolation deployment granularity, which is generally too large.
• Compared with centralized deployment, it is a waste of machines.
• Different service line versions may be inconsistent, which is not conducive to unified management.
• Certain implementation and deployment costs

Solution 4: Dynamic Routing

Method: Use a grayscale policy that can be flexibly configured to affect load balance behavior, so that it can return the IP address and port of the grayscale service according to the grayscale policy.

It is applicable to phased-out services with the background IDL.

Advantages

• Flexible and controllable.

Disadvantages

• Currently, the configuration center and L5 do not consider specifying routing policies and do not have scalability. Therefore, they need to be developed outside of them.
• The sources of API metadata are scattered. Currently, API and IDL metadata are distributed across different data sources, and a gray-scale routing data source needs to be added.

 

 

Gray release generally has three methods: nginx + Lua. nginx is distributed based on cookies, and nginx is allocated based on weights:
Nginx + Lua is differentiated based on the visitor's IP address. Because the company exports an IP address, either the old version or the new version will be used to access the website. This method is not suitable
Nginx is allocated based on the weight, which is easy to implement. You can also try
Nginx is distributed based on cookies, and gray release is more reasonable for users.

Phased release of Internet products