Go to "continuous deployment"

At present, it seems that "do you want to continue integration ?" It is no longer the focus of discussion. Instead, it is "How to Implement continuous integration ?". In the previous article, I introduced the evolution of the continuous integration of the cruise team. Finally, we also mentioned the continuous deployment of the cruise team. This article will share with you the practical experience of continuous deployment based on the actual situation of the team.

Last mile problem

Continuous integration solves some of the problems in software development, but there is still a more important part to be addressed, that is, "What methods can be used to enable the software to run in a real production environment as soon as possible, to realize the value of the software ". During software development, the "Last Mile" is called "starting from the completion of function development until it is deployed to the production environment for normal operation ". If you pay enough attention to product release from the very beginning, it may take several minutes or even several seconds to complete the last mile. However, in fact, most projects will take several weeks at this stage, and, even more, several months.

Why? For complex software, No matter what environment deployment (test environment, test environment, or production) is very difficult. When the software is deployed to a non-development environment for testing for the first time, or when the software
When the component functions and their environments are greatly changed, many problems are usually exposed. During user acceptance testing, more problems are often found, such as non-functional requirements, inconvenient user operations, and functions and usage.
The real difference between things is too far. The development team can re-deploy the test only after fixing these defects. Therefore, this process will be repeated until the software is stable enough to be deployed in the production environment.
.

If deployment to the test environment is so difficult, isn't deployment in the production environment more risky? What's more serious: when there is a problem with the deployment of the production environment, there is almost no choice in front of you (usually roll back to the previous status, the downtime cost for the "rollback" period is quite high ).

The above causes most organizations to adopt a "conservative strategy" for product release, that is, to reduce the frequency of software release, which also leads to a large difference in version features between the two releases. In this way, the release risk is not
The release interval decreases, but increases. When all factors are combined, the software release process becomes expensive and slow. Generally, the "release process and frequency" determine the product in the market.
.

So how can we better solve the "last mile" problem? Continuous deployment!
The continuous integration practice will be extended to automatically build code and deploy it to the test environment frequently and regularly throughout the software lifecycle, and then pass a series of tests, select an appropriate version and deploy it to the preview environment for pilot operation
Finally, select a stable version to deploy it to the production environment, so that the development team can get feedback from the end customer as soon as possible, and the end customer can get the value of the software as soon as possible.

"Continuous deployment" comes from the pain points of deployment

 

 

When we want to upgrade it again in the second week after we use cruise to build the cruise itself, it takes two days to complete the upgrade because we did not consider the Upgrade Method in advance. Since then, we have started the continuous deployment of cruise.

Continuous deployment Environment

Currently, in the development environment of cruise, there is
Testing) "test environment, currently it is composed of a cruise
Server and nearly 20 agents, used for continuous integration and deployment management of the cruise team. Another pre-release production environment called "production ",
By a cruise
Server and nearly 70 agents, which are used by multiple project teams at the same time. The "production" environment is a real production environment. deployment failure means loss. Therefore, although
Work can be done through automated scripts, but we still add the manual switch (Manual
Approval), as shown in. All the first three stages are automatically triggered, and all subsequent stages are manually triggered. Each version to be released will be first deployed in The UAT environment. In fact, we have tried it.
Run. if the version is stable, it is deployed in the "production" environment. In this way, all deployment risks are under control.

 

Key points for successful continuous deployment

1.Full and extensive automated testing coverage

Currently, our tests include unit testing, end2end testing, function testing, and performance testing. The unit test, end2end test, and function test are all in the same pipeline. These tests are run every time the Code is submitted. In another pipeline, performance testing is used to collect performance indicators of the UAT and production environments after each deployment. Because the deployment frequency is sufficient, we can master minor changes in performance data and take corresponding optimization measures accordingly.

Writing Unit Tests has become an indisputable fact. In addition, because Cruise is dealing with many version management software, the end2end test mentioned here refers to testing with these external interfaces. However
Testing refers
Server and Agent run on the test machine before running the twist automated test suite. The principle of functional testing is that each story must have a functional test
Build, QA and developers share the compilation of functional test cases, implemented by developers, and functional testing should be carried out on the basis of real cruise Server and Agent communication. Twist is another product of our company used for automated function testing. Its Testing and editing interface is as follows:

2.Test feedback time as short as possible

Despite the large number of tests and the long absolute test running time, according to the parallel running features provided by cruise, team member Hu Kai, the test-load-balancer tool developed by Derek and Li Yanhui divides all tests into several parts, and cruise distributes them to the agent cluster for running at the same time, make unit tests or other tests complete within an acceptable period of time (unit tests are within 15 minutes and functional tests are within 30 minutes ). Several agents will be added in the near future to further shorten the testing time.

3.Deployment processAutomation

When you deploy complex software, manual processes are used and may take several days. This deployment process is usually complicated and it is difficult to perform repeated operations reliably. Therefore, people may write some documents to help this process, but documents are often not updated in a timely manner. Sometimes it takes several key figures to be present at the same time.

When different people perform deployment operations each time, the probability of errors increases, so there should be as few manual steps as possible.

In the pipeline of cruise, although the deployment in two environments is triggered by a person, the deployment process is automated. During deployment, the cruise installation package is automatically disabled.
Server, update its own program and upgrade the database, and then restart. All agents are automatically updated to the same version as the server, without the need to manually upgrade each
Agent (manual upgrade of 70 agents at a time is also very costly, so we have automatically upgraded this feature ).

4.Ensure data security during deployment

If data loss or errors occur due to continuous deployment, the loss is not worth the candle. Therefore, every time we modify the database structure or the structure of the configuration file, we will write the corresponding migration script and run it during the deployment process.

Currently, we use dbdeploy, an open-source project developed by thoughtworkers, to upgrade the database. We also developed a migration framework for modifying the xml configuration file.

5.Deployment as soon as possible on the premise of stability

Someone may ask, "Why do we need continuous deployment? How do you know whether the deployed version is stable? What should I do if it goes down ?" Indeed, no developer wants continuous integration servers to go down during working hours. Although we cannot ensure that each deployment version is stable, it is enough to ensure stability within the foreseeable range. Otherwise, we will not be able to solve the "last mile" problem.

After the first three iterations (the iteration time is one week), cruise starts to use it as its own continuous Integration Server (that is, the UAT environment ). Let's make it in The UAT Environment
After two weeks of operation, no problem was found. It indicates that the version is relatively stable and is deployed to the "production" environment. Since then, with the increase of users, many people think that the deployment has failed.
The risk of continuous integration server downtime is higher than those of new features and fixing defects. Therefore, our customers require that the new version be deployed
Before the "production" environment, you must run it in The UAT environment. Currently, the frequency of deploying cruise to the UAT environment is not fixed (generally two days to one week),
The frequency of the production environment is one week. That is to say, the production environment version lags behind UAT for one week.

6.Comprehensive risk mitigation measures

As the project progresses, deployment may inevitably fail, so there must be risk mitigation measures. For example:

(1) deployment should be as few users as possible; (2) technical personnel must be present during deployment; (2) Back Up Original data before each deployment; (3) always prepare a rollback script.

7.Deploy the same product to different environments

You can deploy your products in different environments. If the environment configurations of these environments are different, you can exclude the environment configurations from your products. If you have many configuration variables, save these configurations in the same place with default values.
Based on this principle, the only configuration of cruise is the XML file.

8.Constant reflection and Reconstruction

There is nothing to say about this. It applies to all activities.

Summary

Practice shows that creating an automated deployment pipeline has many benefits. In the past few years, thoughtworks has used this method to help many project teams and companies solve their "last mile" problem. For example, in a project, the deployment frequency increases from one day to one day through automated deployment, and the process takes less than 15 minutes (with only one minute of downtime ). This has a positive effect on the entire delivery phase of the software throughout the life cycle. You only need to press the mouse to prepare the required testing environment, which reduces unnecessary losses caused by human errors, significantly reduces the risk of software release. In addition, frequent and easy releases enable developers to focus on what they want to do: developing new features.