Continuous integration Case Analysis Series (2)-one of the continuous integration practices of large-scale project teams

Some days ago, I wrote continuous integration practices for small-scale product teams.
After that, I have been busy with the Project. Today I finally have time to complete this article on the continuous integration of large-scale projects.

 

I. Basics of continuous integration

In typical software projects, the integration phase is generally at the end, so there are also the most problems, and the most likely cause failure to deliver on time. Continuous integration (one of XP 12 practices) can be used to solve this problem. Since we all think that it is a good practice to "frequently build and test software on a code baseline, isn't it better to make the code be successfully built and tested every time it changes?

Through continuous integration, all work will be integrated into a single code base line as much as possible. Each code submission triggers a build and a series of tests. Therefore, developers can get timely feedback and build and test at any time. At the same time, problems may occur during the integration process before software release.

Ii. problems that may occur in large-scale project teams

For small-scale and short-cycle projects, the team and continuous integration will get along very well. At the same time, there won't be too many problems in the early stages of large-scale and long-term projects. In this case, the basic continuous integration mode is build-> test-> package.
However, continuous integration is bad if it takes a little longer. Symptoms include:

1. As a developer

A has to wait a long time to know whether the code can be submitted.

If you follow the "Frequent submission" principle, continuous Submission by a hundred teams will keep the Integration Server busy, and you have to wait for others' builds to complete, to view the submitted results.

B. It takes a long time to know if my submission has passed.

C. If the build fails, it takes a long time to know if it is related to your own modifications.

D. Even if you have submitted the fix, you do not know whether your submission has actually fixed the build.

E build often fails.

2. As a tester

F The tester does not know where to use build for testing.

The G release manager does not know which version has been deployed in various test deployment environments, including new functions or bugs modified.

H. I'm not sure if all the component versions in the same build are correct.

3. As a Project Manager

I'm not sure whether the configuration in each test deployment environment is consistent with the build running on it

J. You are not sure whether the tester has run the correct version in the correct running environment.

4. Other problems

All installation operations are manual.

These problems will bring unlimited problems and risks to your release management.
So, is continuous integration abandoned because of this "continuous annoyance? Of course, the answer is no. Do it more if it hurts you. Don't give up because of the exposure of the problem. On the contrary, you should cheer. This reflects the problems and risks in the release process, and it is time to solve them.

Iii. How to Solve the Problem of continuous integration in large-scale projects

Because of the complexity of a large project, the solution cannot be generalized. The following describes several solutions for a large project.

1. Description of basic project information

This project initially attempted to build a good continuous integration environment and foundation. Although the software can work, the continuous integration fails to meet the expected goal due to the growing team and the changing environment.

Project Background:

A project is a web portal product that is configurable and can be customized for different industries. This project has a legacy code library, and it can be affirmed that it will not be able to get rid of this legacy code library within the next year and a half. In addition, there are a lot of tightly coupled, unnecessary bloated code, and there is no valuable test code at all. Now we are gradually rewriting the code, but we still cannot delete them because some websites still depend on the old code. In fact, this is an SOA-based website on the. NET platform.

Development Team status:

The team is an agile distributed development team (with developers working in three places and time difference ). The entire team has more than 150 people, divided into more than a dozen teams, each team has a completed structure (BA/dev/QA ), one of them is the project continuous Integration Team (there were about five or six people at the beginning, and the workload was heavy). At last, only two people would be enough. Use SVN as a version management tool, and use NAT, msbuild, and Batch scripts on Windows2003 for build management. CC. Net was initially used as a continuous Integration Server.

Continuous integration environment:

The problem of continuous integration described above occurs at the beginning of the project, because the project has a large legacy code library and uses the basic continuous integration method (build-> test-> package) in addition, testers are manually deployed to perform various tests.

Its initial continuous integration environment is as follows:

Step 1: minimize the impact between teams

Method: Convert to zero first and then to zero-code by Team (or by code ).
Means: DVCs + private continuous Integration Server + global continuous Integration Server
Each team uses Git as an intermediate source code management tool. In this way, the team members can submit to git first. Each team has its own continuous integration environment. Once successful, code is submitted to the central source code library and continuous integration of the central source code is triggered. A dedicated team is responsible for global continuous integration result tracking.

Benefits: 1. Every team can submit code every day (if the Code does not make their build fail, it means at least it can pass the preliminary test ).
2. If a team fails to be built, it does not affect the entire project, but only one team.
3. There is a dedicated team responsible for the overall situation, so no team needs to stop.
4. if global continuous integration fails, no team is needed to stop.

Step 2: Improve the feedback speed

Method: Convert to zero and then to whole-test group running.
Means: parallelization and central warehouse (cruise parallelization and central warehouse)
Because of its many functions and complexity, it has many tests and runs very long. With the parallelization feature of cruise, each team divides the test into 28 groups. Once submitted, Cruise runs it on 28 Machines in parallel. After running, upload all test output and results to the same location (the central repository of the cruise server ).
Benefits: 1. The feedback time is greatly shortened (originally over 30 minutes, now within 20 minutes ).
2. Easy Scalability: If the feedback cycle is long due to an increase in testing, more machines can be added.
3. Easy Maintenance: For the increase and decrease of test groups and build machines, it is very easy in cruise, as long as you modify the configuration at the same place.
4. Easy tracing and Debug: All information (including artifacts) is put in the same place and can be accessed through the web.
5. Single View (on the same web management page, you can monitor the building status of all teams ).

Step 3: reduce manual operations
Method: one-click release-automated deployment
Means: Cruise (dependency + story tracker plugin + audit of cruise)
Because there are many teams, each team has multiple test environments. Manual deployment takes a lot of time. Therefore, each team has three build pipelines with the following objectives: (1) tested installer; (2) deployed to the test environment; (3) deploy the tested installer to the demo environment. After the previous build is successful, you can trigger the next one (automatically or manually ).
Benefits: 1. QA can clearly identify the installation package to be tested and the functions contained in the installation package.
2. QA may easily deploy the test environment.
3. History versions of the easy-to-track feature (in the same pipeline, all stages are of the same version, and when using pipeline dependency, version information is passed downstream ).
4. Easy to manage various deployment environments.

 

After using the above methods, continuous integration of the project has already entered the best environment. Currently, you can:

1. All construction and deployment are automated;
2. Developers can get feedback within 20 minutes at most.
3. For each environment, it can be deployed four times a day.
4. Deployment error-free: because it is an automated process, each execution step is the same.
5. Machine Resource Reuse: Cruise automatically distributes work to 165 machines for construction and deployment.
6. Single dashboard view of everything!
7. Developers are very happy: they can submit multiple times without affecting others.
8. Testers are very happy: they can quickly get a good installer and click the button to complete the deployment.
9. Management staff are very happy: they can immediately learn about the current project status (which versions are being tested and those versions have already been demonstrated ).
10. It is easy to know which features are included in each version and which defects are fixed.

In the final continuous integration environment, a total of 260 machines are built and deployed, each with a release every six weeks. Each release requires 15 independent installation packages.