Continuous integration and release behind the ultra-large flow e-commerce platform system

content Source:September 2, 2017, beautiful United Group Technical experts Zhiqiang in the "Seven Qiniu & beautiful United Group Architect Practice Day: CI/CD Landing Best Practices" for "continuous integration and release in the beautiful United group practice" speech sharing. IT Id:itdakashuo, as an exclusive video partner, is released by the organizer and speaker for review.

Read the number of words:2380 | 4 min to read

for guest speech video playback and ppt, please click: Http://t.cn/RscHvFO

Summary

Release as the last step of the application before the launch, has always been the operation of the more frequent and relatively high risk operations, the release system not only to improve the release efficiency, more importantly, to ensure the stability of the release process, reduce the failure caused by the release. This presentation focuses on the evolution of the United States-linked release & Continuous Integration System and the practice of improving release efficiency and ensuring system stability.

Evolution of the publishing system

The evolution of the publishing system represents the evolution of the operation and maintenance system and even the technical structure of the company to some extent.

Technical architecture-Early (2011-2013)

The earliest development language was PHP, the most popular open source runtime environment was LNMP, and code management was SVN.

The first is the human flesh release, and later with the PHP main station release system. It can be used instead of human flesh to publish and operate, support file publishing, rollback functionality, and the simplest approval operation.

Business architecture-Mid-term (2014-2015)

By the year 2014, our business structure has been adjusted to create our own trading platform.

The development language has become Java, and the operating environment is Tomcat, and code management uses Git.

After the business system was changed to Java, more challenges were posed to the release system. There is a big difference between Java Publishing and PHP publishing, so we did a Java publishing system.

The publishing system also changes to Starkplus, which requires support for more release types, release modes, and more publishing features.

The publishing system features many types of support, such as Java, C + +, NodeJS, PHP, Golang, Css_js, and two-party libraries.

There are also many publishing policies, with Batch publishing, group publishing, streaming publishing, auto-publishing, and custom publishing.

More features. The original system can only publish a specific branch at a time, and now there is a case where multiple branches are being developed in parallel, so we need to integrate multiple branches.

Our application is deployed in a number of computer rooms, each machine room configuration may be different, the construction is not the same, so need to build a multi-machine room.

At the beginning, there were only three sets of environments, and it was later discovered that three sets of environments were not enough to meet our research and development needs, requiring multiple sets of environmental deployments.

Docker is currently a very popular container, and we now have some applications on Docker, to do some support for Docker. It also supports the hybrid release of Docker and KBM.

There are integration tests, security scans, performance testing, and jar inspection, which are tools that other business teams do, and we integrate them into our release systems to enhance these capabilities.

The road to Practice

Fundamentals of Operations-standardization

First of all, the basic software and configuration standardization, OS, JDK, Tomcat, Nginx and so on for operations to provide a set of the most standard environment, all applications are running in the same environment.

Standardization of the application itself configuration, application naming, configuration management, start-stop script, detection script, deployment directory, log directory these have a unified standard.

We provide an application template, if the application fully conforms to our standards, there is no need to change the direct access, but also some special applications may be different.

Application Configuration Management

The application type configuration can use our standard templates, or you can do some custom functions, mainly people roles, application types, start-Stop commands and package information. In fact, they are integrated in our publishing system, and later we found that these settings are not only the release system to use, many other operations and maintenance systems, business systems will be used. So we listed it and set up a configuration center.

is a dependency of our release system, where a lap is its core dependency, CMDB Management Server, Configuration Center management application configuration, opsagent deploy an agent on each machine to perform some ongoing operations on the server. Gitlab does code management, and the process engine is used to approve content.

The perimeter is used to enhance our functionality and some external dependencies, such as monitoring, security scanning, and so on.

Publishing the system architecture is very simple, mainly two parts, one is the Java front-end, used to do page and process control. The following is a set of workers written in Pathon to perform specific actions, such as building, merging, and deploying code. In the middle, a task queue and decoupling is done through an MQ, which is passed between them via DB.

Branch Management

is our branch management. All development branches are derived from master, the release system will pull out a release from master when the development branch is finished, and release the feature branch one after the other, and publish the release branch after closing. The release branch is sent to the line successfully and then it is returned to master, creating a baseline.

New & Import Changes

There are two ways to create a change, either a new change, one from master, or an import change that already has a branch from another development branch that needs to be pulled out manually for import.

Integration & Publishing

is our integrated publishing page. The top level is our three set of deployment environments; The underlying information for the publication includes the app name and the branch name of the current publication, and so on, the following is our release process, which differs depending on the type of application; The branch of the integration area is the current branch in the release, and the integration area is already developed, but not yet published.

Enter the line

The pre-send must be successfully deployed, and the checklist of the integrated zone will be fully approved.

Code Merge

Manually resolve conflicts that occur during the code merge process. Release Branch replacement policy is the newly added branch or modify the feature branch code, the release branch will not change, no longer resolve the second conflict.

Different application types are built differently, and build is based on the merge completion of the release branch, like Java, C + +, Golang, Nodejs is built in a Docker container, the build will be output after completion.

Health Check

Every application has a health check Url:/status

When accessing/status, check the core dependencies (DB, cache, dependent applications), and preheat the data.

The execution returned "SUCCESS" successfully, and the remaining conditions were failed.

Release Plans & Approvals

Daily release is from Monday to Thursday working hours, approved by supervisor;

General emergency release in Friday, weekend, by research and development d for approval;

Holidays, such as statutory holidays, operators, such as sealing network, but also research and development d approval;

Special period, such as Big promotion, group launch, etc., in theory is not allowed to publish, if required to be issued by the CTO approval.

Our Features

Research and development process closed loop

Deep integration release systems and project Management Systems (PMO), requirements, projects can be created, associated with changes. After the change is released, the PMO system can be notified to update the requirements and project status so that the purpose of each release can be clarified.

Multi-machine room, multi-grouping construction

The same application has different configurations in different computer rooms, and there are differences in the services provided in different groupings.

Offline & Pre-sending multiple sets of environments

Because of the many requirements and many changes, deployment changes are very frequent and testing always complains about environmental instability. Large projects want to be able to monopolize a set of project environments to address environmental isolation.

Jar Packet Detection &diff

Jar Packet Collision Detection: Jar package collisions can cause inexplicable problems that are difficult to troubleshoot.

Snapshot packet detection: Snapshot version update frequency is high, not controllable.

Jar Package Version limit: Obsolete version, bug version cannot be used.

Jar Package diff: View the jar package differences for this release and baseline version.

Stability

Front-end scanning: For the release of Css_js to do the front-end code quality inspection;

Security Scan: Static security analysis of Java code;

Integration test: Perform unit Test and interface test at the same time as pre-launch environment release;

Performance monitoring & Pressure measurement: Perform performance measurements on beta machines after online Beta release.

I will be here today to share, thank you all!