Xiao: DevOps practice sharing for HPE IT

This article is from HPE and Msup co-hosted the technology Open Day HPE testing technology director Xiao Sharing, compiled by the one-Bai case.

First, the meaning of DevOps analysis

This is the trend graph for devops. The concept of DevOps was probably introduced in 2009, and in 2010 Some companies started experimenting, followed by a continuous increase in devops, which is the amount of search we get from the Google search for devops keywords, a curve that shows the exponential growth of devops heat. So I expect that DevOps will still be a very interesting technology in 2016.

What is DevOps?

We did a lot of research while piloting DevOps, and we did a lot of searching on the web. It is common to say that DevOps is a culture that breaks down the barriers to work between development, operations, and testing. There are barriers to development, testing, operational responsibilities, the purpose of inconsistency. For example, the main purpose of development is to quickly achieve business-level new requirements and functions, to the customer use, and then continue to update according to the feedback; The main purpose of operations is that production must be fast and not problematic, so they are not very fond of "change", because change can bring about risks, followed by a variety of problems, This is the last thing they want to see.

These considerations are right when viewed from their point of view, but the market now demands that the ultimate goal of all developers is to "help business units create business value" rather than narrowly meeting their goals. How to create business value? Fully meet the needs of customers. Customer demand is changeable, and the speed of change, which requires us to keep up with their pace, quickly achieve the latest features and to the customer use, if a system is stable but no one to use, it can not bring us anything.

The explanation for DevOps on Wikipedia is the culture, trend, or practice of communicating and collaborating between software developers and IT operations technicians. Through automated (software-paid) processes, the ability to build, test, and distribute software is faster, more frequent, and more reliable. The previous development, testing, operation and maintenance of the three may be independent, now requires a tighter. In a nutshell, DevOps is: development Testing (Test) (transport) integration, agile and efficient automation. This is also our definition and understanding of devops.

Since the concept of DevOps in 2009, many companies have started to implement DevOps, the more famous abroad are Google, Facebook, etc., the domestic famous Baidu, Huawei, Ali, Flickr, etc., they implement the results of the devops is 10 times a day to deploy, This is very remarkable.

Ii. HPE It practices in DevOps

The HPE IT department introduced the DevOps concept at the end of 2014, found some internal agile projects in 2015, and in 2016 we started to promote devops in more than 400 applications.

1. Introduction of the actual background of DevOps

Now that we want to promote the practice, let's look at the present situation. It has been a long time since it has been using agile in the development process, so it can be seen that development actually employs an agile iterative approach, with each sprint producing a product that can be released. To the OPS side, we have to follow the corporate publishing rules: once every three months, we have to wait for the release window. Depending on the technology used, there are different operations teams to help you deploy and monitor your production environment, so we have to find the appropriate OPS team and submit an RfC 7 days in advance. After these processes have been completed, the OPS team began to deploy, then to write a "deployment document" tells him how to upload to the server, which file deleted, which file backup, and so on, I see the largest possible more than 10 pages, many times OPS personnel to control many projects, deployment and problems. The entire process is cumbersome, agile in the implementation of the development of the application, but in the operational dimension is still insufficient. Finally we see "Deployment" become the last kilometer, the last kilometer if the "agile" can not be achieved, the front of the "agile" may be white.

2. DevOps continuous delivery, continuous deployment
To address these issues, we present a devops approach that is implemented through continuous delivery and continuous deployment. Continuous delivery and continuous deployment broadly includes 4 parts: 1. Related to development and development; 2.QA testing related; 3. User testing related; 4. Production operation and maintenance related. Through continuous integration and continuous delivery of the entire devops, the development, testing, and operations of the three are included.

Continuous integration in the development process
We emphasize that any changes made by each developer must be tested locally through the new functionality of the unit. After the unit tests for the new features are consolidated and then submitted to the Code warehouse, once the code is submitted to the code warehouse, our servers are automatically triggered, automated builds and more comprehensive unit tests are made, and then deployed to ITG, if the new code passes through ITG's test will be shown when the green, The display is red if not passed.

Sometimes the development is very anxious, constantly write new code hope to do faster, do not pay attention to details, QA find it difficult to understand, sometimes even build up, submitted to have no way to deploy. For example, we require that all code be lowercase, one time our development uses some third-party library files, and this third-party library file will have uppercase letters in the middle, these uppercase letters appear in the configuration file, to the deployment can cause the QA environment error. Because our QA environment is a liunx system, it is more sensitive to case.

Development often says: "On my side is good, is your problem, not my problem." "In order to avoid this situation, we have to ask ITG to do the testing, this ITG and production environment is similar, are liunx."

Code branching policy: Single branch structure

Next, we introduce a branching strategy. Our code branching strategy is called a single branch structure, and the benefits are:

(1) If there are too many branches at the time of development, the final merger can be cumbersome. Every time a merge branch developer spends less than half an hour to one hours, and a half-day to one-days, the test staff will do regression testing after the merger. The whole process went down, and the day passed, so we didn't quite approve of having too many branches.

(2) Our continuous Integration services are monitored by the backbone, and any code submissions on the backbone will be found. If we do not look at these branches, these branches do not continue to integrate, so the risk is great.

This is the strategy of our branch, we require each development to submit at least once a day code to the Code warehouse. Submitting every day, testing every day is not so easy, and if you do it for a long time, you will not only find a lot of problems, but also change them for a very long time. We can do about 1.5 times a day/person code submission.

Continuous integration of testing links
Continuous testing

The test is mainly a new functional and non-functional regression test. Every place has a circle with a haircut on it, which means to do it continuously, rather than just once, so we call it "keep doing the test." We try to use the same operating system, data, and so on to make tests, so we have confidence when the regression test is done, because the test environment is consistent with the production environment and it is safe to put it into production after testing. Some of the concerns we have on shift LETF in the test are:

(1) The demand for development, after the development of the test, we do not agree with each other, discussion to discuss to everyone began to talk, in the end is the development and testing of the needs of the understanding is not the same or what? It's a waste of time to pull in the BA and so on, which is not what we want to see. What we want to see is that the entire automated process is over, so that agile and fast delivery can be achieved. So we're going to let the test get involved when we do the need.

(2) The test should be done as soon as possible. When we were agile, the development was agile and then passed on to the test, but the test didn't have to be done right away because it had to be deployed to get someone to deploy to the QA system. The original is manually deployed, such as our front-end has 5 modules, the back end has 1 modules, all deployed by hand to do nearly 20-30 minutes.

At the beginning of everyone's work is not very busy time can let the development of people to help deploy, and in the last few days people are very busy, development there are many needs to modify the place, there is no time to do other things, so we must adopt automated deployment. When we do "continuous integration", we have deployed scripts that are further processed and improved to allow him to deploy directly to this environment. This allows QA to directly select the QA environment to deploy, or to test immediately when any defect is submitted.

Continuous regression

Another concept we introduced in testing is called "continuous regression." If you end up with a regression test, you'll find a lot of flaws, and these flaws can make the development changes and time will be problematic. In fact, this is one of the concepts of shift letf, as soon as possible to do regression testing and continue to do. We will do a regression test in about two days or so, and the good thing about it is that it is relatively easy to do the final round of regression testing in two weeks, because many of the regression problems have been discovered and modified very early on.

Automated testing

Automated testing is important because the frequency of continuous deployment and regression testing is very high, and the continuation of the regression two days, if the QA staff to do manual testing they will certainly say: "Two days at a time how to finish?" "This requires automated testing.

After each automatic deployment to the QA link, our "Continuous Deployment Server" will tune the framework for automated testing, followed by a focus on stable functional modules. I don't want to automate everything that's not realistic, and I don't want our tests to spend too much time maintaining test scripts.

Non-functional also need to pay attention to the previous development and testing more attention to functional requirements, and non-functional demand operation and maintenance of more attention. There are a lot of times when customers complain about performance: I don't log in for two minutes. DevOps is to break down barriers and fuse each teams. Therefore, the development and testing must pay more attention to the non-functional points, concern about operational requirements for performance security.

Continuous integration of operation and maintenance links
Quasi-production environment

We will give the user "acceptance test", but also to ensure that his environment and production environment is more consistent, such as data consistency, deployment files consistent. Starting with continuous integration, are files deployed to ITG or QA or STG or production environments? The deployment script has only one, and we only give him a parameter that says what to target, and then he goes to do the deployment. What are the benefits of doing this? In fact, this file itself is to be tested, or automatically deployed to the production environment will be very dangerous. By constantly testing, including the QA environment to deploy every day, so many deployments determine that this script is no problem. Everything will be placed in the code repository, including scripts, and the deployed scripts are taken from the code repository in real time and deployed, which is a quasi-production environment.

Continuous deployment

Deployment to the production environment, we are faced with a complex problem, to do a lot of manual operation. How to improve this last kilometer? We consulted with OPS: could you not say to me in every deployment that you want to accomplish this, to accomplish that, but that we start working on it before it is done, and that the process I deploy is done with your team. Because you have a requirement for deployment, you confirm that the deployment file is OK, meets the requirements for "security" "stable" in the production environment, including some security tests, performance tests that we have to do before deployment, are automatically triggered in our deployment files, We use this deployment file to deploy, and then all the logs are recorded. The advantage of this is that automation is done without the need to rely on operations personnel to deploy manually.

In the production environment we will have monitoring, the entire monitoring system is divided into three layers: business monitoring, application monitoring, system monitoring. As you can see, we've actually included operations in the process of development and deployment. We now use more cloud services, using the "cloud" after the operation of some basic services to the cloud, such as we use the Amazon or Microsoft cloud, if there is a problem he will send us an email alert.

With a process like devops, we can turn the original high-risk release into a one-time release, but the frequency of the release is high. Business people are also acceptable, sometimes they have to make some urgent business changes, in the previous reply is: Wait three months, three months later to you, so he would not want to, because the emergency on-line, now get the answer is: It's okay, two weeks later, the business people are more willing to accept the latter, Because it can ensure the stability of production, ensure the smooth development. This is what devops can do to help us solve problems, reduce the risk of release, and streamline the whole process automation.

Iii. Benefits of DevOps
1, triggered by the code submission: Eliminate wait time + quick feedback to the business and developers.
2. Each change corresponds to a delivery pipeline: It makes it easy to locate and debug the problem.
3, the whole process automation: stable, fast, predictable delivery results.
4, Continuous automated regression testing: Improve the quality of delivery.
5, the facilities are shared and provided on demand: maximize the use of resources.

This is the implementation of DevOps in agile. Agile people are familiar with, such as four weeks an Agile project: the first week to do agile iteration, planning meetings, Monday began to do, after the meeting development and testing as well as the PO or BA to discuss again, Thursday development, development to the end of the third week, followed by the return, acceptance testing, release on the line such a process. Agile usually finally we will be deployed in STG to the PO to do the demo, in order to find the problem as early as possible, we will be in the second week of Thursday or the third week of Wednesday to deploy the completed parts to STG, sometimes when the business staff to ask for the demand is not very clear, and when you are done, he may feel not very good, So we have to test early, have problems to change as soon as possible, finally we publish to the production environment to do smoke test, and then create a new branch here. All the tests are recycled, and all the code is in the code repository.

Iv. continuous deployment of pipeline and tools, DevOps industry toolchain

Five, this is DevOps

The development of testing operations before the fragmented, now integrated into a collaborative cooperation, happy life together, this is the focus of the DevOps culture.

Xiao: DevOps practice sharing for HPE IT