Dockone Technology Share (39): Continuous integration workflow based on Docker and Git

This is a creation in Article, where the information may have evolved or changed.
"Editor's note" This time the main share of GIT multi-branch pull request works with Docker in continuous integration applications.

First introduce our company and team situation, Kingdee Cloud home, mainly to do corporate social, including IM, Weibo, etc., our services mainly include free public cloud services, and the private cloud deployment of various enterprises, including Vanke, Haier are our customers.

Take a look at our current system and technology stack:

1. There are 449 items present in the system.
2. The technology stack used in the development is Java (Standard container project using Jetty, Play Project), node. js, PHP and so on.
3. Two product lines, nearly 100 orwith developers.
4. We also have a lot of private cloud deployment requirements for our businesses.

With so many projects and people, the product iterations are very fast, which requires us to support the development department. Let's talk about some of the practical problems we have in this situation:

Development Engineer says:

• Who took the data out of my MQ?
• Who changed the code of the API, causing my program code to error?
• Local programs modify the deployment, where more than one problem occurs, and the morning passes.
• ......

The test engineer says:

• The dev environment is always deployed, resulting in no testing.
• When testing a bug, someone else commits the code, causing another bug.

Product manager says:

• The code merge of the Realease branch resulted in a formal environment where other features were not completed.
• Because interdependence leads to self-developed functionality, it must wait for someone else.
• The above problems cause the function progress can not be scheduled.

Each of the above roles have their own pain points, we have to analyze the situation caused by the above reasons, in fact, most of the situation is due to the isolation of the environment is not enough to cause, know why we have to look at our response strategy:

• Dividing the boundaries of each character;
• Adopt a Pull-request approach.

So let's take a look at our infrastructure and our role:

1. Development Engineer
2. Test Engineer
3. Products
4. Other

Then we'll divide the boundaries of each role:

1. Development engineers should only focus on GIt code and JIRA bugs and new feature.
2. The test engineer only focuses on JIRA bugs and new feature tests.
3. Products and others are concerned only with JIRA submitted bugs and the resolution of new feature.

The division of roles, and our infrastructure. Look at the branching view now on Git, assuming we now have two main branches

1. Master: Representative of the Development section of the stable branch
2. Release: Represents a stable branch on the line

Our role boundaries and the basic facilities involved are enough to read, now look at how the development engineer works:

• First step: Fork A corresponding branch iss001 from master based on the bug on Jira or new feature.
• The second step: the development engineer, commits the code on the Fork branch, the git hook notifies the Jenkins build corresponding image.
• The third step: Jenkins will deploy the corresponding image to the server cluster, the developer can iss001.kingdee access the service just corresponding branch, iss001 the corresponding Fork branch name, we require the only.

Let's take a look at how the test engineer works:

• The first step: after the test engineer has been verified, this bug or new feature function, will put the code merge into the Master branch, on this branch to do performance testing and smoke test.
• Step two: After performance testing and smoke testing, you can select the features you want to publish through cherry pick.
• Step three: After the release, the test engineer closes JIRA, we notify the backend Kubernetes cluster by the web hook to delete the resources allocated by the corresponding branch.

This completes the entire system process, where each of these branches can be assigned a resource deployment environment for testing and development.

Q&a

Q: The development of each submission of a bugfix, will trigger Jinkens to build the image, so many developers, is not to build a lot of mirrors?

A: There is no mistake, we are triggering the build image every time, because the image is layered, the underlying existing parent object, is not stored, only the parts of the change, so the disk space is very low, at the beginning of the system, I did statistics, 1000 image is less than 9G, There are many base images.

Q: Would you like to ask a cluster-related, such as Docker deployment, to call the Docker deployment container directly, or through ansible or other tools?

A: With the Kubernetes management cluster, the release of the work is relatively simple, not Ansible. But Ansible still has its uses, such as cleaning up obsolete clusters of
Image, and the container that have exited.

Q: Hello, I have done similar service before. "The third step: Jenkins will deploy the corresponding image to the server cluster, the developer can access the service just corresponding branch by Iss001.kingdee the domain name," a separate branch resolved the corresponding bug, But the actual production is very easy to modify a bug caused by other bugs, how do you control the overall stability? How can you improve the meaning of this single bug branching single test environment?

A: This pull-request works in response to functional development, such as the long-term development of a new feature, you just said a bug produced another bug, we have a regression test, we have a smoke branch, continue to do functional regression testing, Only through the ability to cherry pick to release.

Q: How do you isolate external services such as REDIS/MQ that are dependent on the test environment? Does each test pull up a separate set of externally dependent services?

A: We use multiple means to achieve shared data: Master, Smoke, Release branch testing has its own independent middleware, if you do not have access to shared data, you can deploy such as MQ image, code-level, such as the name of MQ key plus the machine's IP.

Q: Is there any use for Mesos? Is it easy to run into problems? There doesn't seem to be much documentation in this regard.

The A:mesos is a two-level dispatch that can be used to balance resources like there are multiple sets of clusters, such as deploying Hadoop and storm, typically using storm to process real-time requests and Hadoop doing offline work. Night and day there is a possibility that Hadoop is idle, but storm may be very busy, when the Mesos such a two-level scheduling can balance resources, save costs, we do not have such a demand for the time being. As for the documentation, I did not delve into it, and suggested looking at official documents.

The above content is organized according to the December 22, 2015 Night Group sharing content. Share people: Hu Shi, Kingdee, a cloud home engineer, is primarily responsible for architecture and performance tuning, is familiar with Java, JVM-related technologies, focuses on Liunx kernel, has a strong interest in Docker technology stack, and currently studies Docker and Kubernetes in Applications in the actual production environment. Dockone Weekly will organize the technology to share, welcome interested students add: LIYINGJIESX, into group participation, you want to listen to the topic can give us a message.