ci-Continuous Integration (2)-Software industry "pipelining" technology implementation

1 overview

Continuous integration (continuous integration) is a software development practice. The background and theoretical system are described in the previous section of this series of articles. This section is to undertake the above-mentioned theoretical ideas for specific technical implementation.

"Google Software Testing":

"Every day, Google has to test and publish millions of source files, hundreds of thousands of lines of code. Hundreds of millions of build actions trigger millions of automated tests and execute on a good hundreds of thousands of browser instances. In the face of these seemingly impossible tasks, how did Google test it? "

People who want to read this article can find their own answers.

2 main environment and tools

• Git code management System (for example: Gitosc)
• Linux operating system (UBUNTU)
• Jenkins System Software installation package
• The SDK for Jenkins ' Python language
• Python and Tornado Web Framework

The above technology selection is to use Open source and mainstream system as far as possible, the benefits are as follows:

• Save on software acquisition costs
• Have a wealth of references
• Capable enterprises or individuals can do custom extensions as required

3 Principle Analysis

In the previous article, a basic structure diagram of continuous integration was drawn, as follows:

The basic flow is as follows:

1. Developers push code to Git

2. Git notifies Jenkins

3. Jenkins Open Build

4. Build complete to enable subsequent tasks

   • Automated testing
   • Automating deployment

5. Jenkins notifies the automated publishing system

6. Release the system for continuous follow-up tasks ...

The systems involved are:

1. Code Version Management System
2. Automated build system
3. Automated test Systems
4. Automated release system

About the automated release system This is basically the operation and maintenance of the content, in the Internet product area and technical requirements are more complex, temporarily not included in this part of the content. But Jenkins can have an interface to inform the corresponding publishing system, in order to achieve the continuity of the event information flow

This article is mainly from the technical point of view these systems together.

4 Jenkins Installation and configuration 4.1 download and install

For the download and installation of Jenkins , refer to its official website.

1	https://jenkins-ci.org/

Download the Ubuntu/debian version directly and install it on the Ubuntu server.

Because the process is relatively simple, there are a lot of tutorials on the web, no longer repeat here. However, after the general Jenkins installation is complete, the initial permissions configuration will be more cumbersome, so this article focuses on the corresponding use of the scenario, to achieve a complete solution with the rights configuration.

5 Intranet Continuous Integration System

For teams that only use continuous integration within the intranet , the configuration of permissions is relatively straightforward. Because it is only in the intranet, you can relax the requirements of the permission, as long as the people outside the company's network cannot access the Jenkins service.

Note: If you want to form a complete stream of events with the webhook of the Git service, the Git service also needs to be in the intranet, otherwise the build event cannot be triggered by the code push event .

5.1 Permissions Configuration

Do the following with Jenkins :

[System Management]->[configure Global security]->[access Control]->[authorization policy]->[Project matrix authorization Policy]

Configure the anonymous user as follows:

• Overall:read-Enabled
• Job:read-Enabled

The specific configuration interface is as follows:

Note: If this is not configured, then the GIT-based build triggers mentioned later will not trigger the build by invoking the specified URL interface. Because Webhook can only construct a single simple HTTP request and cannot construct a session consisting of multiple requests, it is not possible to invoke an interface that requires identity authorization.

5.2 Building Triggers

In general, the build is based on the release of the code as a starting point, so you need to establish an event association with the GIT server and configure the build trigger in the configuration interface of the Jenkins specific project.

At present, the network is generally introduced this way, specific details, here will not repeat, interested students can self-search on the web.

The basic principle diagram is as follows:

5.3 Final Effect

The following effects can be achieved:

• Developers push code to a GIT server on the intranet
• Git service Webhook sends messages to the intranet and triggers the build
• Jenkins executes build-related commands

The features of the above intranet scheme are as follows:

• Advantages:

  • Simple configuration
  • No need to configure people for any development activities

• Disadvantages

  • Unable to restrict permissions for anonymous users
  • For security reasons, it can only be used in the intranet

Of course, for the development of a small team of relatively scarce resources , recommended by the above method to quickly build their own internal continuous integration system, after all, the first rapid production of their own featured products is the most important thing.

6 Public network Continuous Integration System

For the advantages and disadvantages of the previous intranet continuous Integration System , as an open as the main spirit of the Internet team, it must not be able to meet such a closed internal network system, so the following will be a grand introduction Public network continuous Integration System .

For the intranet system in the configuration of lazy , but actually in other places pay a huge price , these costs include:

• Unable to use the market's already mature public network git services (including but not limited to: Bitbucket,github,[email protected])
• Team members ' workplaces will be restricted and must be confined to the same intranet environment

Therefore, if the configuration personnel have a certain degree of system design capabilities and development capabilities, or proposed to build a public network continuous Integration System .

The public network scheme has the following characteristics:

• Disadvantages:

  • Requires configuration personnel with system design capabilities and development capabilities

• Advantages:

  • Ability to deploy the system to a public network in a login-authorized manner

    • Anonymous users who are not logged in cannot view any project information
    • Logged-on users can configure different permissions

  • Ability to accept events to trigger builds as a server

    • Can be triggered via the webhook of the public web git service
    • Can be triggered by any custom public IT system (buttons, SMS, etc.)

  • Highly scalable, theoretically accessible to any public Internet service

6.1 Permissions Configuration

The public network continuous building system has the following requirements for permission control:

• Anonymous users who are not logged in cannot view any project information
• Logged-on users can configure different permissions

Do the following with Jenkins :

[System Management]->[configure Global security]->[access Control]->[authorization policy]->[Project matrix authorization Policy]

You can set up three types of users and configure permissions:

• Normal User

  Once logged in, you can view the name of the project and the status of the build (primarily the basic viewing function)

• Admin user

  After logging in, you can build operations, add and delete tasks, and so on advanced operations (mainly project management functions)

• Anonymous user

  User not logged in, no permissions, only render login interface

With such a login authorization mechanism, there is no need to use the network for isolation, the system can be safely placed on the public network server.

The main reasons for the solution of the intranet system mentioned above are:

1. Git-based Webhook cannot initiate a valid build request for a build trigger interface request that requires authentication
2. Setting the build trigger interface to do not require authentication can cause the anonymous user to have too much permissions

If you deploy to a public network, you need to resolve the contradictions as above. A better idea is to:

1. Configure the appropriate user rights as required (see Public Network Permissions Configuration scheme)
2. Develop middleware to complete user login authentication for Build API

6.2 Building Triggers

In the context of Git's webhook features and the Jenkins Build feature, you can propose the following solutions:

Use Web services as middleware to impersonate a user login: a session that would otherwise require multiple requests becomes a single HTTP request (you can add an authorized token to the URL of a single request) so that it can be called by Git's webhook.

The basic principle diagram is as follows:

The main communication processes are:

1. Git Server receives the code and initiates a single HTTP request to the Web server (parameter with token)
2. WEB Server initiates authentication authorization requests to Jenkins server First
3. The Web Server then initiates a build request to Jenkins Server , triggering the build

Of course, since Jenkins provides the SDK for the Pyhon language, the above steps 2 and 3 can actually be simplified as a call to its SDK.

Installation method:

1	pip installpython-jenkins

The simplest examples of use are:

123456789101112131415161718192021222324252627

# coding:utf-8"""jenkins相关的工具函数及配置"""fromdtlib.dtlog importdlogimportjenkins__author__ =‘harmo‘jenkins_url = ‘http://jenkins.xxxx.com‘jenkins_user =‘jenkins_user‘jenkins_passwd =‘jenkins_user_password‘defbuild_job(project_name):    """    构建项目    :param project_name: 项目名称    :return:    """    jen =jenkins.Jenkins(jenkins_url, username=jenkins_user, password=jenkins_passwd)    jen.build_job(project_name)    print("build %s succeed"%project_name)if__name__ ==‘__main__‘:    build_job("your-project-name")

Developers only need to integrate this SDK into their Web applications and parameterize them to complete the middleware of Webhook and Jenkins.

6.3 Final Effect

The following effects can be achieved:

• Webhook requests to receive a git service
• Parse the code in the request to provide information, including but not limited to: Commit time, submitter, branch, remark, project name, etc.
• Conditional filtering and triggering Jenkins to build automatically (ex: This article monitors the release branch to trigger the build)

The advantage of using middleware is that it is well-tailored to build events, including branch monitoring, submitter permissions, and so on. Of course, you can use only the simplest features: As long as someone commits the code to the release branch, it triggers the automatic build process, which completes the process.

Of course, after the build succeeds to the release there are some subsequent processes, such as:

1. Developer completes code, after self-test, push code to release branch

2. Trigger automated builds, build success, and build artifacts

3. Publish the build product to the test server

4. Triggering automated test scripts

   • If the test does not pass, send a message to the person concerned to terminate the follow-up process
   • Notify the Automated Publishing system if the test passes

5. The process of publishing a build product to a production server by an automated publishing system

6.4 Other Instructions

Before we know that Jenkins has the Python language SDK, there is actually another way to complete the login authorization and invoke the build interface. Directly through the interface to impersonate the user login behavior (because the Jenkins login does not require a verification code), and then obtain the login successful SessionID , as an authorization token to invoke the built interface.

Specific technical implementation of the Code details are not present in the discussion, as long as the understanding of the principle of login, it is easy to develop.

The purpose of the instructions here is to actually understand the principle, even if the official does not provide some tools, there is still a way to complete the desired function.

7 Scenarios for Continuous integration

Once the platform for continuous integration has been built, there are some simple introductions to the projects it can apply to.

Basically any software project, from production to final release, can be divided into the following processes:

1. Code Development
2. Build build Release artifacts
3. Release online

Different types of projects, is nothing more than the construction process is different, this article is also a simple list of several types of project application process, about the specific technical means by each individual according to their respective projects to do the corresponding custom development.

7.1 Scripting service-side projects

The main representatives are: Php,python and so on.

Their construction artifacts are the source code of their own, so the whole process of continuous integration is as follows:

1. Developer publishes code to git repository
2. Jenkins Sync code to local (backup of release product, easy rollback)
3. Deploy the test server.
4. Execute Automated test scripts
5. Publish to production server or dismiss

7.2 Server-side projects that need to be compiled

The main representatives are: Java and so on.

The process is as follows:

1. Developer publishes code to git repository
2. Jenkins synchronizes the code to the local and uses build tools such as ant to generate the bytecode build product
3. Unified backup of the build product to the appropriate directory, the release product backup, easy rollback
4. Deploying a test server
5. Test......
6. Release......

7.3 Web Front End Project

The main representatives are: Javascript,css and so on.

The process is as follows:

1. Developer publishes code to git repository
2. Jenkins synchronizes the code locally and uses the front-end build tools (such as: grunt, etc.) to build the build artifacts
3. Unified backup of the build product to the appropriate directory, the release product backup, easy rollback
4. Deploying a test server
5. Test......
6. Release......

7.4 Mobile App Project

The main representatives are: Android and so on.

The process is as follows:

1. Developer publishes code to git repository

2. Jenkins synchronizes the code locally and uses the build tool to build the build product APK

3. Unified backup of the build product to the appropriate directory, the release product backup, easy rollback

4. Installing to a device, performing a test

5. Test

   • If the test is passed, publish it to the big application market
   • If the test does not pass, the bug is dismissed after it is found

8 Operating effect

Jenkins system Operating Interface:

Jenkins itself can achieve the following functions:

• Complete project continuous integration and continuous release
• Documenting the build process
• After the completion of the construction can send e-mail or notify the appropriate personnel, to form a good feedback mechanism

Through the system mentioned in the first half of the article, the team can achieve the goal of continuous delivery and continuous deployment , thus achieving a rapid iteration of the project.

Continuous Delivery:

Continuous deployment:

Of course, there are further extensions that need to be tested and implemented by developers.

9 Summary

This article, as a specific technical implementation part of the Continuous Integration series, introduces a general technical framework solution. The rest is that the various engineering features for you are written in various scripts and installed in various environments. This theory and practice, constitutes a continuous integration system capable of providing productivity, everyone enjoy it.

The big internet companies that are currently represented by Google are basically keeping the pace of development. The "Google software testing approach" mentioned in this way of production, but did not give specific technical solutions, this article will be the technology landed on this idea, and publish this program, hoping to give some help to the later.

(If you are interested, you can follow a complete example of continuous integration for a specific project, including the authoring of the build code, the process, and so on.) But if the response is not big, then no longer waste time.

Http://www.cnblogs.com/beer/p/5196438.html

ci-Continuous Integration (2)-Software industry "pipelining" Technology Implementation (EXT)