Golang based on Gitlab CI/CD deployment scenario

This article is reproduced, original: Golang based on Gitlab CI/CD deployment plan

Overview

Continuous integration is a software development practice in which team development members often integrate their work, with at least one integration per member per day, which means multiple integrations per day. Each integration is verified through automated builds (including compilation, publishing, automated testing) to detect integration errors as early as possible.

Continuous deployment is the rapid delivery of high-quality products through automated build, test, and deployment cycles. To some extent, it represents the degree to which a development team is engineered. After all, the fast-running Internet company's labor cost will be higher than that of the machine. The investment machine optimization development process will also improve people's efficiency and maximize engineering productivity.

1. Environmental preparedness

This test is based onCentos 7.3thedocker 17.03.2-ceenvironment. Docker installation here will not repeat, provide the official link bar: Get Docker CE for CentOS

1.1. Docker Boot Gitlab

The start command is as follows:

docker run --detach \
--hostname gitlab.chain.cn \
--publish 8443:443 --publish 8080:80 --publish 2222:22 \
--name gitlab \
--restart always \
--volume /Users/zhangzc/gitlab/config:/etc/gitlab \
--volume /Users/zhangzc/gitlab/logs:/var/log/gitlab \
--volume /Users/zhangzc/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce

Port,hostname, volume according to the specific circumstances of the specific settings

1.2. Docker Boot Gitlab-runner

The start command is as follows:

sudo docker run -d /
--name gitlab-runner /
--restart always /
-v /Users/zhangzc/gitlab-runner/config:/etc/gitlab-runner /
-v /Users/zhangzc/gitlab-runner/run/docker.sock:/var/run/docker.sock /
gitlab/gitlab-runner:latest

Volume according to the specific circumstances of the specific settings

1.3. Image authoring for integrated deployment

Our integration and deployment needs to be carried out in a single container, so we need to create a mirror and install some necessary tools for integrating and deploying related operations. Currently our projects are based on Golang 1.9.2, where a specific image is mapped based on the golang:1.9.2 image.

Dockerfile content is as follows:

# Base image: https://hub.docker.com/_/golang/
FROM golang:1.9.2
USER root
# Install golint
ENV GOPATH /go
ENV PATH ${GOPATH}/bin:$PATH
RUN mkdir -p /go/src/golang.org/x
RUN mkdir -p /go/src/github.com/golang
COPY source/golang.org /go/src/golang.org/x/
COPY source/github.com /go/src/github.com/golang/
RUN go install github.com/golang/lint/golint

# install docker
RUN curl -O https://get.docker.com/builds/Linux/x86_64/docker-latest.tgz \
    && tar zxvf docker-latest.tgz \
    && cp docker/docker /usr/local/bin/ \
    && rm -rf docker docker-latest.tgz

# install expect
RUN apt-get update
RUN apt-get -y install tcl tk expect

Thesegolintare tools for Golang code style checking.
dockerBecause of the need to use the hosted Docker command in the container, it is necessary to install a Docker executable file and then mount the host's/var/run/docker.sock file to the same location in the container when the container is started.
expectis the tool for SSH to automatically log on to the remote server, where the tool is installed to enable remote server-side deployment of the application

In addition, in the installation of Golint, it is necessary to golang.org download the source, because of the relationship between the wall, go get command can not be executed. In order to deal with this problem, first download the relevant source code through other channels, put in the specified path, and then copy into the image, and perform the installation.

The following script is used to generate the image:

#!/bin/bash

Echo "Extract the files needed to build the image"
Source_path="source"
Mkdir -p $source_path/golang.org
Mkdir -p $source_path/github.com
Cp -rf $GOPATH/src/golang.org/x/lint $source_path/golang.org/
Cp -rf $GOPATH/src/golang.org/x/tools $source_path/golang.org/
Cp -rf $GOPATH/src/github.com/golang/lint $source_path/github.com

Echo "Building a Mirror"
Docker build -t go-tools: 1.9.2 .

Echo "Remove the files needed to build the image"
Rm -rf $source_path

After the image is generated, push to the mirror warehouse and pull the mirror on the Gitlab-runner server

The Gitlab and Gitlab-runner of this trial are under Docker running on the same server.

2. Runner Registration and configuration

2.1. Registration

After the environment is ready, execute the following command on the server to register runner:

docker exec -it gitlab-runner gitlab-ci-multi-runner register

Follow the prompts to enter the relevant information

Please enter the gitlab-ci coordinator URL:
# gitlab url, such as: https://gitlab.chain.cn/
Please enter the gitlab-ci token for this runner:
# gitlab->your project->settings -> CI/CD ->Runners settings
Please enter the gitlab-ci description for this runner:
# Example: demo-test
Please enter the gitlab-ci tags for this runner (comma separated):
# Example: demo
Whether to run untagged builds [true/false]:
# true
Please enter the executor: docker, parallels, shell, kubernetes, docker-ssh, ssh, virtualbox, docker+machine, docker-ssh+machine:
# docker
Please enter the default Docker image (e.g. ruby:2.1):
# go-tools:1.9.2 (previously created image)

2.2. Configuration

After successful registration, you also need to do some specific configuration on the original configuration, as follows:

[[runners]]
  name = "demo-test"
  url = "https://gitlab.chain.cn/"
  token = "c771fc5feb1734a9d4df4c8108cd4e"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "go-tools:1.9.2"
    privileged = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
    extra_hosts = ["gitlab.chain.cn:127.0.0.1"]
    network_mode = "host"
    pull_policy = "if-not-present"
    shm_size = 0
  [runners.cache]

Here first explain the process of Gitlab-runner, Gitlab-runner at the time of execution, will start a container according to the above configuration, that is, the configuration of Go-tools:1.9.2,b all the startup parameters will be in the [Runners.docker] Under the node configuration, including Mount Ah, network AH and so on. After the container is successfully started, the container is used to gitlab the pull code, and then the test is performed according to the rules that you define, and then it is deployed when all the tests are successful.

volumes: is a docker command that can execute a host in a container.

extra_hosts: Add a host map to the Gitlab and map to the 127.0.0.1

network_mode: Make the network of the container consistent with the host, so that you can access the Gitlab through 127.0.0.1.

pull_policy: When the specified image does not exist, it is pulled via Docker pull

3. Defining rules

Create a file in the Gitlab project root directory.gitlab-ci.yml, fill in the runner rules, and refer to the official documentation for the specific grammar lesson: https://docs.gitlab.com/ee/ci/yaml/

3.1. Go Integration command

Here are a few Golang common integration commands

1. Package List
   As described in the official documentation, the Go Project is a collection of packages. Most of the tools described below will use these packages, so the first command we need is a way to list the packages. We can do this with the Go list subcommand.

go list ./...

Please note that if we want to avoid applying our tools to external resources and limiting them to our code. Then we need to remove the vendor directory, the command is as follows:

go list ./... | grep -v /vendor/

1. Unit Test
   These are the most common tests that you can run in your code. Each. Go file requires a file that can support unit testing_test.go. You can run tests for all packages by using the following command:

go test -short $(go list ./... | grep -v /vendor/)

1. Data competition
   This is often a difficult problem to avoid, and the Go tool has it by default (but only for use on LINUX/AMD64, FREEBSD/AMD64, Darwin/amd64, and WINDOWS/AMD64)

go test -race -short $(go list . /…| grep - v /vendor/)

1. Code overrides
   This is an essential tool to evaluate the quality of the code and to show which part of the code has been unit tested and which is not.
   To calculate code coverage, you need to run the following script:

PKG_LIST=$(go list ./... | grep -v /vendor/)
for package in ${PKG_LIST}; do
    go test -covermode=count -coverprofile "cover/${package##*/}.cov" "$package" ;
done
tail -q -n +2 cover/*.cov >> cover/coverage.cov
go tool cover -func=cover/coverage.cov

If we want to get an HTML-formatted coverage report, we need to add the following command:

go tool cover -html=cover/coverage.cov -o coverage.html

1. Build
   Finally, once the code is fully tested, we compile the code to build a binary that can execute.

go build .

1. Linter
   This is the first tool we used in the code: Linter. Its role is to check the code style/error. This sounds like an optional tool, or at least a "nice" tool, but it does help to maintain a consistent code style on the project.
   Linter is not part of the go itself, so if you want to use it, you need to install it manually (the previous Go-tools image we have installed).
   The use of the method is fairly straightforward: just run it on the code package (or point to the. go file):

$ golint -set_exit_status $(go list ./... | grep -v /vendor/)

Note the-set_exit_status option. By default, Golint only outputs style problems with a return value (with a 0 return code), so CI is not considered an error. If-set_exit_status is specified, the return code for Golint will not be 0 when any style problems are encountered.

3.2. Makefile

If we do not want to write in the.gitlab-ci.ymlfile too complex, then we can put all the tools used in the continuous integration environment, all packaged in makefile, and call them in a unified manner.

In this case, the.gitlab-ci.ymldocument will be more concise. Of course, makefile can also invoke the*.shscript file

3.3. Configuration examples

3.3.1. gitlab-ci.yml

image: go-tools:1.9.2

stages: 
  - build
  - test
  - deploy

before_script:
  - mkdir -p /go/src/gitlab.chain.cn/ZhangZhongcheng /go/src/_/builds
  - cp -r $CI_PROJECT_DIR /go/src/gitlab.chain.cn/ZhangZhongcheng/demo
  - ln -s /go/src/gitlab.chain.cn/ZhangZhongcheng /go/src/_/builds/ZhangZhongcheng  
  - cd /go/src/_/builds/ZhangZhongcheng/demo

unit_tests:
  stage: test
  script: 
    - make test
  tags: 
    - demo

race_detector:
  stage: test
  script:
    - make race
  tags: 
    - demo

code_coverage:
  stage: test
  script:
    - make coverage
  tags: 
    - demo

code_coverage_report:
  stage: test
  script:
    - make coverhtml
  only:
  - master
  tags: 
    - demo

lint_code:
  stage: test
  script:
    - make lint    

build:
  stage: build
  script:
    - pwd
    - go build .
  tags:
    - demo

build_image:
  stage: deploy
  script:
    - make build_image
  tags:
    - demo

3.3.2. Makefile

PROJECT_NAME := "demo"
PKG := "gitlab.chain.cn/ZhangZhongcheng/$(PROJECT_NAME)"
PKG_LIST := $(shell go list ./... | grep -v /vendor/)
GO_FILES := $(shell find . -name '*.go' | grep -v /vendor/ | grep -v _test.go)

test: ## Run unittests
    @go test -v ${PKG_LIST}

lint: ## Lint the files
    @golint ${PKG_LIST} 

race: ## Run data race detector
    @go test -race -short ${PKG_LIST}

coverage: ## Generate global code coverage report
    ./scripts/coverage.sh;
    
coverhtml: ## Generate global code coverage report in HTML
    ./scripts/coverage.sh html;

build_image:
    ./scripts/buildDockerImage.sh

3.3.3. coverage.sh

#!/bin/bash
#
# Code coverage generation

COVERAGE_DIR="${COVERAGE_DIR:-coverage}"
PKG_LIST=$(go list ./... | grep -v /vendor/)

# Create the coverage files directory
mkdir -p "$COVERAGE_DIR";

# Create a coverage file for each package
for package in ${PKG_LIST}; do
    go test -covermode=count -coverprofile "${COVERAGE_DIR}/${package##*/}.cov" "$package" ;
done ;

# Merge the coverage profile files
echo 'mode: count' > "${COVERAGE_DIR}"/coverage.cov ;
tail -q -n +2 "${COVERAGE_DIR}"/*.cov >> "${COVERAGE_DIR}"/coverage.cov ;

# Display the global code coverage
go tool cover -func="${COVERAGE_DIR}"/coverage.cov ;

# If needed, generate HTML report
if [ "$1" == "html" ]; then
    go tool cover -html="${COVERAGE_DIR}"/coverage.cov -o coverage.html ;
fi

# Remove the coverage files directory
rm -rf "$COVERAGE_DIR";

3.3.4. builddockerimage.sh

#!/bin/bash

#检测GOPATH
Echo "detect GOPATH"
If [ -z "$GOPATH" ];then
Echo "GOPATH not set"
Exit 1
Else
Echo "GOPATH=$GOPATH"
Fi

#Initialization data
Echo "initialize data"
New_version="1.0.0"
Old_version="1.0.0"
Golang_version="1.9.2"
App_name="application"
Projust_root="demo"
DOCKER_IMAGE_NAME="demo"
REGISTRY_HOST="xxx.xxx.xxx.xxx:5000"
Path="/go/src/_/builds/ZhangZhongcheng/demo"


# Current container replaced with old label
Echo "Current container replaced with old label"
Docker rmi $REGISTRY_HOST/$DOCKER_IMAGE_NAME: $old_version

# Based on golang: 1.9.2 image-initiated container instance, compile the binary executable program of this project
Echo "Based on the golang: 1.9.2 image-initiated container instance, compile the binary executable of this project"
Cd $path
Go build -o $app_name

Echo "detect $app_name application"
FILE="$path/$app_name"
If [ -f "$FILE" ];then
Echo "$FILE is ready"
Else
Echo "$FILE application does not exist"
Exit 1
Fi

#dockerBuilding a mirror Disallowing paths outside the build context Adding a copy file
# So here you can use the command to put the required file cp into the same directory as the dockerfile. After the build is complete, delete it with the command.
Cd $path/scripts
Echo "Extract the files needed for the build"
Cp ../$app_name $app_name

#Build a mirror based on the Dockerfile in the current directory
Echo "Building a mirror based on the Dockerfile in the current directory"
Echo "docker build -t $REGISTRY_HOST/$DOCKER_IMAGE_NAME:$new_version ."
Docker build -t $REGISTRY_HOST/$DOCKER_IMAGE_NAME: $new_version .

# Delete the generated executable file and the required files for the build
Echo "Delete the executable file generated this time and the files needed for the build"
Rm -rf $app_name
Rm -rf ../$app_name

#看镜
Echo "View image"
Docker images | grep $DOCKER_IMAGE_NAME

#推镜镜
Echo "push image"
Echo "docker push $REGISTRY_HOST/$DOCKER_IMAGE_NAME:$new_version"
Docker push $REGISTRY_HOST/$DOCKER_IMAGE_NAME: $new_version

Echo "auto deploy"
./automationDeployment.sh $new_version $old_version

3.3.5. automationdeployment.sh

#!/usr/bin/expect
#指定shebang
#Set the timeout period to 3 seconds
Set ip xxx.xxx.xxx.xxx
Set password "xxxxxxx"

Set new_version [lindex $argv 0]
Set old_version [lindex $argv 1]

Spawn ssh root@$ip
Expect {
     "*yes/no" { send "yes\r"; exp_continue}
     "*password:" { send "$password\r" }
}
Expect "#*"
Send "cd /root/demo/\r"
Send "./docker_run_demo.sh $new_version $old_version\r"
Expect eof

3.3.6. Dockerfile

FROM golang: 1.9.2

#defined environment variables for alpine
#ENV TIME_ZONE Asia/Shanghai

ADD application /go/src/demo/

WORKDIR /go/src/demo

ADD run_application.sh /root/
RUN chmod 755 /root/run_application.sh
CMD sh /root/run_application.sh

EXPOSE 8080

3.3.7. run_application.sh

#!/bin/bash

#映射ip
Cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

Cd /go/src/demo/

./application 

4. Results

The following are successful deployments:

Result

Finish

Reprint Please specify source: Golang based on Gitlab CI/CD deployment scenario