This article is reproduced, original: Golang based on Gitlab CI/CD deployment plan
Overview
Continuous integration is a software development practice in which team development members often integrate their work, with at least one integration per member per day, which means multiple integrations per day. Each integration is verified through automated builds (including compilation, publishing, automated testing) to detect integration errors as early as possible.
Continuous deployment is the rapid delivery of high-quality products through automated build, test, and deployment cycles. To some extent, it represents the degree to which a development team is engineered. After all, the fast-running Internet company's labor cost will be higher than that of the machine. The investment machine optimization development process will also improve people's efficiency and maximize engineering productivity.
1. Environmental preparedness
This test is based onCentos 7.3thedocker 17.03.2-ceenvironment. Docker installation here will not repeat, provide the official link bar: Get Docker CE for CentOS
1.1. Docker Boot Gitlab
The start command is as follows:
docker run --detach \
--hostname gitlab.chain.cn \
--publish 8443:443 --publish 8080:80 --publish 2222:22 \
--name gitlab \
--restart always \
--volume /Users/zhangzc/gitlab/config:/etc/gitlab \
--volume /Users/zhangzc/gitlab/logs:/var/log/gitlab \
--volume /Users/zhangzc/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce
Port,hostname, volume according to the specific circumstances of the specific settings
1.2. Docker Boot Gitlab-runner
The start command is as follows:
sudo docker run -d /
--name gitlab-runner /
--restart always /
-v /Users/zhangzc/gitlab-runner/config:/etc/gitlab-runner /
-v /Users/zhangzc/gitlab-runner/run/docker.sock:/var/run/docker.sock /
gitlab/gitlab-runner:latest
Volume according to the specific circumstances of the specific settings
1.3. Image authoring for integrated deployment
Our integration and deployment needs to be carried out in a single container, so we need to create a mirror and install some necessary tools for integrating and deploying related operations. Currently our projects are based on Golang 1.9.2, where a specific image is mapped based on the golang:1.9.2 image.
Dockerfile content is as follows:
# Base image: https://hub.docker.com/_/golang/
FROM golang:1.9.2
USER root
# Install golint
ENV GOPATH /go
ENV PATH ${GOPATH}/bin:$PATH
RUN mkdir -p /go/src/golang.org/x
RUN mkdir -p /go/src/github.com/golang
COPY source/golang.org /go/src/golang.org/x/
COPY source/github.com /go/src/github.com/golang/
RUN go install github.com/golang/lint/golint
# install docker
RUN curl -O https://get.docker.com/builds/Linux/x86_64/docker-latest.tgz \
&& tar zxvf docker-latest.tgz \
&& cp docker/docker /usr/local/bin/ \
&& rm -rf docker docker-latest.tgz
# install expect
RUN apt-get update
RUN apt-get -y install tcl tk expect
Thesegolintare tools for Golang code style checking.
dockerBecause of the need to use the hosted Docker command in the container, it is necessary to install a Docker executable file and then mount the host's/var/run/docker.sock file to the same location in the container when the container is started.
expectis the tool for SSH to automatically log on to the remote server, where the tool is installed to enable remote server-side deployment of the application
In addition, in the installation of Golint, it is necessary to golang.org download the source, because of the relationship between the wall, go get command can not be executed. In order to deal with this problem, first download the relevant source code through other channels, put in the specified path, and then copy into the image, and perform the installation.
The following script is used to generate the image:
#!/bin/bash
Echo "Extract the files needed to build the image"
Source_path="source"
Mkdir -p $source_path/golang.org
Mkdir -p $source_path/github.com
Cp -rf $GOPATH/src/golang.org/x/lint $source_path/golang.org/
Cp -rf $GOPATH/src/golang.org/x/tools $source_path/golang.org/
Cp -rf $GOPATH/src/github.com/golang/lint $source_path/github.com
Echo "Building a Mirror"
Docker build -t go-tools: 1.9.2 .
Echo "Remove the files needed to build the image"
Rm -rf $source_path
After the image is generated, push to the mirror warehouse and pull the mirror on the Gitlab-runner server
The Gitlab and Gitlab-runner of this trial are under Docker running on the same server.
2. Runner Registration and configuration
2.1. Registration
After the environment is ready, execute the following command on the server to register runner:
docker exec -it gitlab-runner gitlab-ci-multi-runner register
Follow the prompts to enter the relevant information
Please enter the gitlab-ci coordinator URL:
# gitlab url, such as: https://gitlab.chain.cn/
Please enter the gitlab-ci token for this runner:
# gitlab->your project->settings -> CI/CD ->Runners settings
Please enter the gitlab-ci description for this runner:
# Example: demo-test
Please enter the gitlab-ci tags for this runner (comma separated):
# Example: demo
Whether to run untagged builds [true/false]:
# true
Please enter the executor: docker, parallels, shell, kubernetes, docker-ssh, ssh, virtualbox, docker+machine, docker-ssh+machine:
# docker
Please enter the default Docker image (e.g. ruby:2.1):
# go-tools:1.9.2 (previously created image)
2.2. Configuration
After successful registration, you also need to do some specific configuration on the original configuration, as follows:
[[runners]]
name = "demo-test"
url = "https://gitlab.chain.cn/"
token = "c771fc5feb1734a9d4df4c8108cd4e"
executor = "docker"
[runners.docker]
tls_verify = false
image = "go-tools:1.9.2"
privileged = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
extra_hosts = ["gitlab.chain.cn:127.0.0.1"]
network_mode = "host"
pull_policy = "if-not-present"
shm_size = 0
[runners.cache]
Here first explain the process of Gitlab-runner, Gitlab-runner at the time of execution, will start a container according to the above configuration, that is, the configuration of Go-tools:1.9.2,b all the startup parameters will be in the [Runners.docker] Under the node configuration, including Mount Ah, network AH and so on. After the container is successfully started, the container is used to gitlab the pull code, and then the test is performed according to the rules that you define, and then it is deployed when all the tests are successful.
volumes: is a docker command that can execute a host in a container.
extra_hosts: Add a host map to the Gitlab and map to the 127.0.0.1
network_mode: Make the network of the container consistent with the host, so that you can access the Gitlab through 127.0.0.1.
pull_policy: When the specified image does not exist, it is pulled via Docker pull
3. Defining rules
Create a file in the Gitlab project root directory.gitlab-ci.yml, fill in the runner rules, and refer to the official documentation for the specific grammar lesson: https://docs.gitlab.com/ee/ci/yaml/
3.1. Go Integration command
Here are a few Golang common integration commands
- Package List
As described in the official documentation, the Go Project is a collection of packages. Most of the tools described below will use these packages, so the first command we need is a way to list the packages. We can do this with the Go list subcommand.
go list ./...
Please note that if we want to avoid applying our tools to external resources and limiting them to our code. Then we need to remove the vendor directory, the command is as follows:
go list ./... | grep -v /vendor/
- Unit Test
These are the most common tests that you can run in your code. Each. Go file requires a file that can support unit testing_test.go. You can run tests for all packages by using the following command:
go test -short $(go list ./... | grep -v /vendor/)
- Data competition
This is often a difficult problem to avoid, and the Go tool has it by default (but only for use on LINUX/AMD64, FREEBSD/AMD64, Darwin/amd64, and WINDOWS/AMD64)
go test -race -short $(go list . /…| grep - v /vendor/)
- Code overrides
This is an essential tool to evaluate the quality of the code and to show which part of the code has been unit tested and which is not.
To calculate code coverage, you need to run the following script:
PKG_LIST=$(go list ./... | grep -v /vendor/)
for package in ${PKG_LIST}; do
go test -covermode=count -coverprofile "cover/${package##*/}.cov" "$package" ;
done
tail -q -n +2 cover/*.cov >> cover/coverage.cov
go tool cover -func=cover/coverage.cov
If we want to get an HTML-formatted coverage report, we need to add the following command:
go tool cover -html=cover/coverage.cov -o coverage.html
- Build
Finally, once the code is fully tested, we compile the code to build a binary that can execute.
go build .
- Linter
This is the first tool we used in the code: Linter. Its role is to check the code style/error. This sounds like an optional tool, or at least a "nice" tool, but it does help to maintain a consistent code style on the project.
Linter is not part of the go itself, so if you want to use it, you need to install it manually (the previous Go-tools image we have installed).
The use of the method is fairly straightforward: just run it on the code package (or point to the. go file):
$ golint -set_exit_status $(go list ./... | grep -v /vendor/)
Note the-set_exit_status option. By default, Golint only outputs style problems with a return value (with a 0 return code), so CI is not considered an error. If-set_exit_status is specified, the return code for Golint will not be 0 when any style problems are encountered.
3.2. Makefile
If we do not want to write in the.gitlab-ci.ymlfile too complex, then we can put all the tools used in the continuous integration environment, all packaged in makefile, and call them in a unified manner.
In this case, the.gitlab-ci.ymldocument will be more concise. Of course, makefile can also invoke the*.shscript file
3.3. Configuration examples
3.3.1. gitlab-ci.yml
image: go-tools:1.9.2
stages:
- build
- test
- deploy
before_script:
- mkdir -p /go/src/gitlab.chain.cn/ZhangZhongcheng /go/src/_/builds
- cp -r $CI_PROJECT_DIR /go/src/gitlab.chain.cn/ZhangZhongcheng/demo
- ln -s /go/src/gitlab.chain.cn/ZhangZhongcheng /go/src/_/builds/ZhangZhongcheng
- cd /go/src/_/builds/ZhangZhongcheng/demo
unit_tests:
stage: test
script:
- make test
tags:
- demo
race_detector:
stage: test
script:
- make race
tags:
- demo
code_coverage:
stage: test
script:
- make coverage
tags:
- demo
code_coverage_report:
stage: test
script:
- make coverhtml
only:
- master
tags:
- demo
lint_code:
stage: test
script:
- make lint
build:
stage: build
script:
- pwd
- go build .
tags:
- demo
build_image:
stage: deploy
script:
- make build_image
tags:
- demo
3.3.2. Makefile
PROJECT_NAME := "demo"
PKG := "gitlab.chain.cn/ZhangZhongcheng/$(PROJECT_NAME)"
PKG_LIST := $(shell go list ./... | grep -v /vendor/)
GO_FILES := $(shell find . -name '*.go' | grep -v /vendor/ | grep -v _test.go)
test: ## Run unittests
@go test -v ${PKG_LIST}
lint: ## Lint the files
@golint ${PKG_LIST}
race: ## Run data race detector
@go test -race -short ${PKG_LIST}
coverage: ## Generate global code coverage report
./scripts/coverage.sh;
coverhtml: ## Generate global code coverage report in HTML
./scripts/coverage.sh html;
build_image:
./scripts/buildDockerImage.sh
3.3.3. coverage.sh
#!/bin/bash
#
# Code coverage generation
COVERAGE_DIR="${COVERAGE_DIR:-coverage}"
PKG_LIST=$(go list ./... | grep -v /vendor/)
# Create the coverage files directory
mkdir -p "$COVERAGE_DIR";
# Create a coverage file for each package
for package in ${PKG_LIST}; do
go test -covermode=count -coverprofile "${COVERAGE_DIR}/${package##*/}.cov" "$package" ;
done ;
# Merge the coverage profile files
echo 'mode: count' > "${COVERAGE_DIR}"/coverage.cov ;
tail -q -n +2 "${COVERAGE_DIR}"/*.cov >> "${COVERAGE_DIR}"/coverage.cov ;
# Display the global code coverage
go tool cover -func="${COVERAGE_DIR}"/coverage.cov ;
# If needed, generate HTML report
if [ "$1" == "html" ]; then
go tool cover -html="${COVERAGE_DIR}"/coverage.cov -o coverage.html ;
fi
# Remove the coverage files directory
rm -rf "$COVERAGE_DIR";
3.3.4. builddockerimage.sh
#!/bin/bash
#检测GOPATH
Echo "detect GOPATH"
If [ -z "$GOPATH" ];then
Echo "GOPATH not set"
Exit 1
Else
Echo "GOPATH=$GOPATH"
Fi
#Initialization data
Echo "initialize data"
New_version="1.0.0"
Old_version="1.0.0"
Golang_version="1.9.2"
App_name="application"
Projust_root="demo"
DOCKER_IMAGE_NAME="demo"
REGISTRY_HOST="xxx.xxx.xxx.xxx:5000"
Path="/go/src/_/builds/ZhangZhongcheng/demo"
# Current container replaced with old label
Echo "Current container replaced with old label"
Docker rmi $REGISTRY_HOST/$DOCKER_IMAGE_NAME: $old_version
# Based on golang: 1.9.2 image-initiated container instance, compile the binary executable program of this project
Echo "Based on the golang: 1.9.2 image-initiated container instance, compile the binary executable of this project"
Cd $path
Go build -o $app_name
Echo "detect $app_name application"
FILE="$path/$app_name"
If [ -f "$FILE" ];then
Echo "$FILE is ready"
Else
Echo "$FILE application does not exist"
Exit 1
Fi
#dockerBuilding a mirror Disallowing paths outside the build context Adding a copy file
# So here you can use the command to put the required file cp into the same directory as the dockerfile. After the build is complete, delete it with the command.
Cd $path/scripts
Echo "Extract the files needed for the build"
Cp ../$app_name $app_name
#Build a mirror based on the Dockerfile in the current directory
Echo "Building a mirror based on the Dockerfile in the current directory"
Echo "docker build -t $REGISTRY_HOST/$DOCKER_IMAGE_NAME:$new_version ."
Docker build -t $REGISTRY_HOST/$DOCKER_IMAGE_NAME: $new_version .
# Delete the generated executable file and the required files for the build
Echo "Delete the executable file generated this time and the files needed for the build"
Rm -rf $app_name
Rm -rf ../$app_name
#看镜
Echo "View image"
Docker images | grep $DOCKER_IMAGE_NAME
#推镜镜
Echo "push image"
Echo "docker push $REGISTRY_HOST/$DOCKER_IMAGE_NAME:$new_version"
Docker push $REGISTRY_HOST/$DOCKER_IMAGE_NAME: $new_version
Echo "auto deploy"
./automationDeployment.sh $new_version $old_version
3.3.5. automationdeployment.sh
#!/usr/bin/expect
#指定shebang
#Set the timeout period to 3 seconds
Set ip xxx.xxx.xxx.xxx
Set password "xxxxxxx"
Set new_version [lindex $argv 0]
Set old_version [lindex $argv 1]
Spawn ssh root@$ip
Expect {
"*yes/no" { send "yes\r"; exp_continue}
"*password:" { send "$password\r" }
}
Expect "#*"
Send "cd /root/demo/\r"
Send "./docker_run_demo.sh $new_version $old_version\r"
Expect eof
3.3.6. Dockerfile
FROM golang: 1.9.2
#defined environment variables for alpine
#ENV TIME_ZONE Asia/Shanghai
ADD application /go/src/demo/
WORKDIR /go/src/demo
ADD run_application.sh /root/
RUN chmod 755 /root/run_application.sh
CMD sh /root/run_application.sh
EXPOSE 8080
3.3.7. run_application.sh
#!/bin/bash
#映射ip
Cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
Cd /go/src/demo/
./application
4. Results
The following are successful deployments:
Result
Finish
Reprint Please specify source: Golang based on Gitlab CI/CD deployment scenario