CI Practice based on GitLab

Last month was invited by the Dockone community to do a CI practice on-line sharing, in this record.
This paper describes the architecture and capabilities of GitLab CI, and analyzes its role in DevOps practice. Analyze the technical details of Docker in Docker and describe in detail the CI practices and optimizations made in the production environment, including but not limited to the mirrored warehouses, to achieve several times performance gains.
This share takes GitLab Community Edition 11.0.4 edb037c as an example.

Why Choose GitLab CI

Meet GitLab CI

What is GitLab CI

GitLab CI is the CI/CD basic function of GitLab in order to enhance its role in software development engineering and to improve the DevOPS concept. Can be easily integrated into the software development process. GitLab CI can be used to define the perfect ci/cd Pipeline.

Advantage

• GitLab CI is included by default in GitLab, and our code is managed using GitLab, which makes it easy to integrate
• GitLab CI's front-end interface is more beautiful and easy to accept.
• Includes real-time build logs for easy tracking
• Using the C/S architecture, can be scaled horizontally, performance will not affect
• With YAML configuration, it's easy for anyone to use.

Key Concepts

Pipeline

Pipeline is equivalent to a build task that can contain multiple processes, such as dependency installation, compilation, testing, deployment, and so on.
Any combination of submissions or merge Request can trigger Pipeline

Stages

Stage represents the phase of the build, the process mentioned above.

• All Stages are executed sequentially, that is, when one stage completes, the next stage will begin
• Any Stage fails, Stages will never execute, Pipeline fail
• Pipeline will succeed only if all Stages have been completed.

Jobs

A job is a task in the Stage.

• Jobs in the same Stage will be executed in parallel
• Any Job fails, then Stage fails, Pipeline fails
• When Jobs in the same stage succeeds, the stage succeeds

OK, the basic concept has been introduced to you, you can find that the concept mentioned above, did not mention the actual executor of the task. Where does that task go?

GitLab Runner

Runner is the actual performer of the task and can be run on a system such as macos/linux/windows. Use Golang for development. Can also be deployed on k8s

Registered

docker run --rm -t -i -v /path/to/config:/etc/gitlab-runner --name gitlab-runner gitlab/gitlab-runner register \  --executor "docker" \  --docker-image alpine:3 \  --url "https://gitlab.com/" \  --registration-token "PROJECT_REGISTRATION_TOKEN" \  --description "docker-runner" \  --tag-list "dev" \  --run-untagged \  --locked="true"

The above example registers runner as a container, and of course you can do it directly on the physical machine. Registration on physical machines is similar to registering as a container

sudo gitlab-runner register \  --non-interactive \  --url "https://gitlab.com/" \  --registration-token "PROJECT_REGISTRATION_TOKEN" \  --executor "docker" \  --docker-image alpine:3 \  --description "docker-runner" \  --tag-list "docker,aws" \  --run-untagged \  --locked="false" \# （这段代码来自官方文档）

Next, let's look at the types of runner so that we can differentiate them when we use them.

Type

• Shared-runner runs jobs from all unassigned projects
• Group-runner runs jobs from all unassigned projects on its group
• Specific-runner runs jobs from assigned projects
• Locked-runner cannot is assigned to other projects
• Paused-runner won't receive any new jobs

Configuration

First, the outermost is the global configuration, and the default is

concurrent = 1check_interval = 0

Two of these. The comparison needs to be focused on the following

Global configuration

• Concurrent: Concurrency number, 0 is unlimited
• SENTRY_DSN: Linkage with Sentry, you can collect the exception to sentry.
• Listen_address: Exposed metrics for Prometheus monitoring

Executor

• Shell
• Docker (Share content this time)
• Docker machine and Docker Machine SSH (autoscaling)
• Parallels
• VirtualBox
• Ssh
• Kubernetes (recommended)

Detailed Docker in Docker

Overview

Docker in Docker abbreviation Dind, in the use of GitLab CI, may often be used in the service section. Dind indicates that a Docker container, or Docker daemon, is actually running in Docker.

In fact, if you just execute Docker commands in Docker, you can install a binary file. However, if you want to run Docker daemon (for example, you need to perform Docker info) or access any device, it is not allowed.

Docker provides two important options in the Run command --privileged --device , and other options, such as --cap-add and --cap-drop permissions, are also relevant, but not the focus of today, not the table.

--deviceoption allows us to access the specified device when we do not use the --privileged option, for example, docker run --device=/dev/sda:/dev/xvdc --rm -it ubuntu fdisk /dev/xvdc but this is only a limited privilege, and we know that Docker's technology implementation is actually based on cgroup resource isolation, but not --device enough to allow us to have sufficient permissions within the container to complete the dock The ER daemon is started.

Around 2013, the--privileged option was added to Docker, which made it possible for us to start the container inside the container. Although--privileged's initial idea is to make container development more convenient, some people may misunderstand it when they use it.

Sometimes, we may just want to be able to make a normal build image within a container, or to interact with Docker daemon, such as a Docker images. Well, we don't really need dind, we need Docker out of Docker, that is Dood, when used, it is actually to mount the docker.sock into the container

For example, use the following command: sudo docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock taobeier/docker /bin/sh you can perform normal Docker images operations within the container, and be aware that the actions within the container will affect the Docker daemon on the host.

How to Achieve

• Create groups and users, and join users to the group. Using the Groupadd and Useradd commands
• Update the Subuid and Subgid files to configure new users and groups to the/etc/subgid and/etc/subuid files. Subuid and Subgid provide a subordinate ID that allows the user to use
• Next you need to mount the/sys/kernel/security for the SECURITYFS type can be tested using the Mountpoint command mountpoint /sys/kernel/security if it is not a mount point, then use mount -t securityfs none /sys/kernel/security mount. If it is not mounted successfully, you can check whether SELinux or AppArmor prevent this behavior. For detailed security questions, refer to the Linux security Modules (LSM)
• Next, allow the Dockerd command to start daemon, and dockerd --host=unix:///var/run/docker.sock --host=tcp://0.0.0.0:2375 you can listen to the Docker daemon to port 2375

Simple procedure

You can directly use the docker:dind image in the Docker's official image warehouse, but at run time, you need to specify --privileged options

CI Practice

Runner Practice

Look at the configuration of the runner section

[[runners]]  name = "docker"  url = "https://gitlab.example.com/"  token = "TOKEN"  limit = 0  executor = "docker"  builds_dir = ""  shell = ""  environment = ["ENV=value", "LC_ALL=en_US.UTF-8"]  clone_url = "http://172.17.0.4"

Clone_url can be configured as an accessible address for network reasons, so the configured address will be used when the code is clone. The actual request ishttp://gitlab-ci-token:TOKEN@172.17.0.4/namespace/project.git

• Look again at the Runners.docker configuration, which will affect the actual operation of Docker

[runners.docker]  host = ""  hostname = ""  tls_cert_path = "/home/tao/certs"  image = "docker"  dns = ["8.8.8.8"]  privileged = false  userns_mode = "host"  devices = ["/dev/net/tun"]  disable_cache = false  wait_for_services_timeout = 30  cache_dir = ""  volumes = ["/data", "/home/project/cache"]  extra_hosts = ["other-host:127.0.0.1"]  services = ["mongo", "redis:3"]  allowed_images = ["go:*", "python:*", "java:*"]

DNS, privileged, extra_hosts, and services are key, especially in the production of a variety of network conditions, need to pay particular attention. As for the devices configuration, it was already spoken at the beginning of today's share, and Allowed_images's words were a limitation.

Some of the above configuration items, used by Docker students, should be easy to understand. Let's take a look at the services this configuration item

Image:registry.docker-cn.com/taobeier/dockervariables:docker_driver:overlay2 # overlay2 is the best bug need kernel ; = 4.2services:-name:registry.docker-cn.com/taobeier/docker:stable-dind alias:dockerstages:-Build-deploy Build_and_test:stage:build Tags:-build script: # change repo #-sed-i ' s/dl-cdn.alpinelinux.org/mirrors.u  Stc.edu.cn/g '/etc/apk/repositories # uses the default official source apk time consuming 7min 30s. Time 18s-ping-c 1 docker-ping-c 1 registry.docker-cn.com__taobeier__docker-ipaddr-apk Add--NO-CAC  He PY-PIP # uses the default time-consuming 1 min 15s. Time-consuming 43s-pip install-i https://mirrors.ustc.edu.cn/pypi/web/simple docker-compose-docker-compose up-d-Do Cker-compose Run--RM Web pytest-s-v tests/test_session.pydeploy:image: "Registry.docker-cn.com/library/centos" stage : Deploy Tags:-deploy script: # Install SSH client-' ssh-agent | | (Yum install-y openssh-clients) ' # Run Ssh-agent-eval $ (ssh-agent-s)-ECHo "$SSH _private_key" | Tr-d ' \ r ' |    Ssh-add->/dev/null # Create SSH dir-mkdir-p ~/.ssh-chmod ~/.ssh # Use Ssh-keyscan to get key -Ssh-keyscan-p $SSH _port $DEPLOY _host >> ~/.ssh/known_hosts-chmod 644 ~/.ssh/known_hosts #-Ssh-p $SSH _ PORT $DEPLOY _user@ $DEPLOY _host ls-rm-rf. Git-scp-r-P $SSH _port. $DEPLOY _user@ $DEPLOY _host:~/we/

The essence of services is actually using Docker's--link, and we look at how it works

How Docker Executor Works

• Create service container (mirror already configured in service)
• Create a cache container (stores the volumes already configured on CONFIG.TOML and the Dockerfile of the build image)
• Create a Build container and link all service containers.
• Start the build container and send the job script into the container.
• The script that executes the job.
• Check out code:/builds/group-name/project-name/.
• Perform the steps defined in. gitlab-ci.yml.
• Check the status code after the script executes, if not 0 build failed.
• Remove the build and service containers.

Private Image source

User authentication requires GITLAB Runner 1.8 or later, the Runner between 0.6 and 1.8 needs to be executed manually on the Runner machine itself.

By default, GitLab Runner uses the Docker_auth_config variable as the credential for authentication if the mirrored warehouse being accessed needs to be taken seriously.

Note: Docker_auth_config is the completed DOCKER AUTH credential, which means that it should be consistent with the content in our ~/.docker/config.json, for example:

{    "auths": {        "registry.example.com": {            "auth": "5oiR5piv5byg5pmL5rab"        }    }}

The simple thing is, we execute the login on the local/server docker login 私有镜像源 successfully, copy the contents of the ~/.docker/config.json file directly, as the value of our variable, or echo -n '用户名:密码' | base64 in such a way to obtain the content of auth, assemble the corresponding format, write GitLab in the value configuration.

CI performance Optimization in a production environment

1. Use domestic sources to speed up container mirroring for example: using Docker China official image Accelerator https://registry.docker-cn.com of course, companies also have to provide mirrored acceleration services.

2. Use a private mirrored warehouse. For example, Docker Registry, or Harbor, we are using Harbor as a private mirror repository.

If the official image is used by default for network reasons, 1. The official image cannot be pulled down; 2. Installing the package in the official image takes a long time; 3. If you change the source, you need to do the same thing for each Dockerfile. Of course, we cannot agree to this. So, we built our own private image. Starting with BusyBox, building Alpine Linux uses a private source to build the other images we need. Users no longer need to switch sources themselves.

Once this is done, we need to install Py-pip (to install docker-compose and our service dependencies) in the process of CI execution and reduce the time from 3min30s to 18s.

Here, we need to say why we are building images from scratch, not based on official images. The main purpose is to reduce the volume of mirrors and to adapt to our needs faster.

Similarly, we have built the basic Docker image, Python/maven and other images, are the default use of our private source, and, users in use, do not need to pay attention to the source of things, reduce the user's mental burden.

3. Standardize Dockerfile, reduce unnecessary dependency on installation and reduce mirror volume. In fact, combined with the above part, we do the thing is to directly build our base image Docker/alpine/maven such as the base image, the default is directly replaced by the source. This makes it easy to use and reduces the number of mirrored layers.

4. Split job, through the way tag can be specified runner, by the different runner to execute the non-strong dependency of some actions. Easy to share pressure.

5. Using the cache, the construction of CI, most of the mirrors, in fact, the change is not big, so using the cache can multiply the speed of CI.

6. May encounter the Pit

There are a variety of services available in the service, whether Dind or MySQL Redis, as mentioned earlier. But if we all do the optimization and use our private sources, we will find the problem.

Because Gitlab ci defaults to docker:dind the service, it will actually choose the host named Docker, as well as Port 2375. When using a private image source, such as

services:    - name: registry.docker-cn.com/taobeier/docker:stable-dind

What is the host of the service?

The host of the service will actually become registry.docker-cn.com__taobeier__docker , then Gitlab runner will not be found, the job will fail to execute

There are two ways to solve this problem. One is to add a variable

variables:    DOCKER_HOST: "tcp://registry.docker-cn.com__taobeier__docker:2375"

But this way is very troublesome, no one can fully remember the encounter/will be converted to _ inevitably there will be problems. Then there is the second way:

services:    - name: registry.docker-cn.com/taobeier/docker:stable-dind    -       alias: docker

Add an alias. This method is rarely used at present, after all, the network is the first to find, but this approach is the simplest.

Q&a

Q: You mentioned that all kinds of dependencies are provided by Service, which way is it? such as Python's dependence, how to make service?

A:service dependence, mainly refers to similar db/mysql/reids and so on. Or Dind is actually providing a 2375 port TCP service. Python dependency, I recommend that you build a Python image that has been replaced by a source. It takes a lot less time to install dependencies. Alternatively, the Venv folder of the virtual environment can be used as the cache when defining Pipeline, and subsequent installations will also check this to avoid unnecessary installation.

Q: Excuse me, why don't you use Jenkins Pipeline while using Gitlab CI?

A: The main reason is that I mentioned a few aspects. Good integration, elegant interface, easy to use (all people with warehouse Write permission can use, just create. Gitlab-ci.yml and configure Runner to use). In other words, we look at the problem of using Jenkins, the configuration of Jenkins for the project and GitLab code is separate, two parts, the user (or our developer) in use, need to have two platforms, and, most of the time, Jenkins's permission is not released Of To the user, that's the equivalent of a black box. What is the possible problem? The build failed, but only Ops knew what was going on, but the development was powerless because there was no authority. The advantage of using the Gitlab, this time is more prominent, the configuration is in the Code warehouse, and the use of YAML configuration, very simple. What is the problem, direct check, directly change.

Q: About the Runner cleaning problem, after a long time of use, the Runner machine back to generate a lot of Cache container, how to clean it up. Can I clear the task automatically?

A: This is relatively simple, first, if your cache container confirmation is useless, each cache container actually has a name, directly by the name of the cache is too small, batch deleted. If you are not sure if it is useful, it is not affected if you delete it directly, because the execution mechanism of Docker excutor is to create the Cache container after the Service container is created. If it is deleted, it will only be created again. If you want to clear the task, there is no relevant practice, after I practice, see if there is a very elegant way.

Q: How do I handle the settings.xml of maven? Where's the local maven repository?

A: We have built a private Maven image, and our private source is used by default in the private image. For users in the project, there is no need to focus on configuring repo in Settings.xml.

Q: In the Gitlab CD scheme, when deploying, you need to configure the private key of the springboard in the variable, how to protect the private key if the project is developed for the whole department of the company?

A: You can use secret variable to write the private key to it, (but the project's administrator, with permission to view the variable) develops a Web server (in fact, as long as the IP port is exposed) to request during CI execution, the server Source to make judgments (such as when performing CI, there will be some specific variables, in order to determine whether the CI is really the request) and then return the private key.

What type of project is Q:gitlab ci suitable for? Is there a small audience at home?

A: The domestic is still relatively small (compared to Jenkins) in fact, as long as the need for CI projects, it is suitable.

The following QR code can be used to subscribe to my article "Moelove" of the public number

Themoelove