Drone support for private image warehouses, and optimization of source code transformation

Drone introduction

Drone is a new generation of CI/CD tools, based on Pipeline+docker mode, can be very flexible to support many business scenarios, currently, done up to 0.8.6, on GitHub, has scored 15K Takaboshi star.

Drone and Gitlab, you can set the. drone.yml file in your project to customize the various processes you need to perform, such as code fetching, mirroring build push, PHP composer package management, Golang build, message notification, automated deployment, automated testing, and more. Plug-in support, as well as plug-in development and usage patterns, make the drone very scalable.

At present, the drone official plugin repository has provided many plugins to extend the functionality of the drone, while implementing a set of plugins is also very simple. Basically, it's enough to write a pipeline configuration file (. drone.yml) with the flexibility to write + plug-in mode, sufficient to cope with infinite scenarios. Personally, the drone is much more flexible than Jenkins (in fact Jenkins has a new generation of tools based on Docker and k8s: Jenkins X).

Drone support for private image warehouses

In our actual use of the drone process, it is possible to require the support of the private image warehouse, the following pipeline as an example:

clone:  git:    image: xxx.com/plugins/drone-plugin-gitpipeline:  build:    image: xxx.com/octocat/hello-image  push_image:    image: xxx.com/plugins/docker    repo: xxx.com/xxx/test

From this example, you can see that this pipeline is divided into 3 steps

The ①:git step pulls the code through the image corresponding to the mirror.

②:build steps to perform code building operations with the corresponding image image.

③:push_image operation, through image corresponding to the image, complete the construction of the image, and the image is pushed to the repo corresponding to the mirror warehouse.

In this process, assuming that our private image warehouse address is xxx.com, and this mirror warehouse has permission to verify, then this pipeline, there are 2 places involved in the private image warehouse permissions processing:

In ①:pipeline, the image corresponds to 3 mirrors that need to be pulled from the private mirror repository.

②: The final step is to mirror the build and mirror the push operation, the image building is actually based on the dockerfile of the project, and this dockerfile base image, it may also be a private image. In addition, Docker push may be pushed to a private mirrored warehouse.

Official programme

From the top, to solve these 2 aspects of the private mirror warehouse requirements. Drone itself provides a set of solutions.

First, "Pipeline, image corresponding image, need to pull from the private mirror repository", this solution, drone provides 2 kinds:

①: Set the mirrored warehouse token for each image, via the Drone client tool

drone secrets add \  --image=octocat/hello-image \  octocat/hello-world REGISTRY_USERNAME octocat

This scheme, which is more troublesome, the advantage is that. drone.yml no need to make any changes

②: Directly to the pipeline in need of a private image warehouse image, in the. Drone.yml, write the authentication information, such as written as:

clone:  git:    image: xxx.com/plugins/drone-plugin-git    auth_config:      username: octocat      password: password      email: octocat@github.compipeline:  build:    image: xxx.com/octocat/hello-image    auth_config:      username: octocat      password: password      email: octocat@github.com  push_image:    image: xxx.com/plugins/docker    repo: xxx.com/xxx/test    auth_config:      username: octocat      password: password      email: octocat@github.com

This scenario does not require the manipulation of client tools, but requires that each project's. DRONE.YML pipeline configuration be written in plain text.

Second, how to resolve the image from the private warehouse, and the image pushed to the private warehouse problem

Drone actually provides the Plugins/docker plugin to do the mirroring build and push. To solve this problem, in fact, this plugin to do this, this plug-in, to solve this problem, you also need to configure the pipeline, clearly specify the authentication information (last three lines)

clone:  git:    image: xxx.com/plugins/drone-plugin-git    auth_config:      username: octocat      password: password      email: octocat@github.compipeline:  build:    image: xxx.com/octocat/hello-image    auth_config:      username: octocat      password: password      email: octocat@github.com  push_image:    image: xxx.com/plugins/docker    repo: xxx.com/xxx/test    auth_config:      username: octocat      password: password      email: octocat@github.com    username: octocat    password: password    email: octocat@github.com

The desired effect of the transformation

We can see that Drone, flexible on the pipeline configuration, but also therefore, a lot of things will be written in the pipeline configuration file. drone.yml, clear, this is not elegant, we can try to make a certain transformation of the source code, To better support the private image warehouse, but does not expose the authentication information.

First, I expect the pipeline configuration effect to look like this:

clone:  git:    image: xxx.com/plugins/drone-plugin-git    auth_config:      innerid: xxx.compipeline:  build:    image: xxx.com/octocat/hello-image    auth_config:      innerid: xxx.com  push_image:    image: xxx.com/plugins/docker    repo: xxx.com/xxx/test    auth_config:      innerid: xxx.com    auth_config_innerid: xxx.com

We compare the previous. Drone.yml, found that there is less clear-text exposure of the private image warehouse certification information, more than a innerid, this innerid, in fact, is the identity of the private image warehouse, this identity, we will be designated as a private mirror warehouse domain name.

My goal is to let the Drone program, through the identification of Innerid, to automatically obtain the corresponding private image warehouse certification information from other places (such as drone-server environment variables), so,. Drone.yml, it is not necessary to specify the text authentication information.

We're going to revamp 2 projects, the first one is the drone project, the second is the Plugins/docker plugin.

SOURCE Modification

Retrofit drone project (used to support image in pipeline using private image warehouse)

1, add a custom environment variable, let drone server know, have this environment variable and have value, this value, is we configure all the private image warehouse of authentication information collection

①: Change the Cmd/drone-server/server.go file, add the configured environment variable name: Registry_auth_inner_config

//pipeline的authconfig默认配置cli.StringFlag{    EnvVar: "REGISTRY_AUTH_INNER_CONFIG",    Name:   "registry-auth-inner-config",    Usage:  "private docker registry authentication username",    Value:  "",},

②: or change the Cmd/drone-server/server.go file, change the server function, increase the use of just the environment variable, the purpose is to parse the string, parse it into the Golang map type

    //初始化全局registry认证数据    registryInnerAuthConfig := c.String("registry-auth-inner-config")    if registryInnerAuthConfig != "" {        authinfo := droneserver.DecodeToMap(registryInnerAuthConfig)        for k, v := range authinfo {            registry := compiler.Registry{}            if e := json.Unmarshal([]byte(v), &registry); e == nil {                droneserver.GlobalRegistryAuthConfig[k] = registry            }        }    }

Here, there is a process of converting a string to a map, using a custom function: Decodetomap, the definition of this function is

//将string解码为mapfunc DecodeToMap(s string) map[string]string {    b := new(bytes.Buffer)    var decodedMap map[string]string    if data, err := base64.StdEncoding.DecodeString(s); err == nil {        b.Write(data)        d := gob.NewDecoder(b)        // Decoding the serialized data        err = d.Decode(&decodedMap)    }    return decodedMap}//将map编码为stringfunc EncodeMapToString(m map[string]string) string {    b := new(bytes.Buffer)    e := gob.NewEncoder(b)    // Encoding the map    err := e.Encode(m)    if err == nil {        return base64.StdEncoding.EncodeToString(b.Bytes())    } else {        return ""    }}

③: Changing the Server/hook.go file, the beginning of this file, requires a package reference, and a variable

//import下面的包"github.com/cncd/pipeline/pipeline/frontend/yaml/compiler"//全局的私有仓库认证信息，这个数据，从环境变量里取（从环境变量，解析为map，key为私有镜像仓库标识，比如 hub.mfwdev.com，value为认证信息json）var GlobalRegistryAuthConfig = make(map[string]compiler.Registry)

④: Our core is to parse the encoded string that is configured in the environment variable into the above variable, which is the process of a string to map. or change the file above:

        //找到下面的这行代码        parsed, err := yaml.ParseString(y)        //在上面行代码下面，补充下面代码        //处理私有镜像仓库认证信息问题        if parsed != nil && err == nil {            if parsed.Pipeline.Containers != nil && len(parsed.Pipeline.Containers) > 0 {                for _, c := range parsed.Pipeline.Containers {                    if c.AuthConfig.Innerid != "" && GlobalRegistryAuthConfig != nil {                        if registrytemp, ok := GlobalRegistryAuthConfig[c.AuthConfig.Innerid]; ok {                            c.AuthConfig.Username = registrytemp.Username                            c.AuthConfig.Password = registrytemp.Password                            c.AuthConfig.Email = registrytemp.Email                        }                    }                }            }        }

⑤: Well, to change the definition of our Innerid data type, modify the file:

Vendor/github.com/cncd/pipeline/pipeline/frontend/yaml/container.go

type (    // AuthConfig defines registry authentication credentials.    AuthConfig struct {        //私有镜像仓库标识，比如：hub.xxx.com，有了这个标识，就可以结合环境变量，取出来这个标识对应的认证信息，取代下面的三个字段的内容        Innerid  string        Username string        Password string        Email    string

Retrofit Plugins/docker plugin (used to retrofit Docker build, Docker push using private image warehouse)

We need to create a new project, clone Plugins/docker as our own project, to transform it, like the drone project

Why do we have to change the drone and change the Plugins/docker project again?

This is because, drone transformation, drone is actually server side, and Plugins/docker, is just a step of pipeline, we can let drone, put this authentication information, pass to Plugins/docker. So, back, we also need to re-transform the drone project, the drone environment variable certification information, passed to Plugins/docker, so that the certification information, passed to Plugins/docker, do not plugins/ What does Docker do to get the authentication information from the outside?

①: Change the Drone project, file: Server/hook.go, in the hook function, set the part of the environment variable, add the following line

    //全局的registry认证信息    envs["REGISTRY_AUTH_INNER_CONFIG"] = os.Getenv("REGISTRY_AUTH_INNER_CONFIG")

Then, go back to the Plugins/docker project.

②: Change Cmd/drone-docker/main.go file, define new type struct

type Registry struct {    // mfwupdate 是否内部验证（pipeline的docker认证，其实是需要在pipeline的yml中配置的，不好的地方是，会导出暴露认证信息    // 这里提供一个 Innerid 表示，优先走内部验证，也就是，通过drone-server的环境变量中取认证username、password、email信息）    Innerid  string    Hostname string    Username string    Password string    Email    string    Token    string}

In the same file, add the part that takes the environment variable

        //use PLUGIN_AUTH_CONFIG_INNERID first，compared to PLUGIN_USERNAME、PLUGIN_PASSWORD、PLUGIN_EMAIL，做docker login        cli.StringFlag{            Name:   "auth_config_innerid",            Usage:  "use auth_config_innerid first，compared to username、password、email for docker login",            EnvVar: "PLUGIN_AUTH_CONFIG_INNERID",        },    }

Or this file, to increase the handling of this environment variable

    //mfwupdate    // login to the Docker registry    authConfigInnerId := c.String("auth_config_innerid")    registryAuthInnerInfo := os.Getenv("REGISTRY_AUTH_INNER_CONFIG")    if registryAuthInnerInfo != "" && authConfigInnerId != "" {        authinfo := DecodeStringToMap(registryAuthInnerInfo)        for k, v := range authinfo {            registry := Registry{}            if e := json.Unmarshal([]byte(v), &registry); e == nil {                if k == authConfigInnerId {                    plugin.Login.Email = registry.Email                    plugin.Login.Username = registry.Username                    plugin.Login.Password = registry.Password                    plugin.Login.Registry = authConfigInnerId                }            }        }    }

At this point, the transformation is complete.

Reference:

http://readme.drone.io/0.5/