Implementation of dependency processing and parallel compilation of Docker image under shell

Implementation of dependency processing and parallel compilation of Docker image under shell

Recently, when making a series of Docker image compilation scripts, I thought we could not speed up the parallel compilation, checked the data, and finally implemented the multiple Docker images in parallel through the shell job control.

The specific objectives to be achieved include:

1. Dockerfile for Docker images within a directory, compiling Docker images on a dependency-by-process basis
2. To speed up, there is no dependency on the mirror can be compiled in parallel (degree of parallelism can be set)

First, if you want to enable job control in the script file, you need to add the following code at the beginning of the script file:

#!/bin/bash# 允许脚本使用 job controlset -m

The shell's job control allows the daemon to execute a program with ' & ' to run the process in the background, with jobs instructions to check how many background tasks are executing, through wait instructions waiting for the process to end, and immediately by $? getting the wait target process The return value.

This series of Docker images is dependent, the dependent relationships are easy to handle, the Dockerfile instructions are checked directly FROM , and the tag of the parent image is processed to get the dependency relationship:

all_dockerfiles=$(find <目录> -name Dockerfile -type f)# 关联数组保存镜像名和对应的路径declare -A all_images=()# 保存镜像名和对应的父镜像名declare -A depends=()for dockerfile in ${all_dockerfiles[@]}; do    image_dir=${dockerfile%/*}    image_name=${image_dir##*/}    all_images[${image_name}]=${image_dir}    parent_image_name=$(cat ${dockerfile} | awk -F ‘[ :]‘ ‘/^FROM/{print $2}‘)    # 这里可做个判断，当镜像名不在目录中时可去除，如 ubuntu:14.04 之类的父镜像不需要记录    depends[${image_name}]=${parent_image_name}done

With the all_images and depends two associative arrays, you can do parallel processing based on these two data, combined with job control. Before deciding whether or not to compile a mirror, it is necessary to determine whether the parent image was compiled successfully, and therefore, add two arrays to record the name of the image that was successfully compiled and failed to compile, using an array to record the current image name in the compilation.

# The maximum number of processes allowed workers=4# a compiled mirror declare-a builts= () # Error Mirroring declare-a errors= () # Running task, [pid]= Mirror name Declare-a runnings= () While [${#all_images [@]}-gt 0] | | [${#runnings [@]-GT 0}]; Do # Check if there is a compile task completed if [$ (Jobs | wc-l)-lt ${#runnings [@]}]; Then # Check which compilation task has completed for PID in ${!runnings[@]}; Do if! (Jobs-l | grep ${pid} >/dev/null);                Then # get process return value wait ${pid} return_code=$? # success or failure if [${return_code}-ne 0]; Then errors= (${errors[@]} ${runnings[${pid}]}) Else builts= (${buil ts[@]} ${runnings[${pid}]}) fi unset runnings[${pid}] fi done fi # Add a new compilation task, pick an image from All_images for image_name in ${!all_images[@]}; Do # is there any worker available if [${#runnings [@]}-ge ${workers}]; Then the break fi # Parent image is compiled Parent_image_name=${depends[${iMage_name}]} # The parent mirror failed with this image failed for I in ${errors[@]}; do if [${i} = = ${parent_image_name}]; Then errors= (${errors[@]} ${image_name}) unset All_images[${image_name}] Co Ntinue Fi Done # The parent image succeeds, the image can start compiling for I in ${builts[@]}; do if [${i} = = ${parent_image_name}]; Then Docker Build-q ${all_images[${image_name}]}-T "${image_name}:latest" & runnings[$ !] =${image_name} unset All_images[${image_name}] fi done doing # sleep a will sleep 0.2 Done

The number of workers can be adjusted based on the performance of the network and the number of CPUs in the host, and when writing scripts, be careful with the problem of parent image compilation errors.

Implementation of dependency processing and parallel compilation of Docker image under shell