Harbor user mechanism, mirroring synchronization, and integration practices with kubernetes

Habor is an open-source container mirroring repository from VMware Corporation. In fact,Habor is an enterprise-scale extension on the Docker registry for a wider range of applications, including: Administrative user interface, role-based access control, AD/LDAP integration, audit logs, and more.

The core of the container is the concept of a mirror, which can be packaged into mirrors and quickly started and stopped, so the container becomes the new hot infrastructure CaaS and provides the underlying support for agile and continuous delivery including DevOps .

The container mirroring repository provided by Habor and Docker registry is the storage and distribution service for container mirroring. There are three reasons why such a service exists:

• Provides a tiered transport mechanism to optimize network transmission
  Docker images are layered, and it is obviously not economical to use a full-volume file for each transfer (so the FTP approach is not appropriate). It is necessary to provide a mechanism for identifying hierarchical transmissions, identifying the objects to be transferred, using the UUID of the layer.

• Provides a web interface to optimize the user experience
  Uploading and downloading with only the name of the image is obviously inconvenient and requires a user interface to support login and search functions, including distinguishing between public and private images.

• Supports horizontally extended clusters
  When a user uploads an image to a server, it needs to decompose the corresponding access pressure.

These are the main work done by Docker Registry, and Habor on this, but also provide users, synchronization and many other features, this article we will do some elaboration, while the example code introduces harbor and k8s integration.

First, the security mechanism of harbor

Software development teams in enterprises are often divided into roles such as project managers, product managers, testing, operations, and so on. In the actual software development and operations process, these roles are different for the use of mirroring. From a security standpoint, there is also a need for a mechanism to control permissions.

For example, developers clearly need to have read-write (Pull/push) permissions to the mirror to update and correct the code, and the tester needs to read (pull) permissions, and the project manager needs to manage the roles described above.

Harbor provides both user and member management concepts for this requirement.

In harbor, the user is divided into two main categories. One is administrator and the other is normal user. Both types of users can be members of a project. The administrator can manage the user.

A member is a concept that corresponds to a project and is divided into three categories: Administrators, developers, and visitors. Administrators can configure and administer permissions for developers and visitors. The test and OPS personnel can read the project image, or the files in the Public Image Library, as guest.
From the project point of view, it is clear that the project administrator has the largest project permissions, if you want to disable or restrict users, you can modify the user in the project member role to implement, and even remove the user out of this project.

650) this.width=650; "class=" Alignnone size-full wp-image-1742 "src=" https://www.kubernetes.org.cn/img/2017/03/ 20170314213441.jpg "alt=" 20170314213441 "width=" 620 "height=" 326 "/>

Second, Harbor Mirror synchronization Why mirror synchronization is required

Since access to the mirror is a core container concept, a mirrored library may not be sufficient in practice, and in the following cases we may need to deploy multiple mirrored warehouses:

• Foreign Public image download too slow, need a transit warehouse for acceleration

• Large container size, a mirrored warehouse overwhelmed

• High system stability requirements requiring multiple warehouses to ensure high availability

• Mirror Warehouse has multi-level planning, subordinate warehouse depends on superior warehouse

A more common scenario is that in an enterprise software environment, there are different mirrored warehouses at different stages of software development,

• In the development environment Library, developers frequently modify the image, and once the code is complete, generating a stable image requires synchronization to the test environment.

• In the test environment library, the tester is read-only to The mirror, and after the test is completed, the image is synchronized to the pre-launch environment library.

• In the pre-launch environment library, OPS is also read-only for mirroring, and once it is running, the mirror is synchronized to the production environment library.

• In this process, mirror synchronization and replication are required between the mirrored libraries in each environment.

Mirroring synchronization mechanism of Harbor

With multiple mirrored warehouses, mirroring synchronization between multiple warehouses becomes a common requirement. There are two ways to compare the traditional image synchronization:

• The first scenario is to use the Rsync service provided by Linux to define the synchronization of mirrored data between two warehouses.

• The second scenario is to use the IaaS Configuration tool to configure the mirroring synchronization for scenarios where the IaaS service is used for mirrored storage.

Both of these scenarios depend on the storage environment in which the warehouse resides, and require different tool policies. The harbor provides a more flexible solution for mirroring synchronization, with three concepts at its core:

• Use Harbor's own API to download and transmit images and decouple them from the underlying storage environment.

• The task scheduling and monitoring mechanism is used to manage replication tasks and ensure the robustness of replication tasks. During the synchronization process, if the source image is deleted, Harbor automatically synchronizes the remote mirror deletion. In the process of mirroring synchronous replication, harbor monitors the entire replication process, encounters network errors, and retries automatically.

• Provides replication policy mechanisms to ensure project-level replication requirements. In harbor, you can create a replication policy in your project to synchronize the mirrors. The difference with Docker registry is that harbor replication is a push strategy, initiated by the source, and the Docker registry replication is a pull strategy, initiated by the target side.

650) this.width=650; "class=" Alignnone size-full wp-image-1743 "src=" https://www.kubernetes.org.cn/img/2017/03/ 20170314213448.jpg "alt=" 20170314213448 "width=" 621 "height=" 366 "/>

Harbor Multi-level deployment

In the actual enterprise-level production operations scene, often need to cross-region, cross-level mirroring synchronous replication, such as the group from the headquarters to the provincial company, by the provincial company re-city company scenario.
This deployment scenario simplifies such as:

650) this.width=650; "class=" Alignnone size-full wp-image-1744 "src=" https://www.kubernetes.org.cn/img/2017/03/ 20170314213455.jpg "alt=" 20170314213455 "width=" 648 "height=" 399 "/>

More complex deployment scenarios such as:

650) this.width=650; "class=" Alignnone size-full wp-image-1745 "src=" https://www.kubernetes.org.cn/img/2017/03/ 20170314213503.jpg "alt=" 20170314213503 "width=" 638 "height=" 401 "/>

Three, harbor and K8s's integration practice

Harbor provides role-based access control mechanisms and controls the ability to organize and access images through projects. Kubernetes in the namespace to isolate resources, in an enterprise-level scenario, the combination of the two can effectively manage and access control of the mirrored resources used by kubernetes, enhancing the security of the image. Especially in multi-tenancy scenarios, the management and access control of multi-tenant mirrored resources can be achieved through a combination of tenants, namespace, and projects.

Core concepts and key steps of integration

The integration of both, a core concept is K8s's secret. As an important resource secret in Kubernetes, it is designed to solve problems that the container validates when accessing external networks or external resources, such as accessing a git repository, connecting a database, setting some password configuration, etc. Scenarios that require additional validation secret store sensitive data, such as permission tokens that allow the container to accept requests. By linking the user information of harbor with the secret of k8s, the integration of the two is achieved. The steps are as follows:

• Create the Create user, project, and set the project to private in harbor.

• Add the created user to the project, set the user's role as a developer or as a project administrator. Make sure that the account has permission to pull the warehouse image.

• Create k8s under Secret, where the user name, password, and mailbox address information in secret are information for the user who was created in harbor.

In this process, it is important to note that the secret created in the third step, the corresponding user must have the right to download the image in the corresponding private library of harbor, otherwise the application deployment will be reported unable to download the mirror.

For example

• A user was created in harbor, such as UserD

• Create a private project in harbor, such as ProjectA

• Use the Docker command line in harbor to log in and upload the image to the private library in step 2

• Create namespace in k8s

• Create the Secretc in K8s namespace, which corresponds to the user account in Harbor UserD

Use the image in the Harbor Private library to deploy the app in the K8s namespace, specifying that the image is downloaded using the secretc created above
If you only need a mirror that can pull harbor to deploy the app in K8s, UserD in harbor needs to have the least privileged guest member role in ProjectA.

Code practices for harbor and K8s integration

Imagepullsecret is used in k8s to save the authentication information of the image warehouse, so as to facilitate kubelet to obtain the authentication information of the Mirror warehouse when the pod is started, and to ensure that it can kubelet enough permissions to download the image required by the pod from the Mirror warehouse.
First, let's take a look at how the Imagepull type of secret in K8s is created. The official documents are: https://kubernetes.io/docs/user-guide/images/#specifying-imagepullsecrets-on-a-pod

The following code practices the harbor version is 0.3.5.

First, we need to select a user in harbor, use its username and password to generate a string, the user name and password in the middle with a colon, and then use Base64 to encrypt it, as follows: [[email protected] ~]# echo "test: tt001″|base64dgvzddp0vdawmqo=

Then, we need to generate a dockerconfig.json, need to use the above generated encryption string, the content is roughly the following, if the corresponding harbor library address is: hub.testharbor.com:

{"Auths": {"hub.testharbor.com": {"auth": "dgvzddp0vdawmqo=", "Email": "}}}

We need to encrypt this entire JSON using Base64. The resulting string may be longer, and you need to add the-W 0 parameter to not wrap it. Save the above JSON as a Dockerconfig.json file, and then execute the command:

[email protected] secret]# cat Dockerconfig.json |base64-w 0 Ewogicjhdxrocyi6ihskicagicjodwiudgvzdghhcmjvci5jb20ioib7ciagicagicjhdxroijogimrhvnpkrhawvkrbd01rbz0ilaogicagicaizw1hawwio iaiigogicagfqogih0kfqo=

Now, we can create the YAML required for secret. When secret is created, you must specify namespace. Secret in multiple namespace can have the same name. If we need to create a secret named Testsecret in the namespace named Hub, the corresponding SECRET.YAML content is as follows. The encryption string generated above needs to be used.
You can create the corresponding secret using the Kubectl create command in k8s:

Kubectl create-f Secret.yaml

Finally, when deploying the application, we need to specify the secret name required to download the image for the pod as follows:

Apiversion:v1 kind:pod metadata:name:httpdpod namespace:hub spec:containers:-Name:httpdpod image:hub.testharbor.c om/project1/httpd:2.2 imagepullsecrets:-Name:testsecret

Note that in order for the deployment to succeed, the test user must be a member of the Project1 project in harbor to have permission to download this httpd:2.2.

User and integration for container cloud

As the container cloud runs, harbor users and K8s's secret can have a more intensive way of consolidating. In our project practice, a user of a container cloud corresponds to project one by one with the same name in a harbor, but this user can create multiple namespace in k8s. In order to simplify the management process, we are currently doing the following:

• In the container cloud boot process, a dedicated user is automatically created in harbor to download the mirrors in each private library.

• Each new private library in harbor will add this dedicated user as its guest member role, so that this dedicated user has permission to download the images in this library

• In k8s, each time a new namespace is created, under this namespace, a fixed-name secret is created using the information from the private user above (namespace with the same name can exist in multiple secret)

• When you deploy the app, specify the name Imagepullsecrets below as the secret name that is fixed above.

Four, two tips

Use the online tool to make harbor interface documents easier to read

Harbor provides a restful form of interface for other system integration, its interface describes the swagger format of the document contained in the source code, the document address is: https://github.com/vmware/harbor/blob/master/ Docs/swagger.yaml.

At present, more and more systems use restful interfaces to expose services, while Swagger has become the de facto standard for describing restful interfaces. Using a text editor to view this swagger document is a bit faint, and it's not easy to see the overall structure of the interface clearly. We can look at these documents with some tools. Online, such as the official Swagger online editor, copy the contents of the document to the left, the right can display the HTML format of the document, can be folded, etc., so that people at a glance at all interfaces. Offline, you can install the plugin in some editors and convert it to HTML as well, such as Vscode, and install the Swagger Viewer plugin. For the use of swagger, you can also read my colleague Li Xiaofei's article "MicroServices Architecture Combat: Swagger Standard restful API."

The documents in both formats are shown below:

650) this.width=650; "class=" Alignnone size-full wp-image-1740 "src=" https://www.kubernetes.org.cn/img/2017/03/ 20170314213511.jpg "alt=" 20170314213511 "width=" 653 "height=" 507 "/>

Tips: Harbor's Java Client open source implementation

For users who are familiar with Java programming, Harbor officially does not provide Java client, but someone on GitHub has written about the project, Project address: https://github.com/grissomsh/harbor-java-client

650) this.width=650; "class=" Alignnone size-full wp-image-1741 "src=" https://www.kubernetes.org.cn/img/2017/03/ 20170314213518.jpg "alt=" 20170314213518 "width=" 638 "height=" 511 "/>

V. Summary

This paper mainly introduces the user mechanism, image synchronization and integration practice of harbor with k8s.

The user mechanism of harbor is divided into two categories: System users and project members. Users can become project members, and different members have different read and write permissions for the mirror.

The synchronization strategy and task scheduling mechanism of harbor provide a flexible mechanism for image synchronization between mirrored libraries.

With K8s's secret associated with harbor users, you can deploy an app by pulling a mirror from the Harbor Private library in k8s. We also use code to give examples.

Finally, share a few small pits encountered during the use of Harbor (0.3.5):

• When you create a user in harbor, the password must be a complex password. However, when modified, this limit is not

• User update password, the original password can not be consistent with the new password, otherwise reported 500 internal error

• When adding a member to Project Harbor, the member role does not have the relevant API, and the ID value that needs to be given does not have a constant definition, so for now, 1 is admin and 2 is devlop,3 for guest.

This article is from the "My Sky" blog, so be sure to keep this source http://sky66.blog.51cto.com/2439074/1934009

Harbor user mechanism, mirroring synchronization, and integration practices with kubernetes