How to use VMware Harbor in rancher Catalog

Harbor Registry is a Docker image management product for VMware. Compared to other mirrored warehouses, Harbor provides identity management capabilities that are more secure and support multiple registry on a single host, which is what many enterprise users need.

On top of the rancher container management platform, VMware Harbor can be added as an entry in the Rancher Store (Catalog). This article will show you how to Docker the harbor online installer and then rancher it to install Harbor on a distributed cluster of Docker hosts.

This author, Massimo, is the cloud architect for VMware Cloud native application business unit.

In order to learn more about Docker containers and their ecosystems, I have been focusing on rancher (open source container management platform) for the past few months.

I am most interested in rancher's Extensible App Store (catalog) and infrastructure services: The library is the official App Store directory (maintained and built by rancher), and a "Community App Store" (maintained by rancher, However, built and supported by the rancher community, users can also add a private store directory (you can add your own private directory entries and maintain them yourself).

Although rancher enables users to connect to cloud Registry, only one Registry (Convoy Registry) in the community directory can be deployed and managed. This will be a good opportunity to add VMware Harbor as a private directory entry option, as shown in:

650) this.width=650; "src=" https://s5.51cto.com/wyfs02/M02/8F/7B/wKiom1jff_SAOWNiAAA0X6G0mTY394.jpg "title=" 1.jpg "alt=" Wkiom1jff_saowniaaa0x6g0mty394.jpg "/>

You should consider all the things you know, literally, in the current form, whether you can copy an idea into production.

Thanks

Before we delve into this, I want to thank Rancher's Raul Sanchez (patiently) for answering my questions (and helping me fix some of the wrong yaml). Without his help, I think this blog will be a lot shorter. Of course, because everyone should be like this, so I also want to thank my wife and my dog (although I don't have a dog).

Difficulty upgrading

First, I need to complete a certain number of tasks to complete the initial goal. In general, a task is dependent on another task.

If you want to create a rancher store entry, instantiate your application from the application definition file (the standard Docker compose file when using the default cattle scheduler) and the rancher compose file. Shell scripts or something like that do not become part of the rancher store entry.

Are you studying how to install Harbor on a Docker host (via the documented "online installer")? It is not really compatible with the Rancher App Store model. You can refer to the following steps:

When installing harbor, you must first download the harbor online tar.gz installer file, set your configuration settings in the Harbor.cfg file, and then run the prepare script. This script enters the Harbor.cfg file and then creates the configuration file and environment variable file. Finally, run the Docker compose file to pass the configuration file and environment variable file as the volume and instructions for the Docker compose (note that some of the procedures occur under the main installation script and occur under the screen). This is the standard online installation program. In fact, the Docker compose file captures the Docker image of the Docker hub and instantiates the harbor based on the configuration input.

Finally, the simple "PoC" project that starts is divided into three "sub-projects":

1. Docker the harbor online installer so that the "prepare" process can be part of the Docker compose and pass the input parameters as variables to the Docker Compose (instead of manually editing the Harbor.cfg file and then performing the entire preparation circus).

2. Rancher the already Docker Harbor online installer and create a rancher private store to simulate typical single-host harbor settings.

3. As a bonus: Rancher the Docker Harbor online installer and create an app template for the rancher private store, allowing us to install Harbor on a distributed cluster of Docker hosts.

Note that I need to create a Docker harbor online installer to match the rancher directory model, and when you can't use manual and interactive methods, only automatically stand up harbor, You can also use this online installer (rancher is one of these user instances).

In the next few sections, I'll detail some of the work I've done to implement these sub-projects.

Subproject 1: containerized the Harbor Online installer

As I write this article, Harbor 0.5.0 can already be installed by using OVA or by installing the program. The installer can be either online (mirroring is dynamically extracted from Docker hub) or offline (mirroring is part of the installer and will be loaded locally).

The next thing we want to focus on is the online installer.

As we have already mentioned, once you have downloaded the online installer, you will need to adjust the parameters of the Harbor.cfg file included with the installation package template to "prepare" your harbor installation.

The resulting configuration set is then entered into the Docker compose file (through the local directory mapped to "volume" and Through the "env_file" Directive).

Wouldn't it be easier or better to pass the harbor settings parameter directly to the Docker compose file without the "prep" process?

Enter the Harbor-setupwrapper.

Harbor-setupwrapper is a Harbor installation package that contains a new Docker image, and (more or less) implements the prepare process in a Docker container. Instead, the harbor configuration parameter is entered into the container as an environment variable. Finally, it is important to run a script in the container to start the prepare routine (which is what all the containers themselves contain).

The dockerfile of this image and the script to start preparing the routines are all issues worth studying.

In fact, what is harbor-setupwrapper.sh and what is install.sh as the standard Harbor online installer, the two issues are very similar.

We now have a new Docker compose file, which is largely based on the original Docker compose file. Additionally, this original Docker compose file is included with the official online installation program. Now you can pass the parameters you have adjusted in the Harbor.cfg file by combining this new Docker compose file.

As shown in the following:

650) this.width=650; "src=" https://s1.51cto.com/wyfs02/M00/8F/7B/wKiom1jfhcHTy7fSAAB2LnQwDzQ292.jpg "title=" 2.jpg "alt=" Wkiom1jfhchty7fsaab2lnqwdzq292.jpg "/>

Be sure to note that this is just a poc!

• I only tested the Harborhostname and harbor Admin_password variables. The other variables should be operational, but I'm not testing them.

• There must be a special case. For example, if you choose to use secure connection (HTTPS), and I haven't found a way to create a certificate. This is the time to implement additional logic in harbor-setupwrapper.sh (hint: some strange things can happen when you enable HTTPS)

• Using the original online installer means running on a single Docker host. I will implement this new installation mechanism under the same model and the same prerequisites

• For these reasons, I have not tried to deploy this compose file on a distributed swarm cluster. In addition, although "legacy Swarm" has been converted to "Swarm mode", but Docker compose and the latter do not seem to be compatible, and I do not want to spend too much time on the former, so I chose not to test it in the SWARM environment

• There may be more warnings, but I have not thought about it (but there is certainly a possibility!). ）

It is not difficult to provide the configuration files in wrapper (generated by the harbor-setupwrapper.sh script) to the application container. I have implemented the "volumes_from" directive, so the application container can get the relevant configuration file directly from the wrapper container.

It is difficult to find a way to pass the environment variable (various files on the wrapper container) to the application container. And I can't run the "env_file" directive in compose because the instructions point to the files that are visible in the system running compose (these files are in the wrapper container when I use them). To make a long story short, I changed the portal of the application container, pointing to a script that would first load these environment variables and then start the original script or the command for the original entry. If you like, you can check all the entrypoint *. sh files in Harbor-setupwrapper GitHub repo.

If you're going to do it this way and set up harbor with this new mechanism, you'll need to clone Harbor-setupwrapper?repo and "Up" the Docker you found in the Harbor-setupwrapper directory Compose file. However, before starting it, export the Harborhostnam and Harbor_admin_password variables first. This is equivalent to adjusting the Harbor.cfg file in the original Setup program. If you forget to export these variables, Docker compose will display:

650) this.width=650; "src=" Https://s5.51cto.com/wyfs02/M00/8F/79/wKioL1jfheXDVTDnAAAl8SDtF34484.png "title=" Code 1.png "alt=" Wkiol1jfhexdvtdnaaal8sdtf34484.png "/>

At least set the Harborhostname variable to set the Harborhostname variable to the IP address or FQDN of the host on which it will be installed (otherwise the setting won't work and I'll explain why later). If you do not set the Harbor_admin_password variable, you will need to use the default Harbor password (Harbor12345).

What you have to do is:

650) this.width=650; "src=" Https://s4.51cto.com/wyfs02/M02/8F/7B/wKiom1jfhj2idalbAAAnd72E4VU060.png "title=" Code 2.png "alt=" Wkiom1jfhj2idalbaaand72e4vu060.png "/>

Note: If you intend to implement the harbor instance repeatedly on the same host and intend to start from scratch, be sure to delete the/data directory on the host (because it will save the instance state and if the new instance finds the directory, it will receive the previous instance state).

Subproject 2: Create a rancher store entry for a single-host deployment

We can use "compose up" to Docker the harbor installer. Now we can focus on the second sub-project. In other words, create the structure of the rancher store entry.

I think this should be easier. After all, we discussed the issue of re-using the new Docker-compose.yml file.

I've learned that it's difficult to be detailed, especially in a container article, where adjustments to "fix" a particular problem usually mean opening some worms somewhere else.

I'll try to write as much as I can about what you need or what you want to know so that you know more about the installation package, and I'm sharing my experience with you, hoping to help you in other situations (I'll record my findings or I'm afraid I'll forget it in 2 weeks).

First of all, in the rancher, you can only do "Volume_from" in sidekick. At the very beginning, I was adding "io.rancher.sidekicks:harbour-setupwrapper" to each container in the compose. However, I suddenly found that this creates a harbor-setupwrapper for each container to assist the container, which is a sidekick. While it all seems to be ready, I finally found that running instances of multiple scripts under a single harbor deployment could result in various configurations being inconsistent (such as tokens signed with untrusted keys, etc.).

I need to change the strategy to become an instance of only one Harbor-setupwrapper container (all the configuration files will be generated consistently in one process) and I have implemented it in the main container and all other application containers. In fact, I just added "Io.rancher.sidekicks:registry,ui,jobservice,mysql,proxy" as a label for the Harbor-setupwrapper container. (Warning: I didn't tell Raul about it because it might scare him or any other rancher expert.) But it worked, so please forgive me. ）

We solve another problem by opening up a problem. Sidekick Container name resolution does not really work as you would expect, so I can only find other solutions (if you're interested, you can get to the problem here and fix it: https://forums.rancher.com/t/ cant-resolve-simple-container-names-within-sidekick-structure/3876).

There are two more issues to be resolved during the creation of the rancher store entry:

• The "harborhostname" variable needs to be set to the exact value so that the user can connect to the harbor instance through it.

• All harbor containers can only be deployed on a single host, which may be one of many host (cattle) clusters.

I might let the rancher expert once again be afraid, in order to paste "Harbor-host = true" label on all the containers on the host, I have already configured the Docker compose file.

This way I can ensure that all containers are deployed on the same host (and, more importantly, have a certain degree of control over a host). And because I know which host my container is going to reach, I can choose the variable "harborhostname" wisely. It can be either the host IP address or the host FQDN.

Finally, the Docker compose file will publish ports 80 and 443 on the agent container on the host (obviously these ports are free on that host, otherwise the deployment will fail). Perhaps this is not a best practice, but it can solve some basic or easier problems.

Note: Because the status is saved in the host's/Data directory, if you are starting and shutting down the harbor instance for testing, you are saving the state in multiple deployments. There's still a big gap between running a real cloud native application, but it shows how harbor is built, and I'm just loyal to the original operating mode of the rancherization scenario on a single host.

Describes the details and relationships among the individual components in a single host deployment:

650) this.width=650; "src=" https://s5.51cto.com/wyfs02/M02/8F/79/wKioL1jfhn-Dl4jNAAB97CSSEd8576.jpg "title=" 3.jpg "alt=" Wkiol1jfhn-dl4jnaab97cssed8576.jpg "/>

is the actual deployment during my run:

650) this.width=650; "src=" https://s2.51cto.com/wyfs02/M00/8F/7B/wKiom1jfho-BfowKAACs6btQaEw301.jpg "title=" 4.jpg "alt=" Wkiom1jfho-bfowkaacs6btqaew301.jpg "/>

The current status of the Harbor Private Store app template for rancher for single-host deployments is as follows:

• It only applies to the cattle scheduler

  There should be no relationship between building the version of the Harbor directory for Swarm and k8s and the version of cattle.

• This application template has all the limitations of the above Docker online installer (for example, it does not support HTTPS, etc.)

• When pushing or pulling a mirror on a Docker host, set the "-insecure-registry" flag on the Docker daemon (because we can only start harbor via HTTP access)

• A host must have a docker-compose "Harbour-host = true" label so that it can work and schedule the container

• Ports 80 and 443 must be available on the host with the "Harbour-host = true" label

You can find the deliverable for this subproject in the extension library of my rancher store: Https://github.com/mreferre/rancher-catalog-extension.

Subproject 3: Rancher store entry creation for distributed deployment

This is a very challenging part of operating a distributed application, and it is also an interesting and useful part.

Although Harbor is a containerized application, for some reason it is not an ideal choice for applying best practices for cloud on-premises application operations. It does not comply with the methodology of the 12 factor application (the Twelve-factor app).

First, the harbor installer has been established under the premise that 6 containers are running on a "well-known" single host. Here are some examples that highlight some of the challenges. We may have mentioned some of them:

• The harbor package comes with an embedded syslog server, which is a session/log for the Docker daemon. If you look at the original Docker compose file, you will find that assuming that the syslog is running on the same host on all other containers, all application containers are logged to 127.0.0.1

• You must enter (as a setup parameter) the exact harbor hostname so that the user can connect to the registry server. Ideally, in a cloud-native environment, an application should be able to use any given IP/FQDN associated with it. Finally, there should be an option to set (after setting) the correct IP/FQDN endpoint that the application will use. For harbor 0.5.0, you need to know what the (pre) IP/FQDN is before starting the setup (making something more difficult to operate in dynamic, self-service, and distributed environments). That is, if your harbor service is exposed to the user in the form of "service123.mycompany.com", you must enter the string as the FQDN at deployment time (and perhaps not even know which hosts of the container to deploy)

• Harbour running on a known single host is a hypothetical part of the product that stores its own state on the local directory of the host to which it is deployed. This is done through different directory mappings in the container configuration.

The goal of this subproject is to have harbor run on a cattle cluster, rather than running on a known host.

To do this, the log image needs to be instantiated on each node of the cluster (requires: Each node must have a label of "Harbor-log = true"). A better solution is to have a separate Syslog server pointing (which completely removes the log service from Docker compose).

In addition, since we do not know which host the proxy server is going to reach (in this case, we want to achieve a low-touch experience with service discovery), we implemented the harbour distributed model by leveraging Traefik. If you're familiar with Docker, you'll find Traefik doing (somewhat) similar to the "HTTP Routing Mesh" out-of-the-box experience that Docker provides through swarm mode. Note that the agent container ports (80 and 443) are not exposed to the host, and Traefik is the only way to expose the service to the outside world (in this particular distributed implementation).

The overall idea is that your DNS can parse the IP that runs Traefik, and then Traefik will "automatically" add the hostname you entered in the Harbor settings to its configuration.

Storage Management is also an interesting part. In a distributed environment, you cannot allow a container to store data on a server that can run in a timely manner at any given point in time.

If the container restarts on another host (due to a failure or upgrade), it needs to access the same set of data. Not to mention that other containers (which may run on different hosts) need access to the same set of data.

To solve this problem, I chose to use the generic NFS service provided by rancher, which is flexible, convenient, and useful. Because it allows you to preconfigure all of the required volumes (in this case, they are re-instantiated by Harbor directory entries), or you can have Docker compose created automatically when instantiated (in this case, they are deleted when the harbor instance is closed). Note that all volumes are mapped to the application container (in addition to the log and proxy containers that do not require the volume). There is a lot of room for optimization (because not all volumes need to be mapped to the container), but I will not consider this issue for the time being.

Because there is no volume directory mapping in Docker compose (all volumes are named volumes on the NFS share), this causes all hosts to be stateless.

Shows detailed information and relationships among the components in a distributed deployment:

650) this.width=650; "src=" https://s1.51cto.com/wyfs02/M01/8F/7B/wKiom1jfhsjQm48nAACu9Z_yIV4359.jpg "title=" 5.jpg "alt=" Wkiom1jfhsjqm48naacu9z_yiv4359.jpg "/>

This picture shows the deployment in the actual operation:

650) this.width=650; "src=" https://s4.51cto.com/wyfs02/M01/8F/79/wKioL1jfht-jsnzdAAEGfvycGZI111.jpg "title=" 6.jpg "alt=" Wkiol1jfht-jsnzdaaegfvycgzi111.jpg "/>

Rancher the current status of the harbor Private store entry for distributed deployment is as follows:

• It only applies to the cattle scheduler

  There should be no relationship between building the version of the Harbor store for Swarm and k8s and the version of cattle.

• This store entry has all the limitations of the above Docker online installer (for example, it does not support HTTPS, etc.)

• When pulling/pushing an image on a docker host, set the "-insecure-registry" flag on the Docker daemon (because we can only start harbor via HTTP access)

• All hosts on the cluster must have a docker-compose "Harbour-host = true" label to function properly and schedule the container

• The Traefik service (located in the Community directory) needs to be up and running to access the harbor externally. This has exposed port 80 (note that the default value of Traefik is 8080)

• The NFS service (located in the Library directory) needs to be started, run, and properly configured to connect to the NFS share. Docker compose file is parameterized, I only tested "RANCHER-NFS" in other drivers

You can find a deliverable for this subproject in my Rancher Private store directory: Https://github.com/mreferre/rancher-catalog-extension.

General challenges and Issues

In this project, I met with some challenges. I will introduce some of them and write them down mainly for future reference.

• Some small things sometimes have some big uses. Sometimes it is a cascading problem (i.e. do a you need to finish B, but do B and need to finish C). For example, Rancher sidekick requires the ability to perform "Volume_from". This basically breaks the full name resolution (see the Single Master section for more information about what the problem is)

• The presence of "full green" in the container does not mean that your application will start and (normally) run. Sometimes, the container starts to determine that there are no errors, but I cannot log on to harbor (due to a mismatch in the certificates generated by running multiple instances of Setup wrapper). Sometimes, I can log on, but I can't push the mirror. Sometimes, I can push mirroring, but they won't show up in the UI (because of the Sidekicks name resolution problem, the registry container cannot resolve the UI container name)

• debugging containers in a distributed environment is difficult. Sometimes I thought I had a random problem, and then I found out because a particular container was dispatched on the misconfigured host (randomly). If the source of the problem is found, it is easy to solve the problem. But often, it is difficult to find the root cause of the problem.

• When wrapping an application to run in a container (most importantly, orchestration deployment), it is critical to understand the internals of the application. One of the reasons I connect all named volumes to all containers in a distributed scenario is because I can't 100% determine which container reads/writes from which volume. In addition, it is not known that the application will make it difficult to package (especially if something does not work correctly when debugging is required). In summary, containers (and orchestrations) are more like how you package and run applications with how you manage infrastructure

• Although container orchestration is about automation and repeatable, it's also a bit like "handmade elf art". Sometimes (random) the proxy container will only display the Nginx Welcome page (not the Harbor user interface). Finally, I resolved this issue by restarting the container (after deployment). I think this is the reason for a startup sequence. I tried to use the "depends_on" command to make the agent container start "toward the end" composition, but this did not succeed. It now seems to be through the use of the "external_links" directive (which theoretically should not require AFAIK). In summary, proper coordination of container start-up is still part of the process (starting in 2014)

• It is difficult to manage the infrastructure (and services) to run containerized applications. When I use some simple services, such as Basic rancher NFS services, I have some problems, I have to address the use of different levels of software, different deployment mechanisms and so on. Upgrading infrastructure from one version of infrastructure to another is also critical

• Another NFS problem I encountered is that when the stack is closed, the volume cannot be properly purged on the NFS share. In the rancher UI, the volume seems to no longer exist, but the NFS share is viewed directly, and some (a random number) seem to be left in the form of the remaining directory. I did not delve into why this was.

Conclusion

As I mentioned, it's just a rough integration, and there's no question that it can be perfected. This is primarily a good learning experience that can be further expanded in the future, such as integration within rancher Kubernetes, enabling HTTPS protocols, and so on.

The dynamic configuration of application services in distributed systems is also one of the challenges of this experiment, but it is not very complicated, but this process will give you a better understanding of how to solve these problems.

From a higher level, it is possible to containerized and deploy an application in a distributed system in two ways: one is that your basic services are built in PAAs, the required service resources (such as scheduling, load balancing, DNS resolution, etc.) are applied to the PAAs, and you may need to deeply transform your application; Using a platform like rancher, which uses the features it provides, only needs to write compose orchestration files, which can be easily implemented. In fact, Ranche is also more to help you quickly implement your needs in the second scenario, and easy-to-use infrastructure services help you quickly implement application containment.

Original source: Rancher Labs

This article is from the "12452495" blog, please be sure to keep this source http://12462495.blog.51cto.com/12452495/1912483

How to use VMware Harbor in rancher Catalog