January 5, 2022
Table of Contents
NSX Application Platform Part 2: Harbor Image Registry
Harbor Image Registry Configuration
The Harbor image registry is critical to the successful deployment of the NSX Application Platform (NAPP). It holds all the images and binaries required for the application platform, which are pulled as pods are being deployed.
There are many ways to deploy Harbor on-premises including, but not limited to:
- Using Helm to deploy a HA instance of Harbor on kubernetes
- Tanzu Application Service with Harbor Integration
- Deploying the Harbor image registry on a virtual machine
This guide will walkthrough deploying Harbor on a Ubuntu virtual machine.
I have recently put together a video that provides clear guidance on deploying NAPP, it can be seen here.
Virtual Machine Configuration
This section details the specifications of the Ubuntu virtual machine that I have deployed for use as my Harbor image registry.
- Ubuntu 20.0.4.3 Focal Fossa
- 2 vCPU/8GB RAM
- 2 Disks attached, one for the OS (40GB) and one for data (150GB)
- Single network interface on my management network (IP address 192.168.63.100)
- VM has internet access to generate certificates and pull files for install
Note: This article will not cover the Ubuntu deployment process.
Prerequisites – Installing Docker
Prior to installing Harbor, the Ubuntu VM must meet the prerequisites, all of which can be found here.
The official steps from Docker can be found here. The commands I ran are provided in the output below.
root@harbor:/# apt-get update
root@harbor:/# apt-get install
> ca-certificates
> curl
> gnupg
> lsb-release
root@harbor:/# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
root@harbor:/# echo
> "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu
> $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
root@harbor:/# apt-get update <--------- DO NOT SKIP THIS
root@harbor:/# apt-get install docker-ce docker-ce-cli containerd.ioNext, check that the service is running.
Finally, I complete the prerequisite installation by installing Docker Compose.
root@harbor:/mnt/data/harbor# sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 664 100 664 0 0 14434 0 --:--:-- --:--:-- --:--:-- 14434
100 11.6M 100 11.6M 0 0 14.6M 0 --:--:-- --:--:-- --:--:-- 25.8M
root@harbor:/mnt/data/harbor# sudo chmod +x /usr/local/bin/docker-compose
root@harbor:/mnt/data/harbor# docker-compose --version
docker-compose version 1.27.4, build 40524192
Harbor Repository Certificate Configuration
In this article I utilize a Lets Encrypt certificate for my Harbor registry.
Note: as of NSX-T 3.2 release (Impactor), self-signed certificates, including Active Directory issued certificates are not supported.
The certificates used for the registry must be signed by a trusted certificate authority. If you do not utilize a trusted certificate, when attempting to deploy NAPP, you will be faced with an x509 certificate issue, similar to the below output.
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned cert-manager/cert-manager-69cc999bb5-khjws to impactorlab-workers-snmxl-dc89f6748-s9p4p
Normal Pulling 10m (x4 over 11m) kubelet Pulling image "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763"
Warning Failed 10m (x4 over 11m) kubelet Failed to pull image "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763": rpc error: code = Unknown desc = failed to pull and unpack image "harbor.shank.com/impactor/clustering/third-part
y/cert-manager-controller:19067763": failed to resolve reference "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763": failed to do request: Head "https://harbor.shank.com/v2/impactor/clustering/third-party/cert-manager-controller/manifests/19067763":
x509: certificate signed by unknown authority
Warning Failed 10m (x4 over 11m) kubelet Error: ErrImagePull
Warning Failed 6m54s (x21 over 11m) kubelet Error: ImagePullBackOff
Normal BackOff 108s (x44 over 11m) kubelet Back-off pulling image "harbor.shank.com/impactor/clustering/third-party/cert-manager-controller:19067763"You will still be presented with this issue if you utilize the workaround here. This is because the underlying docker / containerd still doesn’t trust the certificate and chain. There is potential to work around this issue, however, I will not get into that here. If you absolutely require the use of a self-signed certificate, get in contact via Twitter, LinkedIn, or email.
Generating a Lets Encrypt Certificate
First, I install Lets Encrypt on my virtual appliance. I have listed the commands I ran in the output below. Remember, this command is being run on Ubuntu, you will need to edit the command to suit your operating system.
### Install the package and all its dependencies
root@harbor:/home/harbor# apt install letsencrypt
### Check to ensure it is running
root@harbor:/home/harbor# systemctl status certbot.timer
Now that Lets Encrypt is installed, I generate the certificate for the appliance. Keep in mind that this virtual appliance has internet connectivity, and I have created a DNS A record with my hosting provider for my appliance.
Issue the command below on your Harbor appliance, and follow the prompts on screen.
## Change the value after -d to match the hostname of your Harbor appliance.
root@harbor:/home/harbor# certbot certonly --standalone -d harbor.lab2prod.com.au
As per the prompt, the certificates have been generated and are available in /etc/letsencrypt/live/harbor.lab2prod.com.au/.
Setup and configure Harbor
Download the offline Harbor installer package located here, and transfer it to the Harbor appliance.
Extract the archive using the command below.
root@harbor:/mnt/data# tar xzvf harbor-offline-installer-v2.4.1.tgzBefore continuing, I realised that the Docker data was being stored in /var/lib/docker, this wasn’t going to work for me as it was running on the smaller 40GB drive. I had to move the Docker data to the larger partition, to do so I followed this guide.
Now, copy the Harbor configuration file that is in the uncompressed Harbor installer folder or rename it.
root@harbor:/mnt/data/harbor# cp harbor.yml.tmpl harbor.ymlEdit the yml file to reflect your environment details, the output below are the lines that need to be changed.
root@harbor:/mnt/data/harbor# vi harbor.yml
#### The main options that I changed are
hostname: reg.mydomain.com
certificate: /your/certificate/path
private_key: /your/private/key/path
data_volume: /data
If you would like to see my harbor.yml it is located here.
Next, I install Harbor by running the command in the output below.
Note: Do not skip installing chartmuseum, it is required for NAPP.
root@harbor:/mnt/data/harbor# ./install.sh --with-chartmuseumIt will go through 5 steps to install and configure harbor and you should see something similar to the screen below once it is complete.
You should now be able to access the Harbor registry UI, it is also worthwhile to check that the certificate was properly installed and the site is secure.
Uploading NSX Application Platform Binaries to the Harbor Registry
The final part of this article is to push the images to the Harbor registry.
First I created a new project, to do so, login to Harbor and click on create a new project.
Once the project is created, the images can be uploaded. In my environment I chose to utilize a jumpbox which has all the tools I required pre-installed, as well as the NAPP binaries transferred and uncompressed. Details and configuration of this jump box can be found in the first article of this series.
Navigate to the folder that contains the uncompressed NAPP binaries, there should be a file called upload_artifacts_to_private_harbor.sh. Open this file in a text editor, in my case I used vim.
Change the first three lines to suit your environment, instructions on the VMware website for this can be found here.
DOCKER_REPO=harbor.lab2prod.com.au/impactor
DOCKER_USERNAME=admin
DOCKER_PASSWORD=Harbor12345Once you have made those changes, run the script to start the upload process.
Ignore the errors you see regarding a connection, this is specific to my environment. At this point you can leave the images to be pushed to your Harbor registry, the time this takes can vary depending on your environment.
Once this process completes, you should see something similar on your jumpbox or whatever you chose to upload the images from.
The repository should list 80 repositories and 18 charts.
And that’s it for the Harbor repository deployment!
Next Steps
NSX Application Platform Part 3: NSX-T, NSX-ALB (Avi), and Tanzu
The next part in this series focusses on NSX-T, NSX-ALB, and Tanzu.
NSX Application Platform Part 4: Deploying the Application Platform
The final part of the series demonstrates the deployment process for NSX Application Platform and its security features (NSX Intelligence, Network Detection and Response, and Malware Prevention.
Previous Article
NSX Application Platform Part 1: Environment Overview
This was the first article in the series, it provides an overview to the environment.
