After deploying and configuring the Harbor tile in Pivotal Ops Manager, I ran into a couple of issues with certificates. The first was encountered when I was trying to login to harbor from an Ubuntu VM where I was running all of my PKS and BOSH commands. It was also the VM where I pulled my container images, and the VM from which I now wanted to push them into Harbor. Harbor is our registry server for storing container images. Here is what I got on trying to login:
cormac@pks-cli:~$ sudo docker login -u admin harbor.rainpole.com
Password:
Error response from daemon: Get https://harbor.rainpole.com/v1/users/: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “Pivotal”)
cormac@pks-cli:~$
To resolve this first issue, I had to log into the Harbor UI as the Admin user. From, there I navigated to Administration > Configuration > System Settings, and then I clicked on the Download link associated with the Registry Root Cert, as shown below.
On my Ubuntu VM, the certificate needed to be placed in a particular directory /etc/docker/certs.d/harbor.rainpole.com, where harbor.rainpole.com is obviously the name of my registry that I am trying to login to. With the cert in place, I can now login to my registry, as shown below.
cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ uname -a
Linux pks-cli.rainpole.com4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:36:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ ls
ca.crt
cormac@pks-cli:/etc/docker/certs.d/harbor.rainpole.com$ sudo docker login -u admin harbor.rainpole.com
Password:
Login Succeeded
Cool. At this point, I thought I had solved the certificate issue. I was able to login to Harbor, tag images and push/pull to/from the registry. My next step was to deploy a couchbase app on my Kubernetes cluster, the image of which I had pushed to my registry. However, I got the following issue during the application creation:
root@pks-cli:~/cns-demo# kubectl get pods
NAME READY STATUS RESTARTS AGE
couchbase-0 0/1 ErrImagePull 0 12s
root@pks-cli:~/cns-demo# kubectl describe pods
Name: couchbase-0
Namespace: default
Priority: 0
.
.
.
Events:
Type Reason Age From Message
—- —— —- —- ——-
Normal Scheduled 3s default-scheduler Successfully assigned default/couchbase-0 to 2e2478da-5a3f-4941-90b1-9410f2cebab2
Normal SuccessfulAttachVolume 2s attachdetach-controller AttachVolume.Attach succeeded for volume “pvc-b5eb9ff9-2f2b-11e9-805e-00505682e96b”
Normal Pulling <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 pulling image”harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”
Warning Failed <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 Failed to pull image “harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”: rpc error: code = Unknown desc = Error response from daemon: Get https://harbor.rainpole.com/v2/: x509: certificate signed by unknown authority
Warning Failed <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 Error: ErrImagePull
Normal BackOff <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 Back-off pulling image “harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”
Warning Failed <invalid> kubelet, 2e2478da-5a3f-4941-90b1-9410f2cebab2 Error: ImagePullBackOff
root@pks-cli:~/cns-demo#
After some investigation, I found that I missed a step of integrating Harbor with PKS. In a nutshell, I should have copied the contents of my Harbor Registry CA certificate (same certificate I downloaded to my VM) and add it to the BOSH’s list of Trusted Certificates under Security in the BOSH tile in Pivotal Ops Manager. Once I had added it and applied the changes, I was successfully able to deploy my application.
root@pks-cli:~/cns-demo# kubectl get pods
NAME READY STATUS RESTARTS AGE
couchbase-0 1/1 Running 0 50s
root@pks-cli:~/cns-demo# kubectl describe pods
Name: couchbase-0
Namespace: default
Priority: 0
.
.
.
Events:
Type Reason Age From Message
—- —— —- —- ——-
Warning FailedScheduling 30s (x6 over 37s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
Normal Scheduled 30s default-scheduler Successfully assigned default/couchbase-0 to e47914d4-efa3-4087-87f1-f7feb665b324
Normal SuccessfulAttachVolume 28s attachdetach-controller AttachVolume.Attach succeeded for volume “pvc-8f84d30d-2f8b-11e9-a131-005056821e38”
Normal Pulling 20s kubelet, e47914d4-efa3-4087-87f1-f7feb665b324 pulling image “harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”
Normal Pulled 7s kubelet, e47914d4-efa3-4087-87f1-f7feb665b324 Successfully pulled image “harbor.rainpole.com/library/saturnism/couchbase:k8s-petset”
Normal Created 7s kubelet, e47914d4-efa3-4087-87f1-f7feb665b324 Created container
Normal Started 7s kubelet, e47914d4-efa3-4087-87f1-f7feb665b324 Started container
root@pks-cli:~/cns-demo#
The post Pivotal and Harbor – x509 certificate issues appeared first on CormacHogan.com.