Another product added to the VMware portfolio with the acquisition of Heptio is Sonobuoy. In a nutshell, Sonobuoy will validate the state of your Kubernetes cluster by running a suite of non-destructive tests against your cluster. As part of the end-to-end (e2e) tests that are run by Sonobuoy, there is a also a subset of conformance tests run as well. These include things like best practices and interoperability tests. This will ensure that your Kubernetes cluster (whether is an upstream version or a third-party packaged version) supports all of the necessary Kubernetes APIs. You can read more about conformance here.

To make things slightly more complicated, I decided to implement the air gap solution, since my K8s nodes did not have access to the internet. Thus, I needed to pull down all of the images required for Sonobuoy, and then push them to my internal Harbor repository. Once that step was completed, I then built manifest files which would direct Sonobuoy to pull the necessary images from my Harbor repo and run the test suite. Fortunately Sonobuoy includes a set of steps to make that whole setup fairly seamless. This set of steps details how I deployed and ran Sonobuoy in an air-gap configuration.

1. Initial install

Simply download Sonobuoy, or deploy it by running the go get command outlined here.

2. Pull the test images from external repos

I estimate that you will require around 8GB of disk space to pull down all of the necessary test suite images. The following command will pull down the images for you.

$ sudo sonobuoy images pull
INFO[0000] Pulling image: gcr.io/google-samples/gb-redisslave:v3 ...
INFO[0005] Pulling image: gcr.io/kubernetes-e2e-test-images/ipc-utils:1.0 ...
INFO[0008] Pulling image: gcr.io/kubernetes-e2e-test-images/fakegitserver:1.0 ...
INFO[0010] Pulling image: gcr.io/kubernetes-e2e-test-images/echoserver:2.2 ...
INFO[0014] Pulling image: gcr.io/kubernetes-e2e-test-images/jessie-dnsutils:1.0 ...
.
.
.
INFO[0243] Pulling image: gcr.io/kubernetes-e2e-test-images/liveness:1.1 ...
INFO[0245] Pulling image: docker.io/library/nginx:1.15-alpine ...
INFO[0248] Pulling image: gcr.io/kubernetes-e2e-test-images/resource-consumer/controller:1.0 ...
INFO[0250] Pulling image: docker.io/library/busybox:1.29 ...

3. Create ‘mapping’ file for test images repo location

In order to push the test images to a local or internal repo such as Harbor, we need to create a manifest YAML file which will be used to indiciate where Sunobuoy can find the test images. I called the file custom-repos.yaml.

$ cat custom-repos.yaml
dockerLibraryRegistry: harbor.rainpole.com/library
e2eRegistry: harbor.rainpole.com/library/kubernetes-e2e-test-images
gcRegistry: harbor.rainpole.com/library
etcdRegistry: harbor.rainpole.com/library/coreos
privateRegistry: harbor.rainpole.com/library/k8s-authenticated-test
sampleRegistry: harbor.rainpole.com/library/google-samples

4. Push the test images to an internal repo

We will use the same manifest file to push the images to our local Harbor repo.

$ sudo sonobuoy images push --e2e-repo-config custom-repos.yaml
INFO[0000] Tagging image: gcr.io/kubernetes-e2e-test-images/cuda-vector-add:2.0 as harbor.rainpole.com/library/kubernetes-e2e-test-images/cuda-vector-add:2.0 ...
INFO[0000] Pushing image: harbor.rainpole.com/library/kubernetes-e2e-test-images/cuda-vector-add:2.0 ...
INFO[0000] Tagging image: gcr.io/kubernetes-e2e-test-images/netexec:1.1 as harbor.rainpole.com/library/kubernetes-e2e-test-images/netexec:1.1 ...
.
.
.
INFO[0219] Tagging image: gcr.io/kubernetes-e2e-test-images/nettest:1.0 as harbor.rainpole.com/library/kubernetes-e2e-test-images/nettest:1.0 ...
INFO[0219] Pushing image: harbor.rainpole.com/library/kubernetes-e2e-test-images/nettest:1.0 ...
$

At this point, all of the necessary test suite images are held locally in my Harbor repo.

5. Build the Sonobuoy manifest files

At this point, we need to build a bespoke set of Sonobuoy manifest files which will contain references to our local Harbor repo. This will redirect Sonobuoy to use local images rather than pulling them from external repos.

$ sonobuoy gen --e2e-repo-config custom-repos.yaml > config.yaml

$ ls -ltr
total 28
-rw-rw-r-- 1 cormac cormac  335 Jul 23 10:52 custom-repos.yaml
-rw-rw-r-- 1 cormac cormac 5917 Jul 23 12:19 config.yaml

And on examination, you will notice that the config.yaml contains a special ConfigMap kind that has the list of repositories matching what we created earlier, i.e local Harbor repo:

---
apiVersion: v1
data:
  repo-list.yaml: |
    dockerLibraryRegistry: harbor.rainpole.com/library
    e2eRegistry: harbor.rainpole.com/library/kubernetes-e2e-test-images
    gcRegistry: harbor.rainpole.com/library
    etcdRegistry: harbor.rainpole.com/library/coreos
    privateRegistry: harbor.rainpole.com/library/k8s-authenticated-test
    sampleRegistry: harbor.rainpole.com/library/google-samples
kind: ConfigMap
metadata:
  name: repolist-cm
  namespace: heptio-sonobuoy
---

6. Placing the Sonobuoy components into the local repo

At this point, all of the test suite components that Sonobuoy needs to reference are in the local Harbor repo, so there is no need to go looking for those externally. However, the main Sonobuoy components (Sonobuoy, conformance and systemd-logs plugin) are still fetching their images from an external repo in the newly created config.yaml manifest.

 image: gcr.io/google-containers/conformance:v1.14.3
 image: gcr.io/heptio-images/sonobuoy-plugin-systemd-logs:latest
 image: gcr.io/heptio-images/sonobuoy:v0.15.0

You will now need to pull these images manually, tag them with the local Harbor repo, and push them to the local Harbor repo. For an example on how to do this, please take a look at this Harbor in action post. Finally, modify the manifest to ensure that these images are sourced from the local Harbor repo. Once this has been completed, everything should now be sourced from the local Harbor repo, and your config.yaml image references should look something like this.

 image: harbor.rainpole.com/library/conformance:v1.14.3
 image: harbor.rainpole.com/library/sonobuoy-plugin-systemd-logs:latest
 image: harbor.rainpole.com/library/sonobuoy:v0.15.0

7. Verify supported versions

According to the documentation, Sonobuoy supports 3 Kubernetes minor versions: the current release and 2 minor versions before. It is worth ensuring that you have a supported setup before going any further. First, check the K8s versions as follows:

$ kubectl get nodes -o wide
NAME            STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
csi-master-01   Ready    master   39d   v1.14.3   10.27.51.49   10.27.51.49   Ubuntu 16.04.6 LTS   4.4.0-150-generic   docker://18.6.2
csi-worker-02   Ready    <none>   39d   v1.14.3   10.27.51.61   10.27.51.61   Ubuntu 16.04.6 LTS   4.4.0-150-generic   docker://18.6.2

You can also use Sonobuoy to check versions. If you include the path to the kubeconfig file, it will also check the API version. Looks like we are good to go with our versions.

$ sonobuoy version --kubeconfig ~/.kube/config
Sonobuoy Version: v0.15.0
MinimumKubeVersion: 1.13.0
MaximumKubeVersion: 1.15.99
GitSHA:
API Version:  v1.14.3

 
8. Begin testing

With everything in place, we can now begin the testing. However, since we have had to make changes to the configurations to support the air gap/local repo mechanism, we cannot use the sonobuoy command directly. Instead, we have to use kubectl apply on the manifest built and modified earlier to start the testing.

$ kubectl apply -f config.yaml
namespace/heptio-sonobuoy created
serviceaccount/sonobuoy-serviceaccount created
clusterrolebinding.rbac.authorization.k8s.io/sonobuoy-serviceaccount-heptio-sonobuoy created
clusterrole.rbac.authorization.k8s.io/sonobuoy-serviceaccount created
configmap/sonobuoy-config-cm created
configmap/sonobuoy-plugins-cm created
pod/sonobuoy created
configmap/repolist-cm created
service/sonobuoy-master created

 
9. Monitor tests

By default, two sets of tests are run; the systemd-logs tests and the e2e tests. You can check their progress with the following status command:

$ sonobuoy status
PLUGIN          STATUS  COUNT
e2e             running 1
systemd-logs    running 2

Sonobuoy is still running. Runs can take up to 60 minutes.

$ sonobuoy status
PLUGIN          STATUS          COUNT
e2e             running         1
systemd-logs    complete        2

Sonobuoy is still running. Runs can take up to 60 minutes.

The other useful option for monitoring progress is the ‘logs‘ option. This provides very verbose output, and can be followed if the -f option is used.Here is the output from the very beginning of the tests, showing how many tests will be run:

$ sonobuoy logs -f
namespace="heptio-sonobuoy" pod="sonobuoy-systemd-logs-daemon-set-02274f1712d5490e-bnmkv" container="sonobuoy-worker"
time="2019-07-23T16:17:20Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2019-07-23T16:17:21Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/systemd_logs
namespace="heptio-sonobuoy" pod="sonobuoy-systemd-logs-daemon-set-02274f1712d5490e-lg2cc" container="sonobuoy-worker"
time="2019-07-23T16:17:22Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2019-07-23T16:17:23Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/systemd_logs
namespace="heptio-sonobuoy" pod="sonobuoy-e2e-job-53542f69c61f4dae" container="sonobuoy-worker"
time="2019-07-23T16:17:23Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
namespace="heptio-sonobuoy" pod="sonobuoy" container="kube-sonobuoy"
time="2019-07-23T16:17:21Z" level=info msg="Scanning plugins in ./plugins.d (pwd: /)"
time="2019-07-23T16:17:21Z" level=info msg="Scanning plugins in /etc/sonobuoy/plugins.d (pwd: /)"
time="2019-07-23T16:17:21Z" level=info msg="Directory (/etc/sonobuoy/plugins.d) does not exist"
time="2019-07-23T16:17:21Z" level=info msg="Scanning plugins in ~/sonobuoy/plugins.d (pwd: /)"
time="2019-07-23T16:17:21Z" level=info msg="Directory (~/sonobuoy/plugins.d) does not exist"
time="2019-07-23T16:17:21Z" level=info msg="Filtering namespaces based on the following regex:.*|heptio-sonobuoy"
time="2019-07-23T16:17:21Z" level=info msg="Namespace csi-demo Matched=true"
time="2019-07-23T16:17:21Z" level=info msg="Namespace default Matched=true"
time="2019-07-23T16:17:21Z" level=info msg="Namespace demo Matched=true"
time="2019-07-23T16:17:21Z" level=info msg="Namespace heptio-sonobuoy Matched=true"
time="2019-07-23T16:17:21Z" level=info msg="Namespace kube-node-lease Matched=true"
time="2019-07-23T16:17:21Z" level=info msg="Namespace kube-public Matched=true"
time="2019-07-23T16:17:21Z" level=info msg="Namespace kube-system Matched=true"
time="2019-07-23T16:17:21Z" level=info msg="Namespace svc-demo Matched=true"
time="2019-07-23T16:17:21Z" level=info msg="Namespace velero Matched=true"
time="2019-07-23T16:17:21Z" level=info msg="Starting server Expected Results: [{ e2e} {csi-master-01 systemd-logs} {csi-worker-02 systemd-logs}]"
time="2019-07-23T16:17:21Z" level=info msg="Starting annotation update routine"
time="2019-07-23T16:17:21Z" level=info msg="Starting aggregation server" address=0.0.0.0 port=8080
time="2019-07-23T16:17:21Z" level=info msg="Running plugin" plugin=e2e
time="2019-07-23T16:17:21Z" level=info msg="Running plugin" plugin=systemd-logs
time="2019-07-23T16:17:23Z" level=info msg="received aggregator request" client_cert=systemd-logs node=csi-worker-02 plugin_name=systemd-logs
time="2019-07-23T16:17:24Z" level=info msg="received aggregator request" client_cert=systemd-logs node=csi-master-01 plugin_name=systemd-logs
namespace="heptio-sonobuoy" pod="sonobuoy-e2e-job-53542f69c61f4dae" container="e2e"
+ set +x
+ /usr/local/bin/ginkgo '--focus=\[Conformance\]' '--skip=Alpha|\[(Disruptive|Feature:[^\]]+|Flaky)\]' --noColor=true /usr/local/bin/e2e.test -- --disable-log-dump --repo-root=/kubernetes --provider=local --report-dir=/tmp/results --kubeconfig=
+ tee /tmp/results/e2e.log
I0723 16:17:23.756655      15 test_context.go:405] Using a temporary kubeconfig file from in-cluster config : /tmp/kubeconfig-709992315
I0723 16:17:23.756820      15 e2e.go:240] Starting e2e run "5a22a21f-ad65-11e9-bb39-9288ee65ecf1" on Ginkgo node 1
Running Suite: Kubernetes e2e suite
===================================
Random Seed: 1563898642 - Will randomize all specs
Will run 204 of 3585 specs

Here is an output from the very end of a suite of tests, showing how many tests were run and which ones passed and which ones (if any) failed.

$ sonobuoy logs -f
• [SLOW TEST:8.240 seconds]
[sig-storage] Projected downwardAPI
/workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/common/projected_downwardapi.go:33
  should provide container's cpu limit [NodeConformance] [Conformance]
  /workspace/anago-v1.14.3-beta.0.37+5e53fd6bc17c0d/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:692
------------------------------
SSSSSSSSSSSJul 23 17:40:44.454: INFO: Running AfterSuite actions on all nodes
Jul 23 17:40:44.454: INFO: Running AfterSuite actions on node 1
Jul 23 17:40:44.454: INFO: Skipping dumping logs from cluster

Ran 204 of 3585 Specs in 5000.512 seconds
SUCCESS! -- 204 Passed | 0 Failed | 0 Pending | 3381 Skipped PASS

Ginkgo ran 1 suite in 1h23m21.836284415s
Test Suite Passed

namespace="heptio-sonobuoy" pod="sonobuoy-e2e-job-53542f69c61f4dae" container="sonobuoy-worker"
time="2019-07-23T17:40:45Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/e2e.tar.gz
namespace="heptio-sonobuoy" pod="sonobuoy" container="kube-sonobuoy"
time="2019-07-23T17:40:45Z" level=info msg="received aggregator request" client_cert=e2e plugin_name=e2e
time="2019-07-23T17:40:45Z" level=info msg="Last update to annotations on exit"
time="2019-07-23T17:40:45Z" level=info msg="csidrivers not specified in non-nil Resources. Skipping csidrivers query."
time="2019-07-23T17:40:45Z" level=info msg="csinodes not specified in non-nil Resources. Skipping csinodes query."
time="2019-07-23T17:40:45Z" level=info msg="runtimeclasses not specified in non-nil Resources. Skipping runtimeclasses query."
time="2019-07-23T17:40:45Z" level=info msg="secrets not specified in non-nil Resources. Skipping secrets query."
time="2019-07-23T17:40:45Z" level=info msg="events not specified in non-nil Resources. Skipping events query."
time="2019-07-23T17:40:45Z" level=info msg="horizontalpodautoscalers not specified in non-nil Resources. Skipping horizontalpodautoscalers query."
time="2019-07-23T17:40:45Z" level=info msg="resticrepositories not specified in non-nil Resources. Skipping resticrepositories query."
time="2019-07-23T17:40:45Z" level=info msg="backupstoragelocations not specified in non-nil Resources. Skipping backupstoragelocations query."
time="2019-07-23T17:40:45Z" level=info msg="backups not specified in non-nil Resources. Skipping backups query."
time="2019-07-23T17:40:45Z" level=info msg="downloadrequests not specified in non-nil Resources. Skipping downloadrequests query."
time="2019-07-23T17:40:45Z" level=info msg="podvolumerestores not specified in non-nil Resources. Skipping podvolumerestores query."
time="2019-07-23T17:40:45Z" level=info msg="restores not specified in non-nil Resources. Skipping restores query."
time="2019-07-23T17:40:45Z" level=info msg="deletebackuprequests not specified in non-nil Resources. Skipping deletebackuprequests query."
time="2019-07-23T17:40:45Z" level=info msg="podvolumebackups not specified in non-nil Resources. Skipping podvolumebackups query."
time="2019-07-23T17:40:45Z" level=info msg="serverstatusrequests not specified in non-nil Resources. Skipping serverstatusrequests query."
time="2019-07-23T17:40:45Z" level=info msg="volumesnapshotlocations not specified in non-nil Resources. Skipping volumesnapshotlocations query."
time="2019-07-23T17:40:45Z" level=info msg="schedules not specified in non-nil Resources. Skipping schedules query."
time="2019-07-23T17:40:45Z" level=info msg="Collecting Node Configuration and Health..."
time="2019-07-23T17:40:45Z" level=info msg="Creating host results for csi-master-01 under /tmp/sonobuoy/17fc51fa-6ba7-4f14-bf9d-47dc22e702bc/hosts/csi-master-01\n"
time="2019-07-23T17:40:45Z" level=info msg="Creating host results for csi-worker-02 under /tmp/sonobuoy/17fc51fa-6ba7-4f14-bf9d-47dc22e702bc/hosts/csi-worker-02\n"
time="2019-07-23T17:40:45Z" level=info msg="Running cluster queries"
time="2019-07-23T17:40:45Z" level=info msg="Running ns query (csi-demo)"
time="2019-07-23T17:40:45Z" level=info msg="Collecting Pod Logs (csi-demo)"
time="2019-07-23T17:40:45Z" level=info msg="Running ns query (default)"
time="2019-07-23T17:40:45Z" level=info msg="Collecting Pod Logs (default)"
time="2019-07-23T17:40:45Z" level=info msg="Running ns query (demo)"
time="2019-07-23T17:40:46Z" level=info msg="Collecting Pod Logs (demo)"
time="2019-07-23T17:40:46Z" level=info msg="Running ns query (heptio-sonobuoy)"
time="2019-07-23T17:40:47Z" level=info msg="Collecting Pod Logs (heptio-sonobuoy)"
time="2019-07-23T17:40:47Z" level=info msg="Running ns query (kube-node-lease)"
time="2019-07-23T17:40:47Z" level=info msg="Collecting Pod Logs (kube-node-lease)"
time="2019-07-23T17:40:47Z" level=info msg="Running ns query (kube-public)"
time="2019-07-23T17:40:48Z" level=info msg="Collecting Pod Logs (kube-public)"
time="2019-07-23T17:40:48Z" level=info msg="Running ns query (kube-system)"
time="2019-07-23T17:40:49Z" level=info msg="Collecting Pod Logs (kube-system)"
time="2019-07-23T17:40:57Z" level=info msg="Running ns query (svc-demo)"
time="2019-07-23T17:40:57Z" level=info msg="Collecting Pod Logs (svc-demo)"
time="2019-07-23T17:40:57Z" level=info msg="Running ns query (velero)"
time="2019-07-23T17:40:57Z" level=info msg="Collecting Pod Logs (velero)"
time="2019-07-23T17:40:59Z" level=info msg="Results available at /tmp/sonobuoy/201907231617_sonobuoy_17fc51fa-6ba7-4f14-bf9d-47dc22e702bc.tar.gz"
time="2019-07-23T17:40:59Z" level=info msg="no-exit was specified, sonobuoy is now blocking"

 
10. Querying Results

After completing the test suite, the results can also be queried using a variety of commands. First, verify that every thing has completed using the status command.

$ sonobuoy status
PLUGIN          STATUS          COUNT
e2e             complete        1
systemd-logs    complete        2

Sonobuoy has completed. Use `sonobuoy retrieve` to get results.

As the previous command suggested, we can now retrieve the results of the run, and examine them for failures. Here is a way to do just that.

$ results=$(sonobuoy retrieve)
$ sonobuoy e2e $results
failed tests: 0

 
11. Clean Up

To remove Sonobuoy from your cluster, simply run the following command:

$ sonobuoy delete --wait
INFO[0000] deleted kind=namespace namespace=heptio-sonobuoy
INFO[0000] deleted kind=clusterrolebindings
INFO[0000] deleted kind=clusterroles
$

And that completes our introduction to Sonobuoy. Hopefully you can see how useful this product can be when wishing to verify the integrity of your Kubernetes cluster. Read more about Sonobuoy here. One other nice feature is the ability to write your own custom plugins to extend the test suite. This could be very useful if you wanted to validate some bespoke applications or features, alongside the end-to-end conformance that Sonobuoy already tests.

The post Validating Kubernetes cluster conformance with Sonobuoy appeared first on CormacHogan.com.

Regular readers will be aware that I have been looking at various tools that will allow for the management and monitoring of Kubernetes running on vSphere. In the past, we’ve looked at the vRealize Operations Management Pack for Container Monitoring and vRealize Network Insight for Kubernetes. One of the other VMware products that I really wanted to try out is Wavefront. Wavefront is pretty neat as it has around 200+ pre-built integrations and dashboards. This makes it extremely easy to ingest and visualize performance data. My main issue with getting this up and running is that my Kubernetes cluster (running on PKS, Pivotal Container Service) was isolated from the internet (which I suspect is the same for many customers). Whilst Wavefront provide a Proxy Pod that can get the various K8s metrics out of your K8s cluster, I have another level of indirection where these metrics need to be forwarded to another Proxy (this time in a virtual machine). This VM has access to the internet, and thus can reach my Wavefront cluster URL at https://vmware.wavefront.com. It might be easier to visualize, so I put this diagram together to show the constituent parts.

The following are the steps that I followed in order to get metrics sent from my K8s cluster to my Wavefront cluster.

1. Setup VM (Linux) as a Proxy

Wavefront already provide the steps to do the various integrations necessary to setup a Linux VM as a proxy, as well as for the Linux VM to send metrics (via a Telegraf agent) to the Proxy, and thus back to your Wavefront cluster. If you do not have an existing Wavefront cluster, you can create one with a 30 day free trial. When you login to your Wavefront cluster portal, navigate to the integrations and select the Linux Host integration, then the Setup steps. My Linux VM is running Ubuntu 17.10 (Artful Aardvark). Thus, the Wavefront integration steps provided me  exactly what I need to do to install the Proxy. This involves pulling down the appropriate wavefront proxy package for my distro, and the steps to install the proxy. Finally I was given the command to install the Telegraf agent. Once completed, the configuration file for the Wavefront proxy is located in /etc/wavefront/wavefront-proxy/wavefront.conf and the configuration file for the Telegraf agent is located in /etc/telegraf/telegraf.conf. The next step is to edit the wavefront configuration file, and set 3 parameters, namely server, hostname and token. The server is your Wavefront cluster, the hostname is used to identify stats from this proxy, and the token is an API token for your account. Details on how to retrieve the token are shown in the comments below, as well on the Linux Host integration setup steps.

##############################################################################
# Wavefront proxy configuration file
#
#   Typically in /etc/wavefront/wavefront-proxy/wavefront.conf
#
##############################################################################
# The server should be either the primary Wavefront cloud server, or your custom VPC address.
#   This will be provided to you by Wavefront.
#
server=https://vmware.wavefront.com/api

# The hostname will be used to identify the internal proxy statistics around point rates, JVM info, etc.
#  We strongly recommend setting this to a name that is unique among your entire infrastructure,
#   possibly including the datacenter information, etc. This hostname does not need to correspond to
#   any actual hostname or DNS entry; it's merely a string that we pass with the internal stats.
#
hostname=pks-cli.rainpole.com

# The Token is any valid API Token for your account, which can be generated from the gear icon
#   at the top right of the Wavefront site, under 'Settings'. Paste that hexadecimal token
#   after the '=' below, and the proxy will automatically generate a machine-specific UUID and
#   self-register.
#
token=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx

# Comma separated list of ports to listen on for Wavefront formatted data
pushListenerPorts=2878

#
##############################################################################

While there are a lot of other parameters, these are the only ones you need to concentrate on for the moment. Restart the Wavefront proxy (sudo /etc/init.d/wavefront-proxy restart). That now completes the Proxy VM setup, and at this point the proxy should be visible in the list of active proxies on your Wavefront cluster.

At this point, the Proxy VM has been setup and is sending metrics to the Wavefront cluster. you can look at some of the Linux VM dashboards to verify. The next step is to deploy some components on the K8s cluster, and have it send relevant K8s metrics to this Proxy VM from a Proxy Pod. Then we should have these metrics displayed in Wavefront. Let’s do that next.

2. Deploy and Configure Proxy POD

To get the steps to deploy a wavefront proxy for Kubernetes, select the Kubernetes integration in the Wavefront UI, and once again select the setup steps. Let’s focus on step 1, which is the deployment of the Proxy. In the manifest/YAML file for the Wavefront Proxy Pod, there are a number of environment variables, such as WAVEFRONT_URL, WAVEFRONT_TOKEN and WAVEFRONT_PROXY_ARGS. I had a hard time initially when trying to configure these, especially the WAVEFRONT_PROXY_ARGS where I had read that the value of this environment variable should contain the –proxyHost which pointed back to my Proxy VM (Linux VM). However, this resulted in a lot of error spew in the logs, notably around “[agent:checkin] configuration file read from server is invalid“.  It was hard to tell if this was working or not, so I reached out to Vasily Vorontsov, one of our WaveFront engineers, who recommended that I go with a different approach. He recommended that in the Proxy Pod YAML, I should set the WAVEFRONT_URL to my proxy VM, using port 2879. I could leave the token as is, and that there was no need to set anything in the WAVEFRONT_PROXY_ARGS. This is a snippet of what the environment variable section of my YAML file looked like:

env:
- name: WAVEFRONT_URL
value: http://proxy-vm:2879/api
- name: WAVEFRONT_TOKEN
value: xxxxxxxx-xxxx-xxxx-xxxxxxxxxx

Now, for this to work, I had to make another change to the Proxy VM configuration file. This change was to ad a new entry call pushRelayListenerPort as follows:

# Comma separated list of (proxy) ports to listen on for Wavefront formatted data
pushRelayListenerPorts=2879

And then restart the Proxy service once more. Here are some log snippets from the startup of the Proxy VM and Proxy Pod. Everything seemed much cleaner now:

From Proxy VM logs:

$ tail -f /var/log/wavefront/wavefront.log
2019-08-06 15:27:19,151 INFO  [agent:setupCheckins] scheduling regular check-ins
2019-08-06 15:27:19,152 INFO  [agent:setupCheckins] initial configuration is available, setting up proxy
2019-08-06 15:27:19,392 INFO  [agent:startListeners] listening on port: 2878 for Wavefront metrics
2019-08-06 15:27:19,411 INFO  [agent:startListeners] listening on port: 2003 for graphite metrics
2019-08-06 15:27:19,420 INFO  [agent:startListeners] Not loading logs ingestion -- no config specified.
2019-08-06 15:27:24,424 INFO  [agent:run] setup complete
2019-08-06 15:27:29,163 INFO  [agent:checkin] Checking in: https://vmware.wavefront.com/api
2019-08-06 15:27:29,392 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:29,403 INFO  [AbstractReportableEntityHandler:printStats] [2003] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:29,419 INFO  [AbstractReportableEntityHandler:printStats] [2879] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:39,389 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:39,403 INFO  [AbstractReportableEntityHandler:printStats] [2003] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:39,419 INFO  [AbstractReportableEntityHandler:printStats] [2879] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:49,389 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:49,403 INFO  [AbstractReportableEntityHandler:printStats] [2003] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:49,419 INFO  [AbstractReportableEntityHandler:printStats] [2879] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:59,388 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:59,403 INFO  [AbstractReportableEntityHandler:printStats] [2003] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:27:59,419 INFO  [AbstractReportableEntityHandler:printStats] [2879] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:28:09,388 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:28:09,403 INFO  [AbstractReportableEntityHandler:printStats] [2003] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:28:09,419 INFO  [AbstractReportableEntityHandler:printStats] [2879] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:28:19,389 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 3 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:28:19,389 INFO  [AbstractReportableEntityHandler:printTotal] [2878] Total points processed since start: 232; blocked: 2
2019-08-06 15:28:19,403 INFO  [AbstractReportableEntityHandler:printStats] [2003] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:28:19,403 INFO  [AbstractReportableEntityHandler:printTotal] [2003] Total points processed since start: 0; blocked: 0
2019-08-06 15:28:19,419 INFO  [AbstractReportableEntityHandler:printStats] [2879] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:28:19,419 INFO  [AbstractReportableEntityHandler:printTotal] [2879] Total points processed since start: 0; blocked: 0
2019-08-06 15:28:29,388 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 4 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:28:29,403 INFO  [AbstractReportableEntityHandler:printStats] [2003] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 15:28:29,419 INFO  [AbstractReportableEntityHandler:printStats] [2879] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).

From K8s Pod proxy logs:

$ kubectl logs wavefront-proxy-79568456c6-z82rh -f

2019-08-06 14:28:56,168 INFO  [agent:start] Starting proxy version 4.38
2019-08-06 14:28:56,182 INFO  [agent:parseArguments] Arguments: -h, http://192.50.0.6:2879/api, -t, <HIDDEN>, --hostname, wavefront-proxy-79568456c6-z82rh,\
 --ephemespool/wavefront-proxy/buffer, --flushThreads, 6, --retryThreads, 6
2019-08-06 14:28:56,251 INFO  [agent:parseArguments] Unparsed arguments: true
2019-08-06 14:28:56,300 WARN  [agent:loadListenerConfigurationFile] Loaded configuration file null
2019-08-06 14:28:56,623 INFO  [agent:start] Ephemeral proxy id created: 55a0682a-e23b-4316-b0f7-aa602dfc331e
2019-08-06 14:28:56,992 INFO  [QueuedAgentService:<init>] No rate limit configured.
2019-08-06 14:28:57,202 INFO  [QueuedAgentService:lambda$new$5] retry queue has been cleared
2019-08-06 14:28:57,205 WARN  [QueuedAgentService:lambda$new$5] source tag retry queue has been cleared
2019-08-06 14:28:57,225 INFO  [agent:checkin] Checking in: http://192.50.0.6:2879/api

2019-08-06 14:28:57,387 INFO  [agent:setupCheckins] scheduling regular check-ins
2019-08-06 14:28:57,389 INFO  [agent:setupCheckins] initial configuration is available, setting up proxy
2019-08-06 14:28:57,474 INFO  [agent:startListeners] listening on port: 2878 for Wavefront metrics
2019-08-06 14:28:57,481 INFO  [agent:startListeners] Not loading logs ingestion -- no config specified.
2019-08-06 14:29:02,491 INFO  [agent:run] setup complete
2019-08-06 14:29:07,398 INFO  [agent:checkin] Checking in: http://192.50.0.6:2879/api
2019-08-06 14:29:07,473 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 14:29:17,470 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 14:29:27,470 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 14:29:37,470 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 14:29:47,470 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 14:29:57,470 INFO  [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 0 pps (1 min), 0 pps (5 min), 0 pps (current).
2019-08-06 14:29:57,470 INFO  [AbstractReportableEntityHandler:printTotal] [2878] Total points processed since start: 0; blocked: 0

This looks like proxy chaining is working correctly. To verify, check the Proxy Pod logs – if the “Total points processed” number is not increasing, then that proxy is not receiving any metrics. If it is increasing then it is working as expected.

It should be noted that if you are running PKS and your Kubernetes nodes had access to the internet, it is quite straight-forward to deploy the Kubernetes Pod proxy, since it would be able to reach the Wavefront cluster directly and there would be no need to chain proxies. This can be done with a simple modification of the PKS tile to add the Wavefront cluster URL and Token. However, there is no way to tell PKS about a chained proxy when the nodes cannot reach the internet, so this is how you have to do it.

You can now complete the Pod Proxy deployment by deploying the kube-state-metrics service and the Wavefront Kubernetes Collector. The kube-state-metrics service listens to the Kubernetes API server and generates metrics about the state of Kubernetes objects. The full set of steps are once again include in the Kubernetes integration steps. The only change is that collector needs to be modified to include the name of your Kubernetes cluster in the collector container deployment:

- --sink=wavefront:?proxyAddress=wavefront-proxy.default.svc.cluster.local:2878\
  &clusterName=cork8s-cluster-01&includeLabels=true

This takes us to our last step, which is to verify that we are actually receiving metrics back on our Wavefront cluster.

3. Verify metrics are being received

Back on the Kubernetes integration, this time we select dashboards. The two dashboards which display information about my Kubernetes cluster are Kube-state Metrics and Kubernetes Collector Metrics. When you select the dashboard, you will be given a list of K8s clusters to choose from. You simply choose your one from the list (assuming metrics are being received). In my case, my cluster is cork8s-cluster-01, and when I select this from the list, here is what I see in Kube-state metrics.

I can drill down further and get even further information about my K8s cluster.

The thing to note is that this is a predefined dashboard from Wavefront. It does have a lot of very useful, low-level information about the behaviour of the cluster, but the nice thing is that you can clone this dashboard, and then modify it so it displays only the metrics that are meaningful to you. And the powerful thing about Wavefront is that it can ingest millions of metric data points per second. This means that you are able to monitor over 100,000 containers in real-time. This is why Wavefront is used by the likes of box, reddit and workday.

That concludes the post. If you do have a Kubernetes cluster that you wish to monitor with Wavefront, but it is not connected to the internet, chaining proxies like I’ve showed you to do in this post should help you work around that situation. Kudos once again to Vasily for his help on this solution.

The post Monitoring Kubernetes with Wavefront via Proxy Chaining appeared first on CormacHogan.com.

I have written about Velero a few times on this blog, but I haven’t actually looked at how you would deploy the 1.0.0 version, even though it has been available since May 2019. Someone recently reached out to me for some guidance on how to deploy it, as there are a few subtle differences between previous versions. Therefore I decided to document step-by-step how to do it, but focusing on when your Kubernetes cluster is running on vSphere. I also highlight a gotcha when using Velero to backup applications that are running on Kubernetes deployed via Enterprise PKS, Pivotal Container Service.

To recap, these are the steps that I will cover in detail:

1. Download and extract Velero 1.0.0
2. Download any required images to local repo if K8s nodes cannot access internet
3. Deploy and Configure local Minio S3 Object Store
4. Install Velero via velero install – command should include restic support and Minio publicUrl
5. Modify hostPath setting in restic DaemonSet for Enterprise PKS
6. [New] Create a ConfigMap for the velero-restic-restore-helper
7. Run a test Velero backup/restore

Let’s look at each of steps now.

1. Download and extract Velero 1.0.0

The image can be found here – https://github.com/heptio/velero/releases/tag/v1.0.0. Download and extract it, then copy or move the velero binary to somewhere in your PATH.

2. Pull any required images and push them to local repo (e.g. Harbor)

As mentioned in the introduction, this step is only necessary if your Kubernetes nodes do not have access internet. This is the case in my lab, so I do a docker pull, docker tag, docker push to my Harbor repo. For Velero, there are 3 images that need to be handled. There are 2 Minio images, which also requires a modification to the 00-minio-deployment manifest. Below are the before and after of the manifest file.

$ grep image examples/minio/00-minio-deployment.yaml
        image: minio/minio:latest
        imagePullPolicy: IfNotPresent
        image: minio/mc:latest
        imagePullPolicy: IfNotPresent

$ grep image examples/minio/00-minio-deployment.yaml
        image: harbor.rainpole.com/library/minio:latest
        imagePullPolicy: IfNotPresent
        image: harbor.rainpole.com/library/mc:latest
        imagePullPolicy: IfNotPresent

The third image is referenced during the install. By default, the image used for the Velero and restic server pods comes from “gcr.io/heptio-images/velero:v1.0.0”. We would also need to pull this image and push it to harbor, and then add a –image argument to the velero install to point to the image in my local Harbor repo, which you will see shortly.

3. Deploy and Configure local Minio Object Store

There are a few different steps required here. We have already modified the deployment YAML previously, but only if our images are in a local repo and the K8s nodes have no access to the internet. If they do, then no modification is needed.

3.1 Create a Minio credentials file

A simple credentials file containing the login/password (id/key) for the local on-premises Minio S3 Object Store must be created.

$ cat credentials-velero
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123

3.2 Expose Minio Service on a NodePort

This step is a good idea for 2 reasons. The first is that it gives you a way to access the Minio portal and examine the contents of any backups. The second is that it enables you to specify a publicUrl for Minio, which in turn means that you can access backup and restore logs from the Minio S3 Object Store. To do this, it  requires a modification to the 00-minio-deployment manifest:

spec:
# ClusterIP is recommended for production environments.
# Change to NodePort if needed per documentation,
# but only if you run Minio in a test/trial environment, for example with Minikube.
type: NodePort

3.3 Deploy Minio

 $ kubectl apply -f examples/minio/00-minio-deployment.yaml

3.4 Verify Minio is available on the public URL

If we now go ahead and retrieve the node on which the Minio server is running, as well as the port that it has been exposed on with the changes made in step 3.2, we should be able to verify that Minio is working.

$ kubectl get pods -n velero
NAME                     READY   STATUS      RESTARTS   AGE
minio-66dc75bb8d-95xpp   1/1     Running     0          25s
minio-setup-zpnfl        0/1     Completed   0          25s

$ kubectl describe pod minio-66dc75bb8d-bczf8 -n velero | grep -i Node:
Node:               140ab5aa-0159-4612-b68c-df39dbea2245/192.168.192.5

$ kubectl get svc -n velero
NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
minio   NodePort   10.100.200.82   <none>        9000:32109/TCP   5s

OK. Everything is now in place to allow us to do our velero install.

4. Install Velero

A big difference in Velero 1.0 is that there is a new velero install command. No more messing around with multiple manifest files that we had in previous versions. Now there are a few things to include in the velero install command. Since there is no vSphere plugin at this time, we rely on a third party plugin called restic. The command line must include and option to use restic. As we also mentioned, we have setup a publicUrl for Minio, so we should also include this in our command line. Finally, because my K8s nodes do not have access to the internet, and thus cannot pull down external images, I have a local Harbor repo where I have already pushed the velero image. You can pull the velero image from gcr.io/heptio-images/velero:v1.0.0. I also need to reference this in the install command. With all those modifications, this is what my install command looks like:

$ velero install  --provider aws --bucket velero \
--secret-file ./credentials-velero \
--use-volume-snapshots=false \
--image harbor.rainpole.com/library/velero:v1.0.0 \
--use-restic \
--backup-location-config \
region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000,publicUrl=http://192.168.192.5:32109

After running the command, the following output is displayed:

CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: already exists, proceeding
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: created
DaemonSet/restic: attempting to create resource
DaemonSet/restic: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

LGTM. I also like the little sailboat in the output (Velero is Spanish for sailboat I believe). Let’s take a look at the logs and make sure everything deployed successfully.

time="2019-08-07T15:02:46Z" level=info msg="setting log-level to INFO"
time="2019-08-07T15:02:46Z" level=info msg="Starting Velero server v1.0.0 (72f5cadc3a865019ab9dc043d4952c9bfd5f2ecb)" logSource="pkg/cmd/server/server.go:165"
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/serviceaccount
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/addPVCFromPod
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/addPVFromPVC
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
time="2019-08-07T15:02:46Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/serviceaccount
time="2019-08-07T15:02:46Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:355" namespace=velero
time="2019-08-07T15:02:46Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:361" namespace=velero
time="2019-08-07T15:02:48Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:390"
time="2019-08-07T15:02:48Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:424"
time="2019-08-07T15:02:48Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:431"
time="2019-08-07T15:02:48Z" level=info msg="Starting controllers" logSource="pkg/cmd/server/server.go:535"
time="2019-08-07T15:02:48Z" level=info msg="Starting metric server at address [:8085]" logSource="pkg/cmd/server/server.go:543"
time="2019-08-07T15:02:48Z" level=info msg="Server started successfully" logSource="pkg/cmd/server/server.go:788"
time="2019-08-07T15:02:48Z" level=info msg="Starting controller" controller=gc-controller logSource="pkg/controller/generic_controller.go:76"
time="2019-08-07T15:02:48Z" level=info msg="Waiting for caches to sync" controller=gc-controller logSource="pkg/controller/generic_controller.go:79"
time="2019-08-07T15:02:48Z" level=info msg="Starting controller" controller=backup-deletion logSource="pkg/controller/generic_controller.go:76"
time="2019-08-07T15:02:48Z" level=info msg="Waiting for caches to sync" controller=backup-deletion logSource="pkg/controller/generic_controller.go:79"
time="2019-08-07T15:02:48Z" level=info msg="Starting controller" controller=downloadrequest logSource="pkg/controller/generic_controller.go:76"
time="2019-08-07T15:02:48Z" level=info msg="Waiting for caches to sync" controller=downloadrequest logSource="pkg/controller/generic_controller.go:79"
time="2019-08-07T15:02:48Z" level=info msg="Starting controller" controller=serverstatusrequest logSource="pkg/controller/generic_controller.go:76"
time="2019-08-07T15:02:48Z" level=info msg="Waiting for caches to sync" controller=serverstatusrequest logSource="pkg/controller/generic_controller.go:79"
time="2019-08-07T15:02:48Z" level=info msg="Starting controller" controller=backup-sync logSource="pkg/controller/generic_controller.go:76"
time="2019-08-07T15:02:48Z" level=info msg="Waiting for caches to sync" controller=backup-sync logSource="pkg/controller/generic_controller.go:79"
time="2019-08-07T15:02:48Z" level=info msg="Starting controller" controller=schedule logSource="pkg/controller/generic_controller.go:76"
time="2019-08-07T15:02:48Z" level=info msg="Waiting for caches to sync" controller=schedule logSource="pkg/controller/generic_controller.go:79"
time="2019-08-07T15:02:48Z" level=info msg="Starting controller" controller=restore logSource="pkg/controller/generic_controller.go:76"
time="2019-08-07T15:02:48Z" level=info msg="Waiting for caches to sync" controller=restore logSource="pkg/controller/generic_controller.go:79"
time="2019-08-07T15:02:48Z" level=info msg="Starting controller" controller=restic-repository logSource="pkg/controller/generic_controller.go:76"
time="2019-08-07T15:02:48Z" level=info msg="Waiting for caches to sync" controller=restic-repository logSource="pkg/controller/generic_controller.go:79"
time="2019-08-07T15:02:48Z" level=info msg="Starting controller" controller=backup logSource="pkg/controller/generic_controller.go:76"
time="2019-08-07T15:02:48Z" level=info msg="Waiting for caches to sync" controller=backup logSource="pkg/controller/generic_controller.go:79"
time="2019-08-07T15:02:48Z" level=info msg="Caches are synced" controller=schedule logSource="pkg/controller/generic_controller.go:83"
time="2019-08-07T15:02:49Z" level=info msg="Caches are synced" controller=serverstatusrequest logSource="pkg/controller/generic_controller.go:83"
time="2019-08-07T15:02:49Z" level=info msg="Caches are synced" controller=gc-controller logSource="pkg/controller/generic_controller.go:83"
time="2019-08-07T15:02:49Z" level=info msg="Caches are synced" controller=downloadrequest logSource="pkg/controller/generic_controller.go:83"
time="2019-08-07T15:02:49Z" level=info msg="Caches are synced" controller=backup-sync logSource="pkg/controller/generic_controller.go:83"
time="2019-08-07T15:02:49Z" level=info msg="Caches are synced" controller=backup logSource="pkg/controller/generic_controller.go:83"
time="2019-08-07T15:02:49Z" level=info msg="Caches are synced" controller=restore logSource="pkg/controller/generic_controller.go:83"
time="2019-08-07T15:02:49Z" level=info msg="Caches are synced" controller=restic-repository logSource="pkg/controller/generic_controller.go:83"
time="2019-08-07T15:02:49Z" level=info msg="Syncing contents of backup store into cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:170"
time="2019-08-07T15:02:49Z" level=info msg="Got backups from backup store" backupCount=0 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:178"
time="2019-08-07T15:02:49Z" level=info msg="Caches are synced" controller=backup-deletion logSource="pkg/controller/generic_controller.go:83"
time="2019-08-07T15:02:49Z" level=info msg="Checking for expired DeleteBackupRequests" controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:441"
time="2019-08-07T15:02:49Z" level=info msg="Done checking for expired DeleteBackupRequests" controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:469"
time="2019-08-07T15:03:49Z" level=info msg="Syncing contents of backup store into cluster" backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:170"
time="2019-08-07T15:03:49Z" level=info msg="Got backups from backup store" backupCount=0 backupLocation=default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:178"

Again, this LGTM. There are no errors in the logs. Looks like we are almost ready to take a backup.

5. Modify hostPath in restic DaemonSet for Enterprise PKS

This step is only necessary for Enterprise PKS deployments, the Pivotal Container Service. This is because the path to the Pods on the Nodes in a PKS deployment is different to what we have in native Kubernetes deployments. If you’ve deployed this on PKS, and you query the status of the Pods in the Velero namespace, you will will notice that the restic Pod have a RunContainerError/CrashLoopBackOff error. Typically the path to Pods on native K8s is /var/lib/kubelet/pods, but on PKS, they are located in /var/vcap/data/kubelet/pods. So this step is to point restic to the correct location of Pods for backup purposes, when K8s is deployed by PKS. First, identify the restic DaemonSet.

$ kubectl get ds --all-namespaces
NAMESPACE     NAME             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   vrops-cadvisor   3         3         3       3            3           <none>          5d3h
pks-system    fluent-bit       3         3         3       3            3           <none>          5d3h
pks-system    telegraf         3         3         3       3            3           <none>          5d3h
velero        restic           3         3         0       3            0           <none>          2m21s

Next, edit the DaemonSet and change the hostPath. The before and after edits are shown below.

$ kubectl edit ds restic -n velero

      volumes:
      - hostPath:
          path: /var/lib/kubelet/pods
          type: ""
        name: host-pods

      volumes:
      - hostPath:
          path: /var/vcap/data/kubelet/pods
          type: ""
        name: host-pods

daemonset.extensions/restic edited

This will terminate and restart the restic Pods. At this point, the velero and restic Pods should all be running. Now we are ready to do a test backup/restore.

6. Create a ConfigMap for the velero-restic-restore-helper

This was a step that I missed on the first version of this post. During a restore of Pods with Persistent Volumes that have been backed up with restic, a temporary pod is instantiated to assist with the restore. This image is pulled from “gcr.io/heptio-images/velero-restic-restore-helper:v1.0.0” by default. Since my nodes dod not have access to the internet, I need to tell velero to get this image from my local repo. This is achieved by creating a ConfigMap with the image location, as per the Customize Restore Helper Image instructions found here. After the usual docker pull/tag/push to get the image into my local Harbor repo, I created and applied the following map with the image location at the end:

$ cat restic-config-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  # any name can be used; Velero uses the labels (below)
  # to identify it rather than the name
  name: restic-restore-action-config
  # must be in the velero namespace
  namespace: velero
  # the below labels should be used verbatim in your
  # ConfigMap.
  labels:
    # this value-less label identifies the ConfigMap as
    # config for a plugin (i.e. the built-in restic restore
    # item action plugin)
    velero.io/plugin-config: ""
    # this label identifies the name and kind of plugin
    # that this ConfigMap is for.
    velero.io/restic: RestoreItemAction
data:
  # "image" is the only configurable key. The value can either
  # include a tag or not; if the tag is *not* included, the
  # tag from the main Velero image will automatically be used.
  image: harbor.rainpole.com/library/velero-restic-restore-helper:v1.0.0

$ kubectl apply -f restic-config-map.yaml
configmap/restic-restore-action-config created

This means for any restore that involved restic volumes, the helper can now be successfully pulled. you can now go ahead and check the velero versions of the client and server using the velero version command.

7. Run a test Velero backup/restore

Velero provide a sample nginx application for backup testing. However this once again relies on pulling an nginx image from the internet. If, like me, you are using a local repo, then you will have to do another pull, tag, push and update the sample manifest for the nginx app to get its image from the local repo, e.g.

$ grep image examples/nginx-app/base.yaml
- image: harbor.rainpole.com/library/nginx:1.15-alpine

Again, this is only necessary if your nodes to do not have internet access. With that modification in place, you can go ahead and deploy the sample nginx app so we can try to backup and restore it with Velero.

7.1 Deploy sample nginx app

$ kubectl apply -f examples/nginx-app/base.yaml
namespace/nginx-example created
deployment.apps/nginx-deployment created
service/my-nginx created

$ kubectl get pods --all-namespaces
NAMESPACE             NAME                                     READY   STATUS      RESTARTS   AGE
cassandra             cassandra-0                              1/1     Running     0          23h
cassandra             cassandra-1                              1/1     Running     0          23h
cassandra             cassandra-2                              1/1     Running     0          23h
default               wavefront-proxy-79568456c6-z82rh         1/1     Running     0          24h
kube-system           coredns-54586579f6-f7knj                 1/1     Running     0          5d3h
kube-system           coredns-54586579f6-t5r5h                 1/1     Running     0          5d3h
kube-system           coredns-54586579f6-v2cjt                 1/1     Running     0          5d3h
kube-system           kube-state-metrics-86977fd78d-6tb5m      2/2     Running     0          24h
kube-system           kubernetes-dashboard-6c68548bc9-km8dd    1/1     Running     0          5d3h
kube-system           metrics-server-5475446b7f-m2fgx          1/1     Running     0          5d3h
kube-system           vrops-cadvisor-488p8                     1/1     Running     0          5d3h
kube-system           vrops-cadvisor-cdx5w                     1/1     Running     0          5d3h
kube-system           vrops-cadvisor-wgkkl                     1/1     Running     0          5d3h
nginx-example         nginx-deployment-5f8798768c-5jdkn        1/1     Running     0          8s
nginx-example         nginx-deployment-5f8798768c-lrsw6        1/1     Running     0          8s
pks-system            cert-generator-v0.19.4-qh6kg             0/1     Completed   0          5d3h
pks-system            event-controller-5dbd8f48cc-vwpc4        2/2     Running     546        5d3h
pks-system            fluent-bit-7cx69                         3/3     Running     0          5d3h
pks-system            fluent-bit-fpbl6                         3/3     Running     0          5d3h
pks-system            fluent-bit-j674j                         3/3     Running     0          5d3h
pks-system            metric-controller-5bf6cb67c6-bbh6q       1/1     Running     0          5d3h
pks-system            observability-manager-5578bbb84f-w87bj   1/1     Running     0          5d3h
pks-system            sink-controller-54947f5bd9-42spw         1/1     Running     0          5d3h
pks-system            telegraf-4gv8b                           1/1     Running     0          5d3h
pks-system            telegraf-dtcjc                           1/1     Running     0          5d3h
pks-system            telegraf-m2pjd                           1/1     Running     0          5d3h
pks-system            telemetry-agent-776d45f8d8-c2xhg         1/1     Running     0          5d3h
pks-system            validator-76fff49f5d-m5t4h               1/1     Running     0          5d3h
velero                minio-66dc75bb8d-95xpp                   1/1     Running     0          11m
velero                minio-setup-zpnfl                        0/1     Completed   0          11m
velero                restic-7mztz                             1/1     Running     0          3m28s
velero                restic-cxfpt                             1/1     Running     0          3m28s
velero                restic-qx98s                             1/1     Running     0          3m28s
velero                velero-7d97d7ff65-drl5c                  1/1     Running     0          9m35s
wavefront-collector   wavefront-collector-76f7c9fb86-d9pw8     1/1     Running     0          24h


$ kubectl get ns
NAME                  STATUS   AGE
cassandra             Active   23h
default               Active   5d3h
kube-public           Active   5d3h
kube-system           Active   5d3h
nginx-example         Active   4s
pks-system            Active   5d3h
velero                Active   9m40s
wavefront-collector   Active   24h


$ kubectl get deployments --namespace=nginx-example
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   2/2     2            2           20s


$ kubectl get svc --namespace=nginx-example
NAME       TYPE           CLUSTER-IP       EXTERNAL-IP                 PORT(S)        AGE
my-nginx   LoadBalancer   10.100.200.147   100.64.0.1,192.168.191.70   80:30942/TCP   32s

This nginx deployment assumes the presence of a LoadBalancer for its Service. Fortunately I do have NSX-T deployed, which provides IP addresses for LoadBalancer services. In the output above, the external IP allocated for the nginx service is 192.168.191.70. If I point a browser to that IP address, I get an nginx landing page.

7.2 First backup

$ velero backup create nginx-backup --selector app=nginx
Backup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.

$ velero backup get
NAME           STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
nginx-backup   Completed   2019-08-07 16:13:44 +0100 IST   29d       default            app=nginx

This backup should also be visible in Minio:

7.3 Destroy nginx deployment

Let’s now go ahead and remove the nginx namespace, and then do a restore of our backup. Hopefully our web server will come back afterwards.

$ kubectl get ns
NAME                  STATUS   AGE
cassandra             Active   40h
default               Active   5d20h
kube-public           Active   5d20h
kube-system           Active   5d20h
nginx-example         Active   17h
pks-system            Active   5d20h
velero                Active   17h
wavefront-collector   Active   41h

$ kubectl delete ns nginx-example
namespace "nginx-example" deleted

$ kubectl get ns
NAME                  STATUS   AGE
cassandra             Active   40h
default               Active   5d20h
kube-public           Active   5d20h
kube-system           Active   5d20h
pks-system            Active   5d20h
velero                Active   17h
wavefront-collector   Active   41h

$ kubectl get svc --all-namespaces
NAMESPACE     NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
cassandra     cassandra              ClusterIP   None             <none>        9042/TCP            40h
default       kubernetes             ClusterIP   10.100.200.1     <none>        443/TCP             5d20h
default       wavefront-proxy        ClusterIP   10.100.200.56    <none>        2878/TCP            46h
kube-system   kube-dns               ClusterIP   10.100.200.2     <none>        53/UDP,53/TCP       5d20h
kube-system   kube-state-metrics     ClusterIP   10.100.200.187   <none>        8080/TCP,8081/TCP   41h
kube-system   kubernetes-dashboard   NodePort    10.100.200.160   <none>        443:32485/TCP       5d20h
kube-system   metrics-server         ClusterIP   10.100.200.52    <none>        443/TCP             5d20h
pks-system    fluent-bit             ClusterIP   10.100.200.175   <none>        24224/TCP           5d20h
pks-system    validator              ClusterIP   10.100.200.149   <none>        443/TCP             5d20h
velero        minio                  NodePort    10.100.200.82    <none>        9000:32109/TCP      17h

7.4 First restore

Let’s try to restore our backup.

$ velero backup get
NAME           STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
nginx-backup   Completed   2019-08-07 16:13:44 +0100 IST   29d       default            app=nginx

$ velero restore create nginx-restore --from-backup nginx-backup
Restore request "nginx-restore" submitted successfully.
Run `velero restore describe nginx-restore` or `velero restore logs nginx-restore` for more details.

$ velero restore describe nginx-restore
Name:         nginx-restore
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  Completed

Backup:  nginx-backup

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

7.5 Verify restore succeeded

Now we need to see if the namespace, DaemonSet and service has been restored.

$ kubectl get ns
NAME                  STATUS   AGE
cassandra             Active   40h
default               Active   5d20h
kube-public           Active   5d20h
kube-system           Active   5d20h
nginx-example         Active   17s
pks-system            Active   5d20h
velero                Active   17h
wavefront-collector   Active   41h

$ kubectl get svc --all-namespaces
NAMESPACE       NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP                 PORT(S)             AGE
cassandra       cassandra              ClusterIP      None             <none>                      9042/TCP            40h
default         kubernetes             ClusterIP      10.100.200.1     <none>                      443/TCP             5d20h
default         wavefront-proxy        ClusterIP      10.100.200.56    <none>                      2878/TCP            46h
kube-system     kube-dns               ClusterIP      10.100.200.2     <none>                      53/UDP,53/TCP       5d20h
kube-system     kube-state-metrics     ClusterIP      10.100.200.187   <none>                      8080/TCP,8081/TCP   41h
kube-system     kubernetes-dashboard   NodePort       10.100.200.160   <none>                      443:32485/TCP       5d20h
kube-system     metrics-server         ClusterIP      10.100.200.52    <none>                      443/TCP             5d20h
nginx-example   my-nginx               LoadBalancer   10.100.200.225   100.64.0.1,192.168.191.67   80:32350/TCP        23s
pks-system      fluent-bit             ClusterIP      10.100.200.175   <none>                      24224/TCP           5d20h
pks-system      validator              ClusterIP      10.100.200.149   <none>                      443/TCP             5d20h
velero          minio                  NodePort       10.100.200.82    <none>                      9000:32109/TCP      17h

Note that the nginx service has been restored but it has been assigned a new IP address by the LoadBalancer. This is normal. Now let’s see if we can successfully reach our nginx web service on that IP address. Yes I can! Looks like the restore was successful.

Cool. Backups and Restore are now working on Kubernetes deployed on vSphere+Enterprise PKS using Velero 1.0. If you want to see the steps involved in backing up persistent volumes as well, check back on some of my earlier Velero posts. Also check out the official Velero 1.0 docs. You may also be interested in listening to a recent podcast we had on Velero.

The post Setting up Velero 1.0.0 to backup K8s on vSphere/PKS appeared first on CormacHogan.com.

I was very pleased last week, as I managed to get a bunch of metrics sent from my Kubernetes cluster into Wavefront by chaining proxies together. I was successfully able to see my cluster’s Kube-state Metrics and Kubernetes Collector Metrics in Wavefront. However, on closer inspection, I noticed that a number of the built-in Wavefront Kubernetes dashboards were not being populated (Kubernetes Metrics and Kubernetes Metrics by Namespace), and then I found a number of errors in the Wavefront collector logs in my deployment. This post will describe what these errors were, and how I rectified them.

There were 2 distinct errors related to scraping containers (i.e. gathering logs from containers). First, there were the ones related to the kubelet (the part of Kubernetes that runs on the nodes). I had one of these errors for each of the nodes in the Kubernetes cluster, in my case 3. I was able to view these errors by displaying the logs on the Wavefront Collector Pod via kubectl log :

E0811 09:01:05.002411       1 manager.go:124] Error in scraping containers from \
kubelet_summary:192.168.192.5:10255: Get http://192.168.192.5:10255/stats/summary/: \
dial tcp 192.168.192.5:10255: connect: connection refused
E0811 09:01:05.002573       1 manager.go:124] Error in scraping containers from \
kubelet_summary:192.168.192.3:10255: Get http://192.168.192.3:10255/stats/summary/: \
dial tcp 192.168.192.3:10255: connect: connection refused
E0811 09:01:05.032201       1 manager.go:124] Error in scraping containers from \
kubelet_summary:192.168.192.4:10255: Get http://192.168.192.4:10255/stats/summary/: \
dial tcp 192.168.192.4:10255: connect: connection refused

E0811 09:01:05.008521       1 manager.go:124] Error in scraping containers from \
prometheus_source: http://10.100.200.2:9153/metrics: Get http://10.100.200.2:9153/metrics: \
dial tcp 10.100.200.2:9153: connect: network is unreachable

from:
  - --source=kubernetes.summary_api:''

to:
  - --source=kubernetes.summary_api:''?kubeletHttps=true&kubeletPort=10250&insecure=true

$ kubectl get svc -n kube-system kube-dns -o json | jq .spec.ports
[
  {
    "name": "dns",
    "port": 53,
    "protocol": "UDP",
    "targetPort": 53
  },
  {
    "name": "dns-tcp",
    "port": 53,
    "protocol": "TCP",
    "targetPort": 53
  }
]

$ kubectl get pods -n kube-system --selector k8s-app=kube-dns -o json | jq \
.items[0].spec.containers[].ports
[
  {
    "containerPort": 53,
    "name": "dns",
    "protocol": "UDP"
  },
  {
    "containerPort": 53,
    "name": "dns-tcp",
    "protocol": "TCP"
  },
  {
    "containerPort": 9153,
    "name": "metrics",
    "protocol": "TCP"
  }
]

$ kubectl edit svc -n kube-system kube-dns
service/kube-dns edited

$ kubectl get svc -n kube-system kube-dns -o json | jq .spec.ports
[
  {
    "name": "dns",
    "port": 53,
    "protocol": "UDP",
    "targetPort": 53
  },
  {
    "name": "dns-tcp",
    "port": 53,
    "protocol": "TCP",
    "targetPort": 53
  },
  {
    "name": "metrics",
    "port": 9153,
    "protocol": "TCP",
    "targetPort": 9153
  }
]

I0812 09:30:05.000218       1 manager.go:91] Scraping metrics start: 2019-08-12 09:29:00 +0000 UTC, end: 2019-08-12 09:30:00 +0000 UTC
I0812 09:30:05.000282       1 manager.go:96] Scraping sources from provider: internal_stats_provider
I0812 09:30:05.000289       1 manager.go:96] Scraping sources from provider: prometheus_metrics_provider: kube-system-service-kube-dns
I0812 09:30:05.000317       1 manager.go:96] Scraping sources from provider: prometheus_metrics_provider: kube-system-service-kube-state-metrics
I0812 09:30:05.000324       1 manager.go:96] Scraping sources from provider: prometheus_metrics_provider: pod-velero-7d97d7ff65-drl5c
I0812 09:30:05.000329       1 manager.go:96] Scraping sources from provider: kubernetes_summary_provider
I0812 09:30:05.000364       1 summary.go:452] nodeInfo: [nodeName:fd8f9036-189f-447c-bbac-71a9fea519c0 hostname:192.168.192.3 hostID: ip:192.168.192.3]
I0812 09:30:05.000374       1 summary.go:452] nodeInfo: [nodeName:ebbb4c31-375b-4b17-840d-db0586dd948b hostname:192.168.192.4 hostID: ip:192.168.192.4]
I0812 09:30:05.000409       1 summary.go:452] nodeInfo: [nodeName:140ab5aa-0159-4612-b68c-df39dbea2245 hostname:192.168.192.5 hostID: ip:192.168.192.5]
I0812 09:30:05.006776       1 manager.go:120] Querying source: internal_stats_source
I0812 09:30:05.007593       1 manager.go:120] Querying source: prometheus_source: http://172.16.6.2:8085/metrics
I0812 09:30:05.010829       1 manager.go:120] Querying source: kubelet_summary:192.168.192.3:10250
I0812 09:30:05.011518       1 manager.go:120] Querying source: prometheus_source: http://10.100.200.187:8080/metrics
I0812 09:30:05.034885       1 manager.go:120] Querying source: kubelet_summary:192.168.192.4:10250
I0812 09:30:05.037789       1 manager.go:120] Querying source: prometheus_source: http://10.100.200.2:9153/metrics
I0812 09:30:05.053807       1 manager.go:120] Querying source: kubelet_summary:192.168.192.5:10250
I0812 09:30:05.308996       1 manager.go:179] ScrapeMetrics: time: 308.554053ms size: 83
I0812 09:30:05.311586       1 manager.go:92] Pushing data to: Wavefront Sink
I0812 09:30:05.311602       1 manager.go:95] Data push completed: Wavefront Sink
I0812 09:30:05.311611       1 wavefront.go:122] received metric points: 28894
I0812 09:30:05.947893       1 wavefront.go:133] received metric sets: 91

The post WaveFront Collector Issues: Error in scraping containers appeared first on CormacHogan.com.

I’m delighted to be able to share with you that, coinciding with the release of vSphere 6.7 U3, VMware have also announced Cloud Native Storage (CNS). CNS builds on the legacy of the earlier vSphere Cloud Provider (VCP) for Kubernetes, and along with a new release of the Container Storage Interface (CSI) for vSphere and Cloud Provider Interface (CPI) for vSphere, CNS aims to improve container volume management and provide deep insight into how container applications running on top of vSphere infrastructure are consuming the underlying vSphere Storage. Now, there may be a lot of unfamiliar terminology in that opening paragraph, so let’s try to give a brief introduction about where Kubernetes and vSphere integration started, and how it got to where it is today.

VMware’s first container storage project – Hatchway

​Project Hatchway offered container environments a way to consume vSphere storage infrastructure, from hyper-converged infrastructure (HCI) powered by VMware vSAN to traditional SAN (VMFS) and NAS (NFS) storage. There were two distinct parts to the project initially – one focusing on docker container volumes and the other focusing on Kubernetes volumes. Both aimed to dynamically provision VMDKs on vSphere storage to provide a persistent storage solution for applications running in a Container Orchestrator on vSphere. The VCP is still found in many K8s distributions running on vSphere, but having a volume driver embedded into core Kubernetes code became problematic for a number of reasons, and led to the creation of an out-of-tree driver specification called CSI, the Container Storage Interface. The VCP will eventually be phased out of Kubernetes, in favour of the out-of-tree vSphere CSI.

Introducing CSI, the Container Storage Interface

Kubernetes, in its early days, introduced the concept of volumes to provide persistent storage to containerized applications. When a Pod was provisioned, a Persistent Volume (PV) could be connected to the Pod. The internal “Provider” code would provision the necessary storage on the underlying infrastructure, e.g. vSphere and the VCP. ​However as K8s popularity rose, more and more “providers” were added in-tree. This became unwieldy, making it unsustainable to introduce provider enhancements outside of K8s releases. This led to the creation of the Container Storage Interface (CSI), effectively an API between container orchestrators and storage providers to allow consistent interoperability. ​Eventually all in-tree storage “providers” will be remove from Kubernetes and replaced with an out-of-tree “plugin” format. CSI implements all the volume life-cycle tasks (i.e. create, attach, detach, delete, mount, unmount). VMware has rewritten the VCP to adhere to the CSI specification, and when coupled with the VMware CNS feature (which we will see shortly), provides huge advantages when running and monitoring Kubernetes application consuming persistent storage on vSphere infrastructure.

Introducing CPI, the Cloud Provider Interface

One should note that the CSI does not completely replace the functionality provided by the VCP. Thus, another interface has been developed called the Cloud Provider Interface. This has also been referred to as the Cloud Controller Manager (CCM) in the past. As you can imagine, there are numerous ‘cloud’ providers offering Kubernetes, both in public clouds and on-premises in a private clouds. Since the underlying infrastructure of each cloud is different, it was decided that some of the tasks (control loops) that were previously handled by the core Kubernetes controller should also be moved out of core Kubernetes and into a CPI plugin format, as many of these control loops were cloud provider specific. For the vSphere CPI, it is the node control loops that are of interest. These do a number of tasks such as Initialize a node with cloud specific zone/region labels and other cloud specific instance details such as type and size. It does a number of other tasks as well, but CSI in conjunction with CPI allows us to do intelligent placement of Pods and PVs on vSphere infrastructure (across Datacenters, Clusters, Hosts, etc) .

VMware Cloud Native Storage (CNS)

​Now that we have discussed CSI and CPI, we come to the VMware Cloud Native Storage initiative. VMware CNS is focused on managing and monitoring container volume life-cycle operations on vSphere storage. In a nutshell, Cloud Native Storage is a control plane that manages life-cycle of container volumes on vSphere, independent of virtual machines  and container pod life-cycles. CNS enables health monitoring, policy compliance and visibility of a volume’s metadata (name, cluster name, labels, etc.) through the vSphere UI. It gives a vSphere administrator key insights into vSphere storage resource consumption by containerized applications running Kubernetes on vSphere infrastructure.

CNS Key Components

CNS is already fully integrated into vSphere – there is nothing additional to install. Its purpose is to co-ordinate persistent volume operations (create, attach, detach, delete, mount, unmount) on Kubernetes nodes (which are virtual machines on vSphere), as requested by the CSI driver. Persistent Volumes are instantiated on vSphere storage as First Class Disks, appearing as standard VMDKs on the vSphere storage. What this means is that certain operations, such as snapshot create/delete, backup, restore can now be done on a per Kubernetes persistent volume basis, allowing Kubernetes applications running on vSphere to leverage traditional protection methods that have been available to virtual machines for many years. This is what we mean when we say that the PV life-cycle is outside of the VM or Pod life-cycle; in the past, to snapshot a VMDK, one would have to snapshot the whole of the VM. If this VM is a Kubernetes node, it could be running multiple container based applications as well as having PVs/VMDKs from many applications attached. CNS now facilitates life-cycle operations at the container/Persistent Volume granularity.

From a visibility perspective, the various Kubernetes volume metadata is stored in a local CacheDB on the vCenter server for swift rendering in the vSphere UI. Kubernetes Storage classes can leverage vSphere storage policies (SPBM), allowing attributes of the underlying vSphere storage to be assigned to the volumes, as well as continuously checking the volume for policy compliance. Should the underlying vSphere infrastructure use VMware vSAN, CNS is also integrated into parts of the vSAN UI.

A simplified diagram of the key components of CNS are shown here.

It should be noted that in this first version of CNS, we only offer visibility of block volumes (PVs/VMDKs), but those block volumes can be provisioned on VMFS, NFS or vSAN datastores. Visibility of File Shares is something that we are looking at for a future release.

Kubernetes Volumes visible in the vSphere UI

By way of showing you what sort of information CNS can make visible in the vSphere UI, I borrowed a couple of screenshots from our technical marketing team (thanks guys). Below we have selected a cluster object in the vSphere inventory. Judging by the names of the virtual machines, we can assume that there is a Kubernetes cluster deployed as a set of VMs on the cluster. If we now navigate to the Cluster > Monitor > Cloud Native Storage > Container Volumes view, we can see that the container provider is Kubernetes, and that there is a list of volumes (PVs/VMDKs/FCDs) deployed on a vSAN datastore. The volume name displayed here is exactly the same name used for the volume in Kubernetes, making it easy for the vSphere admin and the Dev-Ops/Site Reliability Engineer to have a sane dialog regarding storage. We can also see that whatever policy was set in the StorageClass in Kubernetes for these persistent volumes is also compliant. There are a number of additional columns that are not shown, such as name of the storage policy, volume capacity, etc.  By clicking on the icon in the second column of the volume, you will be shown a list of Kubernetes objects associated with the volume, such as cluster name, persistent volume claim name and namespace, etc. Also of interest is the fact that for volumes deployed on a vSAN datastore, a click of the volume name will take you to the layout of the object and display the various components that back it. You can also filter displayed volumes based on label identifiers if you want to see only the volumes associated with a particular container application.

The advantage here for vSphere administrators is that if their Kubernetes customers/end-users highlight any storage related issues, it should be extremely easy to map the objects at the Kubernetes layer to vSphere objects, and speed up any monitoring/troubleshooting needed to resolve the issue at hand.

I did want to share one additional screen shot from our technical marketing team, although it is not specifically a CNS enhancement. This is a screenshot showing vSAN Capacity Usage. In vSphere 6.7U3 this has been improved to include the storage consumption of block container volumes which are deployed as First Class Disks on the vSAN datastore. A full list of vSAN 6.7 U3 improvements can be found here.

And that concludes my introduction to CNS. As you can see, it will offer vSphere admins really good insight into how Kubernetes applications are consuming vSphere resources from a storage perspective. Coupled with vSAN and storage policies, vSphere offers an excellent infrastructure platform for your Kubernetes deployments. For more information on CNS, check out this great Virtual Blocks blog post by my good friend Myles Gray. And for a great real-life use case, read how VMware IT used Essential PKS with CNS on vSAN for their cloud native journey.

Survey

One last item. As we continue our efforts around cloud native storage, we are conducting a small survey to further understand container storage use-cases. If you could kindly spend a few minutes completing this survey, we would be very grateful. Survey Link: https://cns-storage.surveyanalytics.com

The post Introducing vSphere Cloud Native Storage (CNS) appeared first on CormacHogan.com.

I put together a few short (7 – 8 minute) videos to show off some new functionality that we’ve recently added in vSphere 6.7U3, as well as our new Velero v1.1 in action.

The first video is on CNS, the VMware Cloud Native Storage feature which we included in vSphere 6.7U3. This demonstration involves the deployment of a Cassandra database on Kubernetes, which incidentally uses the new CSI (Container Storage Interface) driver for persistent volumes. Once the application is deployed, we can see the characteristics of the volumes bubbled up in vSphere. We also see how using CNS, we can see how the persistent volume is configured on vSphere when the volumes is deployed on vSAN. Note that any use of the command ‘k’ is simply an alias to kubectl. Read more about CNS here.

The second demo is of Velero v1.1. Velero v1.1 supports the new CSI driver format. In this demo, we see how the aforementioned Cassandra application is destroyed when the Kubernetes namespace on which it is provisioned is deleted. We see how Velero is able to quickly restore the database, and we also see that the restored Cassandra database contents are intact after the restore. Again, the ‘k’ command in the CLI is simply shorthand for kubectl. Read more about Velero v1.1 here.

The post Two short video demos – CNS and Velero 1.1 appeared first on CormacHogan.com.

Hola! It is only a month or so to go until VMworld 2019 arrives back in Europe. Yet again, we are back in Barcelona for what promises to be a great event.

I missed VMworld in San Francisco this year – too many things happening on the home front. So I’m really looking forward to getting down to Spain in early November and meeting up with everyone again.

I am involved in 2 sessions in Barcelona. One of the sessions is the HCIBU Spotlight Session, HCI3551KE. I am delivering this with our SVP and GM of the Hyper-converged Infrastructure Business Unit at VMware, John Gilmartin. I’ll be demoing some cool new technology and features coming out of our company at this session. You might ask. “Why should I attend?” Well, let me run this scenario by you, and see if something resonates. Let’s say that your organization has just embraced Kubernetes for deployment and managing of modern or next-gen applications. Of course, the best way to run Kubernetes is in virtual machines, and then the best place to run VMs is on vSphere. So you roll out a Kubernetes cluster as a set of VMs on your vSphere infrastructure for your dev-ops team. Everyone is happy! The first modern app goes live, and everyone is really happy. It’s high fives all round. Then a day or two later, you get a call from the dev-ops team – “Hey, the app isn’t performing as well as expected. Customers are complaining. We think it must be an infrastructure issue as our code is perfect.” Sound familiar? OK – so where do you begin, with all of these different layers of abstraction? How do the components in Kubernetes map to resources in vSphere. Are they all healthy and working properly? This is a big part what I will be showcasing at our Spotlight session. This takes place on Wednesday, November 6th at 11:00am.

The other session I am helping out on is HCI2763BE with my buddy Myles Gray. This is again very much focused on Kubernetes running on vSphere, but we will deep dive into the storage side of things. We will also have a bunch of really cool demos ready to roll here again, some of which are going to be hot off the presses. We think you’ll enjoy it, whether you are already well-versed in Kubernetes, or even if you are just starting out and trying to learn a bit more about it. This takes place on Wednesday, November 6th at 3:30pm.

I’ve also been invited along to the VMware User Group (VMUG) Leader Event, to give you a bit of a talk about my career development. I’ve met a lot of the VMUG leaders over the years, many of which have become very good friends of mine. I’m really looking forward to meeting them and catching up.

On the topic of VMUGs, last but not least, I will be speaking at the Barcelona VMUG event which is being held on Monday, November 4th, at the Porta Fira Hotel close to the VMworld event. This event is once again very Kubernetes focused, and I will be doing an overview of Kubernetes and vSphere storage once again. You can find more details here, including registration details. The event runs from 9am to 12:30pm.

If you are coming to VMworld 2019 in Barcelona, I hope to see you there at some point during the week, even if you are unable to attend one of my sessions. Adiós y cuídate.

The post See you at VMworld 2019 (Barcelona) appeared first on CormacHogan.com.

Since the release of the vSphere CSI driver in vSphere 6.7U3, I have had a number of requests about how we plan to migrate applications between Kubernetes clusters that are using the original in-tree vSphere Cloud Provider (VCP) and Kubernetes clusters that are built with the new vSphere CSI driver. All I can say at this point in time is that we are looking at ways to seamlessly achieve this at some point in the future, and that the Kubernetes community has a migration design in the works to move from in-tree providers to the new CSI driver as well.

However, I had also seen some demonstrations from the Velero team on how to use Velero for application mobility. I wanted to see if Velero could also provide us with an interim solution to move applications with persistent storage between a K8s cluster running on vSphere using the in-tree VCP and a native K8s cluster that uses the vSphere CSI driver.

Note that this method requires downtime to move the application between clusters, so the application will be offline for part of this exercise.

It should also be noted the Cassandra application that I used for demonstration purposes was idle at the time of backup (no active I/O), so that should also be taken into account.

tl;dr Yes – we can use Velero for such a scenario, in the understanding that (a) you will need resources to setup the new CSI cluster and (b) there is no seamless migration, and that the application will need to be shutdown on the VCP cluster and restarted on the CSI cluster. Here are the detailed steps.

External S3 object store

The first step is to setup an external S3 object store that can be reached by both clusters. Velero stores both metadata and (in the case of vSphere backups using restic) data in the S3 object store. In my example, I am using MinIO as I have had the most experience with that product. I have a post on how to set this up on vSphere if you want to learn more. In my lab, my VCP K8s cluster is on VLAN 50, and my CSI K8s cluster is on VLAN 51. Thus, for the CSI cluster to access the MinIO S3 object store, and thus the backup taken from the VCP cluster, I will need to re-IP my MinIO VMs to make the backup visible to the CSI cluster. More detail on that later.

VCP StorageClass

Before going any further, it is probably of interest to see how the VCP driver is currently being used. The reference to the provider/driver is placed in the StorageClass. Here is the StorageClass being used by the Cassandra application in the VCP cluster, which we will shortly be backing up and moving to a new cluster.

$ kubectl get sc
NAME PROVISIONER AGE
cass-sc kubernetes.io/vsphere-volume 64d


$ cat cassandra-sc-vcp.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cass-sc
provisioner: kubernetes.io/vsphere-volume
parameters:
    diskformat: thin
    storagePolicyName: raid-1
    datastore: vsanDatastore

Deploy Velero on VCP K8s cluster

At this point, the S3 object store is available on IP 192.50.0.20. It is reachable via port 9000. Thus when I deploy Velero, I have to specify this address:port combination in the s3Url and publicUrl as follows:

$ velero install  --provider aws 
--bucket velero \
--secret-file ./credentials-velero \
--use-volume-snapshots=false \
--use-restic 
--backup-location-config region=minio,s3ForcePathStyle="true",\
s3Url=http://192.50.0.20:9000,publicUrl=http://192.50.0.20:9000

Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

Prepping the Stateful Application

For the purposes of this test, I deployed a Cassandra stateful set with 3 replicas on my VCP cluster. I also populated it with some data so that we can verify that it gets successfully restored on the CSI cluster.

$ kubectl exec -it cassandra-0 -n cassandra -- nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.16.5.2  104.38 KiB  32           66.6%             8fd5fda2-d236-4f8f-85c4-2c57eab06417  Rack1-K8Demo
UN  172.16.5.3  100.05 KiB  32           65.9%             6ebf73bb-0541-4381-b232-7f277186e2d3  Rack1-K8Demo
UN  172.16.5.4  75.93 KiB  32           67.5%             0f5387f9-149c-416d-b1b6-42b71059c2fa  Rack1-K8Demo


$ kubectl exec -it cassandra-0 -n cassandra -- cqlsh
Connected to K8Demo at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 };
cqlsh> use demodb;
cqlsh:demodb> CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint);
cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES (100, 'Cormac', 'Cork', 999, 1000000);
cqlsh:demodb> select * from emp;

emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
    100 |     Cork |   Cormac |       999 | 1000000
(1 rows)

cqlsh:demodb>

Backup Cassandra using Velero

We are now ready to backup Cassandra. The first part is to annotate the volumes so that restic knows that it needs to copy the contents of these volumes as part of the backup process.

$ kubectl -n cassandra annotate pod/cassandra-2 \
backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-2 annotated

$ kubectl -n cassandra annotate pod/cassandra-1 \
backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-1 annotated

$ kubectl -n cassandra annotate pod/cassandra-0 \
backup.velero.io/backup-volumes=cassandra-data
pod/cassandra-0 annotated


$ velero backup create cassandra-pks-1010 --include-namespaces cassandra
Backup request "cassandra-pks-1010" submitted successfully.

Run `velero backup describe cassandra-pks-1010` or `velero backup logs cassandra-pks-1010` for more details.

Restic Backups:
  Completed:
    cassandra/cassandra-0: cassandra-data
    cassandra/cassandra-1: cassandra-data
    cassandra/cassandra-2: cassandra-data

$ velero backup get
NAME                 STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
cassandra-pks-1010   Completed   2019-10-10 10:38:32 +0100 IST   29d       default            <none>
nginx-backup         Completed   2019-10-01 15:12:20 +0100 IST   21d       default            app=nginx

$ kubectl config get-contexts
CURRENT   NAME                CLUSTER             AUTHINFO                               NAMESPACE
*         cork8s-cluster-01   cork8s-cluster-01   d8ab6b15-f7d7-4d20-aefe-5dfe3ecbf63b
          cork8s-csi-01       kubernetes          kubernetes-admin

$ kubectl get nodes -o wide
NAME                                   STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
140ab5aa-0159-4612-b68c-df39dbea2245   Ready    <none>   68d   v1.13.5   192.168.192.5   192.168.192.5   Ubuntu 16.04.6 LTS   4.15.0-46-generic   docker://18.6.3
ebbb4c31-375b-4b17-840d-db0586dd948b   Ready    <none>   68d   v1.13.5   192.168.192.4   192.168.192.4   Ubuntu 16.04.6 LTS   4.15.0-46-generic   docker://18.6.3
fd8f9036-189f-447c-bbac-71a9fea519c0   Ready    <none>   68d   v1.13.5   192.168.192.3   192.168.192.3   Ubuntu 16.04.6 LTS   4.15.0-46-generic   docker://18.6.3

$ kubectl config use-context cork8s-csi-01
Switched to context "cork8s-csi-01".

$ kubectl get nodes -o wide
NAME          STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-master    Ready    master   51d   v1.14.2   10.27.51.39   10.27.51.39   Ubuntu 18.04.3 LTS   4.15.0-58-generic   docker://18.6.0
k8s-worker1   Ready    <none>   51d   v1.14.2   10.27.51.40   10.27.51.40   Ubuntu 18.04.3 LTS   4.15.0-58-generic   docker://18.6.0
k8s-worker2   Ready    <none>   51d   v1.14.2   10.27.51.41   10.27.51.41   Ubuntu 18.04.3 LTS   4.15.0-58-generic   docker://18.6.0

$ velero install  --provider aws \
--bucket velero \
--secret-file ./credentials-velero \
--use-volume-snapshots=false  \
--use-restic \
--backup-location-config region=minio,s3ForcePathStyle="true",\
s3Url=http://10.27.51.49:9000,publicUrl=http://10.27.51.49:9000

Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

$ velero backup get
NAME                 STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
cassandra-pks-1010   Completed   2019-10-10 10:38:32 +0100 IST   29d       default            <none>
nginx-backup         Completed   2019-10-01 15:12:20 +0100 IST   21d       default            app=nginx

$ kubectl get sc
NAME         PROVISIONER              AGE


$ cat cassandra-sc-csi.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cass-sc
  annotations:
    storageclass.kubernetes.io/is-default-class:"false"
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname:"Space-Efficient"


$ kubectl apply -f cassandra-sc-csi.yaml
storageclass.storage.k8s.io/cass-sc created



$ kubectl get sc
NAME         PROVISIONER              AGE
cass-sc      csi.vsphere.vmware.com   3s

$ velero create restore cassandra --from-backup cassandra-pks-1010
Restore request "cassandra" submitted successfully.
Run `velero restore describe cassandra` or `velero restore logs cassandra` for more details.

$ velero restore describe cassandra --details
Name:         cassandra
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  Completed

Backup:  cassandra-pks-1010

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Restic Restores:
  Completed:
    cassandra/cassandra-0: cassandra-data
    cassandra/cassandra-1: cassandra-data
    cassandra/cassandra-2: cassandra-data

 image: harbor.rainpole.com/library/cassandra:v11

 image: gcr.io/google-samples/cassandra:v11

$ kubectl edit pod cassandra-0 -n cassandra
pod/cassandra-0 edited

$ kubectl edit pod cassandra-1 -n cassandra
pod/cassandra-1 edited

$ kubectl edit pod cassandra-2 -n cassandra
pod/cassandra-2 edited

$ kubectl exec -it cassandra-0 -n cassandra -- nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.244.2.61  192.73 KiB  32           100.0%            79820874-edec-4254-b048-eaceac0ec6c8  Rack1-K8Demo
UN  10.244.2.62  157.17 KiB  32           100.0%            ea0e8ef2-aad2-47ee-ab68-14a3094da5be  Rack1-K8Demo
UN  10.244.1.58  139.97 KiB  32           100.0%            110d3212-526b-4a58-8005-ecff802d7c20  Rack1-K8Demo

$ kubectl exec -it cassandra-0 -n cassandra -- cqlsh
Connected to K8Demo at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> use demodb;
cqlsh:demodb> select * from emp;

emp_id | emp_city | emp_name | emp_phone | emp_sal
--------+----------+----------+-----------+---------
    100 |     Cork |   Cormac |       999 | 1000000
(1 rows)

cqlsh:demodb>

The post Moving a Stateful App from VCP to CSI based Kubernetes cluster using Velero appeared first on CormacHogan.com.

After spending some time watching, digesting and then writing about Project Pacific Deep Dive updates from VMworld 2019, the next item on my to-do list was to get up to speed on VMware Tanzu, or to be more specific, Tanzu Mission Control. The reason I am being more specific is that VMware Tanzu is a broad portfolio of products and features which can be categorized into 3 distinct areas. These areas are Build, Run and Manage. The Build category related to initiatives taking place in the developer space, notably with Bitnami and Pivotal, the former having recently been acquired by VMware, whilst the latter is soon to be acquired by VMware, all going well. Run is all about running Kubernetes (K8s) on vSphere, primarily through Enterprise PKS or on Project Pacific (when it becomes available). But I wanted to focus on the Manage category. And this is where Tanzu Mission Control comes in.

I watched a number of VMworld 2019 sessions on Tanzu, and I guess for me the key point was made by Craig McLuckie. Craig made the point (I’m paraphrasing here) that while Kubernetes is a very powerful, production ready platform, it is still a young technology and it is still evolving from a resource management perspective. For that reason, we are not yet at the stage where we can consolidate everything into a single Kubernetes cluster. Thus we are seeing a lot of Kubernetes cluster fragmentation within organizations.  Not only are we seeing multiple teams in the same organization creating and deploying applications on their own Kubernetes clusters, but indeed they are deploying these clusters onto different clouds. There are many reasons for this, some related to geography, some related to security, and others related to politics (e.g. GDPR). This fragmentation is causing significant management overhead for the operations teams. In many cases, they are having to manage K8s clusters on an individual basis, configuring permissions/access, networking and security cluster by cluster – this is unsustainable.

This is where Tanzu Mission Control is positioned – it is a single control point for managing K8s across multiple teams and clouds, providing visibility and control over the environments for IT Ops teams, whilst at the same time not constraining or hindering the developer teams.

On a number of sessions that I watched, Eryn Muetzel gave a demonstration of the planned Tanzu Mission Control capabilities. She highlighted a number of features, such as:

• Ability to deploy K8s clusters directly from Tanzu Mission Control. Whilst the demo showed the deployment of K8s clusters on AWS, she did say that plans were underway to do the same thing with Project Pacific.
• Ability to attach existing K8s clusters to Tanzu Mission Control. This is achieved by installing an agent in the remote K8s cluster, which then provides a secure connection back to Tanzu Mission Control. This means that existing K8s clusters can also be managed by Tanzu MC, even if Tanzu MC did not provision that cluster.
• Cluster Groups enable new clusters to inherit policies at a group level, rather than having to add policies individually to each cluster.
• Workspaces provides the ability to apply access controls to an application that resides in multiple different namespaces, in different clusters on different clouds. This feature provides for the consistent deployment of an application (permission, security, configuration) across different clusters and clouds.

Provisioning a K8s Cluster from Tanzu Mission Control

Eryn showed up the steps involved in provisioning a K8s cluster on AWS from Tanzu Mission Control. She began by providing us with an overview of all the different clusters (AKS from Azure, GKE from Google, PKS from VMware and EKS from Amazon) currently being managed by the Tanzu MC instance in her environment.

She then proceeded to build a new EKS cluster on AWS, filling in some basic details as she went. Here you can see where one could add a Cluster Group to enable this cluster to inherit existing policies that are already associated with the group. This part of Kubernetes is hugely challenging for many organizations as they build out clusters – adding the correct privileges on a cluster by cluster basis becomes a tedious and time consuming operation. Tanzu Mission Control solves this through Cluster Groups.

As Eryn continued with the cluster configuration, we saw the difference between a development cluster and a production cluster (number of control plane nodes and worker nodes) as well as the ability to choose the size of a node. We could also add various labels, which are used extensively throughout Kubernetes. This would help us to easily identify our cluster when we start deploying clusters at scale. Once the cluster was deployed, we were able to see some basic health check information (Components, Agent and Inspection) related to the cluster, as shown here:

She also showed us the nodes view, and how Tanzu Mission Control could be used to drill down into the heath and operations of individual parts of the infrastructure. On an individual worker node, we could see the details about the Kubernetes (kubelet) version, the container run-time and version, and conditions such as memory and disk pressure. And of course, all the Pods running on the worker node were also listed:

So, to conclude, consider managing Kubernetes at scale. Consider managing not only 10s-100s of K8s clusters but also many 1000s of developers!!! This is what Tanzu Mission Control does. It is automating, though policies, many of the tasks that IT operations have to do today when it comes to managing Kubernetes, as well as giving the IT operations team visibility into their whole Kubernetes environment.

Tanzu Mission Control futures

Eryn highlighted that today, Tanzu Mission Control is focused primarily on access control/permissions policies. Going forward the plan is to extend policies to area like the image registry, networking, pod security, quotas and so on. She then shared with us an overview of future plans, which are shown in the diagram below.

For Identity and Access Management, this is basically focused on centralized authentication of K8s clusters. The Security and Configuration initiatives are two-fold: (1) provide the right insights to verify that your clusters are configured and secure, and (2) apply policies so that certain things are not allowed to happen in the cluster, e.g. deployment of privileged containers. The last area that Eryn spoke about was Observability and Diagnostics which is concerned with providing the right high level of health information to see what is going on at the fleet level (fleet == large scale K8s deployments). This will be achieved by building integrations into products such Wavefront (and others) over time.

Of course, there is one thing in the VMware Mission Control futures section which definitely piqued my interest, which is Data Protection. Whilst Eryn did not go into that in any detail during the VMworld 2019 sessions that I watched, it is an area I am going to keep my eye on going forward.

If you are interested in watching the full sessions on Tanzu Mission Control from VMworld 2019, these are the sessions that I watched to gather the above information:

• Modern Apps Keynote: VMware Tanzu and Your Kubernetes Journey (MOD3543KE)
• Introducing VMware Tanzu Mission Control: Manage Kubernetes at Scale (KUB1835BE)

The post Tanzu Mission Control – VMworld 2019 Updates appeared first on CormacHogan.com.

We are nearing the end of our journey with Getting Started with VMware Cloud Foundation (VCF). In this post, we will go through the deployment of Enterprise PKS v1.5 on a Workload Domain created in VCF v3.9. We’ve been through a number of steps to get to this point, all of which can be found here. Now we have some of the major prerequisites in place, notably NSX-T Edge networking and PKS Certificates, so we can proceed with the Enterprise PKS deployment. However, there are still a few additional prerequisites needed before we can start. Let’s review those first of all.

Enterprise PKS Prerequisites

Let’s recap on some of the prerequisites that need to be in place before we can commence the deployment of Enterprise PKS.

• As stated, NSX-T Edge networking must be configured, not just creating internal overlay segments for east-west traffic, but also allowing that traffic route externally north-south. We looked at both the BGP approach as well as the Static Route + SNAT approach.
• You need to ensure that the necessary PKS components (Ops Manager, BOSH, PKS appliance, and optionally Harbor) have IP address allocated, as well as work in forward and reverse DNS lookup.
  • When deploying Enterprise PKS, you will be asked to provide an IP Address exclusion range. The Operations Manager IP address should appear in the exclusion range, but BOSH, PKS appliance and Harbor should all be just outside the exclusion range.
• You need a set of certificates and private keys. We already generated a bunch of PKS Certificates and private keys, so now we have a Root CA PEM, an NSX-T Super User Principal Identity certificate and private key, as well as certificates and private keys for the PKS entities such as Ops Manager, BOSH, PKS appliance, and again, optionally, Harbor.
• In NSX-T Manager, Advanced Networking & Security > Networking > IPAM, a Node Block IP address range needs to be added for the Kubernetes nodes. This address range needs to route externally as Kubernetes master nodes need to talk to vCenter for storage provisioning, among other things.
• Similarly, a Pod Block IP address range needs to be added for the pods (where the containers run). This is an internal address range, and does not need to route.
• In NSX-T Manager, Networking > IP Management > IP Address Pools, a Load Balancing/Floating IP Address Pool should be added. This address pool provide address to Kubernetes Services, e.g. when a Pod/container based application requires external access from it’s allocated Pod Block network.
• Enterprise PKS supports the concept of Availability Zones (AZs). In Enterprise PKS 1.5, these AZs map to vSphere Resource Pools.
  • An AZ (Resource Pool) should be created for the PKS Components (Ops Manager, BOSH, PKS and Harbor).
  • The K8s nodes can be placed in 1 or more AZs. A minimum of 3  AZs are required for availability purposes, placing different K8s masters and nodes in different AZs. For the purposes of demonstration, I built 3 x Resource Pools on the same workload domain cluster. This is to show how, when I deploy a 3 worker node Kubernetes cluster using Enterprise PKS, each K8s nodes is placed in a different AZ/Resource Pool. You can imagine how this would work in production, where resource pools would be configured across multiple clusters/data centers for high availability.

The full range of prerequisites can be found in the official VCF 3.9 documentation.

For this post, here are the Node and Pod IP blocks that I plan on using, 20.0.0.0/16 and 30.0.0.0/16 respectively.

And here is the Floating IP Address Pool which will provide Load Balancer IP addresses to my Kubernetes Services, 147.70.30.0/24.

One final item to note; I have already created a PKS Management network segment and a PKS Services network segments. See step 11 in part 7 of this blog series on VCF for more details on how I did this.

[Update] What became apparent after I completed this deployment is that the PKS Services network segment is that it is not used anywhere. In earlier releases of Enterprise PKS, all the K8s nodes for all the K8s clusters were placed on the service network. In later versions of PKS with NSX-T integration, the capability to support a K8s cluster node on its own LS/T1 was introduced. Thus the service network is now deprecated. So even though you need to provide this service network as part of the deployment, and even though it is still plumbed into the BOSH and PKS tiles in Enterprise PKS 1.5, it is no longer used. In fact, I’ve been informed that this service network entry has been removed from the BOSH and PKS tiles in the latest versions of PKS. Kudos for Francis Guiller for this useful information.

Now, with all the prerequisites in place, we can proceed with the actual Enterprise PKS deployment.

Enterprise PKS deployment on a Workload Domain in VCF

In the VCF SDDC Manager UI, navigate to Workload Domain, click on the +Workload Domain button and choose PKS. The first thing that will popup is a PKS Deployment Prerequisites window. We have covered all of these now, so you can simply click Select All, and continue:

Next, accept the EULA. There are 3 in total to select:

Now you get to General Settings. Here, we must provide the name of the PKS Solution, choose the Workload Domain on which Enterprise PKS will be provisioned from the drop-down list, and populate the various passwords for Operations Manager. Note also the default username of ubuntu, which will come in useful later on.

Now you need to populate some of the NSX-T Settings, such as the Tier-0 Logical Router, Node Block, Pod Block and Load Balancer/Floating IP Pool. If everything has been configured correctly, these should all be available from the drop-downs.

Next, it is time to add some PKS Settings. Here, the fully qualified domain name of the PKS appliance is requested (API FQDN). I am not deploying Harbor in this example, but if the Harbor option is checked, you need to provide the FQDN of the Harbor Registry appliance/VM. The rest of the settings relate to the Operations Manager appliance.

Certificates are next. Refer back to part 8 of this VCF series for information about the format and naming convention required by this part of the installer wizard. As mentioned in that post, there are some nuances here. Once again, if Harbor was selected previously, a certificate and private key for that appliance would also be required.

Now we start to add some information around the Management Availability Zone. The Management Network refers to the PKS Management network overlay that we built earlier (see VCF blog part 7). Note the reserved IP range – it should not include the network gateway, nor the IP address of BOSH, PKS or Harbor VMs, but it should include the IP address of the Operations Manager VM. For the Management AZ, provide a name, select the cluster and choose the correct Resource Pool (which must have been created as part of the prerequisites).

This brings us to the Kubernetes Availability Zone. Here you simply populate network information, similar to what we did for the management network. Note the network is the PKS Service network segment (overlay) built previously. [Update] However, as we later found out, this is no longer used for the K8s nodes. Instead NSX-T creates a Logical Switch/T1 for each K8s cluster. However, you still need to populate this field.

What you might notice in the last window is that we did not provide any information about Availability Zones. This is done in the next window, when we begin to populate the Compute Availability Zones for the Kubernetes cluster. Typically, this is where our Kubernetes masters worker nodes get deployed, but this is all decided by the plan that is used to create the cluster. We will see this in the next part of this series of blog posts. I suppose we could have referred to them as PKS Services Availability Zones to keep it aligned with the Management AZ. As I said, we will create 3 Resource Pools and map those Resource Pools to AZs. When we deploy our Kubernetes cluster, our plan will decide if master and workers are placed in the different AZs by PKS for availability. More on this in the next post.

And that’s it. We can now Review the details before having the values validated.

And if the Review looks good, we can proceed with the Validation step. This checks a whole bunch of things, such as DNS resolution, no IP addresses overlapping the reserved ranges, and of course the validity of the certificates. All going well, there should be no errors and all status should show as INFO.

You can now click on the FINISH button to start the PKS deployment.

Monitoring Enterprise PKS Workload Activation

What you should now notice is that there is a new PKS Workload Domain in the process of activating.

And there are a lot of subtasks as part of this PKS deployment:

While this can offer some insight into what is happening, you can get further details by SSH’ing onto the SDDC Manager and tailing solutionsmanager.log found under /var/log/vmware/vcf/solutionsmanager. This gives some very good details in the early part of the deployment, but once BOSH, PKS and Harbor are being deployed and configured, there is not much visibility into what is going on. When it gets to that stage of the deployment, you can actually connect to the Operations Manager UI, login as the ubuntu user, and monitor what is happening from there. In the Summary tab of the PKS domain, click on the Pivotal Ops Manager link, and log in using the ubuntu credentials that you provided in the General Settings part of the deployment wizard.

From there, you can see that Ops Manager is ‘Applying changes‘ to the configuration. Click on the ‘Show Progress’ button to see more details about what is happening.

Here is the BOSH & PKS deployment and configuration logs as view from my deployment.

Assuming everything has worked successfully (this will take some time to deploy), you should end up with an Active PKS workload domain on VMware Cloud Foundation.

The next step is to use PKS to create a Kubernetes cluster. I will show you how to do that in the next post. For now, sit back and bask in the glory of having successfully deploy Enterprise PKS in a Workload Domain in VMware Cloud Foundation. That’s what I’m doing

The post Getting started with VCF Part 9 – PKS deployment appeared first on CormacHogan.com.

With Enterprise PKS deployed in a Workload Domain in VMware Cloud Foundation, we now come to the point where we can begin to create Kubernetes clusters and deploy some containerized applications. We need access to some tooling to achieve this. One option is to SSH onto the Operations Manager appliance, as it has many of the necessary tools already installed. However, I prefer to do this in my own management/jump desktop rather than use components that are part of the actual product. In this post, I will show you the steps to get setup with the required tool-set, deploy your first Kubernetes (K8s) cluster and then get you on your way to deploying your first containerized application.

Step 1. Download the prerequisite tools

We require the following tools:

• PKS command line interface (pks), for the creation, deletion and querying of K8s clusters
• kubectl command line interface, for communicating to the K8s clusters
• (Optional) Bosh command line interface (bosh), very useful for tracking activities and tasks in Enterprise PKS, though not essential to deploying a K8s cluster

You can get the pks CLI and kubectl from network.pivotal.io, as shown below. You will find these in the Pivotal Container Service (PKS) section. Download the build appropriate to your desktop OS. Add it to your PATH, e.g. /usr/local/bin.

You will also need to pull down the bosh CLI. You can get the latest version (6.2.0) from an AWS repository called bosh-cli-artifacts, using the followings commands in your desktop:

$ wget https://s3.amazonaws.com/bosh-cli-artifacts/bosh-cli-6.2.0-linux-amd64
$ chmod +x bosh-cli-6.2.0-linux-amd64
$ mv bosh-cli-6.2.0-linux-amd64 /usr/local/bin/bosh

Great – the tools are downloaded, and we can start the deployment of our first K8s cluster.

Step 2. Create PKS Credentials

The very next step is to authenticate to the PKS API server appliance. This can be done using the ‘ubuntu‘ user that we provided a password for during the PKS deployment in part 9 of this series. The command will look something like this:

cormac@pks-cli:~$ pks login -a w01-pks-01.rainpole.com -u ubuntu -k

Password: **********
API Endpoint: w01-pks-01.rainpole.com
User: ubuntu

cormac@pks-cli:~$

This has the effect of building a .pks/creds.yml file in your local directory, which is used by further pks CLI commands.

Step 3. Create BOSH Credentials

To be able to run the bosh CLI, we need to get some BOSH credentials from Ops Manager. The easiest way to do this is to login to the Ops Manager UI. You’ll find the link in Workload Domains > PKS (View Details). Click on the PKS Workload Domain, select Service VMs then click the URL to the Pivotal Ops Manager. This will bring up the Ops Manager UI.

After logging into the UI using the ubuntu credentials, click on the BOSH Director for vSphere tile, select the Credentials tab, and located near the bottom of the list are the Bosh Commandline Credentials, as shown below.

Click on ‘Link to Credential‘, and it will display a number of environment variables, such as BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_ENVIRONMENT and BOSH_CA_CERT.

You now need to export these environment variables in your own desktop shell – I typically add them to the .bash_profile so that they are set each time I login. Note that the BOSH_CA_CERT environment variable points to the root_ca_cert location on the Pivotal Ops Manager appliance. You can copy that certificate to your own desktop, then modify that environment variable appropriately. Here is an example on how to do that.

cormac@pks-cli:~$ scp ubuntu@w01-opsmgr-01.rainpole.com:/var/tempest/workspaces/default/root_ca_certificate\
 ~/opsmanager.pem
Unauthorized use is strictly prohibited. All access and activity
is subject to logging and monitoring.
ubuntu@w01-opsmgr-01.rainpole.com's password: **********
root_ca_certificate                                              100% 1208     2.4MB/s   00:00

cormac@pks-cli:~$ export BOSH_CLIENT=ops_manager
cormac@pks-cli:~$ export BOSH_CLIENT_SECRET=SoMeSortOfSecretText
cormac@pks-cli:~$ export BOSH_ENVIRONMENT=147.70.10.11
cormac@pks-cli:~$ export BOSH_CA_CERT=~/opsmanager.pem

The can now run bosh commands, e.g. bosh deployments or bosh vms. We will see how useful this is shortly to track activity, especially the creation of Kubernetes clusters.

We will not use the kubectl command until we have created our first Kubernetes cluster.

Step 4. Create our first Kubernetes cluster

The following pks CLI command is used to create our very first Kubernetes cluster on Enterprise PKS. But before we run it, let’s take a moment to describe it:

pks create-cluster k8s-clus-01 --external-hostname pks-clus-01 --plan small --num-nodes 3

• k8s-clus-01 – this is the name of the Kubernetes cluster
• –external-hostname pks-clus-01 – this is the DNS name for our master node, and should have an DNS entry or an /etc/hosts entry created for it. kubectl will need to communicate to the API server on the master(s) when we start using it. However we will have to wait and see what IP address (from the floating/load balancer pool) is mapped to the master node of the cluster before we can complete this step
• –plan small – the plan as described in the Enterprise PKS tile in Pivotal Ops Manager. By default, this small plan creates a single master/ETCD node. It also builds masters and workers VMs with 2 CPU, 4GiB Memory and 32GB VMDKs. You can enable and change additional plans from the Ops Manager UI. Note that plans also define how Availability Zones should be used by masters and workers. More on this shortly.
• –num-nodes 3 – this is the number of  worker nodes that will be created in this Kubernetes cluster

Let’s run the command:

cormac@pks-cli:~$ pks create-cluster k8s-clus-01 --external-hostname pks-clus-01 --plan small --num-nodes 3

Name:                     k8s-clus-01
Plan Name:                small
UUID:                     23d0fc1f-200b-41c3-9dba-7bdb53b52180
Last Action:              CREATE
Last Action State:        in progress
Last Action Description:  Creating cluster
Kubernetes Master Host:   pks-clus-01
Kubernetes Master Port:   8443
Worker Nodes:             3
Kubernetes Master IP(s):  In Progress
Network Profile Name:

Use 'pks cluster k8s-clus-01' to monitor the state of your cluster

cormac@pks-cli:~$

cormac@pks-cli:~$ pks cluster k8s-clus-01

PKS Version:              1.5.0-build.32
Name:                     k8s-clus-01
K8s Version:              1.14.5
Plan Name:                small
UUID:                     ce2567f3-233e-4ec6-b1e4-fafbbecbadeb
Last Action:              CREATE
Last Action State:        succeeded
Last Action Description:  Instance provisioning completed
Kubernetes Master Host:   pks-clus-01
Kubernetes Master Port:   8443
Worker Nodes:             3
Kubernetes Master IP(s):  147.70.30.2
Network Profile Name:

cormac@pks-cli:~$

cormac@pks-cli:~$ bosh task

Using environment '147.70.10.11' as user 'director'

Task 31

Task 31 | 17:16:11 | Preparing deployment: Preparing deployment
Task 31 | 17:16:13 | Warning: DNS address not available for the link provider instance: pivotal-container-service/cec0ab96-6f39-46f1-a277-8a49181dc244
Task 31 | 17:16:13 | Warning: DNS address not available for the link provider instance: pivotal-container-service/cec0ab96-6f39-46f1-a277-8a49181dc244
Task 31 | 17:16:13 | Warning: DNS address not available for the link provider instance: pivotal-container-service/cec0ab96-6f39-46f1-a277-8a49181dc244
Task 31 | 17:16:23 | Preparing deployment: Preparing deployment (00:00:12)
Task 31 | 17:16:23 | Preparing deployment: Rendering templates (00:00:07)
Task 31 | 17:16:30 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 31 | 17:16:30 | Compiling packages: jq/c6a6daa7f64fc4775d11c0d4441d9fcf49506746
Task 31 | 17:16:30 | Compiling packages: nsx-cni/cab69c27665c0ff1a5210adc44fd97efc6b74ea0
Task 31 | 17:16:30 | Compiling packages: nsx-cni-common/dc5b3b6618f30a09827997245e20288c56dd3f20
Task 31 | 17:16:30 | Compiling packages: nsx-python27/75df9f63298d0d2644c6030b160b9b7486a9c195
Task 31 | 17:18:03 | Compiling packages: nsx-cni-common/dc5b3b6618f30a09827997245e20288c56dd3f20 (00:01:33)
Task 31 | 17:18:03 | Compiling packages: ncp_rootfs/5973b83f6fdd6a4c5286d675fd0729e98acd61b0
Task 31 | 17:18:10 | Compiling packages: jq/c6a6daa7f64fc4775d11c0d4441d9fcf49506746 (00:01:40)
Task 31 | 17:18:13 | Compiling packages: nsx-cni/cab69c27665c0ff1a5210adc44fd97efc6b74ea0 (00:01:43)
Task 31 | 17:18:56 | Compiling packages: ncp_rootfs/5973b83f6fdd6a4c5286d675fd0729e98acd61b0 (00:00:53)
Task 31 | 17:20:14 | Compiling packages: nsx-python27/75df9f63298d0d2644c6030b160b9b7486a9c195 (00:03:44)
Task 31 | 17:20:15 | Compiling packages: openvswitch/2b5d30bcf7b6e19d82dfd7f851701f04e4654dad (00:02:56)
Task 31 | 17:23:44 | Creating missing vms: master/5cb6458c-7f6e-4b1c-acfd-f71ad8b0fab7 (0)
Task 31 | 17:23:44 | Creating missing vms: worker/14ad78ba-d6f5-443c-ae17-1e6b3af4dd1e (0)
Task 31 | 17:23:44 | Creating missing vms: worker/7e544293-cfb6-4291-b348-c6b3aaf6fd5c (1)
Task 31 | 17:23:44 | Creating missing vms: worker/26f89b75-40ec-4972-912f-7f5af600d981 (2) (00:01:23)
Task 31 | 17:25:07 | Creating missing vms: worker/7e544293-cfb6-4291-b348-c6b3aaf6fd5c (1) (00:01:23)
Task 31 | 17:25:15 | Creating missing vms: worker/14ad78ba-d6f5-443c-ae17-1e6b3af4dd1e (0) (00:01:31)
Task 31 | 17:25:16 | Creating missing vms: master/5cb6458c-7f6e-4b1c-acfd-f71ad8b0fab7 (0) (00:01:32)
Task 31 | 17:25:16 | Updating instance master: master/5cb6458c-7f6e-4b1c-acfd-f71ad8b0fab7 (0) (canary) (00:02:31)
Task 31 | 17:27:47 | Updating instance worker: worker/14ad78ba-d6f5-443c-ae17-1e6b3af4dd1e (0) (canary) (00:03:45)
Task 31 | 17:31:32 | Updating instance worker: worker/7e544293-cfb6-4291-b348-c6b3aaf6fd5c (1) (00:03:47)
Task 31 | 17:35:19 | Updating instance worker: worker/26f89b75-40ec-4972-912f-7f5af600d981 (2) (00:03:57)

Task 31 Started  Wed Feb  5 17:16:11 UTC 2020
Task 31 Finished Wed Feb  5 17:39:16 UTC 2020
Task 31 Duration 00:23:05
Task 31 done

Succeeded

Step 5. Use kubectl to query our K8s cluster

Kubernetes Master IP(s):  147.70.30.2

cormac@pks-cli:~$ pks get-credentials k8s-clus-01

Fetching credentials for cluster k8s-clus-01.
Context set for cluster k8s-clus-01.

You can now switch between clusters by using:
$kubectl config use-context <cluster-name>


cormac@pks-cli:~$ kubectl config get-contexts
CURRENT   NAME             CLUSTER          AUTHINFO                               NAMESPACE
*         k8s-clus-01      k8s-clus-01      9b774997-572f-4d22-b6f5-039d74bb4004

cormac@pks-cli:~$

cormac@pks-cli:~$ kubectl get nodes

NAME                                   STATUS   ROLES    AGE   VERSION
2236372c-911e-4784-952b-1fe7bf2067f5   Ready    <none>   21h   v1.14.5
5b1f2193-25fe-4960-9f94-1ea632cf370b   Ready    <none>   21h   v1.14.5
e3a3132b-ae3f-436b-bf01-e485e81701bd   Ready    <none>   21h   v1.14.5

cormac@pks-cli:~$

cormac@pks-cli:~$ bosh vms

Using environment '147.70.10.11' as user 'director'

Task 36
Task 35
Task 36 done

Task 35 done

Deployment 'pivotal-container-service-9be5f01641747546dce7'

Instance                                                        Process State  AZ           IPs           VM CID                                   VM Type     Active
pivotal-container-service/cec0ab96-6f39-46f1-a277-8a49181dc244  running        PKS-MGMT-AZ  147.70.10.12  vm-7776bf04-4eb1-4a7c-881a-7f7e5ef11247  large.disk  true

1 vms


Deployment 'service-instance_ce2567f3-233e-4ec6-b1e4-fafbbecbadeb'

Instance                                     Process State  AZ               IPs       VM CID                                   VM Type      Active
master/5cb6458c-7f6e-4b1c-acfd-f71ad8b0fab7  running        PKS-MGMT-AZ      20.0.0.2  vm-4fb9f77b-8099-4eaf-a191-82db05726640  medium.disk  true
worker/14ad78ba-d6f5-443c-ae17-1e6b3af4dd1e  running        PKS-Compute1-AZ  20.0.0.3  vm-990bc1a5-057e-44f0-a71f-e36714a5a3a5  medium.disk  true
worker/26f89b75-40ec-4972-912f-7f5af600d981  running        PKS-Compute3-AZ  20.0.0.5  vm-3bdc0233-a0b7-4033-9798-a8ae33fd6bad  medium.disk  true
worker/7e544293-cfb6-4291-b348-c6b3aaf6fd5c  running        PKS-Compute2-AZ  20.0.0.4  vm-5cb95724-9929-428d-8b92-182f16a77d17  medium.disk  true

4 vms

Succeeded

Great! We have successfully deployed our first Kubernetes cluster using Enterprise PKS in a Workload Domain in VMware Cloud Foundation. If you want to try building some containerized applications, and become familiar with Kubernetes in general, feel free to look at some of the Kubernetes 101 introductions that I put together previously.

Step 6. Availability Zones

During the PKS deployment in part 9, one of the features we added to the deployment was Availability Zones (AZ). We created a Management AZ for the PKS components and we created 3 x Compute AZs for the Kubernetes masters and workers. By default, the small plan, which was used in step 4 above to create our first K8s cluster, has a single master and 3 workers nodes. This plan places the master in the Management AZ, and each of the workers in its own Compute AZ. Here is a snippet of plan 1 taken from the PKS tile:

And if we look at how that K8s cluster got deployed in vSphere, we see the following in the inventory, with only a single worker in each Compute AZ, and the master placed in the Management AZ. The sc-xxx are stem-cells, best described as templates for the deployment of the K8s nodes by PKS.

 You may enable/modify any of the other plans in Enterprise PKS to meet your Kubernetes needs. For example, if you configured a plan that had 3 masters as well as 3 worker nodes, you could also place each master in its own Compute AZ, as shown here:

Here is how that looks in the vSphere inventory after a K8s cluster with that plan has been deployed. This shows the nodes from both the original small plan deployment of K8s as well as our new plan with 3 x masters. We can now see a new master and a new worker in each of the Compute AZs by comparing it to the previous vSphere inventory screenshot.

Hopefully that shows you the power and simplicity of having Enterprise PKS (and K8s) integrated with Availability Zones on vSphere.

Caveat

I came across one issue with Kubernetes running on Enterprise PKS automatically deployed on top of VCF. When I tried creating my first stateful application on this cluster, utilizing the VMware VCP (vSphere Cloud Provider), I encountered the following error when trying to provision Persistent Volumes (PV):

The first question I had was where was the folder pks_worker_vms defined. I went back and checked my BOSH and PKS tiles in PKS Ops Manager, and I found it here.

It is the Stored VM Folder in the PKS Tile for the Kubernetes Cloud Provider. Now, this should be set to the same as the VM Folder in the BOSH tile which is pcf_vms.

However, it is very easy to resolve. One option is to simply create the expected folder and subfolder pks_worker_vms/xxxx in your vSphere inventory and everything will work as expected. The alternate is to change the Stored VM Folder to pcf_vms instead of pks_worker_vms to make the PKS >  Stored VM Folder the same as the BOSH >  VM Folder, then apply the changes in Ops Manager. Speaking to some PKS experts, the latter is the preferred approach.

This issue has been reported internally. We will get it addressed asap and create a Knowledgebase article on how to resolve it if you do come across it.

Check out the full range of VMware Cloud Foundation blog posts here.

The post Getting started with VCF Part 10 – Kubernetes deployment appeared first on CormacHogan.com.

I recently wanted to deploy a newer versions of Kubernetes to see it working with our Cloud Native Storage (CNS) feature. Having assisted with the original landing pages for CPI and CSI, I’d done this a few times in the past. However, the deployment tutorial that we used back then was based on Kubernetes version 1.14.2. I wanted to go with a more recent build of K8s, e.g. 1.16.3. By the way, if you are unclear about the purposes of the CPI and CSI, you can learn more about them on the landing page, here for CPI and here for CSI.

OK, before we begin I do want to make it clear that the instructions are still completely valid. The only issue is that with the later releases of K8s, some of the “Kinds” (K8s components) have changed. This will become clear as we go through the process. The other thing that has caught me out personally is the requirement to use hard-coded names for the configurations files, both for the CPI and the CSI. I’ll show you how issues manifest themselves when these hard-codes names are not used.

I am going to continue to use the Flannel for my CNI. However, there are a number of modifications required to the flannel YAML. I’ll highlight these to you as we go along.

1. Changes to K8s Master/Control Plane Deployment

The obvious change here is that we need to install K8s tool that is v1.16.3 rather than v1.14.2. This is quite straight forward to do. For the tools, change the apt install to the correct versions:

# apt install -qy kubeadm=1.16.3-00 kubelet=1.16.3-00 kubectl=1.16.3-00

For the correct K8s distribution, modify the master’s kubeadminit.yaml,as shown here:

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
useHyperKubeImage: false
kubernetesVersion: v1.16.3

After deploying the K8s control plane/master as per the tutorial, you will see the following message displayed:

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Until a network has been deployed, the control plane/master node stays in a NotReady state.

root@k8s-master-01:~# kubectl get nodes
NAME            STATUS     ROLES    AGE   VERSION
k8s-master-01   NotReady   master   66s   v1.16.3

You can see the reason for the NotReady via a kubectl describe of the node:

root@k8s-master-01:~# kubectl describe node k8s-master-01
Name:               k8s-master-01
Roles:              master
...

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 06 Mar 2020 09:04:37 +0000   Fri, 06 Mar 2020 09:01:33 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 06 Mar 2020 09:04:37 +0000   Fri, 06 Mar 2020 09:01:33 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 06 Mar 2020 09:04:37 +0000   Fri, 06 Mar 2020 09:01:33 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 06 Mar 2020 09:04:37 +0000   Fri, 06 Mar 2020 09:01:33 +0000   KubeletNotReady              runtime network not ready: NetworkReady=false \
reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

We need to provide a CNI (Container Network Interface), and for the purposes of the tutorial, we have been using flannel. However, with us using a newer version of Kubernetes, this is where things start to get interesting.

2. Changes to Flannel deployment in K8s 1.16.3

Let’s do the very first step from the tutorial and see what happens when we apply the kube-flannel yaml:

root@k8s-master-01:~# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml": \
no matches for kind "PodSecurityPolicy" in version "extensions/v1beta1"
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml": \
no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml": \
no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml": \
no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml": \
no matches for kind "DaemonSet" in version "extensions/v1beta1"
unable to recognize "https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml": \
no matches for kind "DaemonSet" in version "extensions/v1beta1"

root@k8s-master-01:~#

The errors above are as a result of API deprecation in Kubernetes 1.16. PodSecurityPolicy is now is the policy/v1beta API and DaemonSet is now in apps/v1 API. After downloading and making the appropriate changes to the kube-flannel.yaml, I ran it again.

root@k8s-master-01:~# kubectl apply -f kube-flannel-test.yaml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel unchanged
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg unchanged
error: error validating "kube-flannel-test.yaml": error validating data: \
ValidationError(DaemonSet.spec): missing required field "selector" \
in io.k8s.api.apps.v1.DaemonSetSpec; if you choose to ignore these errors, turn validation off with --validate=false

root@k8s-master-01:~#

This error is as a result of changes made to DaemonSet. Because DaemonSet has been updated to use apps/v1 instead of extensions/v1beta1, the apps/v1 API version requires a selector to be provided in the DaemonSet spec. So once again, I modified the flannel YAML file to include a selector in the DaemonSet spec as follows:

Before:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel

After:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-arm
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      name: kube-flannel
  template:
    metadata:
      labels:
        name: kube-flannel
        tier: node
        app: flannel

After making that changed, I deployed my flannel yaml once more. Success!

root@k8s-master-01:~# kubectl apply -f kube-flannel-test.yaml
podsecuritypolicy.policy/psp.flannel.unprivileged configured
clusterrole.rbac.authorization.k8s.io/flannel unchanged
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg unchanged
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

root@k8s-master-01:~#

At least, I thought it was success. However, when I checked on my master node, it still wasn’t ready.

root@k8s-master-01:~# kubectl get nodes
NAME            STATUS     ROLES    AGE   VERSION
k8s-master-01   NotReady   master   23m   v1.16.3


root@k8s-master-01:~# kubectl describe nodes k8s-master-01
Name:               k8s-master-01
Roles:              master
...
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 06 Mar 2020 09:24:37 +0000   Fri, 06 Mar 2020 09:01:33 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 06 Mar 2020 09:24:37 +0000   Fri, 06 Mar 2020 09:01:33 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 06 Mar 2020 09:24:37 +0000   Fri, 06 Mar 2020 09:01:33 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 06 Mar 2020 09:24:37 +0000   Fri, 06 Mar 2020 09:01:33 +0000   KubeletNotReady              runtime network not ready: NetworkReady=false \
reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Hmm. Still not ready. It was time to look at the node’s kubelets logs and see if there is something else wrong. This is what I found:

root@k8s-master-01:~# journalctl -xe | grep kubelet
...
Mar 06 09:25:54 k8s-master-01 kubelet[19505]: W0306 09:25:54.953423   19505 cni.go:202] \
Error validating CNI config &{cbr0  false [0xc0003a53e0 0xc0003a5780]...

After a quick search, I found an issue reported on github. It was entitled network not ready after `kubectl apply -f kube-flannel.yaml` in v1.16 cluster #1178 The solution is to add a cniVersion number to the flannel yaml file. So the following entry was added to my flannel yaml:

Before:

data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "plugins": [
        {
          "type": "flannel",

After:

data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion”: "0.3.1”,
      "plugins": [
        {
          "type": "flannel",

Reapply the flannel yaml once more, and check the status of my master node. Finally – its Ready!

root@k8s-master-01:~# kubectl apply -f kube-flannel-test.yaml
podsecuritypolicy.policy/psp.flannel.unprivileged configured
clusterrole.rbac.authorization.k8s.io/flannel unchanged
clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
serviceaccount/flannel unchanged
configmap/kube-flannel-cfg configured
daemonset.apps/kube-flannel-ds-amd64 unchanged
daemonset.apps/kube-flannel-ds-arm64 unchanged
daemonset.apps/kube-flannel-ds-arm unchanged
daemonset.apps/kube-flannel-ds-ppc64le unchanged
daemonset.apps/kube-flannel-ds-s390x unchanged
root@k8s-master-01:~#

root@k8s-master-01:~# kubectl get nodes
NAME            STATUS   ROLES    AGE   VERSION
k8s-master-01   Ready    master   35m   v1.16.3

root@k8s-master-01:~#

OK – we can now get on with deploying the worker nodes. Of course, if you didn’t want to mess about with all this flannel related stuff, you could of course choose another pod network addon, such as Calico. If you want the new flannel yaml manifest, you can download it with all the changes from here.

3. Changes to Worker Node Deployments

There is very little change required here. You must simply make sure that you deploy the newer kubectl, kubeadm and kubelet versions – 1.1.6.3 rather than 1.14.2. We already seen how to do that for the master node in step 1 – repeat this for the workers. I added to workers to my cluster:

root@k8s-master-01:~# kubectl get nodes
NAME            STATUS   ROLES    AGE   VERSION
k8s-master-01   Ready    master   39m   v1.16.3
k8s-worker-01   Ready    <none>   41s   v1.16.3
k8s-worker-02   Ready    <none>   20s   v1.16.3

root@k8s-master-01:~#

The remainder of the deployment is pretty much the same as before. As you go about deploying the CPI (Cloud Provider Interface) and the CSI (Container Storage Interface), some of the manifests reference  sub-folder called 1.14. It is ok to continue using these manifests, even for later versions (1.16.3) of K8s.

4. A word about CPI and CSI configuration files

The remaining steps involve deploying the CPI and CSI drivers so that you can allow your Kubernetes cluster to consume vSphere storage, and have the usage bubbled up in the vSphere Client. However, something that has caught numerous people out, inclusing myself, is that the CPI and CSI configuration files are hard-coded; for CPI you must use a configuration file called vsphere.conf and for CSI you must use a configuration file called csi-vsphere.conf. I did a quick exercise to show the sorts of failures you would expect to see if different configuraton filenames are used.

4.1 CPI failure scenario

After deploying the CPI yaml manifests, the first thing you would check is to make sure the cloud-controller-manager pod deployed successfully. You wil see something like this if it did not:

root@k8s-master-01:~# kubectl get pods -n kube-system
NAME                                       READY   STATUS             RESTARTS   AGE
..
kube-apiserver-k8s-master-01               1/1     Running            0          13m
kube-controller-manager-k8s-master-01      1/1     Running            0          13m
kube-proxy-58pkq                           1/1     Running            0          9m10s
kube-proxy-gqzpc                           1/1     Running            0          13m
kube-proxy-qx4rd                           1/1     Running            0          7m7s
kube-scheduler-k8s-master-01               1/1     Running            0          13m
vsphere-cloud-controller-manager-4792t     0/1     CrashLoopBackOff   5          5m50s

Let’s look at the logs from the pod. I’ve just cut a few snippets out of the complete log output:

root@k8s-master-01:~# kubectl logs vsphere-cloud-controller-manager-4792t -n kube-system
I0305 13:54:17.317921       1 flags.go:33] FLAG: --address="0.0.0.0"
I0305 13:54:17.318366       1 flags.go:33] FLAG: --allocate-node-cidrs="false"
I0305 13:54:17.318383       1 flags.go:33] FLAG: --allow-untagged-cloud="false"
...
I0305 13:54:17.318492       1 flags.go:33] FLAG: --cloud-config="/etc/cloud/vsphere.conf"
I0305 13:54:17.318496       1 flags.go:33] FLAG: --cloud-provider="vsphere"
...
F0305 13:54:18.517986       1 plugins.go:128] Couldn't open cloud provider configuration \
/etc/cloud/vsphere.conf: &os.PathError{Op:"open", Path:"/etc/cloud/vsphere.conf", Err:0x2}
goroutine 1 [running]:
k8s.io/klog.stacks(0x37e9801, 0x3, 0xc0007fe000, 0xb4)
        /go/pkg/mod/k8s.io/klog@v0.3.2/klog.go:900 +0xb1
k8s.io/klog.(*loggingT).output(0x37e98c0, 0xc000000003, 0xc000423490, 0x3751216, 0xa, 0x80, 0x0)
        /go/pkg/mod/k8s.io/klog@v0.3.2/klog.go:815 +0xe6
k8s.io/klog.(*loggingT).printf(0x37e98c0, 0x3, 0x2008bd6, 0x32, 0xc0005818a0, 0x2, 0x2)
        /go/pkg/mod/k8s.io/klog@v0.3.2/klog.go:727 +0x14e
k8s.io/klog.Fatalf(...)

So ensure that you use the vsphere.conf filename for the CPI.

4.1 CSI failure scenario

CSI is similar to CPI in that it requires a hard-coded configuration filename. Here is what you might observe if the csi-vsphere.conf name is not used. Here is the log snippet taken from the CSI controller pods.

root@k8s-master-01:~# kubectl logs vsphere-csi-controller-0 -n kube-system vsphere-csi-controller
I0305 16:39:14.210288       1 config.go:261] GetCnsconfig called with cfgPath: /etc/cloud/csi-vsphere.conf
I0305 16:39:14.210376       1 config.go:265] Could not stat /etc/cloud/csi-vsphere.conf, reading config params from env
E0305 16:39:14.210401       1 config.go:202] No Virtual Center hosts defined
E0305 16:39:14.210415       1 config.go:269] Failed to get config params from env. Err: No Virtual Center hosts defined
E0305 16:39:14.210422       1 service.go:103] Failed to read cnsconfig. Error: stat /etc/cloud/csi-vsphere.conf: no such file or directory
I0305 16:39:14.210430       1 service.go:88] configured: csi.vsphere.vmware.com with map[mode:controller]
time="2020-03-05T16:39:14Z" level=info msg="removed sock file" path=/var/lib/csi/sockets/pluginproxy/csi.sock
time="2020-03-05T16:39:14Z" level=fatal msg="grpc failed" error="stat /etc/cloud/csi-vsphere.conf: no such file or directory"

root@k8s-master-01:~#

The take-away is to not deviate from the hard-coded configuration filenames for both the CPI and CSI.

The post Deploying flannel, vSphere CPI and vSphere CSI with later versions of Kubernetes appeared first on CormacHogan.com.

On March 10th 2020, we saw a plethora of VMware announcements around vSphere 7.0, vSAN 7.0, VMware Cloud Foundation 4.0 and of course the Tanzu portfolio. The majority of these announcements tie in very deeply with the overall VMware company vision which is any application on any cloud on any device. Those applications have traditionally been virtualized applications. Now we are turning our attention to newer, modern applications which are predominantly container based, and predominantly run on Kubernetes. Our aim is to build a platform which can build, run, manage, connect and protect both traditional virtualized applications and modern containerized applications.

And so we come to vSAN File Services. vSAN File Services, fully integrated into vSAN 7, now has the ability to create file shares alongside block volumes on your vSAN datastore. These file shares are accessible via NFS v3 and NFS v4.1 protocols. It is very simple to setup and use, as you would expect with vSAN. It is also fully integrated with vSAN Virtual Object Viewer and Health, with its own set of checks for both File Server Agents and the actual File Shares themselves. So let’s take a look at vSAN File Service in action.

Disclaimer: “To be clear, this post is based on a pre-GA version of the vSAN Native File Services product. While the assumption is that not much if anything will change between now and when the product becomes generally available, I want readers to be aware that features and UI may still change before that.”

Simple Deployment Mechanism

Let’s begin with a look at how to deploy vSAN File Services. It really is very straight-forward, and is enabled in much the same way as previous vSAN services. We’ll begin with a 3-node vSAN 7 cluster, as shown below.

Next, in the vSphere Client, navigate to the vSAN Services section where you will now see vSAN File Services. It is currently Disabled – let’s Enable it.

This launches the vSAN File Service wizard. The introduction is interesting as it is showing File Shares being consumed by both VMs and Containers. Like I said, we are building a platform for both virtualized and modern applications. In this post, we will concentrate on virtual machines, but we will look at containers consuming vSAN File Service in a future post.

How File Services is implemented on vSAN is that we implement a set of File Server Agents, managed by vSphere ESX Agent Manager.. These are very lightweight, opinionated virtual appliances running Photon OS and Docker. They behave as NFS servers and provide access to the file shares. When File Services is Generally Available, you will be able to download the agent image directly from the internet (My VMware I assume). Alternatively, for sites that do not have internet access, you can download the File Service Agent OVF offline.

Now an interesting item to highlight in the Introduction screen is the Distributed File System sitting between the NFS File Shares and vSAN. This Distributed File System component is how we share the configuration state (file share names, file share redirects, etc) across all of the File Service Agents. If any of the File Service Agents were to fail on one of the ESXi hosts in the vSAN cluster, this enables the agent to be restarted on any other host in the vSAN cluster and continue to have access to all of the metadata around its file shares, etc.

The next step is to provide some information around the domain, such as security mode and DNS. I have called the file service domain “vsan-fs” in this example. At present, in the vSAN 7 release, only AUTH_SYS is supported as the NFS authentication method. This means that the agents enforce file system permissions for users of the NFS file share using Unix-like permissions, the same as it uses for local users.

Now we get to the Networking section, where you select the VM network on which the File Server Agents are deployed, and of course how the file shares are accessed. Usual stuff like Subnet Mask and Gateway are also added at this point.

And the final step is to provide an IP Pool for the File Service Agents. What is nice about this is that you can auto-fill the IP addresses if you have a range of IPs available for the agents. There is also an option to lookup the DNS names of the File Service Agents rather than type them in manually. You can then select which IP address is the primary and this is the address that will be used for accessing all your NFS v4.1 shares. At the back-end, the connection to the share could be redirected to a different File Server Agent using NFSv4 referral. NFSv4 referral is a NFSv4 specific mechanism that allows one File Server Agent to redirect a client connection/request to another File Server Agent as it crosses a directory (hard-coded as /vsanfs/ in Native vSAN File Services). We will see this behaviour in action shortly.

Review your selection and click Finish to start the deployment.

This will lead to a number of tasks being initiated in your vSphere environment, which I’ve captured here:

However, when the tasks complete, you will have a new vSAN File Service Node/Agent per vSAN node (3 agents for my 3 node vSAN cluster).

And if we take a look at the vSAN File Service in vSAN Services, we now see it enabled with various additional information, most of which we provided during the setup process:

One other feature to point out here is the ‘Check Upgrade’ option. Should a new release of the File Service Agent become available, you can use this feature to provide a rolling upgrade of the agents whilst maintaining full availability of all the file shares. And just like we saw with the initial deployment, if your vCenter has internet connectivity, you can pull down the new versions automatically. If your vCenter does not have internet connectivity, you can download the new OVF offline and upload it manually.

Now that we have enabled vSAN File Services, let’s go ahead and create our first file share.

Simple File Share Creation Mechanism

A new menu item now appears in Configure  > vSAN called File Service Shares. You can see the domain (vsan-fs) that we created in the setup, as well as the supported protocols (NFS v3 and 4.1) as well as the Primary IP. One of the File Service Agents was designated as the primary during setup, and this is the IP address used for mounting all NFS v4.1 shares. The connections are redirected internally to the actual IP address of the Agent presenting the share using NFS referral. Click on ADD to create the first share.

In this wizard, we provide a name for the share. We also pick a vSAN storage policy for the share. The policy can include all of the features that we associated with block storage policies for performance and availability. We also have the options of setting a warning threshold and a hard quota. A hard quota means that we will not be able to place any additional data on the share once the quota is reached, even though there may still be free space available on the share. Finally, you can also add labels on the file share. Labels are heavily utilized in the world of Kubernetes, and so it can be useful to mark a share as being used for a particular application. We will discuss how Kubernetes can consume these file shares in another post.

The only other step is to specify which networks can access the file share. You can make the share wide open – accessible from any network, or you can specify which particular networks can access the file share and what permissions they can have e.g. Read Only or Read Write. The Root squash checkbox is a security technique used heavily in NFS which ‘squashes’ the permissions of any root user who mounts and uses the file share.

Review the file share and click Finish to create it.

When the file share is created, it will appear in the File Service Shares as shown below.

Again, very straight-forward and the file shares are created with just a few clicks in the UI. And of course, all shares are deployed on the vSAN datastore, and are configured with both availability and performance through Storage Policy Based Management (SPBM).

I also want to highlight that if you have a need to change the soft or hard quota of a file share, add some labels, or change the network access, simply select the file share, click Edit, and make your changes on the fly. Pretty simple once again.

vSAN Object View and Health Integration

Since the file share is instantiated on vSAN, you can simply click on the name of the file share from the file shares view and see the file share details in the Virtual Objects view as shown below.

You can also click on the “View placement Details” link below to see the layout of the underlying vSAN object to see which hosts and which physical storage devices are used for placing the components of the file share object. And of course, the Heath Service has been extended to include a number of additional health checks specific to the vSAN File Service.

Now that the File Share has been built, lets consume it from an external Linux (Ubuntu 18.04) VM.

Consuming File Shares (NFS v3 and NFS v4.1)

vSAN File Shares can be mounted as either NFS v4.1 or NFS v3. The main point to get across here is how to use the NFS v4.1 primary IP mechanism with NFSv4 referral. If I use the standard showmount command, we do not see the root share folder that is needed for NFSv4 referral (hard-coded to /vsanfs/):

# showmount -e 10.27.51.194
Export list for 10.27.51.194:


# showmount -e 10.27.51.195
Export list for 10.27.51.195:
/first-share 10.27.51.0/24


# showmount -e 10.27.51.196
Export list for 10.27.51.196:

As we can see the file share /first-share is on my second File Service Agent with the IP Address 195. To mount this file share as NFS v4.1 via the Primary IP and the NFSv4 referral mechanism, I need to include the root share (/vsanfs) in the mount path even though it is not shown in the showmount output. This will then refer the client request to mount the file share to the appropriate File Service Agent.

Below is is the command to mount the file share using NFS v4.1. NFS v4.1 is also the default mount version if no protocol if specified. In that case, the client will negotiate the mount protocol with the server and mount with the highest matching protocol, which for vSAN 7 Native File Services is NFS v4.1:

# mount 10.27.51.194:/vsanfs/first-share /newhome/

or

# mount -t nfs -o vers=4.1 10.27.51.194:/vsanfs/first-share /newhome/


# mount | grep newhome
10.27.51.194:/first-share on /newhome type nfs4 (rw,relatime,vers=4.1\
,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2\
,sec=sys,clientaddr=10.27.51.18,local_lock=none,addr=10.27.51.194)

If you want to mount the share using NFSv3, the root share (/vsanfs) is not used in the mount path. In this case, you mount directly from the ‘owning’ File Share Agent which you can determine from the showmount command. There is no referral/redirect mechanism with this protocol. Here is an example of mounting directly from one of the File Share Agents (196, the one which owns the file share) over NFS v3.

# mount -t nfs -o vers=3 10.27.51.196:/second-share /newhome/

# mount | grep newhome
10.27.51.196:/second-share on /newhome type nfs (rw,relatime,vers=3\
,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2\
,sec=sys,mountaddr=10.27.51.196,mountvers=3,mountport=20048,mountproto=udp,\
local_lock=none,addr=10.27.51.196)

And don’t worry if you can’t remember the correct syntax. In the vSphere UI, simply select the file share. Note that a few additional items now appear in the tool bar, such as ‘Copy URL‘:

Next, click Copy Url and decide if you want to mount the share using NFSv3 or NFSv4.

First, lets choose NFSv4. Now you are shown the connection string that should be used to mount this file share using the referral /vsanfs/ root directory:

Or if I decided that I wanted to mount as NFSv3, this is the connection string that should be used:

This time there is no referral directory included; you need to mount the share directly from the ‘owning’ agent.

Quota Alarms and Events

As we saw in the file share creation, we have the ability to set a share warning threshold. If we exceed this threshold, we see the ‘Usage over quota’ field turn orange in the UI.

There is also a share hard quota. Once you reach that you won’t be able to write any more data to the share. It will fail with a Disk quota exceeded error as show below.

If the hard quota is exceed, you’ll also get the following alarm displayed in the vSphere UI:

Conclusion

We’ve been working on a Native File Service solution for vSAN for a very long time. Its great to finally see it make it to release. I think the engineering team have done an excellent job with (a) keeping the simplicity approach that defines vSAN and also (b) the integration with other key vSAN features like the health check and virtual objects view.

We have also seen how easy it is for a virtual machine to consume a file share, and how useful this can be for traditional virtualized applications. But what about containers? Can they consume these file shares just as easily? Watch this space and I’ll share with you how we are planning to extend our CSI driver and CNS (Cloud Native Storage) implementation to consume vSAN File Service, allowing both traditional virtualized applications and newer, modern, containerized applications to consume this service – any app, and cloud, any device.

To learn more about vSAN 7 features, check out the complete vSAN 7 Announcement here. To learn more about VMware Cloud Foundation 4, of which vSAN 7 is an integral part, check out the complete VCF 4 Announcement here.

The post Native File Services for vSAN 7 appeared first on CormacHogan.com.

A few weeks back, just after the vSphere 7.0 launch event, I wrote an article about Native File Services in vSAN 7.0. I had a few questions asking why we decided on NFS support in this initial release, and not something like SMB or some other protocol. The reason is quite straight-forward. We are positioning vSAN as a platform for both traditional virtual machine workloads and newer containerized workloads. We chose NFS to address a storage requirement in Kubernetes, namely a way to share Persistent Volumes between Pods. To date, the vSphere CSI driver only provisioned block based Persistent Volumes which were Read-Write-Once, meaning that only one Pod could consume the volume at a time. With the new vSphere CSI driver, VMware now supports the dynamic provisioning of file shares on vSAN 7.0 with the File Service feature enabled. These shares can then be consumed by container workloads. In this post, I want to show you how this works.

Disclaimer: “To be clear, this post is based on a pre-GA version of the both vSAN 7 File Services and the new vSphere CSI driver. While the assumption is that not much should change between the time of writing and when the products becomes generally available, I want readers to be aware that feature behaviour and the user interface could still change before then.”

I’m not going to spend any time talking about the deployment and configuration of vSAN 7 File Services as this has already been covered in the earlier post. In that post, we saw how to manually create an NFS file share. In this post, we will see how a file share is dynamically instantiated when a Kubernetes application requests a Read-Write-Many (RWX) Persistent Volume using a StorageClass that refers to vSAN file services, and the Kubernetes(K8s) cluster is deployed on vSphere with the new CSI driver.

ReadWriteOnce (Block) Persistent Volumes Revisited

Let’s start with a quick deployment of an application that uses Block Persistent Volumes, just so we can look at how that behaves on vSphere/vSAN. This will show you some of the extensive features we have in the vSphere UI for managing and monitoring containers, as well as demonstrate the issues with trying to share a block PV between two Pods.

What I am doing here is:

1. Create a StorageClass which is using a (block) RAID1 policy, implying the Persistent Volume will be instantiated on my vSAN datastore as a block VMDK.
2. Create a ReadWriteOnce Persistent Volume Claim (PVC), to manually create a Persistent Volume (PV).
3. Create a Pod what uses that PVC which in turn means that it gets the PV associated with the PVC.
4. Launch another Pod with the same PVC which will demonstrate that the Pods cannot share the same RWO block volume.

Here are the YAML manifests that I will use for this demo. The kind field describes what each object is. This first manifest is the StorageClass, which put simply, select a vSphere datastore in which to place any provisioned Persistent Volumes. The provisioner field is a reference to the VMware CSI driver. The storagepolicyname field refers to an SPBM policy is vSphere. In this case, that policy will result in selecting my vSAN datastore.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: vsan-block-sc
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "RAID1"

This is the Persistent Volume Claim manifest. It results in the creation of a 2Gi Persistent Volume (VMDK) on vSphere storage reference by the StorageClassName. Since this StorageClassName refers to the StorageClass above, this PV will be created on my vSAN datastore.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: block-pvc
spec:
  storageClassName: vsan-block-sc
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

This is the Pod manifest. It will create a simple “busybox” Pod which mounts the volume referenced by the claimName block-pvc which is our PVC above. This will result in a 2G VMDK attached and mounted on /mnt/volume1 in the Pod.

apiVersion: v1
kind: Pod
metadata:
  name: block-pod-a
spec:
  containers:
  - name: block-pod-a
    image: "k8s.gcr.io/busybox"
    volumeMounts:
    - name: block-vol
      mountPath: "/mnt/volume1"
    command: [ "sleep", "1000000" ]
  volumes:
    - name: block-vol
      persistentVolumeClaim:
        claimName: block-pvc

This is the second Pod which is identical to the first. It will also try to attach and mount the same PV. However since this is a RWO persistent volume, and it is already attached and mounted to the first Pod, we will see the  attach operation of the PV to this Pod fail.

apiVersion: v1
kind: Pod
metadata:
  name: block-pod-b
spec:
  containers:
  - name: block-pod-b
    image: "k8s.gcr.io/busybox"
    volumeMounts:
    - name: block-vol
      mountPath: "/mnt/volume1"
    command: [ "sleep", "1000000" ]
  volumes:
    - name: block-vol
      persistentVolumeClaim:
        claimName: block-pvc

Let’s create each object, and monitor the vSphere client for changes/updates. Let’s create the StorageClass first.

$ ls
block-pod-a.yaml block-pod-b.yaml block-pvc.yaml block-sc.yaml

$ kubectl apply -f block-sc.yaml
storageclass.storage.k8s.io/vsan-block-sc created

$ kubectl get sc
NAME          PROVISIONER             AGE
vsan-block-sc csi.vsphere.vmware.com  4s

Next step is to create the PVC and query the resulting PV.

$ kubectl apply -f block-pvc.yaml
persistentvolumeclaim/block-pvc created

$ kubectl get pvc
NAME      STATUS VOLUME                                   CAPACITY  ACCESS MODES STORAGECLASS   AGE
block-pvc Bound  pvc-6398f749-f93e-4dfd-8121-cf91926d642e 2Gi       RWO          vsan-block-sc  10s

$ kubectl get pv
NAME                                      CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM              STORAGECLASS   REASON  AGE
pvc-6398f749-f93e-4dfd-8121-cf91926d642e  2Gi       RWO           Delete          Bound   default/block-pvc  vsan-block-sc          13s

And this volume is now visible in the Container Volumes view in the vSphere client, thanks to CNS.

We can see that it has been instantiated as a RAID-1 (Mirror) on the vSAN datastore. If we click on the volume name, and then View Placement Details, we can see that the actual layout of the PV across vSAN hosts and vSAN disks, allowing vSphere admins to track right down and see exactly which infrastructure components are backing a volume.

Another useful feature for the vSphere admin is to identify “stranded” persistent volumes, in other words identify PVs that are not attached to any Pod. We have not yet deployed any Pod, so let’s see how the PV appears in the vSAN > Capacity > Usage breakdown. Here I have expanded User objects. Note that it is identifying “Block container volumes (not attached to a VM)”. By VM, we mean Kubernetes worker node. This is why it appears in the User object report and not the VM report. For a Pod to mount a volume, that volume needs to be attached to the Kubernetes node, and the kubelet running inside the node will then take care of making it available to the Pod. Since we do not have any Pod yet to consume this volume, the PV’s capacity appears under the “not attached” section.

I can click on the User objects in the chart to get a more detailed view.

Now the numbers here are pretty small since I have only provisioned a single PV and vSAN always provisions thin disks. But hopefully it gives you a good idea of the level of detail we can get, and how this can really help a vSphere admin who is managing vSphere infrastructure which is hosting Kubernetes clusters.

Let’s now go ahead and provision our first Pod.

$ kubectl apply -f block-pod-a.yaml
pod/block-pod-a created

$ kubectl get pod
NAME        READY STATUS            RESTARTS AGE
block-pod-a 0/1   ContainerCreating 0        4s

$ kubectl get pod
NAME        READY STATUS  RESTARTS AGE
block-pod-a 1/1   Running 0        16s

The Pod is ready. We can use the following command to examine the events associated with the creation of the Pod.

$ kubectl get event --field-selector involvedObject.name=block-pod-a
LAST SEEN TYPE    REASON                  OBJECT           MESSAGE
1m        Normal  Scheduled               pod/block-pod-a  Successfully assigned default/block-pod-a to k8s-worker7-02
1m        Normal  SuccessfulAttachVolume  pod/block-pod-a  AttachVolume.Attach succeeded for volume "pvc-6398f749-f93e-4dfd-8121-cf91926d642e"
1m        Normal  Pulling                 pod/block-pod-a  Pulling image "k8s.gcr.io/busybox"
1m        Normal  Pulled                  pod/block-pod-a  Successfully pulled image "k8s.gcr.io/busybox"
1m        Normal  Created                 pod/block-pod-a  Created container block-pod-a
1m        Normal  Started                 pod/block-pod-a  Started container block-pod-a

It looks like the volume was successfully attached. Let’s log onto it and check to see if it mounted the volume successfully.

$ kubectl exec -it block-pod-a -- /bin/sh

/ # mount | grep /mnt/volume1
/dev/sdb on /mnt/volume1 type ext4 (rw,relatime,data=ordered)

/ # df /mnt/volume1
Filesystem  1K-blocks  Used  Available  Use%  Mounted on
/dev/sdb    1998672    6144  1871288    0%    /mnt/volume1
/ #

This looks good as well. And now that the volume has been attached to a VM (the K8s worker node where the Pod was scheduled), the Usage breakdown now changes how it is reported. It now appears in the VM category, under Block container volumes (attached to a VM). So it is not longer stranded. Very useful.

Now let’s show you what happens if we try to deploy another Pod that also attempts to attach and mount this ReadWriteOnce block volume:

$ kubectl apply -f block-pod-b.yaml
pod/block-pod-b created

$ kubectl get pod
NAME         READY  STATUS            RESTARTS  AGE
block-pod-a  1/1    Running           0         15m
block-pod-b  0/1    ContainerCreating 0         7s

$ kubectl get event --field-selector involvedObject.name=block-pod-b
LAST SEEN  TYPE     REASON              OBJECT           MESSAGE
16s        Normal   Scheduled           pod/block-pod-b  Successfully assigned default/block-pod-b to k8s-worker7-01
16s        Warning  FailedAttachVolume  pod/block-pod-b  Multi-Attach error for volume "pvc-6398f749-f93e-4dfd-8121-cf91926d642e" Volume is already used by pod(s) block-pod-a

As expected, we failed to attach the volume to Pod B since RWO volumes cannot be simultaneously attached to multiple Pods, and the volume is already in use by Pod A. Since Kubernetes is an eventually consistent system, it continues to try and reconcile this request to create the container. However this will never succeed, and will remain in this ContainerCreating state until we tell it to stop.

Let’s now finish with RWO block volumes and take a closer look at RWX file volumes.

ReadWriteMany (File) Persistent Volumes

Just like we did with block volumes, I am going to do a similar demonstration with file volumes. The steps will be:

1. Create a StorageClass which is using a (file share) RAID1 policy, implying the Persistent Volume will be instantiated on my vSAN datastore as an NFS file share.
2. Create a ReadWriteOnce Persistent Volume Claim (PVC), to manually create a Persistent Volume (PV).
3. Create a Pod what uses that PVC which in turn means that it gets the PV associated with the PVC.
4. Launch another Pod with the same PVC which will demonstrate that these Pods can share the same RWX volume.

Here are the manifests. They are similar to the block manifests in many ways, with a few crucial differences. There are now some NFS specific parameters included in the StorageClass manifest, such as filesystem type, root squash, permissions and which networks are allowed to mount the share. Here we are allowing any network to mount the share with Read-Write permissions. There is also no root squash enabled.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: vsan-file-sc
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "RAID1"
  csi.storage.k8s.io/fstype: nfs4
  allowroot: "true"
  permission: "READ_WRITE"
  ips: "*"

The PersistentVolumeClaim manifest is almost identical to the previous block example. The major difference is that the accessMode is now set to ReadWriteMany whereas previously it was set to ReadWriteOnce.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: file-pvc
spec:
  storageClassName: vsan-file-sc
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi

The Pod manifests are also very similar to before. The idea here is to demonstrate that both Pods can attach, mount and write to the same ReadWriteMany shared volume simultaneously.

apiVersion: v1
kind: Pod
metadata:
  name: file-pod-a
spec:
  containers:
  - name: file-pod-a
    image: "k8s.gcr.io/busybox"
    volumeMounts:
    - name: file-vol
      mountPath: "/mnt/volume1"
    command: [ "sleep", "1000000" ]
  volumes:
    - name: file-vol
      persistentVolumeClaim:
        claimName: file-pvc

This is the second Pod which will try to share the RWX persistent volume with the first Pod.

apiVersion: v1
kind: Pod
metadata:
  name: file-pod-b
spec:
  containers:
  - name: file-pod-b
    image: "k8s.gcr.io/busybox"
    volumeMounts:
    - name: file-vol
      mountPath: "/mnt/volume1"
    command: [ "sleep", "1000000" ]
  volumes:
    - name: file-vol
      persistentVolumeClaim:
        claimName: file-pvc

Let’s begin the same way as we did for block, first creating the storage class, then the PVC. We will then see what changes have occurred on the vSphere client UI. I won’t remove the block objects created previously so that you can compare them to the objects created for file.

$ kubectl apply -f file-sc.yaml
storageclass.storage.k8s.io/vsan-file-sc created


$ kubectl get sc
NAME            PROVISIONER              AGE
vsan-block-sc   csi.vsphere.vmware.com   76m
vsan-file-sc    csi.vsphere.vmware.com   5s


$ kubectl apply -f file-pvc.yaml
persistentvolumeclaim/file-pvc created


$ kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
block-pvc   Bound    pvc-6398f749-f93e-4dfd-8121-cf91926d642e   2Gi        RWO            vsan-block-sc   74m
file-pvc    Bound    pvc-216e6403-fd0c-48ea-bd05-c245a54d72ac   2Gi        RWX            vsan-file-sc    25s


$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS    REASON   AGE
pvc-216e6403-fd0c-48ea-bd05-c245a54d72ac   2Gi        RWX            Delete           Bound    default/file-pvc    vsan-file-sc             17s
pvc-6398f749-f93e-4dfd-8121-cf91926d642e   2Gi        RWO            Delete           Bound    default/block-pvc   vsan-block-sc            74m

Note the access mode on the PVC and PV. The new volume is RWX, meaning ReadWriteMany. Let’s see what changes have occurred in the vSphere client after running the above commands. If we navigate to Configure > vSAN > File Service Shares, we observe that a new dynamically create file share now exists, of type Container Volume.

And if we click on the name of the volume, we once again get taken to the Monitor > vSAN > Virtual Objects view where we can see the object placement details, just like we could for the block volume previously. And the container volume also appears in the Container Volumes view as well.

Let’s now deploy the first of our Pods and see what we get. Notice that the second block Pod is still ‘ContainerCreating’ from earlier, and will continue to do so indefinitely.

$ kubectl apply -f file-pod-a.yaml
pod/file-pod-a created


$ kubectl get pod
NAME          READY   STATUS              RESTARTS   AGE
block-pod-a   1/1     Running             0          68m
block-pod-b   0/1     ContainerCreating   0          52m
file-pod-a    1/1     Running             0          22s


$  kubectl get event --field-selector involvedObject.name=file-pod-a
LAST SEEN   TYPE     REASON                   OBJECT           MESSAGE
51s         Normal   Scheduled                pod/file-pod-a   Successfully assigned default/file-pod-a to k8s-worker7-01
51s         Normal   SuccessfulAttachVolume   pod/file-pod-a   AttachVolume.Attach succeeded for volume "pvc-216e6403-fd0c-48ea-bd05-c245a54d72ac"
42s         Normal   Pulled                   pod/file-pod-a   Container image "gcr.io/google_containers/busybox:1.24" already present on machine
42s         Normal   Created                  pod/file-pod-a   Created container file-pod-a
42s         Normal   Started                  pod/file-pod-a   Started container file-pod-a

The Pod was created and the volume was attached successfully, as per the events above. OK – let’s do something simple on the volume to show that we can read and write to it.

$ kubectl exec -it file-pod-a -- /bin/sh 

/# mount | grep /mnt/volume1 
10.27.51.31:/52890fc4-b24d-e185-f33c-638eabfa5e25 on /mnt/volume1 type nfs4 \
(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,\
timeo=600,retrans=2,sec=sys,clientaddr=192.168.232.1,local_lock=none,addr=10.27.51.31) 

/ # df /mnt/volume1 
Filesystem           1K-blocks      Used Available Use% Mounted on 
10.27.51.31:/52890fc4-b24d-e185-f33c-638eabfa5e25                      
                     4554505216         0 4554402816   0% /mnt/volume1

/ # cd /mnt/volume1

/mnt/volume1 # mkdir CreatedByPodA

/mnt/volume1 # cd CreatedByPodA

/mnt/volume1/CreatedByPodA # echo "Pod A was here" >> sharedfile

/mnt/volume1/CreatedByPodA # cat sharedfile
Pod A was here

The next step is to launch our second Pod, and make sure it can share access and mount the same volume.

$ kubectl get pods
NAME         READY  STATUS            RESTARTS  AGE
block-pod-a  1/1    Running           0         116m
block-pod-b  0/1    ContainerCreating 0         100m
file-pod-a   1/1    Running           0         9m11s


$ kubectl apply -f file-pod-b.yaml
pod/file-pod-b created


$ kubectl get pods
NAME         READY  STATUS            RESTARTS  AGE
block-pod-a  1/1    Running           0         116m
block-pod-b  0/1    ContainerCreating 0         100m
file-pod-a   1/1    Running           0         9m29s
file-pod-b   1/1    Running           0         6s


$ kubectl get event --field-selector involvedObject.name=file-pod-b
LAST SEEN  TYPE    REASON                 OBJECT           MESSAGE
27s        Normal  Scheduled               pod/file-pod-b  Successfully assigned default/file-pod-b to k8s-worker7-02
27s        Normal  SuccessfulAttachVolume  pod/file-pod-b  AttachVolume.Attach succeeded for volume "pvc-216e6403-fd0c-48ea-bd05-c245a54d72ac"
19s        Normal  Pulled                  pod/file-pod-b  Container image "gcr.io/google_containers/busybox:1.24" already present on machine
18s        Normal  Created                 pod/file-pod-b  Created container file-pod-b
17s        Normal  Started                 pod/file-pod-b  Started container file-pod-b

The first thing to notice are the events. These all look good and it would appear that the second Pod, Pod B, has been able to successfully mount the RWX Persistent Volume, even when it is already attached and mounted to Pod A. Excellent! The final step is to log into Pod B to check if we can also read and write to the volume.

$ kubectl exec -it file-pod-b -- /bin/sh

/ # mount | grep /mnt/volume1
10.27.51.31:/52890fc4-b24d-e185-f33c-638eabfa5e25 on /mnt/volume1 type nfs4 \
(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,\
timeo=600,retrans=2,sec=sys,clientaddr=192.168.213.193,local_lock=none,addr=10.27.51.31)

/ # df | grep /mnt/volume1
4554263552 0 4554161152 0% /mnt/volume1

/ # df /mnt/volume1
Filesystem           1K-blocks      Used  Available   Use%  Mounted on
10.27.51.31:/52890fc4-b24d-e185-f33c-638eabfa5e25
                     4554263552     0     4554161152  0%    /mnt/volume1

/ # cd /mnt/volume1

/mnt/volume1 # ls
CreatedByPodA

/mnt/volume1 # cd CreatedByPodA/

/mnt/volume1 # ls
sharedfile

/mnt/volume1/CreatedByPodA # cat sharedfile
Pod A was here

/mnt/volume1/CreatedByPodA # echo "Pod B was here too" >> sharedfile

So we can see the directory and file that were created on the share from Pod A. We just did a simple update to the file originally created by Pod A from Pod B. If we now flip back to the shell session we have open on Pod A, let’s see if we are able to see the update to the sharedfile.

/mnt/volume1/CreatedByPodA # cat sharedfile
Pod A was here
Pod B was here too

Looks like both Pods are successfully able to read and write to this RWX persistent volume, dynamically provisioned by vSAN File Services.

I hope this gives you a good idea about how vSAN File Services can be used for both traditional virtual machine workloads as well as newer containerized workloads. We saw how file shares on vSAN can be dynamically provisioned as persistent volumes, along with a storage class that reflects the desired availability and performance of the volume through storage policies. We also saw some neat UI enhancements although I haven’t shown them all in this post. The main take-away is that is doesn’t matter if a developer is using block based RWO volume or file based RWX volumes provisioned from vSAN, the vSphere administrator has full visibility into how the developer is consuming vSphere storage. This allows good communication to develop between a vSphere Administrator and the Kubernetes persona, whether that is a developer or a K8s admin. Either way, this is enabling a culture of Dev-Ops to happen in the organization.

If you want to learn more about vSAN File Service and the new CSI driver for file shares, check out this blog post from my good pal Myles. It includes a nice demo into how vSAN File Services works with our new CSI driver and integrates with CNS.

The manifests used for this demo are available on this public github repo.

The post Read-Write-Many Persistent Volumes with vSAN 7 File Services appeared first on CormacHogan.com.

In my previous post on VCF 4.0, we looked at the steps involved in deploying vSphere with Kubernetes in a Workload Domain (WLD). When we completed that step, we had rolled out the Supervisor Control Plane VMs, and installed the Spherelet components which allows our ESXi hosts to behave as Kubernetes worker nodes. Let’s now take a closer look at that configuration, and I will show you a few simple Kubernetes operations to get you started on the Supervisor Cluster in vSphere with Kubernetes.

Disclaimer: “Like my earlier posts, I want to be clear, this post is based on a pre-GA version of the vSphere with Kubernetes. While the assumption is that not much should change between the time of writing and when the product becomes generally available, I want readers to be aware that feature behaviour and the user interface could still change before then.”

Supervisor Cluster Overview

Let’s begin with a look at the inventory of my Supervisor Cluster. I have my 3 physical ESXi hosts which now behave as my Kubernetes worker nodes, I have my 3 control plane virtual machines for running the Kubernetes API server and other core K8s components. I also have my NSX-T Edge cluster, deployed to my WLD via VCF 4.0 SDDC Manager.

Another interesting way to view this deployment is via the Kubernetes CLI command, kubectl. Let’s do that next. First, we need to find the Load Balancer IP address assigned to my Supervisor cluster. To find that, navigate to vSphere Client > Workload Management > Clusters. Here we will find the Control Plane Node IP address, highlighted below. This address has been allocated from the Ingress range that was configured during the NSX-T Edge deployment.

If we now point a browser at this IP address, the following landing page is displayed, which importantly for us, include the Kubernetes CLI Tools, kubectl and kubectl-vsphere.

Once these tools are downloaded to a desktop/workstation, we can use them to login to our Supervisor Cluster ‘context’, and query the cluster details.

$ kubectl-vsphere login --vsphere-username administrator@vsphere.local --server=20.0.0.1 --insecure-skip-tls-verify

Password: *********
Logged in successfully.

You have access to the following contexts:
   20.0.0.1

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context <workload name>`

$ kubectl get nodes -o wide

NAME                               STATUS   ROLES    AGE    VERSION                    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                 KERNEL-VERSION      CONTAINER-RUNTIME
422b0e3d39560ed4ea84169e6b77d095   Ready    master   2d2h   v1.16.7-2+bfe512e5ddaaaa   10.244.0.132   <none>        VMware Photon OS/Linux   4.19.84-1.ph3-esx   docker://18.9.9
422b341edb277e79d5ec7da8b50bf31a   Ready    master   2d2h   v1.16.7-2+bfe512e5ddaaaa   10.244.0.130   <none>        VMware Photon OS/Linux   4.19.84-1.ph3-esx   docker://18.9.9
422ba68078f4aaf4c9ba2afb27d4e945   Ready    master   2d2h   v1.16.7-2+bfe512e5ddaaaa   10.244.0.131   <none>        VMware Photon OS/Linux   4.19.84-1.ph3-esx   docker://18.9.9
esxi-dell-g.rainpole.com           Ready    agent    2d2h   v1.16.7-sph-30923be        10.27.51.7     <none>        <unknown>                <unknown>           <unknown>
esxi-dell-j.rainpole.com           Ready    agent    2d2h   v1.16.7-sph-30923be        10.27.51.122   <none>        <unknown>                <unknown>           <unknown>
esxi-dell-l.rainpole.com           Ready    agent    2d2h   v1.16.7-sph-30923be        10.27.51.124   <none>        <unknown>                <unknown>           <unknown>

$

We can see the same 3 x Master VMs and 3 x Worker nodes seen from the vCenter inventory. Another thing we can do is query the namespaces that exist on the Supervisor Cluster, which is quite a bit different to native Kubernetes, and even Enterprise PKS.

$ kubectl get ns
NAME                      STATUS   AGE
default                   Active   2d3h
kube-node-lease           Active   2d3h
kube-public               Active   2d3h
kube-system               Active   2d3h
vmware-system-capw        Active   2d3h
vmware-system-csi         Active   2d3h
vmware-system-kubeimage   Active   2d3h
vmware-system-nsx         Active   2d3h
vmware-system-registry    Active   2d3h
vmware-system-tkg         Active   2d3h
vmware-system-ucs         Active   2d3h
vmware-system-vmop        Active   2d3h

$

And if you were so inclined, you could take a look at all of the currently running Pods on the Supervisor Cluster by running kubectl get pods -A.

Creating our first Namespace

Let’s head back to the vSphere Client, and navigate to Workload Management > Namespaces. This will take us to the following landing page where we can create our first namespace.

A namespace here is simply a way of dividing the resources of the cluster between multiple consumers. For all intents and purpose, we can look at namespaces on the Supervisor Cluster as being very similar to vSphere Resource Pools. When creating the namespace you need to choose a Cluster object from the inventory, and then provide a name and optional description. In this example, I have simply called it cormac-ns.

On successfully creating the namespace, you will be placed into the following view in the vSphere Client. The Status window tells us that the namespace was created successfully. It also has some inventory information, as well as a link to the tools page, which we have already visited. Capacity and Usage allows us to edit CPU, Memory and Storage limits for the namespace. Tanzu Kubernetes is only applicable when we deploy a Tanzu Kubernetes Grid (TKG) guest cluster. We will revisit this in an upcoming post.

I think much of the settings and information displayed here is quite straight-forward to digest. I will mentioned Storage however. You will need to ‘Add Storage’ to the namespace by selecting a Storage Policy from the available list of policies created on this vSphere environment. I am going to keep things simple by selecting the default vSAN storage policy, but of course you can get more granular, depending on the number of hosts that are in the cluster, as well as the data services that you have enabled on the vSAN cluster.

After the storage policy (or policies) has been assigned to the namespace, it becomes available as a Kubernetes Storage Class in our namespace. Let’s return to the kubectl CLI and demonstrate this. First, we will see that our new namespace (cormac-ns) is created. Then we will logout and log back in to the Supervisor Cluster to pick up our new ‘context’. Note that there may be other ways to do this, but this is the way that I found to work for me. When we log back in, we can see that the cormac-ns namespace is already set as the context (* against it), so no need to change the context after logging back in.

$ kubectl get ns
NAME                      STATUS   AGE
cormac-ns                 Active   15m
default                   Active   2d3h
kube-node-lease           Active   2d3h
kube-public               Active   2d3h
kube-system               Active   2d3h
vmware-system-capw        Active   2d3h
vmware-system-csi         Active   2d3h
vmware-system-kubeimage   Active   2d3h
vmware-system-nsx         Active   2d3h
vmware-system-registry    Active   2d3h
vmware-system-tkg         Active   2d3h
vmware-system-ucs         Active   2d3h
vmware-system-vmop        Active   2d3h


$ kubectl config get-contexts
CURRENT   NAME       CLUSTER    AUTHINFO                                   NAMESPACE
*         20.0.0.1   20.0.0.1   wcp:20.0.0.1:administrator@vsphere.local


$ kubectl-vsphere logout
Your KUBECONFIG context has changed.
The current KUBECONFIG context is unset.
To change context, use `kubectl config use-context <workload name>`
Logged out of all vSphere namespaces.


$ kubectl-vsphere login --vsphere-username administrator@vsphere.local --server=20.0.0.1 --insecure-skip-tls-verify

Password:
Logged in successfully.

You have access to the following contexts:
   20.0.0.1
   cormac-ns

If the context you wish to use is not in this list, you may need to try
logging in again later, or contact your cluster administrator.

To change context, use `kubectl config use-context <workload name>`


$ kubectl config get-contexts
CURRENT   NAME        CLUSTER    AUTHINFO                                   NAMESPACE
          20.0.0.1    20.0.0.1   wcp:20.0.0.1:administrator@vsphere.local
*         cormac-ns   20.0.0.1   wcp:20.0.0.1:administrator@vsphere.local   cormac-ns


$ kubectl get sc
NAME                          PROVISIONER              AGE
vsan-default-storage-policy   csi.vsphere.vmware.com   2d2h

$

With the final command above, displaying of Storage Classes, we can see that the default vSAN storage policy is now available for use by Persistent Volumes in the Supervisor Cluster. The Storage Window in the Namespace Summary in the vSphere UI will report the number of Persistent Volume Claims that exist in the namespace. We will see this shortly.

We will leave the namespace for the present and do one more action. We will now enable the Harbor Image Registry on the Supervisor Cluster. Once that is enabled, we will push an image up to the Image Registry and use it to deploy our first application on the Supervisor Cluster.

Enable Harbor Image Registry

A really neat feature of the Supervisor Cluster is that it includes an embedded Harbor Image Registry for container images. To enable the Image Registry, select the Cluster object in the vCenter inventory then navigate to Configure > Namespaces > Image Registry as shown below:

Click on the Enable Harbor button. It prompts you to choose a Storage Policy for the persistent volumes required by the Harbor application (I chose vSAN default once again). Immediately a new namespace is created called vmware-system-registry, and this is where the PodVMs that back the Harbor application are deployed. There are 7 PodVMs  created in total, and when deployment completes, a link to the Harbor UI is provided.

And because the Cloud Native Storage (CNS) is fully integrated with the Supervisor Cluster, we can see the 4 x Persistent Volumes that were deployed on behalf of the Harbor application. Pretty cool, huh?

Let’s also take a look at the Harbor Image Registry namespace as that has some interesting info now.

We can see the Storage window now report 4 PVCs, as well as a Storage Policy (vSAN default) with a 200GB limit. The Capacity and Usage is also showing some usage. If we click on the EDIT LIMITS and expand the Storage, we see that although there is no overall Storage Limit, the amount of storage that can be consumed by the vSAN default storage policy is 200GB.

The last item to point out is that there are now 7 running Pods (blue line). Some Pods were in a state of Pending (yellow line), but now they are running – this is normal. There are no failed Pods (red line).

Push an Image to Harbor

The next step is to push an image to the Harbor Image Registry. The first step is to establish trust. To do that, you will need to login to Harbor UI using SSO credentials, and get the Registry Certificate from the Repository, as shown below.

This certificate will then have to be copied to your docker certificates location on your local laptop/desktop/workstation. On my Ubuntu 17.10, this location is in /etc/docker/certs.d. You then create a directory to match the FQDN or IP address of your Harbor Registry and place the downloaded registry cert there. Otherwise you will hit x509 errors try to communicate to the registry.

With the cert in place, we can pull down an image from an internet repository, tag it and push it to our Harbor registry. The push command format is registry/project/repository:tag. So my Harbor registry is 20.0.0.2 (another Ingress IP address provided by NSX-T). My project is cormac-ns, created automatically for every namespace in the Supervisor Cluster. The repository is busybox and since no tag is provide, :latest is used.

$ docker pull k8s.gcr.io/busybox
Using default tag: latest
latest: Pulling from busybox
a3ed95caeb02: Pull complete
138cfc514ce4: Pull complete
Digest: sha256:d8d3bc2c183ed2f9f10e7258f84971202325ee6011ba137112e01e30f206de67
Status: Downloaded newer image for k8s.gcr.io/busybox:latest
k8s.gcr.io/busybox:latest


$ docker tag k8s.gcr.io/busybox 20.0.0.2/cormac-ns/busybox


$ docker push 20.0.0.2/cormac-ns/busybox
The push refers to repository [20.0.0.2/cormac-ns/busybox]
5f70bf18a086: Pushed
44c2569c4504: Pushed
latest: digest: sha256:d2af0ba9eb4c9ec7b138f3989d9bb0c9651c92831465eae281430e2b254afe0d size: 1146
$

The image is now stored in our Harbor Image Registry and can be referenced by any manifest YAML files that we use to deploy apps on the Supervisor Cluster.

Create our first StatefulSet

Just to complete this first exercise with vSphere with Kubernetes, I’ll create a small application in my own namespace. I will download the image, push it to Harbor and then use it within my own manifest file. The application will be my trusty Cassandra StatefulSet application that I’ve used many times before. You’ve seen how to do the push and pull to Harbor, so I won’t repeat the steps here. Instead, I will simple create the service and the 3 x replica StatefulSet for the application and query the StatefulSet, including its Service, Pods and PVCs, after deployment.

Let’s use kubectl to query this StatefulSet after it has been deployed:

$ kubectl get sts
NAME        READY   AGE
cassandra   3/3     6m57s

This is the Service:

$ kubectl get svc
NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
cassandra   ClusterIP   10.96.0.35   <none>        9042/TCP   8m38s

These are the Pods:

$ kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          6m56s
cassandra-1   1/1     Running   0          6m8s
cassandra-2   1/1     Running   0          91s

And finally the PVCs, which also appear in CNS of course:

$ kubectl get pvc
NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
cassandra-data-cassandra-0   Bound    pvc-019d8f46-ccc8-402c-a038-6acd36abd103   1Gi        RWO            vsan-default-storage-policy   7m46s
cassandra-data-cassandra-1   Bound    pvc-3143a4d4-0d23-482a-8720-5e2912cd3e33   1Gi        RWO            vsan-default-storage-policy   6m58s
cassandra-data-cassandra-2   Bound    pvc-a4c5e528-7868-46d0-9df0-b5304ed90925   1Gi        RWO            vsan-default-storage-policy   3m7s

Here are the volumes in CNS, where I filtered on the app:cassandra, so we do not display the Harbor volumes that we already looked at earlier:

Let’s now return to the Namespace view in vSphere Client.  We can now see some more interesting details around Storage, Capacity and Usage, and Pods. There are now 3 running Pods (I requested 3 replicas in my StatefulSet manifest) along with 3 Persistent Volume Claims, which we saw in the CNS view previously.

Pretty neat. OK – I hope this has given you a reasonable appreciation of some of the things that can be done in the Supervisor Cluster. Check back through my previous VCF 4.0 posts if you want to see how easy it was to deploy it in the first place. Stay tuned, and we will take a close look at the Tanzu Kubernetes Grid guest cluster, deployed in a namespace, in a future post.

The post A first look at vSphere with Kubernetes in action appeared first on CormacHogan.com.

Last week I updated my version of VMware Fusion to 11.5.6. If you don’t know about VMware Fusion, it is a VMware product that gives Mac users the ability to run guest virtual machines. One of the new features that I noticed was the inclusion of a new vctl utility (IIRC, it became available first in v11.5.5.).  This is a command line utility for the Nautilus Container Engine which is now part of VMware Fusion. It allows you to work on OCI (Open Container Initiative) containers from your desktop. I decided to take a closer look, and do a few simple container tasks just to see it in action. From what I understand, starting the Nautilus Container Engine will launch a very lightweight virtual machine (CRX) based on VMware Photon OS. This is very similar to how PodVMs are constructed in vSphere with Kubernetes (You can find more CRX details in this earlier blog post on Project Pacific). Once that is up and running, you will be able to use vctl to do various container operations. Let’s have a look.

Getting started

vctl is installed with VMware Fusion. You don’t need to do anything else. Simply open a terminal on your host system and run vctl without any arguments to display the help output.

chogan@chogan-a01 ~ % vctl

vctl - A CLI tool for the Nautilus Container Engine powered by VMware Fusion
vctl Highlights:
• Build and run OCI containers.
• Push and pull container images between remote registries & local storage.
• Use a lightweight virtual machine (CRX VM) based on VMware Photon OS to host a container. Use 'vctl system config -h' to learn more.
• Easy shell access into virtual machine that hosts container. See 'vctl execvm’.

USAGE:
  vctl COMMAND [OPTIONS]

COMMANDS:
  build       Build a container image from a Dockerfile.
  create      Create a new container from a container image.
  describe    Show details of a container.
  exec        Execute a command within a running container.
  execvm      Execute a command within a running virtual machine that hosts container.
  help        Help about any command.
  images      List container images.
  ps          List containers.
  pull        Pull a container image from a registry.
  push        Push a container image to a registry.
  rm          Remove one or more containers.
  rmi         Remove one or more container images.
  run         Run a new container from a container image.
  start       Start an existing container.
  stop        Stop a container.
  system      Manage the Nautilus Container Engine.
  tag         Tag container images.
  version     Print the version of vctl.

Run 'vctl COMMAND --help' for more information on a command.

OPTIONS:
  -h, --help   Help for vctl

chogan@chogan-a01 ~ %

First we need to start the Container runtime using the vctl system start command.

chogan@chogan-a01 ~ % vctl system start
Preparing storage...
Container storage has been prepared successfully under /Users/chogan/.vctl/storage
Preparing container network, you may be prompted to input password for administrative operations...
Password:*************
Container network has been prepared successfully using vmnet: vmnet9
Launching container runtime...
Container runtime has been started.
chogan@chogan-a01 ~ %

Starting the Container Runtime mounts a volume called “Fusion Container Storage” to the host’s desktop. Now that the runtime is started, let’s try to use vctl to deploy a container image.

Pull & Run a Container Image

For the purpose of this test, I’ll use everyone’s favorite stateless application, nginx. I will first pull it down from the docker hub/registry, and then I will run it using the —name option to give it a unique name, and the -t (tty) option to open a terminal to it, and finally a -d (detach) option so that it runs in the background . Finally I’ll connect the the IP address associated with the image and we should see the default nginx landing page. The container’s rootfs is also mounted to the host so you will observe another volume related to nginx container appear on your desktop.

chogan@chogan-a01 ~ % vctl pull nginx
INFO Pulling from index.docker.io/library/nginx:latest
───                                                                                ──────   ────────
REF                                                                                STATUS   PROGRESS
───                                                                                ──────   ────────
index-sha256:b0ad43f7ee5edbc0effbc14645ae7055e21bc1973aee5150745632a24a752661      Done     100% (1862/1862)
manifest-sha256:179412c42fe3336e7cdc253ad4a2e03d32f50e3037a860cf5edbeb1aaddb915c   Done     100% (1362/1362)
layer-sha256:a5e4a503d449887a0be28a2969149e647460aa6013f9ca90e88491aedf84f24e      Done     100% (666/666)
layer-sha256:cb9a6de05e5a22241e25960c7914f11b56c9070ce28b8f69e899236e0d265c50      Done     100% (26401375/26401375)
layer-sha256:bf59529304463f62efa7179fa1a32718a611528cc4ce9f30c0d1bbc6724ec3fb      Done     100% (27092121/27092121)
config-sha256:4bb46517cac397bdb0bab6eba09b0e1f8e90ddd17cf99662997c3253531136f8     Done     100% (7512/7512)
layer-sha256:9513ea0afb9372e5cabc4070c7adda0e8fc4728e0ad362b349fe233480f2e7d8      Done     100% (600/600)
layer-sha256:b49ea07d2e9310b387436861db613a0a26a98855372e9d5e207660b0de7975a7      Done     100% (899/899)
INFO Unpacking nginx:latest...
INFO done

chogan@chogan-a01 ~ % vctl images
────           ─────────────               ────
NAME           CREATION TIME               SIZE
────           ─────────────               ────
nginx:latest   2020-09-03T10:08:16+01:00   51.0 MiB

chogan@chogan-a01 ~ % vctl run --name cornginx -t -d nginx
INFO container cornginx started and detached from current session

chogan@chogan-a01 ~ % vctl ps
────       ─────          ───────                   ──                ─────   ──────    ─────────────
NAME       IMAGE          COMMAND                   IP                PORTS   STATUS    CREATION TIME
────       ─────          ───────                   ──                ─────   ──────    ─────────────
cornginx   nginx:latest   /docker-entrypoint.s...   192.168.143.128   n/a     running   2020-09-03T10:11:07+01:00

And now if we connect to the IP address assigned to our nginx container, we should see the nginx landing page.

Looks good. Now, since the rootfs has been mounted to the desktop, you can very easily modify some of the container’s file contents. For example, if I wanted to change the landing page above, I could open the nginx volume for the container cornginx on my host desktop and navigate to /usr/share/nginx/html and modify the index.html file:

And now if I modify that index.html to something slightly different to the default, simply adding a newline of text to the bottom of the landing page, and saving it:

I have now updated my landing page in the easiest way possible:

Push an image using vctl to a remote container repository

Let’s now look at how we can use vctl to push images to a remote container repository. I have my own personal repo on docker hub that I will use for this exercise. The steps you will see below are the tagging of my local image with the name of the image I want in my repository. I will then push the image to my repo, and then we will examine my repo to see that the image is indeed there.

chogan@chogan-a01 ~ % vctl ps
────       ─────          ───────                   ──                ─────   ──────    ─────────────
NAME       IMAGE          COMMAND                   IP                PORTS   STATUS    CREATION TIME
────       ─────          ───────                   ──                ─────   ──────    ─────────────
cornginx   nginx:latest   /docker-entrypoint.s...   192.168.143.128   n/a     running   2020-09-03T10:11:07+01:00


chogan@chogan-a01 ~ % vctl tag nginx cormachogan/nginx

chogan@chogan-a01 ~ % vctl push cormachogan/nginx -u cormachogan
Password for cormachogan:**********
INFO pushing image: cormachogan/nginx:latest to index.docker.io/cormachogan/nginx:latest
───                                                                                ──────   ────────
REF                                                                                STATUS   PROGRESS
───                                                                                ──────   ────────
manifest-sha256:179412c42fe3336e7cdc253ad4a2e03d32f50e3037a860cf5edbeb1aaddb915c   Done     100% (1362/1362)
layer-sha256:a5e4a503d449887a0be28a2969149e647460aa6013f9ca90e88491aedf84f24e      Done     100% (666/666)
layer-sha256:9513ea0afb9372e5cabc4070c7adda0e8fc4728e0ad362b349fe233480f2e7d8      Done     100% (600/600)
layer-sha256:cb9a6de05e5a22241e25960c7914f11b56c9070ce28b8f69e899236e0d265c50      Done     100% (26401375/26401375)
config-sha256:4bb46517cac397bdb0bab6eba09b0e1f8e90ddd17cf99662997c3253531136f8     Done     100% (7512/7512)
layer-sha256:bf59529304463f62efa7179fa1a32718a611528cc4ce9f30c0d1bbc6724ec3fb      Done     100% (27092121/27092121)
layer-sha256:b49ea07d2e9310b387436861db613a0a26a98855372e9d5e207660b0de7975a7      Done     100% (899/899)
chogan@chogan-a01 ~ %

And the final check to see that the image made it to my docker hub repo:

Looks good to me.

Pull an image using vctl from a remote container repository

OK – let’s now use the image I just pushed to my personal docker hub repo and use that as a container image rather than pulling directly from docker registry. First, lets remove the current container and the 2 images that currently exist, one locally and one on docker hub.

chogan@chogan-a01 ~ % vctl images
────                       ─────────────               ────
NAME                       CREATION TIME               SIZE
────                       ─────────────               ────
cormachogan/nginx:latest   2020-09-03T10:08:16+01:00   51.0 MiB
nginx:latest               2020-09-03T10:08:16+01:00   51.0 MiB


chogan@chogan-a01 ~ % vctl ps -a
────       ─────          ───────                   ──    ─────   ──────    ─────────────
NAME       IMAGE          COMMAND                   IP    PORTS   STATUS    CREATION TIME
────       ─────          ───────                   ──    ─────   ──────    ─────────────
cornginx   nginx:latest   /docker-entrypoint.s...   n/a   n/a     stopped   2020-09-03T10:11:07+01:00

chogan@chogan-a01 ~ % vctl rm cornginx
────       ──────    ──────
NAME       RESULT    REASON
────       ──────    ──────
cornginx   REMOVED

chogan@chogan-a01 ~ % vctl ps -a
────   ─────   ───────   ──   ─────   ──────   ─────────────
NAME   IMAGE   COMMAND   IP   PORTS   STATUS   CREATION TIME
────   ─────   ───────   ──   ─────   ──────   ─────────────

chogan@chogan-a01 ~ % vctl images
────                       ─────────────               ────
NAME                       CREATION TIME               SIZE
────                       ─────────────               ────
cormachogan/nginx:latest   2020-09-03T10:08:16+01:00   51.0 MiB
nginx:latest               2020-09-03T10:08:16+01:00   51.0 MiB

chogan@chogan-a01 ~ % vctl rmi nginx:latest
────           ──────    ──────
NAME           RESULT    REASON
────           ──────    ──────
nginx:latest   REMOVED

chogan@chogan-a01 ~ % vctl images
────                       ─────────────               ────
NAME                       CREATION TIME               SIZE
────                       ─────────────               ────
cormachogan/nginx:latest   2020-09-03T10:08:16+01:00   51.0 MiB

chogan@chogan-a01 ~ % vctl rmi cormachogan/nginx:latest
────                       ──────    ──────
NAME                       RESULT    REASON
────                       ──────    ──────
cormachogan/nginx:latest   REMOVED

chogan@chogan-a01 ~ % vctl images
────   ─────────────   ────
NAME   CREATION TIME   SIZE
────   ─────────────   ────
chogan@chogan-a01 ~ %

At this point, we have no running containers, and the only image that is available is on my personal docker hub repo. Let’s now pull the image from my docker hub, and run it.

chogan@chogan-a01 ~ % vctl pull cormachogan/nginx:latest
INFO Pulling from index.docker.io/cormachogan/nginx:latest
───                                                                                ──────   ────────
REF                                                                                STATUS   PROGRESS
───                                                                                ──────   ────────
manifest-sha256:179412c42fe3336e7cdc253ad4a2e03d32f50e3037a860cf5edbeb1aaddb915c   Done     100% (1362/1362)
layer-sha256:a5e4a503d449887a0be28a2969149e647460aa6013f9ca90e88491aedf84f24e      Done     100% (666/666)
layer-sha256:cb9a6de05e5a22241e25960c7914f11b56c9070ce28b8f69e899236e0d265c50      Done     100% (26401375/26401375)
config-sha256:4bb46517cac397bdb0bab6eba09b0e1f8e90ddd17cf99662997c3253531136f8     Done     100% (7512/7512)
layer-sha256:bf59529304463f62efa7179fa1a32718a611528cc4ce9f30c0d1bbc6724ec3fb      Done     100% (27092121/27092121)
layer-sha256:9513ea0afb9372e5cabc4070c7adda0e8fc4728e0ad362b349fe233480f2e7d8      Done     100% (600/600)
layer-sha256:b49ea07d2e9310b387436861db613a0a26a98855372e9d5e207660b0de7975a7      Done     100% (899/899)
INFO Unpacking cormachogan/nginx:latest...
INFO done

chogan@chogan-a01 ~ % vctl images
────                       ─────────────               ────
NAME                       CREATION TIME               SIZE
────                       ─────────────               ────
cormachogan/nginx:latest   2020-09-03T14:31:57+01:00   51.0 MiB

chogan@chogan-a01 ~ % vctl run --name mynewnginx -t -d cormachogan/nginx:latest
INFO container mynewnginx started and detached from current session

chogan@chogan-a01 ~ % vctl ps -a
────         ─────                      ───────                   ──                ─────   ──────    ─────────────
NAME         IMAGE                      COMMAND                   IP                PORTS   STATUS    CREATION TIME
────         ─────                      ───────                   ──                ─────   ──────    ─────────────
mynewnginx   cormachogan/nginx:latest   /docker-entrypoint.s...   192.168.143.129   n/a     running   2020-09-03T14:37:54+01:00

Again, this all looks good. A nice addition to VMware Fusion for container workloads.

Running a command on a running container

Previously we saw how a container’s volume appears on the desktop when launch via vctl, and how easy it was to access the container’s file and make changes. But let’s say we wanted to run a command on a container. How would we do that? Well, vctl also comes with an exec argument that allows us to run commands on the selected container. Let’s once again use our nginx container, and use the exec argument run a bash shell on it. In this example I include the -i (interactive) option as well as the -t (tty) option. Check out the help on exec for more details. Like before, I am navigating to the location of the index.html nginx landing page, and displaying the contents.

chogan@chogan-a01 ~ % vctl exec -it mynewnginx /bin/bash
root@mynewnginx:/# cd /usr/share/nginx/html
root@mynewnginx:/usr/share/nginx/html# ls
50x.html  index.html
root@mynewnginx:/usr/share/nginx/html# cat index.html
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

root@mynewnginx:/usr/share/nginx/html#

One other nice feature that I have read about is in the Fusion 12 Announcement is an extension to vctl which will allow the deployment of a Kind cluster, which is in essence Kubernetes cluster running in containers. I’m very much looking forward to trying that out when I get a chance.

A number of months back, I wrote an article which looked at how we now provide a Kubernetes in Docker (KinD) service in VMware Fusion 12. In a nutshell, this allows us to very quickly stand up a Kubernetes environment using the Nautilus Container Engine with a very lightweight virtual machine (CRX) based on VMware Photon OS. In this post, I wanted to extend the experience, and demonstrate how we can stand up a simple Nginx deployment. First, we will do a simple deployment.  Then we will extend it to use a Load Balancer service (leveraging MetalLB).

This post will not cover how to launch the Container Engine or KinD with Fusion, since these are both covered in the previous post. Instead we will focus on deploying an Nginx web server. First, let’s look at a sample deployment and service for the Nginx application. Here is a simple manifest, which describes 2 objects; a deployment with 2 replicas (Pods), and a service. These are linked through the use of spec.selector.matchLabels. There is a single container image which presents its web service via port 80.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      app: my-nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    app: my-nginx
spec:
  ports:
    - port: 80
      protocol: TCP
  selector:
    app: my-nginx

Assuming that I have once again used VMware Fusion to launch the Container Engine and KinD,  I can apply the above manifest via kubectl create or kubectl apply via my MacOS terminal. Next, I will look at what objects are created. I should see a deployment, two Pods, two endpoints and a service.

% kubectl apply -f nginx.yaml
deployment.apps/my-nginx created
service/my-nginx created


% kubectl get deploy my-nginx
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
my-nginx   2/2     2            2           25s


% kubectl get pods -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
my-nginx-74b6849986-glqbs   1/1     Running   0          45s   10.244.0.13   kind-control-plane   <none>           <none>
my-nginx-74b6849986-r4vf4   1/1     Running   0          45s   10.244.0.14   kind-control-plane   <none>           <none>


% kubectl get endpoints my-nginx
NAME       ENDPOINTS                       AGE
my-nginx   10.244.0.13:80,10.244.0.14:80   37s


% kubectl get svc my-nginx
NAME       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
my-nginx   ClusterIP   10.97.25.54   <none>        80/TCP    47s

As we can see, the deployment and two Pods are up and running. What is interesting to observe is the networking configuration. The idea behind a deployment is that there can be multiple Pods to provide the service, in this case an Nginx web server. If one of the Pods fails, the other Pod continues to provide the functionality.

Each of the Pods gets its own IP addresses (e.g. 10.244.0.13, 10.244.0.14) from the Pod network range. These IP addresses are also assigned to the endpoints, which can be referenced by the service.

Similarly, the idea of creating a Service is to provide a “front-end” or “virtual” IP address from the Service network range to access the deployment (e.g. 10.97.25.54).  It gets its own unique IP address so that clients of the web server can avoid using the Pod IP/Endpoints. If clients use the Pod IP addresses, then they would lose connectivity to the application (e.g. web server) if that Pod failed. If connectivity is made via the Service, then there is no loss of connectivity if the Pod fails as the Service would redirect the connection to the other Pod IP address/Endpoint.

When a service is created, it typically gets (1) a virtual IP address, (2) a DNS entry and (3) networking rules that ‘proxy’ or redirects the network traffic to the Pod/Endpoint that actually provides the service. When that virtual IP address receives traffic, the traffic is redirected to the correct back-end Pod/Endpoint.

Let’s test the deployment, and see if we can verify that the web service is running. At present, there is no route from my MacOS to either the Pod network (10.244.0.0) or the service network (10.97.25.0). In order to reach them, I can add a static route to them using the IP address of the KinD node as the gateway. You can get the KinD node IP address by simply running a docker ps as shown below:

% docker ps
────                 ─────                                                                                  ───────                   ──               ─────            ──────    ─────────────
NAME                 IMAGE                                                                                  COMMAND                   IP               PORTS            STATUS    CREATION TIME
────                 ─────                                                                                  ───────                   ──               ─────            ──────    ─────────────
kind-control-plane   kindest/node@sha256:98cf5288864662e37115e362b23e4369c8c4a408f99cbc06e58ac30ddc721600   /usr/local/bin/entry...   172.16.255.128   54785:6443/tcp   running   2021-02-10T12:43:03Z

% sudo route add -net 10.244.0.0 -gateway 172.16.255.128
Password: *****
add net 10.244.0.0


% curl 10.244.0.13:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>


% curl 10.244.0.14:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

This looks good – we can get the Nginx web server landing page from both Pods. Let’s now check accessibility via the service. First, let’s remove the route to the Pods, and then add the route to the Service.

% sudo route delete -net 10.244.0.0
delete net 10.244.0.0


% sudo route add -net 10.97.25.0 -gateway 172.16.255.128
add net 10.97.25.0


% curl 10.97.25.54:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

% kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
namespace/metallb-system created


% kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created


% cat config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.1-192.168.1.250


% kubectl apply -f config.yaml
configmap/config created
%

apiVersion: apps/v1 
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: my-ngin
  replicas: 2 
  template:
    metadata:
      labels:
        app: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    app: my-nginx
spec:
  type: LoadBalancer
  ports:
    - port: 80
      protocol: TCP
  selector:
    app: my-nginx

% kubectl apply -f nginx.yaml
deployment.apps/my-nginx created
service/my-nginx created


% kubectl get deploy my-nginx
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
my-nginx   2/2     2            2           4s


% kubectl get pods
NAME                        READY   STATUS    RESTARTS   AGE
my-nginx-74b6849986-z77ph   1/1     Running   0          10s
my-nginx-74b6849986-zlwdh   1/1     Running   0          10s


% kubectl get endpoints
NAME         ENDPOINTS                       AGE
kubernetes   172.16.255.128:6443             98m
my-nginx     10.244.0.18:80,10.244.0.19:80   16s


% kubectl get svc my-nginx
NAME       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
my-nginx   LoadBalancer   10.96.90.176   192.168.1.1   80:31374/TCP   27s

% sudo route add -net 192.168.1.0 -gateway 172.16.255.128
add net 192.168.1.0

% curl 192.168.1.1:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Everything is working as expected. Hopefully that has given you a good idea of how you can use KinD in VMware Fusion (and indeed VMware Workstation) to become familiar with Kubernetes.

I’m delighted to report that my new book, Kubernetes for vSphere Administrators, is now available. It is available in both paper form and as a Kindle eBook. Links to both are provided below.

The links above direct you to Amazon.com. However the book is available in other Amazon marketplaces as well. (If you can’t see the text+images above, this is a link to the paper book and this is a link to the eBook.) I hope readers of this book find it useful. I would be delighted to receive feedback and reviews on the content.

The post New book: Kubernetes for vSphere Administrators now available appeared first on CormacHogan.com.