IBM Cloud Private - 502 Bad Gateway

I saw this: -

after resuming three VMs that host my IBM Cloud Private (ICP) cluster.

Assuming that the VM sleep had borked things, I followed this: -

What is the proper way to shut down ICp?

to shut down and restart Docker and Kubelet: -

sudo systemctl stop kubelet

sudo systemctl stop docker

sudo systemctl start kubelet

sudo systemctl start docker

but to no avail.

I even rebooted the VMs ( having again stopped Docker and Kubelet ), but still no dice :-(

Running this: -

sudo journalctl -e -u kubelet

on the Boot node ( VM ) showed me this: -

Feb 28 09:13:54 icpboot.uk.ibm.com hyperkube[856]: W0228 09:13:54.072049     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: I0228 09:14:04.003480     856 kuberuntime_manager.go:500] Container {Name:iam-policy-decision Image:ibmcom/iam-policy-decision:2.1.0.1 Command:[] Args:
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: n:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: I0228 09:14:04.004425     856 kuberuntime_manager.go:739] checking backoff for container "iam-policy-decision" in pod "auth-pdp-bpk7h_kube-system(b5ddc
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:04.190524     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:04.193761     856 kuberuntime_container.go:191] Non-root verification doesn't support non-numeric user (iam)
Feb 28 09:14:14 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:14.313716     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:24 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:24.350819     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:34 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:34.484130     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:44 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:44.564230     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:54 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:54.641818     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:04 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:04.763905     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:15 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:15.138129     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:25 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:25.268208     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:35 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:35.303902     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:45 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:45.377637     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:55 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:55.508744     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available

However, it may have just been timing, because, after ~10 minutes, things started to improve.

I again hit the URL: -

https://192.168.1.100:8443/console/

and got a login page and, post-login, this: -

Patience is the key !