Huddo Boards & Minio problems – Read before you restart!

Posted on by

Last week I got the unpleasant surprise of a no longer working Huddo Boards for Component pack installation at a customer after I had rebooted my Kubernetes environment. I had to reboot this environment after I updated the Kubernetes certificates. Of course, after a change you immediately think that your problem is related to the change you just made, but in this case the only relation was with the restart, which means that this can happen to everyone running Huddo/Kudos Boards for Component pack or Huddo Boards Docker.

The cause

Huddo Boards uses an open source component for their file storage: Minio. They simply get the latest version of the Minio image from the Docker hub. At least, they did so till recently. Huddo Boards uses the filesystem gateway of Minio. The people behind the Minio project, however, have decided in 2020 to sunset this gateway. They deprecated it in February 2022 with the message that they would remove this gateway in October. On the 29th of October 2022, they did. The resulting messaging in the log file is:

ERROR Unable to initialize backend: found backend fs, expected xl

The message that Minio would be remove the Filesystem gateway was in the Minio logs for quite a while already. But who checks logs of components that are working fine? Not many people apparently as for many organisations, including ISW/Huddo themselves, this change seems to have come as a surprise.

The solution

For now, ISW have chosen the easy way out. They changed their Helm chart to no longer download the latest version of Minio, but the last version that still contained the filesystem gateway. I think it would be good to look for a more future proof solution. Either by replacing Minio with a similar product that contains a filesystem gateway or rewriting their code to use Minio without needing this fs gateway. I don’t know if they’re working on this.

So to prevent this outage of Boards after a restart, make sure to download their latest Helm chart (3.1.4), and run:

helm upgrade kudos-boards-cp <path-to>/microservices_connections/hybridcloud/helmbuilds/kudos-boards-cp-3.1.4.tgz -i -f <path-to>/microservices_connections/hybridcloud/support/kudos-boards/boards-cp.yaml --namespace connections --recreate-pods

Note: this will only work if you’re currently on a Boards version that uses the 3.x Helm charts.

I’m on an old version of Boards and I don’t want to upgrade

This was the case at my customer. The solution would be to backport the fix. This is relatively simple. Untar the helm chart which you used to install your current boards version and look for the file charts/kudos-minio/templates/deployment.yaml. In this file, look for the line:

image: docker.io/minio/minio

and replace it with:

image: docker.io/minio/minio:RELEASE.2022-10-24T18-35-07Z

Tar the file to a tgz again and use this Helm chart to upgrade your installation.

A warning on upgrading

This is not too obvious, so a quick warning on upgrading. If you upgrade to a newer version of Boards, Boards will migrate the current Boards data to a newer structure. Both in the Mongo database and in the Minio storage. Of course, not all newer versions have these schema updates, but you never know which do. This means that upgrading is a one-way process. Once upgraded, you can’t go back by simply downgrading your pods. You will have to restore a backup of the Mongo database and the Minio filesystem if you want to downgrade.

More thoughts on Minio

Another interesting warning message in the Minio logs is:

WARNING: Detected Linux kernel version older than 4.0.0 release, there are some known potential performance problems with this kernel version. MinIO recommends a minimum of 4.x.x linux kernel version for best performance

Many organisations still use CentOS 7 or RHEL 7 as base OS for their Connections environment. HCL is one of them. This message just shows that it’s really time to stop that. RHEL 7/CentOS 7 will go out of support in 2024, but don’t wait for that. If you need to rebuild your Connections environment, go at least for a RHEL 8 install or derrivative.