Replacing Docker with Podman – first steps
In 2018, after I finally figured out how I could run both containers and KVM VMs on my Fedora server with both being reachable, I jumped on the container bandwagon. Docker inc was still in its prime, so I invested quite some time to learn everything about Docker containers, Docker swarm, docker-compose scripts etc. My first container was this blog, but I have since containerized most applications which previously were running on my host. Times change though. In 2020 Docker Inc. is in decay and Red Hat dropped support for Docker in RHEL 8 in favour of their own (open source) offerings: OpenShift and Podman. No support in RHEL 8 obviously also means that Docker isn’t necessarily first choice any more in Red Hats’ Fedora. Red Hat promotes Podman as a drop-in replacement for Docker, so it made sense to try and use my current containers with Podman. Let’s start with saying, Podman is not a drop-in replacement. If you noticed that this blog was down during the weekend, you now know why.
So what is Podman? According to podman.io, Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System. Containers can either be run as root or in rootless mode. If we compare that with Docker, Docker has a daemon and Docker can not run containers rootless. Podman can use “Docker” containers, as Docker containers aren’t actually Docker container, but containers which adhere to the Open Container Initiative (OCI) standards. Any container engine that follows these standards, therefore should be able to deal with these containers. In that sense, podman is a drop-in replacement, but as usual the devil is in the details. Docker, being a daemon, starts containers itself. You can create a container from a docker-compose file or a stack file and tell in the script that the container should start when Docker starts. The latter you arrange by a simple ‘systemctl enable docker-ce’ and you’re done. Podman doesn’t work that way. If you want your containers to start on boot, you have to write a systemd service for each of them.
Podman is still very young. It was created in 2018 and only started maturing half way 2019. Still there are many plans to enhance functionality and make it more Docker-like, but it’s not there yet. Being so young, that also means that information is still scarce and scattered. podman.io is a great resource, Red Hat also has documentation on their site on on their developer’s blog. There’s also an IRC channel (#podman) on Freenode where you can ask questions. Still, I had to figure out myself how to autostart my containers which I created from a docker-compose script.
To understand my struggle, I’ll explain a bit more on how Podman works. As an example, a WordPress installation consists of two containers. A container with, usually, Apache and PHP with all wordpress scripts and files and a container with the backend SQL database. In my case MariaDB. A docker-compose script lists these as 2 separate services and docker would create 2 separate containers. I ran my wordpress installation in Docker swarm mode (on a single node, but anyway) so I would see with ‘docker services ls’ the services from my stack file, how many replica’s they had and if they were up etc. Podman doesn’t have a swarm mode. The idea is that you use Kubernetes for that. You can use your stack file as a docker compose file and create your containers with this command:
podman-compose -t 1podfw -f wordpress.yml up
For a docker-compose script, this usually works without problem, but I had to change some parts of my stack file to get this to work. podman container ls / podman ps then shows 2 containers, while underneath it also created a 3rd ‘pause’ container. podman pod ps shows that there’s a podman wordpress pod. The pod consists of the 3 containers. You can stop and start the service with podman stop wordpress and podman start wordpress. podman pod stops and starts both containers at once, while with podman container stop wordpress_app, I can stop just the Apache/PHP container. On boot, I of course want both containers to nicely start. Podman has a built-in command to create a systemd script to start a container:
podman generate systemd <containername or podname>
If I do a podman generate systemd wordpress, podman generates the following systemd file:
# pod-16327ab1be215ef9f7471311f54f7678231eb2f6fe769d822b0eddfe7daad8e8.service # autogenerated by Podman 1.7.0 # Mon Jan 13 14:09:12 CET 2020 [Unit] Description=Podman pod-16327ab1be215ef9f7471311f54f7678231eb2f6fe769d822b0eddfe7daad8e8.service Documentation=man:podman-generate-systemd(1) Requires=container-a0f4a09e9584121f67c67305db748e58d82db3682b78fc3802fd838944f627c8.service container-b917dc29dbce43dabe4e27bbb2a12ac2688ce87604a41cecaad3c84d9a7acefa.service Before=container-a0f4a09e9584121f67c67305db748e58d82db3682b78fc3802fd838944f627c8.service container-b917dc29dbce43dabe4e27bbb2a12ac2688ce87604a41cecaad3c84d9a7acefa.service [Service] Restart=on-failure ExecStart=/usr/bin/podman start 34e8b42572e9dad4b1562def2560a9ef3d7fe906280fad817d2a5fe4a360b7d7 ExecStop=/usr/bin/podman stop -t 10 34e8b42572e9dad4b1562def2560a9ef3d7fe906280fad817d2a5fe4a360b7d7 KillMode=none Type=forking PIDFile=/var/run/containers/storage/overlay-containers/34e8b42572e9dad4b1562def2560a9ef3d7fe906280fad817d2a5fe4a360b7d7/userdata/conmon.pid [Install] WantedBy=multi-user.target # container-b917dc29dbce43dabe4e27bbb2a12ac2688ce87604a41cecaad3c84d9a7acefa.service # autogenerated by Podman 1.7.0 # Mon Jan 13 14:09:12 CET 2020 [Unit] Description=Podman container-b917dc29dbce43dabe4e27bbb2a12ac2688ce87604a41cecaad3c84d9a7acefa.service Documentation=man:podman-generate-systemd(1) RefuseManualStart=yes RefuseManualStop=yes BindsTo=pod-16327ab1be215ef9f7471311f54f7678231eb2f6fe769d822b0eddfe7daad8e8.service After=pod-16327ab1be215ef9f7471311f54f7678231eb2f6fe769d822b0eddfe7daad8e8.service [Service] Restart=on-failure ExecStart=/usr/bin/podman start b917dc29dbce43dabe4e27bbb2a12ac2688ce87604a41cecaad3c84d9a7acefa ExecStop=/usr/bin/podman stop -t 10 b917dc29dbce43dabe4e27bbb2a12ac2688ce87604a41cecaad3c84d9a7acefa KillMode=none Type=forking PIDFile=/var/run/containers/storage/overlay-containers/b917dc29dbce43dabe4e27bbb2a12ac2688ce87604a41cecaad3c84d9a7acefa/userdata/conmon.pid [Install] WantedBy=multi-user.target # container-a0f4a09e9584121f67c67305db748e58d82db3682b78fc3802fd838944f627c8.service # autogenerated by Podman 1.7.0 # Mon Jan 13 14:09:12 CET 2020 [Unit] Description=Podman container-a0f4a09e9584121f67c67305db748e58d82db3682b78fc3802fd838944f627c8.service Documentation=man:podman-generate-systemd(1) RefuseManualStart=yes RefuseManualStop=yes BindsTo=pod-16327ab1be215ef9f7471311f54f7678231eb2f6fe769d822b0eddfe7daad8e8.service After=pod-16327ab1be215ef9f7471311f54f7678231eb2f6fe769d822b0eddfe7daad8e8.service [Service] Restart=on-failure ExecStart=/usr/bin/podman start a0f4a09e9584121f67c67305db748e58d82db3682b78fc3802fd838944f627c8 ExecStop=/usr/bin/podman stop -t 10 a0f4a09e9584121f67c67305db748e58d82db3682b78fc3802fd838944f627c8 KillMode=none Type=forking PIDFile=/var/run/containers/storage/overlay-containers/a0f4a09e9584121f67c67305db748e58d82db3682b78fc3802fd838944f627c8/userdata/conmon.pid [Install] WantedBy=multi-user.target
So we have a systemd file that starts a pod with it’s specific ID. This pod requires 2 containers with their respective IDs. Sure, we can have this systemd file start on boot and start the pod, but if you know anything about containers, you’ll know that a systemd file containing container IDs and pod ids is ridiculous. It goes completely against the whole concept of containerisation where a container is a disposable unit. Every time a container is recreated, it gets a new container ID. I simply don’t get what the idea of the developers was to build this function this way as, in my view, it’s just crap. To improve the systemd file a bit, you could replace the container IDs with the container names (like in my case wordpress for the pod and wordpress_app and wordpress_db for the containers). On recreating the containers, those names at least stay the same so this works. I however ran into a nasty bug in podman which is summarised here. Despite the fact that in this thread it says that the problem was fixed, it wasn’t for me. As a result my pod would not start on boot and would only start if I manually deleted the ip address from /var/lib/cni/networks/podman/. That’s not a workable solution obviously. Another problem was that if there was a problem with a container, it would not get auto-repaired (like it would with Docker in swarm mode and a stack file) and that if a container was missing, the pod would simply not start.
So I had to rethink my strategy on how to auto start a podman pod. In the end I came up with this systemd file:
# container for wordpress.service [Unit] Description=Podman container for wordpress (apache / php) Documentation=https://hub.docker.com/_/wordpress After=wordpressdb [Service] Type=simple TimeoutStartSec=5m #ExecStartPre=-/usr/bin/podman pod prune ExecStartPre=/usr/bin/podman rm -i wordpress_app ExecStartPre=/usr/bin/podman rm -i wordpress_db ExecStartPre=/usr/bin/podman pod rm -i wordpress ExecStart=/usr/bin/podman-compose -t 1podfw -f /virtual/buildfiles/wordpress/docker-compose.yml up ExecStop=-/usr/bin/podman pod stop -t 10 wordpress ExecStop=-/usr/bin/podman rm wordpress_app ExecStop=-/usr/bin/podman stop -t 10 wordpress_db ExecStop=-/usr/bin/podman rm wordpress_db ExecStop=-/usr/bin/podman pod rm wordpress ExecReload=-/usr/bin/podman pod stop -t 15 wordpress ExecReload=-/usr/bin/podman rm wordpress_app ExecReload=-/usr/bin/podman stop -t 10 wordpress_db ExecReload=-/usr/bin/podman rm wordpress_db ExecReload=-/usr/bin/podman pod rm wordpress ExecReload=-/usr/bin/podman-compose -t 1podfw -f /virtual/buildfiles/wordpress/docker-compose.yml up KillMode=none Restart=always RestartSec=30 [Install] WantedBy=multi-user.target
The beauty (my personal opinion) of this script is that it really takes the idea of disposable containers seriously. Every time the service is stopped, the pod with it’s containers is simply removed and recreated when the service starts. This also means that my pod on every reboot is updated to the latest version from the registry (yes, I am foolish enough to simply use wordpress:latest as my image). This script solved my problem of auto-starting my containerised services that consisted of more than one container (wordpress and nextcloud). Next challenge will be to see if my containers that directly use a device on my host (either a usb device or a dvb card) still work.
Just to be clear, the given solution works for me but I don’t know of this is the best solution. Let me know what you think! Do you think it makes sense what I made or are other ways preferable?
