Moving To Quadlets

-Ben

I moved my website from running caddy as a systemd service to running a container as a systemd service! Otherwise known as Podman Quadlets, podman has a systemd generator that will let you run containers as a systemd service using Unit files that resemble podman run commands (or docker compose yaml) in systemd’s ini file syntax.

The result is a container with systemd running as its supervisor. systemd will start the container when the system starts. It can manage the lifecycle and more. You can interact with your container with typical systemd commands.

Motivation

My deployment method was simple, but very bespoke. I had written my original Caddyfile on the server and that was the only place it lived. My deployment system was a bespoke set of scripts for every site/repo I hosted there (four and counting) and it was all tied together by scp and rsync’ing into the right directories. I would have never been able to recreate it. Maybe, given enough time, but the configurations were so scattered and that Caddyfile still only lived on the single server.

That worked great for me. I was barely comfortable with bash at the time, but caddy made it so simple that I was able to get a website up and keep it running for 3+ years this way. I don’t want to understate this, it was my beginning on a journey running my own operations.

But it’s been three years and that box is out of date. I’ve become very comfortable with bash and containers. I think containers are really great; their runtimes, like Docker and Podman, offer a standard interface to start them, stop them, and more. A Containerfile (or Dockerfile) is an easy way to write down the configuration in a way that makes it trivially repeatable to build.

I want to have a repeatable build that coordinates across my repos and gives me a single command (or small set) to deploy my website. I still want to deploy it from my local machine. But I want to limit my bespoke bash when I can.

My Decisions

I decided to go super simple with the image and bake everything, including the Caddyfile, into my image. I’m coordinating all the different repos I deploy from by making them submodules of my single deployment repo, this lets me deploy easily from my local machine still and has git handle the coordination between repos.

I now have a bespoke script for setting up a brand new Fedora machine to put the newest version of podman on it. It’s still bespoke, but I didn’t have a script at all for that before. My deploy script basically just copies files to the remote machine then runs a couple of systemctl and podman commands. Bespoke, technically, but also basically four lines of bash.

With an image ready to be deployed, how do I deploy it? Docker is fine, but podman seems lighter weight. I’ve been looking at kubernetes for work, and I hear K3s aren’t just for Black Friday but I just don’t feel comfortable with that yet. I know I need a supervisor to make sure my container keeps running. I used systemd with my old configuration (basically without knowing it).

Podman has done some really cool stuff to bridge a container runtime with kubernetes. They’re where I learned what a Pod was and why use one. They can run kube yaml, although I’ve never tried. And they have this thing called Quadlets that lets you run a declarative configuration under systemd. Their unit files are declarative configuration, like a docker compose file or kube yaml, but in systemd’s ini file syntax. Honestly, ini is so super simple to read and write that wasn’t a problem. Podman should give me the easiest way to deploy!

Quadlets

Like most of what I learn from Podman, I really misunderstand the concepts involved at first. When I first learned about Pods, I expected them to behave like a docker compose file. I didn’t realize all those containers shared a network namespace (plus more, of course). When I first tried to use Quadlets, I didn’t realize how much control podman ceded to systemd. Once I run podman quadlet install, I’m basically done with podman. I think that command might even just copy the files to the right place and run systemd daemon-reload. Now, I think podman is providing the generator to turn those Units into Service Units that systemd understands. But having podman is enough, so the quadlet subcommand feels like icing on the cake.

The Trouble

It’s all rosy now that you’re reading this from a quadlet I have deployed, but getting a version of podman with the right support is tough. You need 5.6 to get the quadlet subcommand at all. And 5.7 to get the --replace flag on the install command, useful for making my deploy scripts simpler. Plus, I couldn’t seem to run a container with 5.6 on CentOS Stream. And my local machine is Ubuntu and only has 4.9.3 which doesn’t support quadlets at all! Your options seem to be Arch or Fedora. And Digital Ocean only supports Fedora officially. Then you need to install the specific new version of podman from their updates repo because Fedora only ships with 5.6.2…. Trouble and misunderstandings aside, I learned some stuff and quadlets are a fantastic way to run containers.

Future Plans

Socket Activation for my caddy container. Caddy is already really lightweight, but it’s a powerful seeming technique I’d like to try.

Rolling deployments. Technically, I’ve introduced downtime by restarting a container. With socket activation, systemd can hold the connections for me. And podman auto-update will manage starting the new container with a health check then rolling back to the old one if the health check fails. This means I also need to introduce a health check and maybe better tag versions, but neither will be hard if I find the time.

Some dynamic content. Now I can put anything in a container. I could use multiple containers! The sky feels like the limit and I don’t have to write a whole new set of scripts for every one. Just small updates to what I have. If I add a site visitor counter, you’ll know I’ve done just that. This was another motivator for me.

Why Not Kamal

I use Kamal for deplying things internally, I’ve written a bunch about it here. But it uses docker and it has an extra kamal proxy container for rolling deployments. And it’s a big local dependency. And it’s defaults are very much geared towards a whole web application. It requires a lot more ssh’ing into specific containers to debug. It doesn’t have anything like journald.

My biggest issue: Kamal requires a container registry. I’m just not interested in setting one up or paying for one. With podman, I can podman save | ssh podman load to copy my 80 MB image over. It’s not terribly efficient, but it’s direct and doesn’t require any additional infrastructure. Otherwise, I’ve taken a lot of inspiration from their interface in writing my little bash scripts. I think they’re easy to read.