From Containers And Back

-Ben

A month or two ago I revamped my deployment process. I was manually scp’ing files on to a server and I wanted something that felt less fragile than that. I’m familiar with containers so I figured I would build a container! I wrapped up my caddy server and decided to use podman because I thought I was familiar with their quadlets and that would be an easier than something like Kubernetes. That sent me down a whole rabbit hole learning about systemd and socket activation then OpenTofu and sops and age. It turns out I didn’t know quadlets very well, but I know them much better now!

TL;DR: infra and apps/sites have different lifecycles; treat them differently. Make sure you can deploy all your apps/sites separately.

I didn’t fully appreciate how scp’ing static files to the server gave me separate deployments for free. I didn’t have my Caddyfile separated out, so they were still interconnected. But my Caddyfile changed so seldom that it wasn’t a big deal.

Deploying a container was a pretty nice artifact to deploy, but I realized then I had to deploy all my sites and services together, the way I had set it up. That was a bad idea, I couldn’t leave anything half finished! I needed to be able to deploy them separately again. Four separate deployments meant I couldn’t have a single artifact to push up. I needed something in front to coordinate my sites. Something that didn’t actually bundle them together so tightly, but each deployment could affect. Enter back in caddy and systemd.

I’m using caddy to front all my other sites/services. This is basically the same way I was doing it before, but this time around I broke the Caddyfile apart. I can have caddy include all the Caddyfiles in a particular directory. Leveraging this, each site can “install” its own caddyfile. If it’s a service and shipping a container, maybe that holds a reverse_proxy directive. Maybe it’s just some static files with file_server or even uses the templates directive. Caddy can terminate tls and then the sky is the limit! [^1]

I also learned a good deal about packaging. Debian packages are basically a couple of tarballs archived together. They hold a tarball that maps on to your system directory and will place files where they appear. It has a postinstall script (among others) that can do anything, including loading container images or reloading systemd’s daemon. A tool like nfpm makes it easy to define how your repo directories map to the tarball. Having a package makes for an easy “handoff” between an site repo and my infra repo. This artifact can affect more of the system it runs on than a container can.

Containers can’t describe how they’re deployed, so you always need something like this. Podman and systemd via quadlets is a great method for deployment. I can wrap those unit files up in a deb package and podman will pull them! The postinstall script can reload the systemd daemon to make sure they’re picked up. Security is only as good as my unit files I guess, but at least I didn’t have to hand write the caddy one.

I now have an infrastruture repo with OpenTofu that sets up the caddy server which will front the rest of my sites and podman because I’m sure I’ll use a container runtime. Debian packages make it easy to install my sites and other softwares as I need them. Having a separate repo is important because I will very seldomly need to change much about the server, but I want to deploy my sites very often!

I can put all the deployment scripts in the infra repo. That includes scripts for all my sites. The sites build their artifact, the deb package, then I can deploy from the infra repo using a mise task. I looked into Ansible and others but didn’t find anythign I liked enough for my very simple scripts, so these are just bash with a slightly nicer handling. This splut should make it easy for me to turn this into GitOps if I ever wanted to. Build the package on a push, then pull from the remote and install. But I don’t have any plans to do that just yet.

Lastly, secrets can go right in deployment artifacts. Using sops and age, I can write unit files that decode them on the server in a very ephermeral way. I’m not sure that it’s as good as podman secrets, but I’m sure I’ll learn how to make it even better in time.

My old stack: caddy, systemd (unwittingly), bash

My new stack: caddy, systemd (knowledgeably), bash, opentofu, mise, podman, nfpm, sops, age, cloud-init

What I lost: nothing, in the end. This was a big knowledge gain and massive deployment improvement

What I gained: the ability to deploy as many sites as I want easily. Knowledge about the Linux Filesystem Hierarchy (FHS). A container runtime on my server. Better secret management

For the time being, this system is flexible enough that I can continue to use my container setup as I migrate my other repos back and break apart that container.

[^1]: The caddy file is really simple: import /etc/caddy/othercaddyfiles/*.caddyfile I can have cloud-init drop it on the new machine