Handoff Between Release and Production

Most days as software engineers we don’t think of how our release actually makes it to production. If we have one of numerous GitOps integrations, we just push to Github. Maybe someone wrote a script that runs or an action is manually fired. Occasionally you might find a slower release cadence with some more ceremony around it. Only once have I ever “thrown it over the fence” to another engineer dedicated to the job (it was a very old product by that point).

Lots of companies have made it their business to help you deploy because you want to develop an app, not develop a deployment pipeline, right? At least, that’s the general salse pitch I hear. Using their process means releasing on their infrastructure, paying their prices. You may or may not be able to develop well with their system. It might be somewhat of a black box. I don’t want to attribute anything malicious to them, their goal is to make it seem seamless and magical to you. But as engineers, we know that it isn’t magical. And we pride ourselves in being able to take apart the internals when we need (sometimes when we want!).

What I’ve Tried

If you aren’t using a provider, how might you set up a deployment pipeline? For years now with this personal site I had a Digital Ocean Droplet I had set up manually with Caddy, then I just built a Jekyll site on my computer and rsync’d the files up. As rudimentary as it might sound, it worked great for me. I never had any downtime (I would’ve noticed, I think, because it wouldn’t have come back up on its own) and I could focus on writing these posts.

That doesn’t “scale” past static sites, and it doesn’t really tell you what was released when. For some personal projects, I started using Kamal. That worked really great for Rails applications once I got a registry set up, but it didn’t work very well for my static sites. Working with containers was very nice.

Looking at something like Kubernetes, I ended up going with podman quadlets instead. I thought it would be just a bit easier, and I still think it is. And it introduced me to the power of systemd. I had all the power of container deployments and all the power of systemd, so I wrapped all my static sites in a caddy container and shipped it up to my Droplet. This worked great, until I realized I had to deploy all my sites in order to deploy any of them. I couldn’t have multiple containers all sharing ports 80 and 443, so back to the drawing board I went.

Using caddy as a reverse proxy, I can now deploy all my sites individually. Combining the powers of systemd, podman, and caddy, I can deploy static sites and containers with equal ease, and manage them with the same commands. I don’t think I would want to deploy a rails app directly on to the machine, but I shouldn’t ever need to. Containers cover that case amazingly well. Caddy also has some fastcgi directies I might try eventually, to add interactivity to my static sites without a whole web app.

Moving Deployment Control to the App Repo

With those in place, I didn’t want to go back to just rsync’ing the static files back to the Droplet. I wanted something a little more controlled than that. The droplet runs Ubuntu and nfpm makes it trivial to build debain packages. Using a debian package, I can decide where all my files should be placed and installation will run a postinstall script. I can put my caddyfile in place, and systemd unit files, podman quadlets, static files, etc into the debian package. I can rsync that to the server then run dpkg -i to install.

How’s that different that rsync’ing the files up directly? It moves where the control of the release is. My “deploy” scripts now live in an infrastructure repo. But my apps need to be released more often than that. I want to be able to change what the app needs with impunity, without having to modify a different repo. I’ve taken a hard dependency on the expectation of caddy and podman being present, but that doesn’t seem different than other deployment processes. There are usually some release assumptions baked into the app repo. I’ll only need to change the deploy scripts if I change something about the computer it’s running on. Worst case, I change distros, but nfpm makes it trivial to build a rpm or other distro packages.

With this setup, I can even add container images to the debian package and load them with the postinstall script. Or pull them from a registry.

Using systemd and caddy, I can install a sensitive file in /etc then decrypt it at runtime into /run.

I can set up systemd timers or path files to perform backups on a schedule or reload services when files change.

From what I’ve read about Kubernetes, I have 90% of what I would use on my personal sites. And even if I’m not in K8s, I think I’m getting a taste for what they provide and why, if not necessasrily how. But I also don’t need to run the whole service K8s, or even minikube, require.

Versioning

With containers under your control, you might pin to a latest tag so an auto-update picks it up. Debian packages are the first deployment strategy I’ve used that feels like it strongly benefits from version numbers. Maybe I feel I need versioning now because debian pacakge don’t have sha’s of any kind like containers do (checksums, I suppose). Mostly I feel like it’s an important way to talk about releases, so I wanted to have something similar.

I’m using CalVer because nobody is consuming these sites based on SemVer rules. I’m using git tags to keep track of it. 5 lines of bash is all I need to query and calculate the next version.

# find the current/latest tag
git tag -l 'v*' | sed 's/^v//' | sort -V | tail -1

# create next tag
prefix=$(date +%Y.%m)
last=$(git tag -l "v${prefix}.*" | sed "s/^v${prefix}\.//" | sort -n | tail -1)
next=$((${last:-0} + 1))
version="${prefix}.${next}"

Release

Creating a release is a mise task that checks for a clean git worktree, calculates the new version number, runs my build task with the version number, then creates the git tag. The file is not very long, even with checks and confirmation. It builds a file to my local ~/.cache directory, which I think it appropriate since the release could easily be recreated by running the build pipeline again. In a larger system, or if you ran this on your git forge, this might become an artifact you download during deployment. Since my deployment is from my machine, I use ~/.cache as my place to share the file.

Deploy

After I have a file with the version number as the name, my deployment script can simply look in the shared folder, grab the lastest version number, then deploy it. It’s as simple as collecting the necessary information out of my OpenTofu state, resync’ing the package up, and running dpkg -i. If you had a CI pipeline, the rsync might be a wget command. If you wanted more control, you could set up a apt repository. Then your actual deployement would just be an apt update; apt upgrade <pkg>. I use rsync because I’m keeping everything local for now.