diff --git a/.gitignore b/.gitignore index 8e72e46..3f3c437 100644 --- a/.gitignore +++ b/.gitignore @@ -5,7 +5,7 @@ hugo_stats.json # Temporary lock file while building -/.hugo_build.lock +.hugo_build.lock # Files for testing pagination **/example copy* diff --git a/content/posts/podman-quadlets-and-really-tall-trees.md b/content/posts/podman-quadlets-and-really-tall-trees.md new file mode 100644 index 0000000..865521a --- /dev/null +++ b/content/posts/podman-quadlets-and-really-tall-trees.md @@ -0,0 +1,291 @@ +--- +title = "Podman, quadlets, and really tall trees" +date = 2026-01-04T16:32:07+10:30 +lastmod = +draft: false +--- + +Over this last Christmas break I had nothing but time as I was stuck down with +covid. All my Christmas and new years plans were abruptly cancelled or moved and +I was bed-ridden for a week. So, I decided to make good on a promise that I made +myself some time ago to refresh my homelab. It's something I've been meaning to +do for a while as I had some 13 different VMs some of which running a single + service, many of which could have simply been containers instead. And now + I finally had the time to commit to such an endeavour. + +One of the virtual machines in my homelab (I named it bristlecone for it's age +and gnarled nature) has been around as long as I've been playing with linux. At +some point in 2016 I took the old family desktop and installed Ubuntu server on +it. Originally I only used it as a web server, but as time went on I installed +ownCloud, which I later upgraded to Nextcloud, then I added this service, then +I added that. Before long I wanted more than just a single logical machine to +play with. My solution was to get a new HDD, install Proxmox on it, and then +image the original Ubuntu installation onto a virtual disk to run as a VM. This +was probably a year or so after the initial install, for context. So, for the +past 9 years (8 as a virtual machine) this Ubuntu server installation has trudged +along. A port forward pointing port 80 and 443 at this machine has existed in at +least 5 different routers as I've moved around. + +Yesterday, I shut it down. + +I've managed to pull all but a couple of services in my homelab onto a single +machine to be run as [podman +quadlets](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html), +a ingenious little piece of technology I learnt about from a lovely French +Canadian colleague of mine recently. I had used docker in the past to run +services like Bitwarden (vaultwarden) and Gitlab but, although I'd heard of it, +I'd never used podman. I had certainly never come across quadlets anyhow. So +I dove in, starting with the previously linked documentation (which is +a wonderous resource if you're planning on doing anything like what I've done), +and then with some good old fashioned googling. I figured that I could simply as +I pleased, set up quatlets for the services that I wanted, and then migrate my +old running instances across to the new containers. And that's basically how it +went down. + +I'd like to outline the basic premise, how I went about setting everything up, +and some of the tips and tricks I learnt along the way. Also I want to be clear +that my exposure to podman has been almost solely through quadlets. I have had +to interact with the podman cli somewhat throughout this but primarily I've been +defining configs for quadlets in files and then using `systemd` to orchestrate +them. + +I'm running on podman version 5.4.1, which is the version available in the +Ubuntu 25.04. I have 24.04 installed and will until 26.04 is released as I want +to tie myself only to LTS releases. However, 24.04 has podman 4.x.x which is +missing some substantial feature development for quadlets as quadlets are still +under active development. To solve this problem I found and reviewed a script, +see `plucky-pinning.sh` below, which pins podman and a collection of +dependencies to the 25.04 releases. + +One of the features that I was particularly interested in was podman quadlets, +as I'm sure is clear. A quadlet is a `systemd` unit which defines a container or +some other piece of podman infrastructure (network, pod, volume, etc). These +unit files can then be symlinked into a set of predefined locations and +`systemd` +can then be used to start, stop, inspect, etc, them. You can even use `journalctl` +to view logs. The primary ways I set up services were through standalone +containers, and pods. Pods, which I was under the impression I needed podman v5+ +for, are a way to logically group containers and make it easier to manage groups + of containers supporting the same service. The general idea is, as far as + I engaged with it, that you create a `my-service.pod` file in which you can + specify your network and published ports, among other things. Again, + I pushed this as far as networking as that is all I needed, I'm sure there + is more to learn here. + +For example, the following is the definition for my +[Planka](https://planka.app/) service that I'm running. + +`planka.pod`: +``` +[Pod] +PodName=planka +Network=planka.network +PublishPort=1337:1337 +``` + +`planka-server.container`: +``` +[Unit] +Description=Planka - Server +Requires=planka-db.service +After=planka-db.service + +[Container] +Pod=planka.pod +ContainerName=planka-server +Image=ghcr.io/plankanban/planka:2.0.0-rc.4 + +Environment=BASE_URL=planka.example.com +Environment=DATABASE_URL=postgresql://:@planka-db:5432/ +# this is the host name of the db container. ^ +Environment=SECRET_KEY=<64-character-secret-key> + +Volume=//favicons:/app/public/favicons +Volume=//user-avatars:/app/public/user-avatars +Volume=//background-images:/app/public/background-images +Volume=//attachments:/app/private/attachments + +[Service] +Restart=always +TimeoutStartSec=300 + +[Install] +WantedBy=default.target + +``` + +`planka-db.container`: +``` +[Unit] +Description=Planka - DB + +[Container] +Pod=planka.pod +ContainerName=planka-db +Image=docker.io/postgres:16-alpine + +Environment=POSTGRES_PASSWORD= +Environment=POSTGRES_USER= +Environment=POSTGRES_DB= + +Volume=//postgresql:/var/lib/postgresql/data +Volume=/etc/timezone:/etc/timezone:ro +Volume=/etc/localtime:/etc/localtime:ro + +... + +... + +[Service] +Restart=always +TimeoutStartSec=300 + +[Install] +WantedBy=default.target +``` + +`planka-network.container`: +``` +[Unit] +Description=Planka network +After=network-online.target + +[Network] +NetworkName=planka-network +Subnet=10.1.0.0/24 +Gateway=10.1.0.1 +DNS= + +[Install] +WantedBy=default.target +``` + +Thees files were soft linked with `ln -s ` into the +`$HOME/.config/containers/systemd` directory. After which they could be started +with `systemd` commands. + +While setting all this up I ended up using some commands a lot, for instance +`systemctl --user restart ` so I ended up creating a list of aliases +to help with this, these are for `bash`: + +```bash +alias \ + usta="systemctl --user start" \ + usto="systemctl --user stop" \ + ures="systemctl --user restart" \ + ustatus="systemctl --user status" \ + ureload="systemctl --user daemon-reload" \ + usvc="systemctl --user --type=service" \ +``` + +One of the services I'm running is [homepage](https://gethomepage.dev/) which +integrates with podman to get container information directly. You can specify +homepage related data as Labels, this is not dissimilar to how one would do the +same with docker I believe. For instance, the following is what I had for +Planka: + +``` +Label=homepage.group=Productivity +Label=homepage.name=Planka +Label=homepage.icon=planka.png +Label=homepage.href=planka.example.com +Label=homepage.description="Kanban Board" +``` + +This makes it very easy to keep track of your services, all in one place, each +service defining its important data itself. + +In my setup I had all my containers running in userspace as rootless containers. +The only time this became an issue was when a container needed access to the +podman (or docker) socket. This usually requires root privileges. Fortunately, +podman comes with a solution to this problem. + +You can make a userspace version of the socket available with the followig +commands: + +```bash +systemctl --user enable podman.socket + +# check with +systemctl --user status podman.socket +``` + +The socket should then be available to be mounted as in the following: + +``` +Volume=/run/user/1000/podman/podman.sock:/run/podman/podman.sock + +``` + +In my case my user has the uid 1000. + +All in all, I am very happy with how this turned out. I have all my original +services setup as quadlets and their data migrated across (with basically no +issues at all, Nexcloud was the worst but nothing broke), and I have a host of +new services to try out. It's going to be very simple for me to add services in +the future, and manage those that I have set up and I'm very happy about it. + +There are however a couple of caveats to this setup. The first is that every +container, thus every service, is tied to the same machine. Admittedly, that +machine has a low upgrade surface area so updates and reboots should be +infrequent, but regardless, when the machine goes down, so do ALL of the +services. I see this as an acceptable price to pay for the ease of setup and +administration though. Secondly, is backups. I have setup +a [`restic`](https://github.com/restic/restic) profile for the data diretories +of the containers which I run as a cronjob every week but for instance, an image +of a running database is not a backup. I'm going to need to investigate how to +properly automate backups of this data using a method that doesn't leave room +for error. That being said, all of the configuration for the containers (to get + them set up and running) is stored in a git repository, and most of the data + is contained in the files one way or another which is all being backed up as + well. So, I'm not overly worried right now. + +The machine that all of my new services ended up on has been named rimu as an +ode to my kiwi heritage and the fact that now instead of having multiple small +VMs hosting separate services, I now have one towering VM to host all my +services. + +> P.S. In the future, I will migrate the containers to using environment files +> and +then publish a sanitised copy of the repository for public perusal. + +- plucky-pinning.sh +```bash +#!/bin/bash + +# Must be run as root +if [ "$EUID" -ne 0 ]; then + echo "Please run as root (e.g., sudo $0)" + exit 1 +fi + +# Define file paths +PINNING_FILE="/etc/apt/preferences.d/podman-plucky.pref" +SOURCE_LIST="/etc/apt/sources.list.d/plucky.list" + +# Write Plucky APT source list +echo "Adding Plucky repo to $SOURCE_LIST..." +echo "deb http://archive.ubuntu.com/ubuntu plucky main universe" > "$SOURCE_LIST" + +# Write APT pinning rules +echo "Writing APT pinning rules to $PINNING_FILE..." +cat < "$PINNING_FILE" +Package: podman buildah golang-github-containers-common crun libgpgme11t64 libgpg-error0 golang-github-containers-image catatonit conmon containers-storage +Pin: release n=plucky +Pin-Priority: 991 + +Package: libsubid4 netavark passt aardvark-dns containernetworking-plugins libslirp0 slirp4netns +Pin: release n=plucky +Pin-Priority: 991 + +Package: * +Pin: release n=plucky +Pin-Priority: 400 +EOF + +# Update APT cache +echo "Updating APT package list..." +apt update + +echo "Plucky pinning setup complete." +```