Ansible for updating LXCs

The gap in my patching philosophy/system

Up until today I had two reliable habits for keeping the homelab patched. When Proxmox flags updates, I open the Updates tab. When a Docker service falls behind, Komodo redeploys it.

But… most of the homelab is neither Docker nor the hypervisor. It’s a dozen-odd LXC containers, each a full Debian install with its own packages: Pi-hole, Traefik, the internal CA, the git server, the backup server, and a handful more. The Proxmox Updates tab patches the hypervisor those containers run on. It does nothing to the containers themselves. There is no button anywhere that means “update all my guests”. I hadn’t really given it much thought either until recently.

The way updates actually split up looks like this:

flowchart LR
    subgraph layers["Three things that need patching"]
        direction TB
        host["Proxmox hosts\ncorvus + corax"]
        docker["Docker services\non the VM"]
        guests["14 LXC guests + 1 VM\neach its own Debian"]
    end

    host -->|"PVE Updates tab"| ok1["✅ handled"]
    docker -->|"Komodo redeploy"| ok2["✅ handled"]
    guests -->|"???"| gap["⚠️ nobody"]

The first two were handled. The third was the biggest by host count and the one I’d quietly ignored. “Patch all my guests” means SSHing into each box and running apt update && apt full-upgrade, fifteen times over, which I did not feel like doing… mostly out of my laziness and desire for automating everything boring in my life.

The obvious solution… Ansible

This is a textbook job for Ansible, a config automation engine.

The shape is honestly pretty simple: an inventory of every guest, and a play that connects over SSH and runs the upgrade. The apt module does the work, and because it reports whether anything actually changed, I get a clean per-host result instead of fifteen screens of scrolling apt output.

The core of the play:

- name: Patch OS on all guests
  hosts: guests
  become: true
  serial: 3                       # three at a time, not all at once
  environment:
    DEBIAN_FRONTEND: noninteractive
    NEEDRESTART_MODE: a           # auto-restart services, never prompt
  tasks:
    - name: apt update + full-upgrade + autoremove
      ansible.builtin.apt:
        update_cache: true
        upgrade: full
        autoremove: true

Two choices in there are important. serial: 3 patches three hosts at a time instead of the whole fleet at once, so a bad upgrade can break three boxes at most and I notice before the rest run. NEEDRESTART_MODE: a stops the run hanging forever on Debian’s interactive “which services should restart?” prompt.

It is also idempotent so you can run it against a host that is already current and Ansible checks, finds nothing to do, and moves on. That is what makes it safe to run whenever I like or to run on a timer.

I also attached a summary onto the end so each run finishes with something readable:

  pbs              updated
  timemachine      updated
  jellyfin         up-to-date  (reboot required)
  step-ca          up-to-date
  ...

That was the plan. Then I ran it for the first time, and of course it broke in three different places.

What broke

The first run died partway through downloading, with 400 Bad Request coming back from security.debian.org. A different package failed on each host, every time after a stretch of unusually slow transfer. The mirror was fine; fetching the exact file it choked on by hand returned a clean 200. The reason is that security.debian.org sits behind a Fastly CDN, and apt reuses one connection to request many packages back to back. Fastly occasionally answers one of those pipelined requests with a 400, and apt treats it as fatal. Turning pipelining off fixes it, so the playbook now writes a small apt config onto each guest before upgrading:

Acquire::Retries "3";
Acquire::http::Pipeline-Depth "0";

“It works when I curl it” and “it works under apt’s connection reuse” turned out to be different.

The backup server failed differently, with a 401 from enterprise.proxmox.com. Proxmox ships with its enterprise repository switched on, and that repo needs a paid subscription; without one, apt update refuses to go any further. Disabling the enterprise repo and leaning on the free no-subscription one is the standard fix, but it is a per-host bit of setup. I’d forgotten to disable it on PBS, as I had already done in the past on both of my nodes.

Another issue I ran into was that my containers do not all log in the same way. Some of my older ones still take root over SSH. The newer, hardened ones refuse root and only allow a nick user with sudo, so the inventory carries a per-host override:

timemachine:
  ansible_host: 10.0.1.208
  ansible_user: nick      # root SSH disabled on this one

One thing that did not become a problem: reboots. An LXC container runs on the host’s kernel, so a kernel update inside one does nothing on its own, and rebooting the container changes nothing. The only guest that genuinely needs a reboot is my Docker VM. So the playbook leaves reboots off by default: when I run it by hand it just lists the hosts that asked for one and I deal with them when it suits me. The weekly run is the exception however.

Making it run itself

A playbook I have to remember to run every so often still requires my action. Since I am both forgetful and lazy I wanted this to happen on its own, so I hung it off Forgejo Actions, the CI built into my self-hosted git server, on a weekly timer:

on:
  schedule:
    - cron: '0 4 * * 0'          # Sundays, 04:00
  workflow_dispatch: {}          # or trigger it by hand

Originally I had been running the playbook from my laptop, which already has SSH access to every guest. My CI stuff however, runs on a dedicated runner container with none of that access and no Ansible installed. Scheduling it really meant giving the runner its own dedicated SSH key, its public half authorized on every guest (with one more small Ansible play), and the private half handed to the runner as an encrypted CI secret via Forgejo.

That does hand the runner a key with broad access across the fleet, which on a private network behind Tailscale is a trade I am happy with. The scheduled job also skips the runner itself, so it is not trying to patch and reboot the machine it is running on.

The weekly run is also the one place I let it reboot on its own. Running by hand, I keep reboots off and handle them later, but at four on a Sunday morning nothing is in use, so the scheduled job passes auto_reboot=true and restarts whatever asked for it without me watching. In practice though that is usually just the one VM.

Of course there was one last small issue to sort out. The bare runner had no locale generated, and Ansible’s Python refuses to start without one. Pinning LC_ALL=C.UTF-8 in the workflow sorted it.

What it’s like now

It runs every Sunday at four in the morning, works through the guests in threes, reboots the one VM, and leaves a summary in the CI log.

Patching is the least interesting thing I do to this setup, but it is the part that keeps everything alive. I’m glad I don’t have to worry about it as much any longer, and have delegated this worry to automation… Also once you set this up properly once, it just keeps going without you needing to think about it.