corvus

In-Place Still Reboots

· 6 min read

One variable

Back in August I finally retired Pi-hole. It had been doing DNS and DHCP for the whole house from a single container on a single node, which is a silly place to put the thing that every other thing depends on. It got replaced by a pair of Technitium servers, one on each Proxmox node, so losing a node no longer costs me name resolution.

The cutover went fine. What the cutover did not do was move my containers. Every LXC in the fleet has a static resolver written into it by OpenTofu at creation. They never used DHCP, so flipping DHCP over to the new pair didn’t touch a single one of them. They were all still pointing at a machine I was about to switch off.

The fix is one variable:

variable "lan_dns_servers" {
  type        = list(string)
  default     = ["10.0.1.213", "10.0.1.214"]
  description = "LAN DNS resolvers (the Technitium pair, dns1 + dns2)."
}

which lands in every container’s initialization block:

initialization {
  hostname = each.key

  dns {
    domain  = try(each.value.search_domain, null)
    servers = var.lan_dns_servers
  }
  ...
}

Guest provisioning here is GitOps. tofu plan runs on the pull request, tofu apply runs on merge to main, both on a Forgejo Actions runner, and I never run apply by hand. So: branch, edit one line, open a PR, read the plan.

What the plan said

16 to change, 0 to add, 0 to destroy.

I have a rule in my own docs about this, written after a .gitignore PR of all things took the backup server down. Read every plan for must be replaced before you merge. Mount and device changes force destroy-and-recreate in the Proxmox provider, and a merge is a live apply, so the plan is the only gate there is.

There was no must be replaced. Sixteen guests, all update in-place. I merged it.

What “in-place” actually means

Terraform’s update in-place doesn’t mean “without downtime”. It means “not destroyed and recreated”. That is the entire promise. What happens to the resource on the way there is up to the provider, and the bpg Proxmox provider applies a change to the dns block by restarting the container.

Which it has to. Proxmox writes /etc/resolv.conf when the container starts. Change the config underneath a running container and the file on disk stays exactly as it was, because there’s no reload path. The restart is the mechanism. A guest that doesn’t restart keeps pointing at the dead server, which is the opposite of the thing I was trying to do.

So the reboots were correct. Every one of them was the provider doing the only thing that would make the change real. They just weren’t what “in-place” had put in my head when I read the plan, and I merged on the strength of the verb rather than on what the verb costs.

Sixteen containers went down and came back. In a homelab that’s a non-event; nothing here has an SLA.

Fourteen seconds

Except one of those sixteen containers is forgejo-runner.

It’s LXC 205, and it’s in hosts.yaml like everything else, because it is a guest like everything else. Two cores, a gig of RAM, a Debian rootfs, provisioned the same way as the git server and the media box. It also happens to be the machine executing the apply.

Fourteen seconds in, the apply reached it and restarted it. The job died mid-run.

What it left behind:

  • 14 guests updated
  • 2 guests not
  • a state lock in MinIO reading Who: root@forgejo-runner, held by a process that no longer existed

The lock is the part that actually bites. OpenTofu takes it so two applies can’t stomp on each other and releases it when the apply finishes. This one didn’t finish, it was shot, so the lock just sat there. Every run after that refused to start — correctly, as far as the backend could tell somebody else was mid-apply.

force-unlock, and knowing when you’re allowed

tofu force-unlock is the command everybody tells you not to run. The warning is a real one: break a lock that a live apply is holding and you get two applies writing the same state file, which is how you lose track of real infrastructure.

The warning also assumes you can’t tell the difference. Here I could. The lock claimed to be held by a process on the runner, and the runner had a fresh uptime, because it had just been rebooted out from under itself. That’s the whole diagnosis: a holder that booted after it took the lock isn’t holding anything.

Confirm the uptime, tofu force-unlock <ID>, re-run the workflow. The second pass finished clean, and it was always going to. The runner had already been changed on the first pass, so the two stragglers were the only diff left and neither of them was the machine doing the work.

The fix

I had already solved this once, in a different tool, and forgotten. My weekly Ansible patching job skips the runner, and I wrote that back in June for exactly this reason: so it wouldn’t try to patch and reboot the box it was running on. Terraform got the same problem and none of the same care, because Terraform’s plan looked like it was telling me everything.

The apply workflow asks the question up front now. Before it applies anything it renders a plan and checks whether the runner is in it:

target = sys.argv[1]
plan = json.load(sys.stdin)

for change in plan.get("resource_changes", []):
    if change.get("address") == target:
        actions = change.get("change", {}).get("actions", [])
        print("true" if actions != ["no-op"] else "false")
        break
else:
    print("false")

The bit worth pointing at is resource_changes. The plan JSON also embeds the current state of everything, so a naive grep for the runner’s address matches on every plan ever rendered, including the ones that don’t touch it. You want the changes, and you want to discount a no-op.

If the answer comes back true, the apply splits in two:

- name: tofu apply (everything except the runner)
  run: tofu apply -auto-approve -exclude="$RUNNER_ADDR"

- name: tofu apply (the runner itself, may kill this job)
  run: tofu apply -auto-approve -target="$RUNNER_ADDR"

Everything else first, the runner last, deliberately.

flowchart TB
    plan["🧾 tofu plan → JSON"] --> q{"Does the plan change\nforgejo-runner?"}
    q -->|no| all["✅ tofu apply\none pass, job survives"]
    q -->|yes| rest["tofu apply -exclude=runner\n15 guests, job survives"]
    rest --> last["⚠️ tofu apply -target=runner\nlast; may kill this job"]
    last --> done["✅ done"]
    last -.->|"job dies"| recover["uptime → force-unlock → re-run\n1 resource left to reconcile"]
    recover -.-> done

The failure mode hasn’t gone anywhere. The second step can still kill the job it’s running inside, and it will, every time the runner’s own config changes. What’s changed is the price. A self-kill costs the last step of the run instead of an arbitrary 2 of 16, and the re-run has one resource left to reconcile instead of a puzzle.

Two things

Read a plan’s semantics, not its verb. update in-place is a statement about the state file, not about the machine. What it costs in real downtime is a property of the provider, and the only way to know is to have read the provider, because nothing in the plan output is ever going to tell you.

And a control plane that manages the machine it runs on sits inside its own blast radius. Mine does, on purpose. The runner is a guest, and taking it out of the fleet would leave one box that nobody provisions, which is a worse problem than a job that occasionally kills itself. So it goes last in the apply, and I expect it to die.

Discussion