Fable clean up

src/content/work/backup-container-btrfs-borg.md

@@ -25,11 +25,11 @@ project:
     alt: Placeholder thumbnail for the backup container project.
 ---
 
-Once you self-host a few services with live databases, the backup question stops being theoretical. A Postgres or SQLite file half-written when `tar` reads it goes into the archive in a state nothing on Earth will replay; you just don't find out until the restore. Two years in, with multiple incidents I had to actually recover from (including the photos behind the [e-ink frame](/articles/frame-eink-photo-display/)), I trust this stack precisely because the correctness argument is short: BTRFS gives me an atomic snapshot, and everything above it can be a shell script. One Alpine container, ~75 lines of Bash, pushes that snapshot to one or more [Borg](https://borgbackup.readthedocs.io/) repositories on a fixed interval. Multi-target is numeric env vars (`BORG_REPO_0`, `BORG_REPO_1`, ...). No config format, no DSL; the env file is the configuration.
+Once you self-host a few services with live databases, the backup question stops being theoretical: everything on the box is mid-write at every moment of the day. This container is my answer, two years and several real restores in (including the photo library behind the [e-ink frame](/articles/frame-eink-photo-display/)). One Alpine container and four short shell scripts (the longest is 84 lines) push a BTRFS snapshot to one or more [Borg](https://borgbackup.readthedocs.io/) repositories on a fixed interval. Multi-target is numeric env vars (`BORG_REPO_0`, `BORG_REPO_1`, ...); there's no config format and no DSL, because the env file is the configuration. The design has exactly one clever moment, the snapshot, and I've worked to keep everything else too simple to break.
 
 ## The problem the snapshot solves
 
-I self-host several databases that are mid-write at every moment of the day. `tar | borg create` against the live volume is a race: a Postgres or SQLite file that's half-written when borg reads it goes into the archive in a state nothing on Earth can replay. The "right" answer is to coordinate a quiesce with every database: a fan-out of `pg_dump`, SQLite `.backup`, Redis `BGSAVE`, and so on, all with retry, timeouts, and per-app credentials.
+`tar | borg create` against a live volume is a race: a Postgres or SQLite file that's half-written when borg reads it goes into the archive in a state nothing on Earth can replay, and you find out at restore time, which is the one moment you can least afford the discovery. The "right" answer is to coordinate a quiesce with every database: a fan-out of `pg_dump`, SQLite `.backup`, Redis `BGSAVE`, and so on, each with retries, timeouts, and per-app credentials.
 
 The cheaper answer, if you've put everything on one BTRFS volume, is `btrfs subvolume snapshot`. It returns instantly with a copy-on-write fork of the entire filesystem. Every file is now atomically consistent at exactly the same instant. Run borg against the snapshot, not against the live volume.
 
@@ -71,13 +71,13 @@ while true; do
 done
 ```
 
-A comment in the file says it out loud: "Using a simple sleep loop to schedule backups instead of cron to avoid concurrency issues." Cron with a one-hour cadence and a backup that occasionally takes 70 minutes will eventually overlap itself. The sleep-loop can't: the next run starts when the previous one is done, plus the interval. One process, one snapshot, one borg invocation. Concurrency bugs you can't have are concurrency bugs you don't have.
+A comment in the file says it out loud: "Using a simple sleep loop to schedule backups instead of cron to avoid concurrency issues." Cron with a one-hour cadence and a backup that occasionally takes 70 minutes will eventually overlap itself, and two borg processes writing the same repo is a bad afternoon. The sleep loop can't overlap: the next run starts when the previous one finishes, plus the interval. One process, one snapshot, one borg invocation, and a whole category of bug that has nowhere to live.
 
 ## Healthcheck is a file mtime
 
 `borg create` succeeded? Write `date > /health/backup_completion_time.log`. The Docker healthcheck shells out every 10 seconds and compares that mtime against `MAX_BACKUP_AGE_SECONDS` (default 86400). Older than that, container is unhealthy and whatever's watching containers (in my case a notification hook) finds out.
 
-Two subtleties worth naming:
+Two subtleties hide in there:
 
 - **First-boot grace period.** If `backup_completion_time.log` doesn't exist yet (fresh container, first backup still running), fall back to `container_start_time.log` so the container isn't reported unhealthy during the first scheduled run.
 - **Partial success is not success.** In multi-target mode, the completion log is only written if _every_ target succeeded. One repo failing means the healthcheck stays red even if the other two are fine. Stale-but-quiet was the failure mode I wanted to make impossible.
@@ -91,6 +91,7 @@ Two subtleties worth naming:
 - **`--files-cache=ctime,size,inode`.** The default `mtime,size,inode` re-hashes files when their mtime changes; on BTRFS, ctime is the more honest signal of "this content actually changed."
 - **`compression=zstd,12`.** The sweet spot for backup data on my hardware: substantially better than zlib, not so slow it dominates the run.
 - **`borg compact --threshold=5 --cleanup-commits`.** Reclaims space from pruned archives whenever the segment-file fragmentation crosses 5%.
+- **Retention: 6 daily, 3 weekly, 48 monthly, 10 yearly.** Four years of monthly archives sounds extravagant until the question becomes "when exactly did this file change?", which is the question every incident eventually asks.
 - **`IGNORE_GIT_UNTRACKED=true`.** Optional. Walks every `.git` dir under the snapshot, runs `git ls-files --others --exclude-standard`, and feeds the result into `--exclude-from`. Skips `target/`, `node_modules/`, build caches; anything the repo already knows isn't worth keeping.
 - **`SYS_ADMIN` capability on the container.** Needed for `btrfs subvolume snapshot` and `delete` from inside the namespace. The narrower capability set didn't have a way through.
 
@@ -100,4 +101,4 @@ Two subtleties worth naming:
 - **A failure notifier separate from the healthcheck.** Docker healthcheck-unhealthy is one signal; I'd also want an explicit push (ntfy, email, Telegram) on first failure of a run, so I don't have to be watching the container state.
 - **Parallel targets when network and disk don't compete.** The current loop is strictly sequential: rsync.net then local HDD. They share neither bandwidth nor spindles; they could run in parallel and halve the wall-clock. Sequential made the wrapper trivial; the trade was knowable and I made it.
 
-Two years in, the part I'd defend hardest is the snapshot. Everything above it is a wrapper that could be rewritten in an afternoon. The snapshot is what makes the wrapper allowed to be one.
+Two years in, the part I'd defend hardest is still the snapshot. Everything above it is a wrapper anyone could rewrite in an afternoon, and that's not an apology for the wrapper, it's the property I now optimise for on purpose.