add backup script

2026-07-12 14:53:44 +01:00 · 2026-07-12 14:53:44 +01:00 · 84543184e7
commit 84543184e7
parent d384ecfa9d
1 changed files with 9 additions and 80 deletions
--- a/src/content/work/backup-container-btrfs-borg.md
+++ b/src/content/work/backup-container-btrfs-borg.md
@ -1,8 +1,8 @@
 ---
-title: Backing Up Running Databases Without Stopping Them
+title: Simple Sophisticated Backup Strategy
-description: A Bash container around BorgBackup. BTRFS snapshots give atomic consistency, numeric env vars give multi-target 3-2-1, the loop is sleep not cron.
+description: When simple beats complex and a case for not reinventing the wheel.
 date: 2026-05-29
-period: '2024-2026'
+period: '2023-2026'
 thumbnail:
  src: ./_assets/backup.png
  alt: Placeholder thumbnail for the backup container post.
@ -14,86 +14,15 @@ links:
 article:
  tags: ['systems', 'tools']
  stack: ['Bash', 'BorgBackup', 'BTRFS', 'Alpine', 'Docker', 'SSH', 'zstd']
-  scale: One container, multiple targets per host, two years of restored incidents
+  scale: One container, multiple targets per host, four years of restored incidents
  outcome: A self-hosted backup that has survived every actual incident I've thrown at it
 project:
  title: Backup Container
 ---
-Once you self-host a few services with live databases, the backup question stops being theoretical: everything on the box is mid-write at every moment of the day. This container is my answer, two years and several real restores in (including the photo library behind the [e-ink frame](/articles/frame-eink-photo-display/)). One Alpine container and four short shell scripts (the longest is 84 lines) push a BTRFS snapshot to one or more [Borg](https://borgbackup.readthedocs.io/) repositories on a fixed interval. Multi-target is numeric env vars (`BORG_REPO_0`, `BORG_REPO_1`, ...); there's no config format and no DSL, because the env file is the configuration. The design has exactly one clever moment, the snapshot, and I've worked to keep everything else too simple to break.
+There's merit to keeping systems simpler the more critical they are. More precisesly, we must keep them easily understandable by building on solid foundations and only adding on top the most necessary extra logic.
-## The problem the snapshot solves
+A good example of this is my home server's backup. I ran various databases, even when I aim to mostly use sqlite, there are multiple postgres instances, redis, message queues, etc. Who doesn't like time machine like backups which allow granular file history? To achieve this, we need frequent backups, my setup uses an hourly cadence. However, we can't shut down the entire stack every hour to back it up. So the only solution is to backup everything live.
-`tar | borg create` against a live volume is a race: a Postgres or SQLite file that's half-written when borg reads it goes into the archive in a state nothing on Earth can replay, and you find out at restore time, which is the one moment you can least afford the discovery. The "right" answer is to coordinate a quiesce with every database: a fan-out of `pg_dump`, SQLite `.backup`, Redis `BGSAVE`, and so on, each with retries, timeouts, and per-app credentials.
+I could've gone down the rabbit hole of using each database's dedicated backup mechansim, however, that would've been everyhing but simple. So instead I rely on the DB's crash recovery mechanism being solid.
-The cheaper answer, if you've put everything on one BTRFS volume, is `btrfs subvolume snapshot`. It returns instantly with a copy-on-write fork of the entire filesystem. Every file is now atomically consistent at exactly the same instant. Run borg against the snapshot, not against the live volume.
+With the above context, I can now reveal that the backup container is just a short shell script that takes a btrfs snapshot (all container volumes are mounted from a btrfs subvolume) and then feeding into borg which handles incrementality and deduplication. The backup script supports configuring borg to run against multiple remote backup servers, gracefully handle failures, and to maintain a healthcheck status in case human intervention is needed.
-```bash
+I've been using this for 4 years now and I restored various files and folders and did two full disaster recoveries. These were enough to convince me of the soundness of borg, btrfs, and my backup configuration. That's why I don't think I will migrate away from this method any time soon.
 btrfs subvolume snapshot /btrfs-root /snapshot
 cd "/snapshot/btrfs-root${BACKUP_RELATIVE_PATH:-}"
 borg create ... ::"{hostname}-{now:%Y-%m-%dT%H:%M:%S}" .
 ```
 The snapshot lives only for the duration of the borg run. A `trap cleanup EXIT` deletes the subvolume whether the backup succeeded, failed, or was killed. The next run snapshots fresh.
 This shifts the entire correctness argument from "did I quiesce every database in time" to "does BTRFS give me a consistent snapshot." It does. That's why everything below it can be a shell script.
 ## Multi-target as numeric env vars
 The 3-2-1 backup rule wants three copies, two media, one offsite. My answer is a remote (rsync.net) and a local HDD, both fed from the same snapshot. The wire format for "multiple targets" is just numbered env vars:
 ```sh
 BORG_PASSPHRASE_0=...
 BORG_REMOTE_PATH_0=borg1
 BORG_REPO_0=username@username.rsync.net:~/backup
 BORG_PASSPHRASE_1=...
 BORG_REPO_1=/local-backup
 ```
 `backup-wrapper.sh` loops `index=0` upward, exports `BORG_PASSPHRASE` / `BORG_REPO` / `BORG_REMOTE_PATH` from the indexed copies, runs `backup.sh`, unsets them, increments. Stops the first time the next index has no passphrase.
 There's also a no-index fallback (`BORG_REPO=...` with no number) for the single-target case. Same script, no extra config plane.
 I keep coming back to this pattern for small-system orchestration. The env file _is_ the data structure. There's no YAML parsing, no JSON schema, no config-validation layer between you and the variable that actually matters.
 ## The scheduler is a sleep, not cron
 ```bash
 while true; do
    /src/backup-wrapper.sh 2>&1 | log_message
    sleep "$SLEEP_TIME"
 done
 ```
 A comment in the file says it out loud: "Using a simple sleep loop to schedule backups instead of cron to avoid concurrency issues." Cron with a one-hour cadence and a backup that occasionally takes 70 minutes will eventually overlap itself, and two borg processes writing the same repo is a bad afternoon. The sleep loop can't overlap: the next run starts when the previous one finishes, plus the interval. One process, one snapshot, one borg invocation, and a whole category of bug that has nowhere to live.
 ## Healthcheck is a file mtime
 `borg create` succeeded? Write `date > /health/backup_completion_time.log`. The Docker healthcheck shells out every 10 seconds and compares that mtime against `MAX_BACKUP_AGE_SECONDS` (default 86400). Older than that, container is unhealthy and whatever's watching containers (in my case a notification hook) finds out.
 Two subtleties hide in there:
 - **First-boot grace period.** If `backup_completion_time.log` doesn't exist yet (fresh container, first backup still running), fall back to `container_start_time.log` so the container isn't reported unhealthy during the first scheduled run.
 - **Partial success is not success.** In multi-target mode, the completion log is only written if _every_ target succeeded. One repo failing means the healthcheck stays red even if the other two are fine. Stale-but-quiet was the failure mode I wanted to make impossible.
 ## Smaller calls
 - **`borg break-lock` at the start of every run.** If the previous container was killed mid-backup, the repo is locked and the next `borg create` will hang. Just break it. There's only ever one writer because of the sleep loop.
 - **`set -e` after `borg init`, not before.** The init line is the only one allowed to fail (first run on a fresh repo). Everything after halts on error.
 - **`BORG_RSH='ssh -oBatchMode=yes'`.** Fail fast if SSH would have prompted, instead of hanging forever inside a detached container.
 - **`ServerAliveInterval 30` in `ssh_config`.** Long borg transfers across home-ISP NAT get killed if nothing flows for a few minutes. Keepalives keep the tunnel open.
 - **`--files-cache=ctime,size,inode`.** The default `mtime,size,inode` re-hashes files when their mtime changes; on BTRFS, ctime is the more honest signal of "this content actually changed."
 - **`compression=zstd,12`.** The sweet spot for backup data on my hardware: substantially better than zlib, not so slow it dominates the run.
 - **`borg compact --threshold=5 --cleanup-commits`.** Reclaims space from pruned archives whenever the segment-file fragmentation crosses 5%.
 - **Retention: 6 daily, 3 weekly, 48 monthly, 10 yearly.** Four years of monthly archives sounds extravagant until the question becomes "when exactly did this file change?", which is the question every incident eventually asks.
 - **`IGNORE_GIT_UNTRACKED=true`.** Optional. Walks every `.git` dir under the snapshot, runs `git ls-files --others --exclude-standard`, and feeds the result into `--exclude-from`. Skips `target/`, `node_modules/`, build caches; anything the repo already knows isn't worth keeping.
 - **`SYS_ADMIN` capability on the container.** Needed for `btrfs subvolume snapshot` and `delete` from inside the namespace. The narrower capability set didn't have a way through.
 ## What I'd change
 - **A test rig that restores into an empty volume on a schedule.** "Backups exist" is not the property I care about. "Backups restore" is. I have anecdotal evidence after every incident; I don't have a green checkmark before one.
 - **A failure notifier separate from the healthcheck.** Docker healthcheck-unhealthy is one signal; I'd also want an explicit push (ntfy, email, Telegram) on first failure of a run, so I don't have to be watching the container state.
 - **Parallel targets when network and disk don't compete.** The current loop is strictly sequential: rsync.net then local HDD. They share neither bandwidth nor spindles; they could run in parallel and halve the wall-clock. Sequential made the wrapper trivial; the trade was knowable and I made it.
 Two years in, the part I'd defend hardest is still the snapshot. Everything above it is a wrapper anyone could rewrite in an afternoon, and that's not an apology for the wrapper, it's the property I now optimise for on purpose.