From b9b0cae0d2261508bcd3f47fb746b9223807994b Mon Sep 17 00:00:00 2001 From: Andras Schmelczer Date: Sun, 2 Jun 2024 12:40:54 +0100 Subject: [PATCH] Add config folder --- Dockerfile | 4 ++-- README.md | 24 +++++++++++++----------- .env => config/default.env | 0 config/exclude.conf | 9 +++++++++ ssh_config => config/ssh_config | 0 exclude.conf | 6 ------ 6 files changed, 24 insertions(+), 19 deletions(-) rename .env => config/default.env (100%) create mode 100644 config/exclude.conf rename ssh_config => config/ssh_config (100%) delete mode 100644 exclude.conf diff --git a/Dockerfile b/Dockerfile index d086498..c90c4e4 100644 --- a/Dockerfile +++ b/Dockerfile @@ -4,8 +4,8 @@ USER root RUN apk --no-cache add btrfs-progs openssh borgbackup bash coreutils -COPY ssh_config /etc/ssh/ COPY src /src -COPY exclude.conf /exclude.conf +COPY config/ssh_config /etc/ssh/ +COPY config/exclude.conf /exclude.conf ENTRYPOINT ["sh", "-c", "/src/schedule.sh"] diff --git a/README.md b/README.md index 9dbcce2..8f56751 100644 --- a/README.md +++ b/README.md @@ -5,11 +5,13 @@ Snapshot a [BTRFS](https://docs.kernel.org/filesystems/btrfs.html) volume from a ## Quick start - Check out [docker-compose.yml](docker-compose.yml) and set the environment variables according to your needs -- Update the password in the [.env](.env) file +- Update the password in the [.env](config/.env) file +- Adjust the [exclude.conf](conifg/exclude.conf) file to match your needs +- Run `docker compose --env-file ./config/default.env up` ## Background -I've been using this setup for the past years and have managed to avoid disaster by restoring all my data. This is all thanks to the excellent backup tool called Borg. The scripts in the this repository serve as an opinionated thin wrapper around it. The Docker image I provide in this repository is generic enough to be used verbatim in many self-hosting setups. However, it's just a thin wrapper and a set of configs that work together well. It's been a long journey for me to find the backup setup that works for me, so I make this available to save others from pitfalls and serve as a potenrial starting point for your own backup container. +I've been using this setup for the past year and have managed to avoid disaster by restoring all my data. This is all thanks to the excellent backup tool called BorgBackup. The scripts in the this repository serve as an opinionated thin wrapper around it. The Docker image I provide here is generic enough to be used verbatim in many self-hosting setups. However, it's just a thin wrapper and a set of configs that work together well. It's been a long journey for me to find a backup setup that works for me, so I make this available to save others from pitfalls and serve as a potenrial starting point for your own self-hosted backup container. ## Features @@ -17,15 +19,15 @@ I've been using this setup for the past years and have managed to avoid disaster > I self-host multiple databases and this is the most feasible way of avoiding data corruption - Schedule the backup so that it keeps running automatically - Keep a weekly rotated log of all previous runs -- Back up the same source to multiple borg repositories +- Back up the same source to multiple BorgBackup repositories ### Multi-target backups -In case we don't have disk-level redundancy and would still wish to follow the [3-2-1 rule](https://en.wikipedia.org/wiki/Backup), we have to back up the same source folder to multiple targets. For instance, I don't have a RAID setup but instead back up an SSD into rsync.net and a local HDD at the same time. +In case we don't have disk-level redundancy and would still wish to follow the [3-2-1 rule](https://en.wikipedia.org/wiki/Backup), we have to back up the same source folder to multiple targets. For instance, I don't have a RAID setup but instead back up an SSD to [rsync.net](rsync.net) and a local HDD at the same time. -You can see in the [docker-compose.yml](docker-compose.yml) file that we can set `BORG_REPO_0`, `BORG_REPO_1`, `BORG_PASSPHRASE_0`, `BORG_PASSPHRASE_1`, etc. +You can see in the [docker-compose.yml](docker-compose.yml) file that we can configure multi-target backups by setting the environment variables: `BORG_REPO_0`, `BORG_REPO_1`, `BORG_PASSPHRASE_0`, `BORG_PASSPHRASE_1`, etc. -The backup script first takes `BORG_REPO_0` and the corresponding env vars and sets up the [`BORG_REPO`](https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls), `BORG_REMOTE_PATH`, and `BORG_PASSPHRASE` environment variables for borg. Once that finished (successfully or otherwise), the script checks whether `BORG_REPO_1` exists, if so, it sets `BORG_REPO` and the other env vars to their expected values and backs up again. The script keeps going to `BORG_REPO_2`, `BORG_REPO_3` and so on as long as these are set. Otherwise, it unsets the previous `BORG_REPO` and corresponding env vars and goes to sleep. +The backup script first takes `BORG_REPO_0` and the corresponding env vars and sets up the [`BORG_REPO`](https://borgbackup.readthedocs.io/en/stable/usage/general.html#repository-urls), `BORG_REMOTE_PATH`, and `BORG_PASSPHRASE` environment variables for `borg`. Once the backup finished (successfully or otherwise), the script checks whether `BORG_REPO_1` exists, if so, it sets `BORG_REPO` and the other env vars to their expected values and backs up again. The script keeps going to `BORG_REPO_2`, `BORG_REPO_3` and so on as long as these are set. Otherwise, it unsets the previous `BORG_REPO` and corresponding env vars and goes to sleep. Thus, the following sets of environment variables are valid for multi-target backups: @@ -67,12 +69,12 @@ Thus, the following sets of environment variables are valid for multi-target bac ## Repository layout - src - - backup.sh: Creates a Borg repository if it doesn't exist, snapshots the BTRFS volume, backs up, and removes old backups - - backup-wrapper.sh: Runs backup.sh for every Borg repository provided through environment variables + - backup.sh: Creates a BorgBackup repository if it doesn't exist, snapshots the BTRFS volume, backs up your data, and removes old backups. + - backup-wrapper.sh: Runs backup.sh for every BorgBackup repository provided through environment variables. - schedule.sh: Persists the logs of backup-wrapper.sh and runs it in a while True. ## Related resources -- I like this video on how to install Debian with BTRFS https://www.youtube.com/watch?v=MoWApyUb5w8. Of course, a btrfs disk can be created after installation as well. -- rsync.net has a special discount when using borgbackup: https://www.rsync.net/products/borg.html -- Checking out the documentation of BorgBackup is worth it: https://www.borgbackup.org/demo.html +- I like this video on how to install Debian with BTRFS https://www.youtube.com/watch?v=MoWApyUb5w8. Of course, a BTRFS disk can be created after installation as well. +- rsync.net has a special discount when using BorgBackup: https://www.rsync.net/products/borg.html +- Checking out the documentation of BorgBackup is worth it: https://www.borgbackup.org/demo.html, especially the description of [borg create](https://borgbackup.readthedocs.io/en/stable/usage/create.html#description) diff --git a/.env b/config/default.env similarity index 100% rename from .env rename to config/default.env diff --git a/config/exclude.conf b/config/exclude.conf new file mode 100644 index 0000000..7d977cc --- /dev/null +++ b/config/exclude.conf @@ -0,0 +1,9 @@ +# https://borgbackup.readthedocs.io/en/stable/usage/create.html#description +# https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-patterns + +*/node_modules +*/.env +*/.dev.env +*/__pycache__ +*/.venv +/snapshot diff --git a/ssh_config b/config/ssh_config similarity index 100% rename from ssh_config rename to config/ssh_config diff --git a/exclude.conf b/exclude.conf deleted file mode 100644 index ea5c11f..0000000 --- a/exclude.conf +++ /dev/null @@ -1,6 +0,0 @@ -*/node_modules -*/.env -*/.dev.env -*/__pycache__ -*/.venv -/snapshot