I had been running a password vault and a git mirror forge on a VPS for a while with no backups at all. Not a partial backup, not an untested one. None. The disk was a single 40 GB volume and the only copy of anything lived on it.

This is the more likely way to lose data than any attacker, and it is worth stating plainly because the failure mode of self-hosting is almost never a breach. It is a disk, or a fat-fingered rm, or a migration that goes sideways.

What follows is what I built. Two jobs, two mechanisms, and a couple of design decisions that are not obvious.

Encrypt before it leaves the machine

The vault's items are already encrypted client-side. But its database also contains the master password hash, which is offline-crackable. That should not sit in third-party storage in the clear, no matter how much you trust the provider.

I used age, which is a keypair rather than a passphrase:

sudo age-keygen -o /etc/backup/age-identity.key
sudo chmod 600 /etc/backup/age-identity.key
sudo sh -c 'age-keygen -y /etc/backup/age-identity.key > /etc/backup/age-recipient.txt'

The key lives on the box, and that is a deliberate limit worth being explicit about. It does not protect against a compromise of the box, which can read the live database anyway. It protects against exposure at the storage provider. Since an attacker with root already has the plaintext, keeping the key locally costs nothing against that threat and lets the jobs run unattended.

Two mechanisms, because the data has two shapes

The vault database is a few hundred kilobytes. The git mirror tree is 1.7 GB across 71 repositories, of which roughly 200 files change on any given day.

Shipping a 1.7 GB encrypted tarball every night to protect 200 changed files is absurd. But there is no good way to incrementally update an encrypted archive, either. So:

• Small things ship as whole age-encrypted archives, one per run, with dated retention.
• The git tree goes through rclone sync to an encrypted remote. Git objects are immutable and content-addressed, so after the first push each run transfers only genuinely new objects. Filenames are encrypted too, since the archive is not a single opaque blob.

The tradeoff to understand: rclone sync is a mirror, not history. A repo deleted locally is deleted remotely on the next run. That is acceptable here because the repos are themselves mirrors of upstreams, but if you need point-in-time recovery, use --backup-dir or a real snapshotting tool.

Use the database's own backup command

sqlite3 "$LIVE_DB" ".backup '$STAGING/db.sqlite3'"

Not cp. Both databases here run in WAL mode, and when I looked there was a live 247 KB -wal file next to the main one. Copying just the file captures a torn state and silently loses committed transactions. .backup takes a proper read lock without stopping the service.

The same care applies on the way back: when restoring over a live database, delete the stale -wal and -shm files, because they belong to the database you just replaced.

Every run restores its own output

This is the part I would keep if I threw away everything else.

A backup that has never been restored is not a backup, it is a guess. So the job does not finish by uploading. It finishes by decrypting what it just wrote, unpacking it, and checking the result:

age -d -i "$IDENTITY" "$ARCHIVE" | tar xzf - -C "$VERIFY" \
  || die "verification failed: could not decrypt and untar"

INTEG=$(sqlite3 "$VERIFY/db.sqlite3" 'PRAGMA integrity_check;')
[ "$INTEG" = "ok" ] || die "verification failed: integrity_check said '$INTEG'"

COUNT=$(sqlite3 "$VERIFY/db.sqlite3" 'SELECT COUNT(*) FROM items;')
[ "$COUNT" = "$LIVE_COUNT" ] || die "verification failed: $COUNT in backup vs $LIVE_COUNT live"

Three separate claims get checked: it decrypts, the database is structurally intact, and it contains the same number of rows as the live one. If any fails, the job exits non-zero and does not upload, so a bad archive never displaces a good one in the retention window.

The same paranoia applies to the upload. rclone copy succeeding is not evidence the object is there:

LOCAL=$(stat -c%s "$ARCHIVE")
REMOTE=$(rclone size "$REMOTE_PATH/$(basename "$ARCHIVE")" --json \
         | sed -n 's/.*"bytes":\([0-9]*\).*/\1/p')
[ "$REMOTE" = "$LOCAL" ] || die "size mismatch (local $LOCAL, remote ${REMOTE:-none})"

It also refuses to ship an obviously wrong snapshot at all:

[ "$COUNT" -gt 0 ] || die "snapshot has zero items, refusing to ship it"

An empty backup that overwrites a good one is worse than a failed job.

Two rclone details that cost me time

rclone config create hangs on OAuth backends. Supplying a token on the command line does not stop it trying to run the interactive browser flow, and with stdin not a terminal it blocks forever. Write the config file by hand:

[remote]
type = <provider>
hostname = <region-specific api host>
token = {"access_token":"...","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}

Providers with no token expiry can be wrapped like that from a bare access token.

Region matters and is not auto-detected when you supply a token by hand. rclone normally learns the correct API host during its own OAuth dance. Skip that and it uses the default host, which fails to authenticate for accounts in another region, with an error that looks exactly like a bad token. The provider's docs for the backend usually say which option to set.

Also worth knowing: rclone obscure is reversible obfuscation, not encryption. A crypt password in rclone.conf is readable by anyone with root. Same trade-off as the age key, and worth stating rather than assuming.

And prefer rclone sync --checksum for a git tree. The default mtime-and-size comparison misses repacks that rewrite a file to the same size, and consumer storage often has coarse mtime granularity.

The circular dependency

This is the failure mode that turns a working backup system into encrypted noise, and it is easy to walk into.

The obvious place to store the age key is a password manager. But if the password manager is one of the things being backed up, then recovering the backup requires the key, which requires the password manager, which requires the backup. Circular.

Mine is subtler and I nearly missed it. The key went into pass, which is fine, because pass lives on my laptop and pushes to a git host, both independent of the VPS. But that same pass repo is mirrored onto the forge, and the forge is backed up to the cloud under the key stored inside it. So the cloud copy of pass is not a recovery route for the key, even though it looks like one.

Two rules that fall out of this:

1. Write down which copies of a secret are genuinely independent of the thing it decrypts. "It's in three places" means nothing if two of them are downstream of the encrypted backup.
2. Keep one copy on a medium that has no dependency on any of it. An age key is 184 bytes over three lines. It fits on a piece of paper.

There is a hardware version of the same trap: if your password store is GPG-encrypted to a key on a hardware token, then losing the token and the laptop together leaves the store unreadable, and the backups undecryptable with it.

Shape of the result

Two systemd timers, half an hour apart so they do not contend for two vCPUs and one uplink, both Persistent=true so a missed run catches up after a reboot. Local retention of 14 days, remote 90, with the git tree mirrored rather than dated. First full push of 1.7 GB took about 17 minutes; nightly runs move the couple of hundred files that actually changed.

The uncomfortable thing I would flag to anyone doing this: my scripts live in /usr/local/bin and my configs in /etc, and neither is backed up. The data is safe and the machinery that protects it is not. For now the mitigation is that both scripts are reproduced in full in my notes, which is a real answer but not a good one. Backing up /etc is the obvious next job.