zli(1): blog

Self-hosting Vaultwarden, and the one path that ruins it

2026-08-03T00:00:00.000Z

Vaultwarden is a Rust reimplementation of the Bitwarden server API, small enough to run comfortably on a 2 vCPU box alongside other services. It inherits Bitwarden's client-side encryption: the master password never reaches the server, the vault key is derived on the device, and the server only ever stores ciphertext plus a KDF-derived hash. A full server breach does not hand over your passwords.

That property is what makes self-hosting defensible. It is also easy to accidentally give it away, which is most of what this post is about.

The deployment

services:
  vaultwarden:
    image: vaultwarden/server:1.37.1
    container_name: vaultwarden
    restart: unless-stopped
    ports:
      # Loopback only; nginx terminates TLS and proxies to it.
      - "127.0.0.1:8082:80"
    volumes:
      - ./vaultwarden-data:/data
    environment:
      # Must match the public URL exactly or the clients break.
      - DOMAIN=https://vault.example.com
      - SIGNUPS_ALLOWED=false
      - SHOW_PASSWORD_HINT=false
      - WEB_VAULT_ENABLED=false

No ADMIN_TOKEN, so the admin panel does not exist. One fewer credential to leak on a box that is reachable from the internet.

nginx in front needs two things beyond the usual proxy headers: client_max_body_size 128m for attachments and full exports, and WebSocket upgrade headers, because /notifications/hub is how clients learn about changes made elsewhere. Without the upgrade headers, everything works except live sync, which is a confusing thing to debug later.

Do not put HTTP basic auth in front of it. The browser extension and mobile clients cannot answer that prompt.

Closing signups, and actually checking

SIGNUPS_ALLOWED=false is the setting. Verifying it is less obvious, because there are three registration paths and they do not all report the same way.

V=https://vault.example.com

curl -s -X POST "$V/identity/accounts/register" \
  -H 'Content-Type: application/json' \
  --data '{"email":"probe@example.com","name":"p","masterPasswordHash":"x","key":"x","kdf":0,"kdfIterations":600000}'

curl -s -X POST "$V/identity/accounts/register/send-verification-email" \
  -H 'Content-Type: application/json' --data '{"email":"probe@example.com","name":"p"}'

Both return Registration not allowed or user already exists. The third, /identity/accounts/register/finish, returns Registration is missing required parameters for a payload it cannot deserialise. That is a validation error raised before the signup check, so on its own it tells you nothing about whether signups are open. Reading the source, it delegates to the same register() handler that enforces CONFIG.is_signup_allowed(&email), so it is covered.

The check that settles it regardless of endpoint semantics:

sqlite3 /path/to/vaultwarden-data/db.sqlite3 'SELECT email FROM users;'

One more wrinkle: the web vault UI keeps showing a "Create account" link even with signups closed. It is cosmetic. The server refuses.

Turning off the web vault

This is the recommendation I would push hardest, and it is the one that costs you something.

The threat model of client-side encryption assumes the client is trustworthy. The browser extension and desktop apps ship their own code; they never download it from your server. The web vault does. So if someone gets code execution on your box, they can modify the JavaScript it serves and capture your master password the next time you log in through a browser. That converts a server compromise, which the encryption model is designed to survive, into total vault compromise.

- WEB_VAULT_ENABLED=false

After recreating the container, the root path is gone but the API is untouched:

GET /                                      404
GET /alive                                 200
GET /api/config                            200
POST /identity/accounts/prelogin/password  200
POST /identity/connect/token   (bad pw)     400

A 404 on / with a 200 on /alive is the healthy state. Extensions and desktop and mobile clients keep working exactly as before. What you lose is browser access from a machine where you have not installed a client, which for most people is rarer than it sounds.

The error message that tells you nothing

Here is the failure that cost me the most time, and the lesson generalises.

The Firefox extension refused to log in to the self-hosted server with "an unexpected error has occurred". Nothing else. I checked the obvious things: the certificate chain verified with return code 0, /alive and /api/config both returned 200, prelogin returned proper KDF parameters, and connect/token correctly rejected a deliberately wrong password. Every endpoint I tested by hand behaved perfectly.

The server log had the answer immediately:

POST /identity/accounts/prelogin/password  =>  404 Not Found

Current Bitwarden clients call /identity/accounts/prelogin/password as the first step of login. The version I had pinned did not implement that route. It 404s, and the client surfaces that as a generic error with no indication of which request failed.

Read the server log before touching anything else. Client-side error strings in this ecosystem are close to useless, and the server names the exact route.

The deeper mistake was mine: I had pinned the image version out of habit, the way you would pin a database to avoid an unwanted schema migration. But this is a service whose clients update themselves continuously, from browser stores and app stores you do not control. Pinning it far behind guarantees that eventually a client speaks an API your server does not. For a self-hosted server with auto-updating clients, staying current is the safer default.

Since schema migrations are one-way, back up before upgrading:

sqlite3 vaultwarden-data/db.sqlite3 ".backup '/tmp/vw.sqlite3'"
tar czf /path/to/backups/vaultwarden-manual-$(date -u +%Y%m%dT%H%M%SZ).tar.gz \
    -C /path/to vaultwarden-data
# then bump the tag, docker compose pull && up -d, and check:
docker logs vaultwarden 2>&1 | grep -iE 'error|panic|migrat'
sqlite3 vaultwarden-data/db.sqlite3 'SELECT COUNT(*) FROM ciphers;'

Two things about backups

If you back this up, and you should, two details are easy to get wrong.

Use sqlite3 .backup, not cp. The database runs in WAL mode. When I looked, there was a live 247 KB db.sqlite3-wal sitting next to the main file. Copying just the file captures a torn state and silently loses committed data. .backup takes a proper read lock without stopping the service.

Include rsa_key.pem. It signs the JWTs. Restore without it and every client is silently logged out, which looks like a much worse failure than it is.

And encrypt the backup before it leaves the machine. The vault items are already encrypted client-side, but the database also contains the master password hash, which is offline-crackable. That should not sit unencrypted in someone else's object storage.

Where it actually stands

Setting	State
signups	closed, verified against all three endpoints
web vault	off
admin panel	disabled, no token set
2FA	TOTP enabled
KDF	PBKDF2-SHA256, 600,000 iterations
exposure	loopback only, nginx with TLS 1.2/1.3 and HSTS

The remaining upgrade is switching the KDF to Argon2id, a one-time client-side change, since it is materially harder to attack on GPUs than PBKDF2. Worth knowing that 2FA does nothing for that threat: an attacker cracking a stolen database never touches your API, so only the master password's strength matters there. 2FA protects the online path.

Is it safe?

Cryptographically, self-hosted and hosted Bitwarden are the same design. What differs is who runs it. The hosted service has a security team, audits, DDoS absorption and reliable backups.

For most people hosted is safer in practice, not because of the cryptography but because the failure mode of self-hosting is operational: backups you never took, and patches you never applied. Both of my problems above were exactly that. Self-hosting is a reasonable choice if you own those two things honestly.

Metrics and a Homepage

2026-08-03T00:00:00.000Z

Two additions to a small VPS that was already running a git forge, an RSS reader and a password vault: a metrics stack for actual history, and a homepage for the at-a-glance view. Combined footprint is around 250 MB resident, on a box with 3.7 GB total.

VictoriaMetrics instead of Prometheus

Same query language, same scrape config format, and Grafana treats it as a Prometheus datasource. On this workload it sits at about 157 MB resident where Prometheus would want considerably more, and retention is a single flag (-retentionPeriod=6 for six months, which at a 30 second scrape interval of one node is tens of megabytes).

Nothing about the setup is VictoriaMetrics-specific. If you already know Prometheus, you already know this.

Host networking, deliberately

All three components use network_mode: host and bind their own loopback addresses:

  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.112.0
    network_mode: host
    command:
      - '-httpListenAddr=127.0.0.1:8428'
      - '-promscrape.config=/etc/prometheus.yml'
      - '-retentionPeriod=6'

  node-exporter:
    image: prom/node-exporter:v1.9.1
    network_mode: host
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--web.listen-address=127.0.0.1:9100'
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'

  grafana:
    image: grafana/grafana:11.6.1
    network_mode: host
    environment:
      - GF_SERVER_HTTP_ADDR=127.0.0.1
      - GF_SERVER_HTTP_PORT=3001
      - GF_SERVER_ROOT_URL=https://metrics.example.com
      - GF_USERS_ALLOW_SIGN_UP=false

The alternative, bridge networking with published ports, means VictoriaMetrics cannot reach node_exporter on 127.0.0.1:9100 and Grafana cannot reach VictoriaMetrics. You end up putting them on a shared bridge and hardcoding gateway addresses that change whenever a network is recreated. Host networking with explicit loopback binds is less machinery and exposes nothing; nginx in front is the only thing on a public port.

The catch is that you must set each listen address, because the defaults bind 0.0.0.0. Host networking means a default-bound service is immediately public.

Two smaller traps. grafana-data has to be owned by uid 472, the user Grafana runs as in the official image, or it cannot create its own sqlite database. And despite --path.rootfs=/rootfs, node_exporter still labels the root filesystem mountpoint="/", so queries use that, not /rootfs. I wrote a dashboard panel against /rootfs first and got a silent no-data panel.

Provisioned, not clicked

Datasource and dashboards as files, so the whole thing is reproducible:

# provisioning/datasources/vm.yml
apiVersion: 1
datasources:
  - name: VictoriaMetrics
    type: prometheus
    access: proxy
    uid: victoriametrics
    url: http://127.0.0.1:8428
    isDefault: true

Every panel in the dashboard JSON references {"type": "prometheus", "uid": "victoriametrics"}. If that uid does not match the provisioned datasource, panels render empty with no error at all, which is a miserable thing to debug. It is the first thing to check when a provisioned dashboard looks broken.

Change the default password before the cert exists

Grafana starts on admin/admin. Worth thinking about the ordering here: issuing a TLS certificate publishes the hostname to public Certificate Transparency logs within seconds, and scanners read those logs continuously. In my case automated probes arrived at brand-new hostnames within minutes of the certificate being issued.

So anything with a first-run setup state is exposed from the moment the cert exists, not from the moment you tell someone the URL. That applies to Grafana's default credentials and, more sharply, to any service where the first visitor becomes the administrator, which is how several self-hosted dashboards bootstrap. Set credentials before or immediately after issuing the certificate.

One footgun if you script it: GF_SECURITY_ADMIN_PASSWORD is applied on container start, so a stale value left in your compose file silently resets the password back to it on the next recreate. Either manage the password entirely through that variable, or do not set it at all.

The homepage layer

Separately from Grafana, a Glance instance for the things you want on one page: clock, service up/down monitors, RSS, repository releases, bookmarks, host stats. About 8 MB resident, which makes it hard to argue with.

Two things I would tell anyone setting it up.

Set server.host to 0.0.0.0 inside the container. The default of localhost binds the container's own loopback, which makes a published port unreachable. Docker does the loopback restriction from the outside; the app should not.

Validate the config before restarting. It refuses to start on a bad config, so a typo takes the dashboard down. Validating against the config directory without touching the running container costs nothing:

docker run --rm -v /path/to/config:/app/config:ro glanceapp/glance config:validate

Silence means valid. It reports one error at a time with a line number. This caught two mistakes for me before they mattered: an indentation problem, and two widget types that appear in the current documentation but do not exist in the released binary. When a widget name is rejected, trust the binary over the docs:

docker run --rm --entrypoint sh glanceapp/glance -c \
  'strings "$(command -v glance)" | grep -oE "\b(rss|monitor|todo|calendar-legacy)\b" | sort -u'

Also set slug: explicitly on every page. Auto-derived slugs from titles containing punctuation do not produce the URL you would guess, and the page 404s while the root still works.

Almost every widget needs outbound network

Obvious in hindsight, and worth a sentence because it made the dashboard look broken for reasons unrelated to the dashboard. Glance is mostly a feed aggregator: RSS, releases, monitors, weather, all of it is outbound HTTP from the container. On this host, container egress was silently blackholed by an unrelated firewall problem, so the dashboard rendered clocks and bookmarks and nothing else.

If widgets are uniformly empty, test the container's network before reading any config:

docker exec glance sh -c 'wget -q -T 8 -O /dev/null https://api.github.com && echo OK || echo FAIL'

Encrypted offsite backups that verify themselves

2026-08-03T00:00:00.000Z

I had been running a password vault and a git mirror forge on a VPS for a while with no backups at all. Not a partial backup, not an untested one. None. The disk was a single 40 GB volume and the only copy of anything lived on it.

This is the more likely way to lose data than any attacker, and it is worth stating plainly because the failure mode of self-hosting is almost never a breach. It is a disk, or a fat-fingered rm, or a migration that goes sideways.

What follows is what I built. Two jobs, two mechanisms, and a couple of design decisions that are not obvious.

Encrypt before it leaves the machine

The vault's items are already encrypted client-side. But its database also contains the master password hash, which is offline-crackable. That should not sit in third-party storage in the clear, no matter how much you trust the provider.

I used age, which is a keypair rather than a passphrase:

sudo age-keygen -o /etc/backup/age-identity.key
sudo chmod 600 /etc/backup/age-identity.key
sudo sh -c 'age-keygen -y /etc/backup/age-identity.key > /etc/backup/age-recipient.txt'

The key lives on the box, and that is a deliberate limit worth being explicit about. It does not protect against a compromise of the box, which can read the live database anyway. It protects against exposure at the storage provider. Since an attacker with root already has the plaintext, keeping the key locally costs nothing against that threat and lets the jobs run unattended.

Two mechanisms, because the data has two shapes

The vault database is a few hundred kilobytes. The git mirror tree is 1.7 GB across 71 repositories, of which roughly 200 files change on any given day.

Shipping a 1.7 GB encrypted tarball every night to protect 200 changed files is absurd. But there is no good way to incrementally update an encrypted archive, either. So:

Small things ship as whole age-encrypted archives, one per run, with dated retention.
The git tree goes through rclone sync to an encrypted remote. Git objects are immutable and content-addressed, so after the first push each run transfers only genuinely new objects. Filenames are encrypted too, since the archive is not a single opaque blob.

The tradeoff to understand: rclone sync is a mirror, not history. A repo deleted locally is deleted remotely on the next run. That is acceptable here because the repos are themselves mirrors of upstreams, but if you need point-in-time recovery, use --backup-dir or a real snapshotting tool.

Use the database's own backup command

sqlite3 "$LIVE_DB" ".backup '$STAGING/db.sqlite3'"

Not cp. Both databases here run in WAL mode, and when I looked there was a live 247 KB -wal file next to the main one. Copying just the file captures a torn state and silently loses committed transactions. .backup takes a proper read lock without stopping the service.

The same care applies on the way back: when restoring over a live database, delete the stale -wal and -shm files, because they belong to the database you just replaced.

Every run restores its own output

This is the part I would keep if I threw away everything else.

A backup that has never been restored is not a backup, it is a guess. So the job does not finish by uploading. It finishes by decrypting what it just wrote, unpacking it, and checking the result:

age -d -i "$IDENTITY" "$ARCHIVE" | tar xzf - -C "$VERIFY" \
  || die "verification failed: could not decrypt and untar"

INTEG=$(sqlite3 "$VERIFY/db.sqlite3" 'PRAGMA integrity_check;')
[ "$INTEG" = "ok" ] || die "verification failed: integrity_check said '$INTEG'"

COUNT=$(sqlite3 "$VERIFY/db.sqlite3" 'SELECT COUNT(*) FROM items;')
[ "$COUNT" = "$LIVE_COUNT" ] || die "verification failed: $COUNT in backup vs $LIVE_COUNT live"

Three separate claims get checked: it decrypts, the database is structurally intact, and it contains the same number of rows as the live one. If any fails, the job exits non-zero and does not upload, so a bad archive never displaces a good one in the retention window.

The same paranoia applies to the upload. rclone copy succeeding is not evidence the object is there:

LOCAL=$(stat -c%s "$ARCHIVE")
REMOTE=$(rclone size "$REMOTE_PATH/$(basename "$ARCHIVE")" --json \
         | sed -n 's/.*"bytes":\([0-9]*\).*/\1/p')
[ "$REMOTE" = "$LOCAL" ] || die "size mismatch (local $LOCAL, remote ${REMOTE:-none})"

It also refuses to ship an obviously wrong snapshot at all:

[ "$COUNT" -gt 0 ] || die "snapshot has zero items, refusing to ship it"

An empty backup that overwrites a good one is worse than a failed job.

Two rclone details that cost me time

rclone config create hangs on OAuth backends. Supplying a token on the command line does not stop it trying to run the interactive browser flow, and with stdin not a terminal it blocks forever. Write the config file by hand:

[remote]
type = <provider>
hostname = <region-specific api host>
token = {"access_token":"...","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}

Providers with no token expiry can be wrapped like that from a bare access token.

Region matters and is not auto-detected when you supply a token by hand. rclone normally learns the correct API host during its own OAuth dance. Skip that and it uses the default host, which fails to authenticate for accounts in another region, with an error that looks exactly like a bad token. The provider's docs for the backend usually say which option to set.

Also worth knowing: rclone obscure is reversible obfuscation, not encryption. A crypt password in rclone.conf is readable by anyone with root. Same trade-off as the age key, and worth stating rather than assuming.

And prefer rclone sync --checksum for a git tree. The default mtime-and-size comparison misses repacks that rewrite a file to the same size, and consumer storage often has coarse mtime granularity.

The circular dependency

This is the failure mode that turns a working backup system into encrypted noise, and it is easy to walk into.

The obvious place to store the age key is a password manager. But if the password manager is one of the things being backed up, then recovering the backup requires the key, which requires the password manager, which requires the backup. Circular.

Mine is subtler and I nearly missed it. The key went into pass, which is fine, because pass lives on my laptop and pushes to a git host, both independent of the VPS. But that same pass repo is mirrored onto the forge, and the forge is backed up to the cloud under the key stored inside it. So the cloud copy of pass is not a recovery route for the key, even though it looks like one.

Two rules that fall out of this:

Write down which copies of a secret are genuinely independent of the thing it decrypts. "It's in three places" means nothing if two of them are downstream of the encrypted backup.
Keep one copy on a medium that has no dependency on any of it. An age key is 184 bytes over three lines. It fits on a piece of paper.

There is a hardware version of the same trap: if your password store is GPG-encrypted to a key on a hardware token, then losing the token and the laptop together leaves the store unreadable, and the backups undecryptable with it.

Shape of the result

Two systemd timers, half an hour apart so they do not contend for two vCPUs and one uplink, both Persistent=true so a missed run catches up after a reboot. Local retention of 14 days, remote 90, with the git tree mirrored rather than dated. First full push of 1.7 GB took about 17 minutes; nightly runs move the couple of hundred files that actually changed.

The uncomfortable thing I would flag to anyone doing this: my scripts live in /usr/local/bin and my configs in /etc, and neither is backed up. The data is safe and the machinery that protects it is not. For now the mitigation is that both scripts are reproduced in full in my notes, which is a real answer but not a good one. Backing up /etc is the obvious next job.

PGP and SSH keys on a YubiKey

2026-08-01T00:00:00.000Z

Two separate things that both end up on the same piece of hardware: a PGP keypair whose private half lives on the card, and FIDO2-backed SSH keys. They solve different problems and are worth keeping straight.

Generating and moving a PGP key

gpg --expert --full-gen-key

Export everything before it goes anywhere near the card. keytocard is a move, not a copy: it writes the key to the card and removes the private half from your keyring. Without a backup at this point the key is gone the moment the YubiKey is lost.

gpg --export-secret-key --armor $KEYID > privkey.armor
gpg --export-secret-key $KEYID | paperkey > privkey.paperkey
gpg --export $KEYID > pubkey.armor

paperkey strips out everything reconstructible from the public key and leaves only the secret bits, which is what makes a printed copy a reasonable size. Store the armored copy somewhere offline.

Then move it:

gpg --edit-key $KEYID

and keytocard. Select the key first by entering key 1, otherwise the command operates on the primary key rather than the subkey you meant.

Restoring on another machine

The card holds the private key, but a fresh machine still has no idea the public key exists. Fetch it from a keyserver:

gpg --keyserver hkps://keys.openpgp.org --search-keys "you@example.com"

Then pull the card's public key stubs, which tell GnuPG that the private key lives on hardware:

gpg --edit-card

and fetch.

Finally, trust it:

gpg --edit-key $KEYID

and trust. Note that this is --edit-key, not --edit-card. Getting these two confused is easy and the error messages do not point at the mistake.

git credentials through pass

pass-git-helper lets git pull credentials out of a pass store, which is itself encrypted to the PGP key on the card:

yay -S pass-git-helper
git config --global credential.helper /usr/bin/pass-git-helper
git config credential.useHttpPath true

Create ~/.config/pass-git-helper/git-pass-mapping.ini:

[github.com]
target=github.com/<username>

credential.useHttpPath true matters if you keep per-repository credentials, since without it every host collapses to a single entry.

FIDO2-backed SSH keys

Separate mechanism, same token. OpenSSH 8.2 added the ed25519-sk and ecdsa-sk key types, where the private key is bound to the hardware token and useless without it. libfido2 is required.

Both ends need to understand the key type, so an old server will reject these outright.

ssh-keygen -t ed25519-sk

You will be asked for the PIN and a touch to confirm generation, and normally a touch on every connection afterwards.

Point your config at the key explicitly:

Host SERVER1
    IdentitiesOnly yes
    IdentityFile ~/.ssh/id_ed25519_sk

IdentitiesOnly yes is worth setting. Without it, ssh offers every key it can find, which on a token means a touch prompt for keys you did not intend to use.

Skipping the touch requirement

ssh-keygen -O no-touch-required -t ed25519-sk

Three caveats, in increasing order of annoyance:

Not every token supports it. On a YubiKey, ed25519-sk needs firmware 5.2.3 or newer.
sshd rejects no-touch-required keys by default. Allow it per key in authorized_keys:
```
no-touch-required sk-ssh-ed25519@openssh.com AAAAInN... user@example.com
```
or system-wide in /etc/ssh/sshd_config with PubkeyAuthOptions none.
GitHub and GitLab do not support no-touch-required at all. If these keys are for pushing to a forge, you are touching the token on every push regardless of what you generated.

An ecdsa-sk keypair works the same way, with the usual reasons to prefer ed25519 over ECDSA still applying.

Reference: the Arch wiki's SSH keys page and Yubico's importing keys guide.

eSIM and WWAN on a ThinkPad P14s under Arch

2026-08-01T00:00:00.000Z

Every piece of this is documented somewhere, and almost none of it is documented together. This is the whole path from a dead WWAN card to an eSIM profile installed and enabled on a ThinkPad P14s Gen 5 AMD running Arch, plus the failure modes that cost the most time.

Fair warning up front: the profile downloads and enables, but the modem still does not attach to the network. The open problem is at the bottom, unresolved. Everything above it is verified working.

Hardware

Component	Value
Laptop	ThinkPad P14s Gen 5 AMD
Modem	Quectel RM520N-GL (5G, Qualcomm)
USB ID	`2c7c:0801`
Firmware	`RM520NGLAAR03A03M4G`
SIM slots	1 = physical nano-SIM, 2 = embedded eSIM (eUICC)
eUICC free memory	~411 KB
AT port	`/dev/wwan0at0`
MBIM port	`/dev/wwan0mbim0`

The two device nodes matter. lpac talks to the eUICC over the AT port, while radio state is set over MBIM. Reaching for the wrong one produces errors that look like hardware faults.

1. Firmware

Firmware comes through fwupdmgr, but current fwupd will not do it. You need to downgrade to 1.9.x from the Arch Archive. Stop ModemManager and unload libmbim/libqmi first, or the update fails holding a busy device.

fwupdmgr refresh --force
fwupdmgr get-devices
fwupdmgr update

Confirm the modem still answers afterwards:

minicom -D /dev/wwan0at0

Type AT, expect OK.

2. FCC unlock

Lenovo ships the modem FCC-locked. The radio stays off until an unlock runs. ModemManager has the script, it just is not wired up:

sudo ln -s /usr/share/ModemManager/fcc-unlock.available.d/1eac \
           /etc/ModemManager/fcc-unlock.d/1eac:1007

If the automatic unlock does not take, force the radio on directly:

mbimcli --device-open-proxy --device="/dev/wwan0mbim0" \
        --quectel-set-radio-state=on

There is also an AUR package that handles this as a service:

yay -S lenovo-wwan-unlock
sudo systemctl enable --now lenovo-wwan-unlock

3. Switch to the eSIM slot

The eUICC is slot 2. Stop ModemManager before touching the AT port, otherwise the two fight over it:

sudo systemctl stop ModemManager
sudo minicom -D /dev/wwan0at0

AT+QUIMSLOT=2

AT+QUIMSLOT? reports the current slot.

4. Install lpac

lpac is the eSIM profile manager, the LPA in the specification's terms.

paru -S lpac-git
sudo pacman -S pcsclite ccid
sudo systemctl enable --now pcscd.socket

The gotcha that wasted the most time. The environment variables must be passed through sudo, on the same side as the command:

sudo LPAC_APDU=at LPAC_APDU_AT_DEVICE=/dev/wwan0at0 lpac chip info

Setting them before sudo means lpac never sees them, falls back to the PC/SC backend, and fails with SCardEstablishContext, which looks like a smartcard daemon problem and is not one.

Also note the older names AT_DEBUG and AT_DEVICE are deprecated. Use LPAC_APDU_AT_DEBUG and LPAC_APDU_AT_DEVICE.

5. Download a profile

An eSIM QR code encodes LPA:1$<SMDP_ADDRESS>$<ACTIVATION_CODE>. Split it on the $ and feed the halves in:

sudo LPAC_APDU=at LPAC_APDU_AT_DEVICE=/dev/wwan0at0 \
     lpac profile download -s <SMDP_ADDRESS> -m "<ACTIVATION_CODE>"

This needs working internet over some other interface, Wi-Fi being the obvious one, since the download happens over IP and not over the cellular link you are trying to provision.

6. Enable it

Downloading is not enabling. A downloaded but disabled profile reports as esim-without-profiles, which reads like the download failed when it did not:

sudo LPAC_APDU=at LPAC_APDU_AT_DEVICE=/dev/wwan0at0 \
     lpac profile enable <ICCID>

Then reset the modem from minicom so it re-reads the eUICC:

AT+CFUN=1,1

7. Connect

sudo systemctl start ModemManager
mmcli -m 0 --enable
nmcli connection add type gsm ifname "*" con-name <name> apn <apn>
nmcli connection up <name>

Known issues

Collected from the actual attempts, in rough order of how much time each cost:

SCardEstablishContext: LPAC_APDU=at did not reach lpac through sudo. See above.
esim-without-profiles after a successful download: the profile is there but disabled. lpac profile enable.
lpac hangs: the AT port can lock up during long operations. Reboot or rebind the USB device, then check state with lpac profile list.
PhoneFailure on a network scan: radio is off. Re-run the mbimcli radio-state command and retry.
EID mismatch on download: the profile was bound to a different device's eUICC. There is no fix on the Linux side; the carrier has to re-provision against the right EID.
Activation code already consumed: once a profile has been downloaded to any device, the code is spent. It cannot be reused, including after a failed install on the same machine.
Profiles do not transfer between devices: once activated on one eUICC the binding is permanent. Confirmed with the provider. Moving to another laptop means buying another profile, which is worth knowing before you buy the first one.
General modem instability: the RM520N-GL resets itself fairly often, and the radio-state command frequently needs repeating afterwards.

The part that does not work

The profile is installed and enabled. The modem does not attach.

mmcli -m 0 --3gpp-scan fails with PhoneFailure even directly after a successful radio-state-on, and the correct APN for the provider is still unknown; the plausible candidates each fail the same way, so the scan failure is probably the real blocker rather than the APN.

Two open threads: whether the repeated self-resets are a firmware problem that a newer RM520NGL build fixes, and whether the profile needs a carrier-side activation step that was never performed. I will update this post when the link comes up.

Command reference

Task	Command
List modems	`mmcli -L`
Modem details	`mmcli -m 0`
Network scan	`mmcli -m 0 --3gpp-scan`
Radio on	`mbimcli --device-open-proxy --device="/dev/wwan0mbim0" --quectel-set-radio-state=on`
eUICC info	`sudo LPAC_APDU=at LPAC_APDU_AT_DEVICE=/dev/wwan0at0 lpac chip info`
List profiles	`sudo LPAC_APDU=at LPAC_APDU_AT_DEVICE=/dev/wwan0at0 lpac profile list`
Enable profile	`sudo LPAC_APDU=at LPAC_APDU_AT_DEVICE=/dev/wwan0at0 lpac profile enable <ICCID>`
Modem reset	`AT+CFUN=1,1` (minicom)
Check SIM slot	`AT+QUIMSLOT?` (minicom)
Switch to eSIM	`AT+QUIMSLOT=2` (minicom)

Your own IMEI, EID, and ICCID appear throughout lpac and mmcli output. They identify your hardware and your SIM subscription, so keep them out of anything you paste into a forum thread or an issue tracker.

Linux tips and tricks

2026-08-01T00:00:00.000Z

Accumulated notes that are individually too small to be posts and collectively annoying to rediscover. Arch on a desktop and a ThinkPad, Hyprland, fish.

Dotfiles across two machines

The interesting problem is not "put configs in git," it is keeping two machines mostly identical without either one's quirks leaking into the other. Three long-lived branches:

Branch	Role
`base`	shared, machine-neutral config, byte-identical across machines
`master`	desktop, `base` plus desktop bits
`thinkpad`	laptop, `base` plus laptop bits

A shared change is committed on base and propagated:

git switch master   && git merge base
git switch thinkpad && git merge base

A machine-specific change is committed on that machine's branch only. Catching a shared change on a machine branch means cherry-picking it back to base.

Deployment is GNU Stow:

stow -t ~/.config -S <module>

Per-machine files are not tracked at all. They are gitignored and seeded from a tracked *.example template, which is what keeps base genuinely identical everywhere rather than merely similar:

cp hypr/hypr/monitors.conf.example      hypr/hypr/monitors.conf
cp kitty/kitty/kitty-local.conf.example kitty/kitty/kitty-local.conf

The generated-file problem

Some tracked files get rewritten at runtime. Fish toggles a helix theme file between light and dark, for instance. You want the committed copy to exist so a fresh clone works, but you do not want the local churn showing up in every git status.

.gitignore cannot do this. It only affects untracked files, and these are tracked on purpose. The tool is skip-worktree:

git update-index --skip-worktree <file>

To update the committed baseline later, reverse it, commit, and re-apply:

git update-index --no-skip-worktree <file>

Worth keeping this in a script, because the list grows and reapplying it is a per-clone step that is easy to forget.

Networking

dnsmasq under NetworkManager

Local caching resolver, with NetworkManager managing it:

# /etc/NetworkManager/conf.d/dns.conf
[main]
dns=dnsmasq

sudo nmcli general reload

NetworkManager starts dnsmasq itself and points /etc/resolv.conf at 127.0.0.1. The upstream servers it actually forwards to end up in /run/NetworkManager/no-stub-resolv.conf.

To confirm caching is real rather than assumed, run the same lookup twice and compare query times:

drill example.com

See the Arch wiki.

mDNS and `.local`

sudo pacman -S avahi nss-mdns
sudo systemctl enable --now avahi-daemon.service

Then /etc/nsswitch.conf:

hosts: mymachines mdns_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] files myhostname dns

Order matters here. mdns_minimal has to come before resolve and dns, or .local names go to the upstream resolver and fail.

DNS inside Docker containers

If name resolution fails inside a container, check whether you are running a DNS resolver container. If so, point the host's /etc/resolv.conf at something external such as 1.1.1.1. The container inherits the host's resolver, and a resolver that is itself containerised produces a loop.

Keyboard

Caps lock, via keyd

Install and enable keyd, then /etc/keyd/default.conf:

[ids]

*

[main]

# Escape when tapped, control when held.
capslock = overload(control, esc)

leftshift+leftmeta = layer(meta)

Works at the evdev layer, so it applies under both X and Wayland, unlike the various per-compositor remapping options.

Vial and QMK keyboards

A udev rule is needed before Vial can talk to the board. Get the vendor and product IDs out of lsusb first:

echo 'KERNEL=="hidraw*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="3434", ATTRS{idProduct}=="0350", MODE="0660", GROUP="users", TAG+="uaccess", TAG+="udev-acl"' \
  | sudo tee /etc/udev/rules.d/99-vial.rules

Replace both IDs with your own.

Power and boot

AMD P-state

Add amd_pstate=active to the kernel command line. In /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 quiet nvidia-drm.modeset=1 amd_pstate=active"

systemd-boot entry

title   Arch Linux (linux)
linux   /vmlinuz-linux
initrd  /initramfs-linux.img
options root=PARTUUID=<partuuid> rootflags=subvol=@ rw rootfstype=btrfs nvidia-drm.modeset=1 quiet splash

Get the PARTUUID from blkid. To dual boot Windows, copy EFI\Microsoft to /boot/EFI.

Recovering a lost boot entry

From a live image:

mount /dev/nvme0n1p1 /boot
mount /dev/nvme0n1p2 /mnt
arch-chroot /mnt

grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=arch
grub-mkconfig -o /boot/grub/grub.cfg

Storage

blkid gives the UUIDs for /etc/fstab:

# <file system> <dir> <type> <options> <dump> <pass>
UUID=<uuid>  /                     ext4  rw,relatime  0 1
UUID=<uuid>  /home/<user>/Storage  ext4  rw,relatime  0 1
UUID=<uuid>  /boot                 vfat  rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro  0 2

btrfs subvolume mount options worth reusing:

mount -o noatime,compress=zstd,space_cache=v2,subvol=@ /dev/sdX /mnt

rclone over SFTP

~/.config/rclone/rclone.conf:

[remote]
type = sftp
host = xfer.hpc.example.edu
user = <username>
key_file = ~/.ssh/id_ed25519
key_file_pass = <obscured>
md5sum_command = md5sum
sha1sum_command = sha1sum
shell_type = unix

key_file_pass is not plaintext. Generate it with echo "passphrase" | rclone obscure -, and note that obscuring is obfuscation, not encryption; the config file still deserves 600.

rclone mount remote:/some/path ~/mnt/path
rclone rcd --rc-web-gui --rc-no-auth   # local web UI

Desktop odds and ends

Default file manager for directories:

xdg-mime default org.gnome.Nautilus.desktop inode/directory

A custom .desktop entry for a terminal program:

[Desktop Entry]
Type=Application
Name=Lynx
Exec=lynx %u
Terminal=true

Packages that come up on every fresh install:

yay -S nerd-fonts-meta noto-fonts-emoji bibata-cursor-theme-bin ncpamixer

SMB in Nautilus needs both samba and gvfs-smb, then a restart.
Bluetooth headphones need pulseaudio-bluetooth.
gnome-keyring is what lets VS Code use keyring storage.
Chinese input: Rime.

Bluetooth controllers

bluetoothctl
scan on
trust  AA:BB:CC:DD:EE:FF
connect AA:BB:CC:DD:EE:FF

If an Xbox controller refuses to pair, its firmware probably needs updating, which unfortunately requires Windows.

A Windows VM without libvirt

qemu-system-x86_64 -enable-kvm -cpu host -smp $(nproc) -m 16G \
  -vga virtio -display gtk,gl=on -usb -device usb-tablet \
  -device ich9-intel-hda -device hda-output \
  -boot d -cdrom ~/Downloads/windows.iso \
  -drive format=qcow2,file=windows

gnome-boxes is the easier route and prefers qcow2 over raw.

Forgejo on a small VPS: a backup forge

2026-07-31T00:00:00.000Z

Every repository I care about lives on Codeberg, GitLab or GitHub. None of those is a backup, they are just someone else's disk. This is the full deployment of a self-hosted Forgejo that pull-mirrors all three onto a VPS I control, including every config file and script, plus the handful of things that are genuinely easy to get wrong.

Two boxes went through this, one with 950 MiB of RAM and one vCPU, and a later one with 3.7 GB and two. The configs below are the larger box, with the smaller box's numbers noted where they differ, because the memory tuning is the one part you should not copy between machines.

Throughout: git.example.com is the forge hostname, myuser the account, myorg an upstream organisation.

Install

Verify the binary. A forge is the last place to run an unverified download:

V=16.0.2
curl -fsSL --http1.1 --retry 3 -C - -O \
  https://codeberg.org/forgejo/forgejo/releases/download/v$V/forgejo-$V-linux-amd64
curl -fsSL --http1.1 -O \
  https://codeberg.org/forgejo/forgejo/releases/download/v$V/forgejo-$V-linux-amd64.sha256
curl -fsSL --http1.1 -O \
  https://codeberg.org/forgejo/forgejo/releases/download/v$V/forgejo-$V-linux-amd64.asc

sha256sum -c forgejo-$V-linux-amd64.sha256
gpg --keyserver hkps://keys.openpgp.org \
    --recv-keys EB114F5E6C0DC2BCDD183550A4B61A2DC5923710
gpg --verify forgejo-$V-linux-amd64.asc forgejo-$V-linux-amd64

sudo install -m 755 forgejo-$V-linux-amd64 /usr/local/bin/forgejo

The checksum file sits on the same server as the binary, so on its own it proves nothing. The signature is the part that matters, and the key should come from somewhere other than where the artifact lives.

--http1.1 is not decoration. Codeberg's HTTP/2 aborts large transfers from some hosts with curl 92 ... stream not closed cleanly: CANCEL, and a 119 MB download is large enough to hit it. Same flag, same reason, shows up again later for git itself.

Then the service account:

sudo apt-get install -y nginx certbot git sqlite3 nftables zstd gnupg
sudo adduser --system --shell /bin/bash --gecos 'Forgejo' \
     --group --disabled-password --home /home/git git
sudo mkdir -p /var/lib/forgejo/{custom,data,log} /etc/forgejo
sudo chown -R git:git /var/lib/forgejo
sudo chown root:git /etc/forgejo && sudo chmod 770 /etc/forgejo

/etc/forgejo is root:git and app.ini is mode 640, so the service reads its config but can never write it. That is deliberate, and it is why the OAuth2 JWT secret has to be set by hand below.

Generate the secrets before writing the config:

for s in SECRET_KEY INTERNAL_TOKEN LFS_JWT_SECRET JWT_SECRET; do
  echo "$s = $(forgejo generate secret $s)"
done

app.ini

Skip the web installer entirely: set INSTALL_LOCK = true and create the admin from the CLI, otherwise the setup page is exposed until someone clicks through it.

; Forgejo configuration for the backup forge (Debian 13, 2 vCPU, 3.7 GB RAM,
; 40 GB disk). Role: pull-mirror of Codeberg, GitLab and GitHub. Not a primary
; forge, so search/CI/packages stay off and the tuning favours reliable large
; clones over interactive responsiveness.

APP_NAME = backup forge
RUN_USER = git
RUN_MODE = prod
WORK_PATH = /var/lib/forgejo

[server]
PROTOCOL         = http
DOMAIN           = git.example.com
ROOT_URL         = https://git.example.com/
; Loopback only; nginx terminates TLS and proxies in.
HTTP_ADDR        = 127.0.0.1
HTTP_PORT        = 3000
DISABLE_SSH      = false
; Built-in SSH server, so the host sshd and /home/git/.ssh stay untouched.
START_SSH_SERVER = true
SSH_LISTEN_HOST  = 0.0.0.0
SSH_LISTEN_PORT  = 2222
SSH_DOMAIN       = git.example.com
SSH_PORT         = 2222
; Forgejo generates RSA only, and will not generate a type it was not told
; about. The ed25519 key is created alongside this file.
SSH_SERVER_HOST_KEYS = ssh/gitea.rsa,ssh/gitea.ed25519
LFS_START_SERVER = true
LFS_JWT_SECRET   = <forgejo generate secret LFS_JWT_SECRET>
OFFLINE_MODE     = true
APP_DATA_PATH    = /var/lib/forgejo/data
; nginx handles compression.
ENABLE_GZIP      = false

[database]
DB_TYPE  = sqlite3
PATH     = /var/lib/forgejo/data/forgejo.db
SQLITE_JOURNAL_MODE = WAL
MAX_OPEN_CONNS = 10
MAX_IDLE_CONNS = 5
CONN_MAX_LIFETIME = 0

[repository]
ROOT = /var/lib/forgejo/data/forgejo-repositories
DEFAULT_BRANCH = main
; A sink, not a source. Discourage accidental local work.
DEFAULT_PRIVATE = private
DEFAULT_PUSH_CREATE_PRIVATE = true
DISABLE_HTTP_GIT = false

[repository.upload]
FILE_MAX_SIZE = 64
MAX_FILES = 10

[repository.pull-request]
DEFAULT_MERGE_STYLE = merge

[mirror]
ENABLED = true
DISABLE_NEW_PULL = false
; Pushing mirrors out of here is not the point; off to avoid surprises.
DISABLE_NEW_PUSH = true
DEFAULT_INTERVAL = 24h
MIN_INTERVAL = 10m

[cron]
ENABLED = true
RUN_AT_START = false

[cron.update_mirrors]
ENABLED = true
; Every 6h; each run only syncs mirrors whose own interval has elapsed.
SCHEDULE = @every 6h
; 2 vCPU, so a little more parallelism than the previous single-core host.
PULL_LIMIT = 4
PUSH_LIMIT = 0

[cron.git_gc_repos]
ENABLED  = true
SCHEDULE = @every 168h
TIMEOUT  = 180m
ARGS     = --aggressive=false

[cron.repo_health_check]
ENABLED  = true
SCHEDULE = @every 168h
TIMEOUT  = 60m

[cron.archive_cleanup]
ENABLED  = true
SCHEDULE = @every 24h
OLDER_THAN = 24h

[cron.cleanup_actions]
ENABLED = true

[cron.delete_old_system_notices]
ENABLED = true
SCHEDULE = @every 168h
OLDER_THAN = 720h

[git]
MAX_GIT_DIFF_LINES = 1000
MAX_GIT_DIFF_LINE_CHARACTERS = 5000
MAX_GIT_DIFF_FILES = 100
GC_ARGS = --no-cruft

[git.timeout]
DEFAULT  = 360
MIGRATE  = 3600
MIRROR   = 1800
CLONE    = 1800
PULL     = 1800
GC       = 600

[git.config]
; Codeberg's HTTP/2 aborts mid-fetch on large repos (curl 92 / stream CANCEL),
; which surfaces as a failed clone or a confusing "unable to rename temporary
; '*.pack' file" once Forgejo cleans up behind it. Pin HTTP/1.1.
http.version = HTTP/1.1
http.postBuffer = 524288000
; Bounded repack cost. Roomier than a 1 GB host needs, still bounded so one
; huge repo cannot swallow all 3.7 GB.
pack.threads = 2
pack.windowMemory = 256m
pack.packSizeLimit = 1g
pack.deltaCacheSize = 64m
core.bigFileThreshold = 32m
core.packedGitWindowSize = 32m
core.packedGitLimit = 512m
receive.maxInputSize = 0
user.name  = myuser
user.email = me@example.com

[indexer]
; bleve is the heaviest component in a default install and buys nothing on a
; forge whose issues nobody searches. Affordable here, still not worth it.
ISSUE_INDEXER_TYPE = db
REPO_INDEXER_ENABLED = false

[queue]
TYPE = level
DATADIR = /var/lib/forgejo/data/queues/
LENGTH = 200
BATCH_LENGTH = 20
MAX_WORKERS = 2

[cache]
ENABLED = true
ADAPTER = memory
INTERVAL = 60
HOST =
ITEM_TTL = 8h

[session]
PROVIDER = file
PROVIDER_CONFIG = /var/lib/forgejo/data/sessions
COOKIE_SECURE = true
SESSION_LIFE_TIME = 604800

[security]
INSTALL_LOCK = true
SECRET_KEY = <forgejo generate secret SECRET_KEY>
INTERNAL_TOKEN = <forgejo generate secret INTERNAL_TOKEN>
PASSWORD_HASH_ALGO = argon2
PASSWORD_COMPLEXITY = lower,upper,digit
MIN_PASSWORD_LENGTH = 12
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = 127.0.0.0/8,::1/128
LOGIN_REMEMBER_DAYS = 30

[service]
DISABLE_REGISTRATION              = true
ALLOW_ONLY_EXTERNAL_REGISTRATION  = false
REGISTER_EMAIL_CONFIRM            = false
ENABLE_NOTIFY_MAIL                = false
REQUIRE_SIGNIN_VIEW               = true
DEFAULT_KEEP_EMAIL_PRIVATE        = true
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
ENABLE_TIMETRACKING               = false
NO_REPLY_ADDRESS                  = noreply.git.example.com
ENABLE_CAPTCHA                    = false

[oauth2]
; Set explicitly, so Forgejo never needs to write to this file. It treats a
; failed write of a generated JWT secret as fatal.
ENABLED = false
JWT_SECRET = <forgejo generate secret JWT_SECRET>

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[actions]
ENABLED = false

[packages]
; Package storage would compete with repository data for the same disk.
ENABLED = false

[ui]
EXPLORE_PAGING_NUM = 20
ISSUE_PAGING_NUM   = 20
FEED_MAX_COMMIT_NUM = 5
GRAPH_MAX_COMMIT_NUM = 100
DEFAULT_THEME = forgejo-auto

[log]
MODE = console
LEVEL = info
ROOT_PATH = /var/lib/forgejo/log

[log.console]
STDERR = false

[metrics]
ENABLED = false

[federation]
ENABLED = false

[other]
SHOW_FOOTER_VERSION = false
SHOW_FOOTER_TEMPLATE_LOAD_TIME = false

The parts worth explaining:

ISSUE_INDEXER_TYPE = db. A default install runs bleve, the heaviest component in the whole process, to make issues searchable. On a forge whose issues nobody reads, that is pure overhead. On the 950 MiB box it was the single biggest win.

[oauth2] JWT_SECRET set explicitly. Leave it out and Forgejo generates one at startup, tries to write it back into app.ini, fails because the file is not writable by git, and treats that as fatal. The symptom is a restart loop with save oauth2.JWT_SECRET failed: permission denied, which does not obviously point at file permissions.

SSH_SERVER_HOST_KEYS names ed25519. Forgejo generates RSA only, and will not generate a key type it was not told about, so create it yourself:

sudo mkdir -p /var/lib/forgejo/data/ssh
sudo chown -R git:git /var/lib/forgejo/data/ssh
sudo chmod 700 /var/lib/forgejo/data/ssh
sudo -u git ssh-keygen -t ed25519 -N '' -f /var/lib/forgejo/data/ssh/gitea.ed25519

Create and chown the directory before running ssh-keygen as git, or it fails with permission denied.

http.version = HTTP/1.1 under [git.config]. The same Codeberg HTTP/2 problem as the download. Without it, large mirror clones fail, sometimes as the genuinely baffling unable to rename temporary '*.pack' file to ...: No such file or directory, which is a secondary symptom: git's transfer died, Forgejo removed the half-built repository, and git then tried to finish writing into a directory that no longer existed.

[git.config] at all. Forgejo writes that section into its own .gitconfig, so it applies to every git it forks. That matters because the process which exhausts a small box is never Forgejo, it is a git repack on a large mirror.

The unit file

[Unit]
Description=Forgejo (backup mirror forge)
Documentation=https://forgejo.org/docs/
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=git
Group=git
WorkingDirectory=/var/lib/forgejo/
RuntimeDirectory=forgejo
RuntimeDirectoryMode=0750

ExecStart=/usr/local/bin/forgejo web --config /etc/forgejo/app.ini
Restart=always
RestartSec=10s

Environment=USER=git HOME=/home/git GITEA_WORK_DIR=/var/lib/forgejo
# 3.7 GB box: give the heap real room, but still bounded.
Environment=GOMEMLIMIT=1400MiB
Environment=GOMAXPROCS=2

# --- Resource containment -------------------------------------------------
# MemoryHigh throttles and swaps first; MemoryMax is the hard wall. A runaway
# `git repack` dies inside this cgroup instead of letting the kernel OOM killer
# pick a victim by badness score (which could just as easily be sshd).
MemoryHigh=2000M
MemoryMax=2800M
MemorySwapMax=1500M

# --- Hardening -----------------------------------------------------------
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectClock=true
ProtectHostname=true
ProtectProc=invisible
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
LockPersonality=true
# Loose enough not to break git repacks.
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
CapabilityBoundingSet=
AmbientCapabilities=

# ProtectSystem=strict makes / read-only; punch through only what we own.
ReadWritePaths=/var/lib/forgejo
ReadWritePaths=/etc/forgejo
ReadWritePaths=/home/git

[Install]
WantedBy=multi-user.target

MemoryHigh throttles and pushes to swap first, MemoryMax is the hard wall. The point is not to be stingy, it is that without a cgroup limit the kernel OOM killer picks its victim by badness score, and on a small box that is as likely to be sshd as the process that actually misbehaved. Bounding the service means a runaway repack dies alone and everything else survives.

Sized for 3.7 GB above. The 950 MiB box used GOMEMLIMIT=380MiB, MemoryHigh=520M, MemoryMax=760M, and correspondingly pack.threads = 1, pack.windowMemory = 96m, PULL_LIMIT = 2. Idle footprint was about 230 MB there and 110 MB here. Do not copy these numbers between machines in either direction.

If the box has no swap, add some, or MemorySwapMax has nowhere to spill:

sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

One firewall, not two

#!/usr/sbin/nft -f
# Single source of truth for inbound filtering on this host.
#
# Deliberately the ONLY input firewall here: multiple nftables base chains can
# attach to the same hook at the same priority and all of them run, so a second
# tool (ufw, firewalld) layered on top means every future port has to be opened
# in both places or it silently fails.
#
# Apply with: sudo nft -f /etc/nftables.conf   (nftables.service does this at boot)

table inet filter
delete table inet filter

table inet filter {
	chain input {
		type filter hook input priority filter; policy drop;

		iifname "lo" accept
		ct state established,related accept
		ct state invalid drop

		icmp type { echo-request, destination-unreachable, time-exceeded, parameter-problem } accept
		icmpv6 type { echo-request, destination-unreachable, time-exceeded, parameter-problem, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept

		tcp dport 22 accept   # host sshd
		tcp dport 80 accept   # nginx: ACME HTTP-01 + redirect to https
		tcp dport 443 accept  # nginx -> Forgejo
		tcp dport 2222 accept # Forgejo built-in SSH (git over ssh)

		udp dport 68 accept   # DHCP lease renewal (v4)
		udp dport 546 accept  # DHCP lease renewal (v6)
	}

	chain forward {
		type filter hook forward priority filter; policy drop;
	}

	chain output {
		type filter hook output priority filter; policy accept;
	}
}

sudo nft -f /etc/nftables.conf
sudo nft list chain inet filter input   # confirm the port 22 rule BEFORE logging out
sudo systemctl enable --now nftables

This is the part I got wrong, and it cost the most time, so it is worth being precise about why.

Multiple nftables base chains can attach to the same hook at the same priority, and every one of them runs. A packet has to be accepted by all of them, and a drop in any single chain is final. There is no first-match-wins across chains and no precedence to reason about.

So on a host that already had a policy drop table from an unrelated service, my freshly configured ufw reported 443 ALLOW IN and was completely powerless, because the other table only ever allowed udp dport 443 and never tcp. The service answered perfectly over loopback and was unreachable from the internet.

The counters say so plainly, if you look:

chain ufw-user-input {
  tcp dport 22   counter packets 40 bytes 2320 accept
  tcp dport 443  counter packets 0  bytes 0    accept
}

Zero packets on a rule that should be busy means that rule is not in the path. Read the whole ruleset with nft list ruleset, not just the table you wrote.

TLS

Get the certificate before writing a vhost that references it, since nginx will not start on a missing certificate file:

sudo certbot certonly --webroot -w /var/www/html -d git.example.com \
     --non-interactive --agree-tos -m me@example.com


# git.example.com -> Forgejo on 127.0.0.1:3000
#
# Works both DNS-only and behind the Cloudflare proxy. When the orange cloud
# is on, real client IPs arrive in CF-Connecting-IP and are restored by
# /etc/nginx/conf.d/cloudflare-realip.conf.

upstream forgejo {
    server 127.0.0.1:3000;
    keepalive 4;
}

server {
    listen 80;
    listen [::]:80;
    server_name git.example.com;

    # Leave room for certbot renewals over HTTP-01.
    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name git.example.com;

    ssl_certificate     /etc/letsencrypt/live/git.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/git.example.com/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/git.example.com/chain.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:2m;
    ssl_session_timeout 1d;

    add_header Strict-Transport-Security "max-age=31536000" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    # Git pushes and LFS uploads can be large; do not cap them at the proxy.
    # (Note: the Cloudflare free plan enforces its own 100 MB request cap.)
    client_max_body_size 0;
    # Stream request bodies straight through instead of spooling to disk;
    # /tmp is a 476 MB tmpfs on this host, so buffering a big push would
    # consume RAM.
    proxy_request_buffering off;
    proxy_buffering off;
    proxy_http_version 1.1;

    # Large clones on 1 vCPU are slow. Be patient.
    proxy_connect_timeout 60s;
    proxy_send_timeout    900s;
    proxy_read_timeout    900s;

    location / {
        proxy_pass http://forgejo;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_set_header Connection        "";
    }
}

client_max_body_size 0 and proxy_request_buffering off are both about git. Unbuffered matters more than it looks: if /tmp is a tmpfs, which it is by default on recent Debian and Ubuntu, then buffering a large push spools it into RAM.

certonly installs no reload hook, so a renewal two months out would quietly keep serving the old certificate until something restarted nginx:

sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh >/dev/null <<'EOF'
#!/bin/sh
set -e
nginx -t && systemctl reload nginx
EOF
sudo chmod 755 /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
sudo certbot renew --dry-run

Delete any ssl_stapling lines from your boilerplate too. Let's Encrypt no longer runs OCSP responders, so they only produce a warning.

Behind Cloudflare

If the domain is already on Cloudflare, restore real client IPs at the origin or every log line and rate limit sees the edge instead of the client:

#!/bin/bash
# Refresh the nginx set_real_ip_from list from Cloudflare published ranges.
# Safe to run while DNS-only: set_real_ip_from only trusts CF-Connecting-IP
# when the request actually arrives from one of these ranges.
set -euo pipefail
OUT=/etc/nginx/conf.d/cloudflare-realip.conf
TMP=$(mktemp)
cleanup() { rm -f "$TMP"; }
trap cleanup EXIT
{
  echo "# Generated by cloudflare-realip-update on $(date -Is). Do not edit."
  for u in https://www.cloudflare.com/ips-v4 https://www.cloudflare.com/ips-v6; do
    curl -fsS --max-time 20 "$u" | sed "s/^/set_real_ip_from /; s/$/;/"
  done
  echo "real_ip_header CF-Connecting-IP;"
  echo "real_ip_recursive on;"
} > "$TMP"
grep -q "^set_real_ip_from" "$TMP" || { echo "no ranges fetched; keeping existing" >&2; exit 1; }
install -m 644 "$TMP" "$OUT"
nginx -t && systemctl reload nginx

Safe to install before enabling the proxy, because set_real_ip_from only trusts CF-Connecting-IP when the request actually arrives from one of those ranges, so a direct client cannot forge it. Refresh it monthly from cron.

Then orange-cloud the record and set SSL/TLS to Full (strict), which works because the origin has a real certificate. Three limits to know first:

The free plan caps request bodies at 100 MB. That is a push limit, not a clone limit, and pull mirrors are unaffected because the server dials upstream directly rather than through the edge.
SSH is not proxied without Spectrum, so git over SSH connects to the origin address directly, which also discloses the IP the orange cloud would otherwise conceal.
A slow upload-pack can exceed the 100 s edge timeout and surface as a 524.

Confirm the edge actually reaches your origin by forcing a Cloudflare address rather than trusting DNS:

curl -sI --resolve git.example.com:443:104.21.12.54 https://git.example.com/ \
  | grep -iE 'HTTP|server|cf-ray'

A cf-ray header means you went through the edge. And once real-IP restoration is working, Cloudflare's own addresses stop appearing in the nginx access log, which is the feature working rather than a failure.

Admin user and token

FJ() { sudo -u git env GITEA_WORK_DIR=/var/lib/forgejo HOME=/home/git \
       /usr/local/bin/forgejo "$@" --config /etc/forgejo/app.ini; }

FJ admin user create --admin --username myuser --email me@example.com \
   --password "$PW" --must-change-password=false

FJ admin user generate-access-token --username myuser \
   --token-name mirror-bootstrap \
   --scopes write:repository,write:user,write:organization --raw

write:organization is required if any upstream repositories live under an organisation. write:repository does not imply it, and the failure arrives as token does not have at least one of required scope(s) only once you are already mid-run.

Creating the mirrors

Doing dozens of repositories by hand is not the intent. This enumerates each upstream account and creates a pull mirror per repository:

#!/usr/bin/env python3
"""Bulk-create Forgejo pull mirrors of a GitHub / GitLab / Codeberg account.

Reads credentials from /etc/forgejo/mirror-tokens.env (mode 600).

    forgejo-add-mirrors codeberg
    forgejo-add-mirrors github
    forgejo-add-mirrors gitlab
    forgejo-add-mirrors all
    DRY_RUN=1 forgejo-add-mirrors all     # show a plan, change nothing

Idempotent: a repo that already exists locally is skipped, so re-running after
creating new upstream repos only adds the new ones.

Two things this gets right that are easy to get wrong:

  * Upstream credentials are passed as `auth_token` ONLY. Supplying
    `auth_username` alongside it makes Forgejo present the *username* to the
    upstream as the token, which fails with a confusing auth cascade
    ("access token does not exist [sha: <username>]").

  * Upstream organisation structure is preserved. A local org is created for
    each upstream owner that is not you, so `myorg/thing` mirrors to
    `myorg/thing` rather than colliding with your own `thing`.
"""

import json
import os
import subprocess
import sys
import time
import urllib.error
import urllib.request

ENV_FILE = "/etc/forgejo/mirror-tokens.env"
FORGEJO_API = os.environ.get("FORGEJO_API", "http://127.0.0.1:3000/api/v1")
DRY_RUN = os.environ.get("DRY_RUN", "0") == "1"
# Seconds to pause between migrations. This box has 1 vCPU; each migration
# clones synchronously, so back-to-back requests just queue up behind git.
CREATE_DELAY = float(os.environ.get("CREATE_DELAY", "3"))
INTERVAL = os.environ.get("INTERVAL", "24h")
TIMEOUT = int(os.environ.get("HTTP_TIMEOUT", "1800"))


def log(*a):
    print(time.strftime("[%H:%M:%S]"), *a, flush=True)


def load_env(path):
    vals = {}
    try:
        text = open(path).read()
    except OSError as e:
        sys.exit("cannot read %s: %s" % (path, e))
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        k, v = line.split("=", 1)
        vals[k.strip()] = v.strip().strip("\"'")
    return vals


ENV = load_env(ENV_FILE)


def need(key):
    v = ENV.get(key)
    if not v:
        sys.exit("set %s in %s" % (key, ENV_FILE))
    return v


def request(url, token=None, method="GET", data=None, header="token"):
    req = urllib.request.Request(url, method=method)
    if token:
        if header == "token":
            req.add_header("Authorization", "token " + token)
        elif header == "bearer":
            req.add_header("Authorization", "Bearer " + token)
        else:
            req.add_header(header, token)
    req.add_header("Accept", "application/json")
    if data is not None:
        req.add_header("Content-Type", "application/json")
        data = json.dumps(data).encode()
    try:
        with urllib.request.urlopen(req, data, timeout=TIMEOUT) as r:
            body = r.read().decode()
            return r.status, (json.loads(body) if body.strip() else None)
    except urllib.error.HTTPError as e:
        body = e.read().decode()
        try:
            return e.code, json.loads(body)
        except Exception:
            return e.code, {"message": body[:400]}
    except Exception as e:
        return 0, {"message": str(e)}


# --------------------------------------------------------------------------
# Local Forgejo side
# --------------------------------------------------------------------------
FJ_TOKEN = need("FORGEJO_TOKEN")

status, me = request(FORGEJO_API + "/user", FJ_TOKEN)
if status != 200 or not me:
    sys.exit("cannot authenticate to Forgejo (%s): %s" % (status, me))
FJ_USER = me["login"]
log("authenticated to Forgejo as %s" % FJ_USER)

_known_orgs = set()


def ensure_org(name):
    """Make sure a local org exists to hold an upstream owner's repos."""
    if name.lower() == FJ_USER.lower() or name in _known_orgs:
        return
    status, _ = request("%s/orgs/%s" % (FORGEJO_API, name), FJ_TOKEN)
    if status == 200:
        _known_orgs.add(name)
        return
    if DRY_RUN:
        log("  DRY_RUN would create local org %s" % name)
        _known_orgs.add(name)
        return
    status, resp = request(FORGEJO_API + "/orgs", FJ_TOKEN, "POST", {
        "username": name,
        "visibility": "private",
        "description": "Mirrored organisation",
    })
    if status in (200, 201):
        log("  created local org %s" % name)
        _known_orgs.add(name)
    else:
        log("  WARNING could not create org %s (%s): %s"
            % (name, status, (resp or {}).get("message")))


def local_repo(owner, name):
    status, body = request("%s/repos/%s/%s" % (FORGEJO_API, owner, name), FJ_TOKEN)
    return body if status == 200 else None


def create_mirror(clone_url, owner, name, token, service, upstream_empty=False):
    """Create one pull mirror. `token` is the UPSTREAM credential."""
    existing = local_repo(owner, name)
    if existing:
        # A failed migration leaves a database record with no git data behind.
        # Skipping that forever would silently keep a broken mirror, so rebuild
        # it -- but only when upstream actually has commits to fetch.
        if existing.get("empty") and not upstream_empty:
            if DRY_RUN:
                log("  DRY_RUN would REBUILD %s/%s (exists but empty)" % (owner, name))
                return "planned"
            log("  rebuilding %s/%s (exists but empty)" % (owner, name))
            status, resp = request("%s/repos/%s/%s" % (FORGEJO_API, owner, name),
                                   FJ_TOKEN, "DELETE")
            if status not in (200, 204):
                log("  FAILED to delete broken %s/%s (%s): %s"
                    % (owner, name, status, (resp or {}).get("message")))
                return "failed"
        else:
            log("  skip %s/%s (exists)" % (owner, name))
            return "skipped"

    if DRY_RUN:
        log("  DRY_RUN would mirror %s -> %s/%s" % (clone_url, owner, name))
        return "planned"

    ensure_org(owner)
    payload = {
        "clone_addr": clone_url,
        "repo_name": name,
        "repo_owner": owner,
        "service": service,
        "mirror": True,
        "mirror_interval": INTERVAL,
        "private": True,
        "wiki": True,
        "labels": True,
        "issues": True,
        "milestones": True,
        "releases": True,
        "pull_requests": False,
        "lfs": False,
        "description": "Backup mirror of " + clone_url,
        # auth_token ONLY -- see module docstring.
        "auth_token": token,
    }
    status, resp = request(FORGEJO_API + "/repos/migrate", FJ_TOKEN, "POST", payload)
    if status in (200, 201):
        log("  mirrored %s/%s" % (owner, name))
        time.sleep(CREATE_DELAY)
        return "created"
    msg = (resp or {}).get("message", "")
    log("  FAILED %s/%s (%s): %s" % (owner, name, status, str(msg).replace("\n", " | ")[:300]))
    return "failed"


def local_owner_for(upstream_owner, upstream_user):
    """Own repos go under your account; everything else under a local org."""
    if upstream_owner.lower() == upstream_user.lower():
        return FJ_USER
    return upstream_owner.replace("/", "-")


# --------------------------------------------------------------------------
# Upstream enumeration
# --------------------------------------------------------------------------
def paged(url_tmpl, token, header, per_page=50, limit_key="limit"):
    page = 1
    while True:
        url = url_tmpl % {"page": page, "per": per_page, "limit": limit_key}
        status, body = request(url, token, header=header)
        if status != 200 or not body:
            if status != 200:
                log("  enumeration stopped at page %d (HTTP %s): %s"
                    % (page, status, (body or {}).get("message")))
            return
        if not isinstance(body, list) or not body:
            return
        for item in body:
            yield item
        if len(body) < per_page:
            return
        page += 1


def do_codeberg(stats):
    token = need("CODEBERG_TOKEN")
    user = need("CODEBERG_USER")
    log("enumerating Codeberg repos for %s" % user)
    url = "https://codeberg.org/api/v1/user/repos?limit=%(per)d&page=%(page)d"
    for r in paged(url, token, "token"):
        owner = local_owner_for(r["owner"]["login"], user)
        stats[create_mirror(r["clone_url"], owner, r["name"], token, "gitea",
                            bool(r.get("empty")))] += 1


def do_github(stats):
    token = need("GITHUB_TOKEN")
    user = need("GITHUB_USER")
    log("enumerating GitHub repos for %s" % user)
    url = ("https://api.github.com/user/repos?per_page=%(per)d&page=%(page)d"
           "&affiliation=owner,organization_member")
    for r in paged(url, token, "bearer"):
        owner = local_owner_for(r["owner"]["login"], user)
        stats[create_mirror(r["clone_url"], owner, r["name"], token, "github",
                            bool(r.get("size") == 0))] += 1


def do_gitlab(stats):
    token = need("GITLAB_TOKEN")
    log("enumerating GitLab projects")
    status, me_gl = request("https://gitlab.com/api/v4/user", token, header="PRIVATE-TOKEN")
    gl_user = (me_gl or {}).get("username", "")
    url = ("https://gitlab.com/api/v4/projects?membership=true&per_page=%(per)d"
           "&page=%(page)d")
    for r in paged(url, token, "PRIVATE-TOKEN"):
        ns = r.get("namespace", {}).get("full_path", "")
        owner = local_owner_for(ns, gl_user)
        stats[create_mirror(r["http_url_to_repo"], owner, r["path"], token, "gitlab",
                            bool(r.get("empty_repo")))] += 1


def disk_free():
    try:
        out = subprocess.run(["df", "-h", "--output=avail", "/"],
                             capture_output=True, text=True).stdout
        return out.strip().splitlines()[-1].strip()
    except Exception:
        return "?"


def main():
    if len(sys.argv) != 2 or sys.argv[1] not in ("github", "gitlab", "codeberg", "all"):
        sys.exit("usage: forgejo-add-mirrors {github|gitlab|codeberg|all}")
    which = sys.argv[1]
    if DRY_RUN:
        log("DRY_RUN=1, nothing will be created")
    log("disk free before: %s" % disk_free())

    stats = {"created": 0, "skipped": 0, "failed": 0, "planned": 0}
    for name, fn in (("codeberg", do_codeberg), ("github", do_github), ("gitlab", do_gitlab)):
        if which in (name, "all"):
            if which == "all" and not any(
                    ENV.get(k) for k in ("%s_TOKEN" % name.upper(),)):
                log("skipping %s (no token configured)" % name)
                continue
            fn(stats)

    log("done: %d created, %d skipped, %d failed, %d planned"
        % (stats["created"], stats["skipped"], stats["failed"], stats["planned"]))
    log("disk free after: %s" % disk_free())
    if stats["failed"]:
        log("check failures with: journalctl -u forgejo --since '1 hour ago' | grep -i migrat")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

sudo DRY_RUN=1 forgejo-add-mirrors codeberg   # plan
sudo forgejo-add-mirrors codeberg             # execute
sudo forgejo-add-mirrors all

Three traps in there, each of which I hit:

Pass auth_token alone. Supplying auth_username next to it makes Forgejo present the username to the upstream as the credential. The upstream rejects it and its error comes back wrapped in your own API's error envelope, complete with a "url" pointing at your own swagger endpoint, so it reads like a local authentication failure:

token is malformed: token contains an invalid number of segments
user's password is invalid [uid: NNNNNN, name: myuser]
access token does not exist [sha: myuser]

That last line is the giveaway: the upstream is looking up a token whose value is your username. What makes it confusing is that both plain forms work fine against the upstream API directly, and only the mixed form fails.

Derive local names from the full upstream path, not basename. If some repositories live under an organisation, myorg/thing and myuser/thing both reduce to thing and the second silently collides with the first. Create a local organisation per upstream owner and mirror into it.

Treat an empty local repository as a failure, not as done. Forgejo inserts the repository row before it clones, so an interrupted migration leaves a record with empty: true and no git data behind it. An existence check alone skips that forever, and the backup that appears to be present is not there at all. Audit for it:

curl -sS -H "Authorization: token $TOKEN" \
  "$API/repos/$OWNER/$NAME" | jq '{empty, mirror, size}'

Backing up the backup

Metadata only: the database, app.ini, and the SSH host keys. Repository content is deliberately excluded, because every repository here is a mirror and the recovery path is re-cloning from upstream rather than restoring:

#!/bin/bash
# Back up the Forgejo *metadata* only: the SQLite DB, app.ini and SSH host keys.
#
# Repository content is deliberately excluded. These repos are themselves
# mirrors of Codeberg/GitLab/GitHub, so re-cloning from upstream is the
# recovery path. That keeps each backup a few MB instead of many GB, which
# matters on a 23 GB disk.
#
# Restore: stop forgejo, untar over /var/lib/forgejo/data + /etc/forgejo,
# chown -R git:git /var/lib/forgejo, start forgejo.

set -euo pipefail

DEST=/var/lib/forgejo/backups
KEEP=4
DB=/var/lib/forgejo/data/forgejo.db

mkdir -p "$DEST"
chmod 700 "$DEST"

STAMP=$(date +%Y%m%d-%H%M%S)
OUT="$DEST/forgejo-meta-$STAMP.tar.zst"
TMP=$(mktemp -d)
cleanup() { rm -rf "$TMP"; }
trap cleanup EXIT

# Consistent snapshot of a live, WAL-mode SQLite database.
if command -v sqlite3 >/dev/null 2>&1; then
    sqlite3 "file:${DB}?mode=ro" ".backup '$TMP/forgejo.db'"
else
    cp -a "$DB" "$TMP/" 2>/dev/null || true
    cp -a "${DB}-wal" "${DB}-shm" "$TMP/" 2>/dev/null || true
fi

cp -a /etc/forgejo/app.ini "$TMP/app.ini"
# Built-in SSH server host keys: preserving these avoids host-key warnings
# for anyone who has cloned over ssh://git@git.example.com:2222.
cp -a /var/lib/forgejo/data/ssh "$TMP/ssh" 2>/dev/null || true

tar -C "$TMP" -cf - . | zstd -q -19 -o "$OUT"
chmod 600 "$OUT"

# Prune all but the newest $KEEP archives.
ls -1t "$DEST"/forgejo-meta-*.tar.zst 2>/dev/null | tail -n +$((KEEP + 1)) | xargs -r rm -f

logger -t forgejo-backup "wrote $OUT ($(du -h "$OUT" | cut -f1))"

That produces tens of KB per archive instead of gigabytes. .backup rather than cp, because the database is live and in WAL mode, and a plain copy gives a torn snapshot with a stale -wal sibling. Keeping the host keys means nobody sees a host-key warning after a restore.

Restoring is the DB plus one script run, since the mirror list lives in the database and the rebuild-empty logic above re-clones the content:

sudo systemctl stop forgejo
# unpack, copy forgejo.db / app.ini / ssh into place, chown -R git:git
sudo systemctl start forgejo
sudo forgejo-add-mirrors all

Watching the disk

Mirroring several accounts onto a small volume is exactly the kind of thing that fills up quietly:

#!/bin/bash
# Disk and memory guard for the Forgejo backup forge.
#
# This box has 23 GB total / ~16 GB free and 950 MiB RAM, so mirroring three
# accounts can plausibly fill it. Run hourly from cron; findings go to
# journald (tag forgejo-diskcheck) and to the status file below.
#
#   journalctl -t forgejo-diskcheck --since today
#   cat /var/lib/forgejo/diskcheck.status

set -uo pipefail

WARN_PCT=80
CRIT_PCT=90
REPO_DIR=/var/lib/forgejo/data/forgejo-repositories
STATUS_FILE=/var/lib/forgejo/diskcheck.status

say() {
    local level="$1"; shift
    logger -t forgejo-diskcheck -p "daemon.${level}" -- "$*"
    printf '%s [%s] %s\n' "$(date -Is)" "$level" "$*"
}

pct=$(df --output=pcent / | tail -1 | tr -dc '0-9')
avail=$(df -h --output=avail / | tail -1 | tr -d ' ')
repo_size=$(du -sh "$REPO_DIR" 2>/dev/null | cut -f1)
repo_count=$(find "$REPO_DIR" -maxdepth 2 -name '*.git' -type d 2>/dev/null | wc -l)
mem_avail=$(free -m | awk '/^Mem:/ {print $7}')
swap_used=$(free -m | awk '/^Swap:/ {print $3}')

{
    say info "disk ${pct}% used, ${avail} free | repos ${repo_count} using ${repo_size:-0} | mem ${mem_avail}MiB avail, swap ${swap_used}MiB used"

    if (( pct >= CRIT_PCT )); then
        say crit "CRITICAL: root filesystem ${pct}% full (${avail} left). Mirror syncs will start failing. Free space or resize the volume."
        # Stop scheduling new mirror work so a half-written repo does not wedge
        # the disk completely. Existing data is untouched.
        if systemctl is-active --quiet forgejo; then
            say warning "leaving forgejo running; disable the update_mirrors cron in the admin UI if the disk does not recover"
        fi
    elif (( pct >= WARN_PCT )); then
        say warning "WARNING: root filesystem ${pct}% full (${avail} left)."
    fi

    if (( mem_avail < 80 )); then
        say warning "low memory: only ${mem_avail}MiB available, ${swap_used}MiB swap in use"
    fi

    # Largest repos, useful when deciding what to drop.
    if (( pct >= WARN_PCT )) && [[ -d "$REPO_DIR" ]]; then
        say info "largest mirrors: $(du -s "$REPO_DIR"/*/* 2>/dev/null | sort -rn | head -5 | awk '{printf "%s(%.0fMB) ", $2, $1/1024}')"
    fi
} > "$STATUS_FILE" 2>&1

exit 0

And the cron that ties it together:

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Disk/memory guard: journalctl -t forgejo-diskcheck | /var/lib/forgejo/diskcheck.status
17 *  * * * root /usr/local/bin/forgejo-diskcheck >/dev/null 2>&1
# Metadata-only backup (DB + app.ini + host keys), keeps 4
23 3  * * 0 root /usr/local/bin/forgejo-backup-config >/dev/null 2>&1
# Refresh Cloudflare IP ranges for real-IP restoration
43 4  3 * * root /usr/local/bin/cloudflare-realip-update >/dev/null 2>&1

Verify from somewhere else

This is the other mistake worth confessing, because it invalidated everything I thought I had checked. My first HTTPS test returned a clean 200 and meant nothing: my shell had https_proxy set to a local SOCKS listener whose exit node was that same server, so the request hairpinned internally and never crossed the public internet. The evidence was sitting in the access log the whole time, where the client address was the server's own.

If a reachability test shows the origin talking to itself, it has tested nothing. Test from a third host:

for p in 22 80 443 2222; do
  timeout 8 bash -c "exec 3<>/dev/tcp/$IP/$p" 2>/dev/null \
    && echo "port $p open" || echo "port $p blocked"
done

curl -s -o /dev/null -w '%{http_code} tlsverify=%{ssl_verify_result}\n' \
     https://git.example.com/
echo | openssl s_client -connect git.example.com:443 -servername git.example.com \
     2>/dev/null | openssl x509 -noout -subject -dates
ssh-keyscan -p 2222 -T 10 $IP

git clone https://USER:TOKEN@git.example.com/USER/repo.git

The last one is the only test that actually proves the thing works, so do not stop before it.

Summary of the traps

symptom	cause
restart loop, `save oauth2.JWT_SECRET failed: permission denied`	`[oauth2] JWT_SECRET` unset while `app.ini` is not writable by the service account
works on loopback, unreachable from outside	a second nftables chain on the same hook with `policy drop`
firewall rule present but 0 packets	that rule is not in the path, something earlier drops
reachability test passes suspiciously easily	proxy hairpin, check the client IP in the access log
migrate fails, error names your username as a token	`auth_username` sent alongside `auth_token`
org repositories fail with `required scope(s)`	token lacks `write:organization`
large clone dies with `curl 92 ... stream CANCEL`	HTTP/2, pin `http.version = HTTP/1.1`
`unable to rename temporary '*.pack' file`	secondary symptom of the above, cleanup raced the dying clone
repository exists locally with no commits	failed migration left `empty: true`, delete and recreate
`ssh-keygen` as the service account fails	parent directory created by root, `chown` it first
certificate renews but the old one is still served	`certonly` installs no reload hook

MATLAB

2026-07-30T00:00:00.000Z

"The glibc problem" on arch + MATLAB isn't one bug — it's a category. Two different releases, two different root causes, found by actually reading the crash instead of guessing.

`glibc 2.41` executable-stack bug

glibc 2.41 stopped honoring a library's request for an executable stack via dlopen/dlmopen. Some of MATLAB's bundled .so files ask for exactly that, so they segfault on load. Clear the flag with patchelf:

sudo patchelf --clear-execstack ~/.MathWorks/ServiceHost/-mw_shared_installs/v2025.1.1.2/bin/glnxa64/libmwfoundation_crash_handling.so
sudo patchelf --clear-execstack ~/.MathWorks/ServiceHost/-mw_shared_installs/v2025.1.1.2/bin/glnxa64/mathworksservicehost/rcf/matlabconnector/serviceprocess/rcf/service/libmwmshrcfservice.so

Check whether a given install even has the problem before chasing this fix — later releases (R2026a) ship with zero executable-stack libraries, MathWorks fixed it upstream:

find "$MATLAB_ROOT" -name '*.so' -print0 \
  | xargs -0 -I{} sh -c 'readelf -lW "{}" | grep -q "GNU_STACK.*RWE" && echo {}'

`gnutls` ABI break (R2026a activation)

Fresh R2026a install: the activation window never appears, no error, process just dies. Not the exec-stack bug above — confirmed with a real backtrace instead of assuming:

gdb -q -batch -ex run -ex bt --args \
  "$MATLAB_ROOT/bin/glnxa64/MathWorksProductAuthorizer"

Crash is inside libgnutls.so.30, called from lc_init() in libmwinstall_activationwsclientimpl.so. MATLAB's activation/licensing plugin framework dlopen()s the system gnutls at runtime, and Arch's current gnutls is ABI-incompatible with what the client expects.

Fix: stage a pinned, private gnutls + nettle build and point LD_LIBRARY_PATH at it for MATLAB's own processes — no need to touch the system gnutls or the MATLAB install itself. Version pairing matters: gnutls 3.8.9-1 needs a libhogweed exporting nettle_rsa_oaep_sha384_decrypt under symbol version HOGWEED_6; nettle 3.9.1 is missing that symbol, nettle 3.10-1 has it under the same soname.

mkdir -p ~/.local/lib/matlab-gnutls-compat
cd /tmp && mkdir gnutls-old nettle-310

curl -sL -o gnutls-old/g.pkg.tar.zst \
  https://archive.archlinux.org/packages/g/gnutls/gnutls-3.8.9-1-x86_64.pkg.tar.zst
tar --zstd -xf gnutls-old/g.pkg.tar.zst -C gnutls-old

curl -sL -o nettle-310/n.pkg.tar.zst \
  https://archive.archlinux.org/packages/n/nettle/nettle-3.10-1-x86_64.pkg.tar.zst
tar --zstd -xf nettle-310/n.pkg.tar.zst -C nettle-310

cp -a gnutls-old/usr/lib/libgnutls* ~/.local/lib/matlab-gnutls-compat/
cp -a nettle-310/usr/lib/libnettle* nettle-310/usr/lib/libhogweed* \
  ~/.local/lib/matlab-gnutls-compat/

Then launch MATLAB with that on LD_LIBRARY_PATH:

env LD_LIBRARY_PATH=~/.local/lib/matlab-gnutls-compat \
  "$MATLAB_ROOT/bin/matlab" -nodesktop -nosplash -nosoftwareopengl

To pop the activation window on its own — e.g. to (re)activate without launching the full app — run the authorizer client directly with the same LD_LIBRARY_PATH:

env LD_LIBRARY_PATH=~/.local/lib/matlab-gnutls-compat \
  "$MATLAB_ROOT/bin/glnxa64/MathWorksProductAuthorizer.sh"

Watch out for stale LD_PRELOADs from older fixes lying around in shell config (LD_PRELOAD=/usr/lib/libstdc++.so forcing the system C++ runtime over MATLAB's bundled one is its own separate, documented crash cause) — they're easy to leave behind across MATLAB version upgrades and mask what's actually going on.

"Graphics acceleration hardware is unavailable" (R2026a)

Same bundled-runtime theme as above, different victim. R2026a no longer draws figures with desktop OpenGL; it renders WebGL inside an embedded Chromium (MATLABWindow + libcef.so) through ANGLE. So the fallback is no longer MATLAB's own software OpenGL, it's SwiftShader, and on arch you land there by default.

opengl('data') is gone in R2026a. The replacement is rendererinfo:

r = rendererinfo;
disp(r.RendererDevice)
disp(r.Details.HardwareSupportLevel)

Broken looks like this, on a Radeon 780M that has a perfectly good amdgpu + mesa stack:

ANGLE (Google, Vulkan 1.3.0 (SwiftShader Device (Subzero) (0x0000C0DE)), SwiftShader driver)

Root cause

ANGLE picks its Vulkan backend, asks the loader to enumerate devices, and the only ICD that answers is the SwiftShader one MATLAB ships (bin/glnxa64/vk_swiftshader_icd.json). The real driver never loads. Reproduce the failure directly, outside MATLAB, with a five-line dlopen harness:

#include <dlfcn.h>
#include <stdio.h>
int main(int c, char **v) {
  void *h = dlopen(v[1], RTLD_NOW);
  printf("%s\n", h ? "OK" : dlerror());
  return !h;
}

gcc -o dlopentest dlopentest.c
./dlopentest /usr/lib/libvulkan_radeon.so                      # OK
LD_LIBRARY_PATH=$MATLAB_ROOT/sys/os/glnxa64 \
  ./dlopentest /usr/lib/libvulkan_radeon.so
# .../sys/os/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.32' not found
#   (required by /usr/lib/libSPIRV-Tools.so)

There it is. radv links libSPIRV-Tools, which is built against current mesa's toolchain and needs GLIBCXX_3.4.32. MATLAB puts its bundled libstdc++.so.6.0.30 (GCC 12.3) ahead of the system 6.0.36, so radv fails to load, Vulkan enumeration comes back with SwiftShader only, and MATLAB dutifully warns that hardware acceleration is unavailable. The GPU is fine; the C++ runtime under it is too old.

Why `LD_LIBRARY_PATH` cannot fix this

The obvious move is to prepend a directory holding a symlink to the system libstdc++. The matlab launcher even documents a variable for it, LDPATH_PREFIX, which lands at the very front of LD_LIBRARY_PATH. It changes nothing, and the reason is worth knowing:

readelf -d "$MATLAB_ROOT/bin/glnxa64/MATLABWindow" | grep -E 'RPATH|RUNPATH'
# 0x000000000000000f (RPATH) Library rpath: [$ORIGIN:$ORIGIN/../../sys/os/glnxa64]

DT_RPATH, not DT_RUNPATH. The dynamic linker searches DT_RPATH before LD_LIBRARY_PATH, so no amount of path ordering wins. Confirm what a live process actually mapped rather than what you hoped it would:

grep -oE '/[^ ]*libstdc\+\+[^ ]*' /proc/$(pgrep -f MATLABWindow | head -1)/maps
# /usr/local/MATLAB/R2026a/sys/os/glnxa64/libstdc++.so.6.0.30

LD_PRELOAD is loaded before anything the RPATH can reach, so that is the lever that works.

Fix

env LD_PRELOAD=/usr/lib/libstdc++.so.6 "$MATLAB_ROOT/bin/matlab"

Newer libstdc++ keeps the old symbol versions, so MATLAB's own GCC-12-era libraries are satisfied by 6.0.36 too. The launcher appends to LD_PRELOAD (LD_PRELOAD="${LD_PRELOAD:+${LD_PRELOAD}:}...") instead of clobbering it, so an inherited value survives alongside MATLAB's own shims. Result:

ANGLE (AMD, AMD Radeon 780M Graphics (radeonsi phoenix ACO), OpenGL 4.6)
HardwareSupportLevel: Full

Note this is the narrow, versioned libstdc++.so.6, not the libstdc++.so linker script blamed for crashes at the end of the previous section. Same family of hack, and it's still the first thing to rip out when MATLAB later misbehaves for unrelated reasons.

If you'd rather not carry the variable, MATLAB's bundled libstdc++.so.6 is already just a symlink and the pristine copies live in sys/os/glnxa64/orig/, so repointing it at /usr/lib/libstdc++.so.6 does the same thing install-wide, at the cost of a change a MATLAB update can quietly revert.

Two dead ends worth naming

strings over the CEF glue libraries turns up MW_ENABLE_GPU_RENDERING sitting next to disable-gpu and enable-unsafe-swiftshader in libmwcef_common.so, which looks exactly like the switch you want. It isn't; setting it made no difference, because the GPU was never being blocklisted, just left driverless. bin/glnxa64/gpu_info is likewise a red herring that reports the right device (0x1002:0x1900) with driver version 0.0.0.0.

Also, -nosoftwareopengl is now inert: R2026a answers with Warning: OpenGL Startup options have been removed. Drop it from launcher aliases.

Verify

rendererinfo alone can lie by omission, so render something that leans on the GPU (lighting plus transparency) and confirm it survives:

f = figure('Visible','off');
[X,Y,Z] = peaks(120);
surf(X,Y,Z,'EdgeColor','none'); camlight; lighting gouraud; alpha 0.7
exportgraphics(f, 'hwtest.png', 'Resolution', 150)

nixos

2025-01-10T00:00:00.000Z

A few key commands:

nix-store --verify --check-contents --repair

nix-collect-garbage -d

nixos-rebuild switch

Networking in linux

2025-01-10T00:00:00.000Z

`nmcli`

To import:

nmcli connection import type wireguard file school.conf

To activate/deactivate:

nmcli connection up school / nmcli connection down school

To disable autoconnect:

nmcli connection modify school connection.autoconnect no

`GlobalProtect`, using Northeastern as an example.

nmcli connection add connection.id school \
connection.type vpn vpn.service-type openconnect \
vpn.data cookie-flags=1,gateway=vpn.northeastern.edu,protocol=gp \
vpn.secrets gateway=vpn.northeastern.edu,gwcert=

`eduroam` and `802-1x` authentication

nmcli con add \
  type wifi \
  ifname wlp3s0 \
  con-name eduroam \
  ssid eduroam \
  ipv4.method auto \
  802-1x.eap peap \
  802-1x.phase2-auth mschapv2 \
  802-1x.identity "<your ldap username>" \
  802-1x.password "<your ldap password>" \
  wifi-sec.key-mgmt wpa-eap

zli(1): blog

Self-hosting Vaultwarden, and the one path that ruins it

The deployment

Closing signups, and actually checking

Turning off the web vault

The error message that tells you nothing

Two things about backups

Where it actually stands

Is it safe?

Metrics and a Homepage

VictoriaMetrics instead of Prometheus

Host networking, deliberately

Provisioned, not clicked

Change the default password before the cert exists

The homepage layer

Almost every widget needs outbound network

Encrypted offsite backups that verify themselves

Encrypt before it leaves the machine

Two mechanisms, because the data has two shapes

Use the database's own backup command

Every run restores its own output

Two rclone details that cost me time

The circular dependency

Shape of the result

PGP and SSH keys on a YubiKey

Generating and moving a PGP key

Restoring on another machine

git credentials through pass

FIDO2-backed SSH keys

Skipping the touch requirement

eSIM and WWAN on a ThinkPad P14s under Arch

Hardware

1. Firmware

2. FCC unlock

3. Switch to the eSIM slot

4. Install lpac

5. Download a profile

6. Enable it

7. Connect

Known issues

The part that does not work

Command reference

Linux tips and tricks

Dotfiles across two machines

The generated-file problem

Networking

dnsmasq under NetworkManager

mDNS and .local

DNS inside Docker containers

Keyboard

Caps lock, via keyd

Vial and QMK keyboards

Power and boot

AMD P-state

systemd-boot entry

Recovering a lost boot entry

Storage

rclone over SFTP

Desktop odds and ends

Bluetooth controllers

A Windows VM without libvirt

Forgejo on a small VPS: a backup forge

Install

app.ini

The unit file

One firewall, not two

TLS

Behind Cloudflare

Admin user and token

Creating the mirrors

Backing up the backup

Watching the disk

Verify from somewhere else

Summary of the traps

MATLAB

glibc 2.41 executable-stack bug

gnutls ABI break (R2026a activation)

"Graphics acceleration hardware is unavailable" (R2026a)

Root cause

Why LD_LIBRARY_PATH cannot fix this

mDNS and `.local`

`glibc 2.41` executable-stack bug

`gnutls` ABI break (R2026a activation)

Why `LD_LIBRARY_PATH` cannot fix this

`nmcli`

`GlobalProtect`, using Northeastern as an example.

`eduroam` and `802-1x` authentication