Every repository I care about lives on Codeberg, GitLab or GitHub. None of those is a backup, they are just someone else's disk. This is the full deployment of a self-hosted Forgejo that pull-mirrors all three onto a VPS I control, including every config file and script, plus the handful of things that are genuinely easy to get wrong.

Two boxes went through this, one with 950 MiB of RAM and one vCPU, and a later one with 3.7 GB and two. The configs below are the larger box, with the smaller box's numbers noted where they differ, because the memory tuning is the one part you should not copy between machines.

Throughout: git.example.com is the forge hostname, myuser the account, myorg an upstream organisation.

Install

Verify the binary. A forge is the last place to run an unverified download:

V=16.0.2
curl -fsSL --http1.1 --retry 3 -C - -O \
  https://codeberg.org/forgejo/forgejo/releases/download/v$V/forgejo-$V-linux-amd64
curl -fsSL --http1.1 -O \
  https://codeberg.org/forgejo/forgejo/releases/download/v$V/forgejo-$V-linux-amd64.sha256
curl -fsSL --http1.1 -O \
  https://codeberg.org/forgejo/forgejo/releases/download/v$V/forgejo-$V-linux-amd64.asc

sha256sum -c forgejo-$V-linux-amd64.sha256
gpg --keyserver hkps://keys.openpgp.org \
    --recv-keys EB114F5E6C0DC2BCDD183550A4B61A2DC5923710
gpg --verify forgejo-$V-linux-amd64.asc forgejo-$V-linux-amd64

sudo install -m 755 forgejo-$V-linux-amd64 /usr/local/bin/forgejo

The checksum file sits on the same server as the binary, so on its own it proves nothing. The signature is the part that matters, and the key should come from somewhere other than where the artifact lives.

--http1.1 is not decoration. Codeberg's HTTP/2 aborts large transfers from some hosts with curl 92 ... stream not closed cleanly: CANCEL, and a 119 MB download is large enough to hit it. Same flag, same reason, shows up again later for git itself.

Then the service account:

sudo apt-get install -y nginx certbot git sqlite3 nftables zstd gnupg
sudo adduser --system --shell /bin/bash --gecos 'Forgejo' \
     --group --disabled-password --home /home/git git
sudo mkdir -p /var/lib/forgejo/{custom,data,log} /etc/forgejo
sudo chown -R git:git /var/lib/forgejo
sudo chown root:git /etc/forgejo && sudo chmod 770 /etc/forgejo

/etc/forgejo is root:git and app.ini is mode 640, so the service reads its config but can never write it. That is deliberate, and it is why the OAuth2 JWT secret has to be set by hand below.

Generate the secrets before writing the config:

for s in SECRET_KEY INTERNAL_TOKEN LFS_JWT_SECRET JWT_SECRET; do
  echo "$s = $(forgejo generate secret $s)"
done

app.ini

Skip the web installer entirely: set INSTALL_LOCK = true and create the admin from the CLI, otherwise the setup page is exposed until someone clicks through it.

; Forgejo configuration for the backup forge (Debian 13, 2 vCPU, 3.7 GB RAM,
; 40 GB disk). Role: pull-mirror of Codeberg, GitLab and GitHub. Not a primary
; forge, so search/CI/packages stay off and the tuning favours reliable large
; clones over interactive responsiveness.

APP_NAME = backup forge
RUN_USER = git
RUN_MODE = prod
WORK_PATH = /var/lib/forgejo

[server]
PROTOCOL         = http
DOMAIN           = git.example.com
ROOT_URL         = https://git.example.com/
; Loopback only; nginx terminates TLS and proxies in.
HTTP_ADDR        = 127.0.0.1
HTTP_PORT        = 3000
DISABLE_SSH      = false
; Built-in SSH server, so the host sshd and /home/git/.ssh stay untouched.
START_SSH_SERVER = true
SSH_LISTEN_HOST  = 0.0.0.0
SSH_LISTEN_PORT  = 2222
SSH_DOMAIN       = git.example.com
SSH_PORT         = 2222
; Forgejo generates RSA only, and will not generate a type it was not told
; about. The ed25519 key is created alongside this file.
SSH_SERVER_HOST_KEYS = ssh/gitea.rsa,ssh/gitea.ed25519
LFS_START_SERVER = true
LFS_JWT_SECRET   = <forgejo generate secret LFS_JWT_SECRET>
OFFLINE_MODE     = true
APP_DATA_PATH    = /var/lib/forgejo/data
; nginx handles compression.
ENABLE_GZIP      = false

[database]
DB_TYPE  = sqlite3
PATH     = /var/lib/forgejo/data/forgejo.db
SQLITE_JOURNAL_MODE = WAL
MAX_OPEN_CONNS = 10
MAX_IDLE_CONNS = 5
CONN_MAX_LIFETIME = 0

[repository]
ROOT = /var/lib/forgejo/data/forgejo-repositories
DEFAULT_BRANCH = main
; A sink, not a source. Discourage accidental local work.
DEFAULT_PRIVATE = private
DEFAULT_PUSH_CREATE_PRIVATE = true
DISABLE_HTTP_GIT = false

[repository.upload]
FILE_MAX_SIZE = 64
MAX_FILES = 10

[repository.pull-request]
DEFAULT_MERGE_STYLE = merge

[mirror]
ENABLED = true
DISABLE_NEW_PULL = false
; Pushing mirrors out of here is not the point; off to avoid surprises.
DISABLE_NEW_PUSH = true
DEFAULT_INTERVAL = 24h
MIN_INTERVAL = 10m

[cron]
ENABLED = true
RUN_AT_START = false

[cron.update_mirrors]
ENABLED = true
; Every 6h; each run only syncs mirrors whose own interval has elapsed.
SCHEDULE = @every 6h
; 2 vCPU, so a little more parallelism than the previous single-core host.
PULL_LIMIT = 4
PUSH_LIMIT = 0

[cron.git_gc_repos]
ENABLED  = true
SCHEDULE = @every 168h
TIMEOUT  = 180m
ARGS     = --aggressive=false

[cron.repo_health_check]
ENABLED  = true
SCHEDULE = @every 168h
TIMEOUT  = 60m

[cron.archive_cleanup]
ENABLED  = true
SCHEDULE = @every 24h
OLDER_THAN = 24h

[cron.cleanup_actions]
ENABLED = true

[cron.delete_old_system_notices]
ENABLED = true
SCHEDULE = @every 168h
OLDER_THAN = 720h

[git]
MAX_GIT_DIFF_LINES = 1000
MAX_GIT_DIFF_LINE_CHARACTERS = 5000
MAX_GIT_DIFF_FILES = 100
GC_ARGS = --no-cruft

[git.timeout]
DEFAULT  = 360
MIGRATE  = 3600
MIRROR   = 1800
CLONE    = 1800
PULL     = 1800
GC       = 600

[git.config]
; Codeberg's HTTP/2 aborts mid-fetch on large repos (curl 92 / stream CANCEL),
; which surfaces as a failed clone or a confusing "unable to rename temporary
; '*.pack' file" once Forgejo cleans up behind it. Pin HTTP/1.1.
http.version = HTTP/1.1
http.postBuffer = 524288000
; Bounded repack cost. Roomier than a 1 GB host needs, still bounded so one
; huge repo cannot swallow all 3.7 GB.
pack.threads = 2
pack.windowMemory = 256m
pack.packSizeLimit = 1g
pack.deltaCacheSize = 64m
core.bigFileThreshold = 32m
core.packedGitWindowSize = 32m
core.packedGitLimit = 512m
receive.maxInputSize = 0
user.name  = myuser
user.email = [email protected]

[indexer]
; bleve is the heaviest component in a default install and buys nothing on a
; forge whose issues nobody searches. Affordable here, still not worth it.
ISSUE_INDEXER_TYPE = db
REPO_INDEXER_ENABLED = false

[queue]
TYPE = level
DATADIR = /var/lib/forgejo/data/queues/
LENGTH = 200
BATCH_LENGTH = 20
MAX_WORKERS = 2

[cache]
ENABLED = true
ADAPTER = memory
INTERVAL = 60
HOST =
ITEM_TTL = 8h

[session]
PROVIDER = file
PROVIDER_CONFIG = /var/lib/forgejo/data/sessions
COOKIE_SECURE = true
SESSION_LIFE_TIME = 604800

[security]
INSTALL_LOCK = true
SECRET_KEY = <forgejo generate secret SECRET_KEY>
INTERNAL_TOKEN = <forgejo generate secret INTERNAL_TOKEN>
PASSWORD_HASH_ALGO = argon2
PASSWORD_COMPLEXITY = lower,upper,digit
MIN_PASSWORD_LENGTH = 12
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = 127.0.0.0/8,::1/128
LOGIN_REMEMBER_DAYS = 30

[service]
DISABLE_REGISTRATION              = true
ALLOW_ONLY_EXTERNAL_REGISTRATION  = false
REGISTER_EMAIL_CONFIRM            = false
ENABLE_NOTIFY_MAIL                = false
REQUIRE_SIGNIN_VIEW               = true
DEFAULT_KEEP_EMAIL_PRIVATE        = true
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
ENABLE_TIMETRACKING               = false
NO_REPLY_ADDRESS                  = noreply.git.example.com
ENABLE_CAPTCHA                    = false

[oauth2]
; Set explicitly, so Forgejo never needs to write to this file. It treats a
; failed write of a generated JWT secret as fatal.
ENABLED = false
JWT_SECRET = <forgejo generate secret JWT_SECRET>

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[actions]
ENABLED = false

[packages]
; Package storage would compete with repository data for the same disk.
ENABLED = false

[ui]
EXPLORE_PAGING_NUM = 20
ISSUE_PAGING_NUM   = 20
FEED_MAX_COMMIT_NUM = 5
GRAPH_MAX_COMMIT_NUM = 100
DEFAULT_THEME = forgejo-auto

[log]
MODE = console
LEVEL = info
ROOT_PATH = /var/lib/forgejo/log

[log.console]
STDERR = false

[metrics]
ENABLED = false

[federation]
ENABLED = false

[other]
SHOW_FOOTER_VERSION = false
SHOW_FOOTER_TEMPLATE_LOAD_TIME = false

The parts worth explaining:

ISSUE_INDEXER_TYPE = db. A default install runs bleve, the heaviest component in the whole process, to make issues searchable. On a forge whose issues nobody reads, that is pure overhead. On the 950 MiB box it was the single biggest win.

[oauth2] JWT_SECRET set explicitly. Leave it out and Forgejo generates one at startup, tries to write it back into app.ini, fails because the file is not writable by git, and treats that as fatal. The symptom is a restart loop with save oauth2.JWT_SECRET failed: permission denied, which does not obviously point at file permissions.

SSH_SERVER_HOST_KEYS names ed25519. Forgejo generates RSA only, and will not generate a key type it was not told about, so create it yourself:

sudo mkdir -p /var/lib/forgejo/data/ssh
sudo chown -R git:git /var/lib/forgejo/data/ssh
sudo chmod 700 /var/lib/forgejo/data/ssh
sudo -u git ssh-keygen -t ed25519 -N '' -f /var/lib/forgejo/data/ssh/gitea.ed25519

Create and chown the directory before running ssh-keygen as git, or it fails with permission denied.

http.version = HTTP/1.1 under [git.config]. The same Codeberg HTTP/2 problem as the download. Without it, large mirror clones fail, sometimes as the genuinely baffling unable to rename temporary '*.pack' file to ...: No such file or directory, which is a secondary symptom: git's transfer died, Forgejo removed the half-built repository, and git then tried to finish writing into a directory that no longer existed.

[git.config] at all. Forgejo writes that section into its own .gitconfig, so it applies to every git it forks. That matters because the process which exhausts a small box is never Forgejo, it is a git repack on a large mirror.

The unit file

[Unit]
Description=Forgejo (backup mirror forge)
Documentation=https://forgejo.org/docs/
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=git
Group=git
WorkingDirectory=/var/lib/forgejo/
RuntimeDirectory=forgejo
RuntimeDirectoryMode=0750

ExecStart=/usr/local/bin/forgejo web --config /etc/forgejo/app.ini
Restart=always
RestartSec=10s

Environment=USER=git HOME=/home/git GITEA_WORK_DIR=/var/lib/forgejo
# 3.7 GB box: give the heap real room, but still bounded.
Environment=GOMEMLIMIT=1400MiB
Environment=GOMAXPROCS=2

# --- Resource containment -------------------------------------------------
# MemoryHigh throttles and swaps first; MemoryMax is the hard wall. A runaway
# `git repack` dies inside this cgroup instead of letting the kernel OOM killer
# pick a victim by badness score (which could just as easily be sshd).
MemoryHigh=2000M
MemoryMax=2800M
MemorySwapMax=1500M

# --- Hardening -----------------------------------------------------------
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectClock=true
ProtectHostname=true
ProtectProc=invisible
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
LockPersonality=true
# Loose enough not to break git repacks.
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
CapabilityBoundingSet=
AmbientCapabilities=

# ProtectSystem=strict makes / read-only; punch through only what we own.
ReadWritePaths=/var/lib/forgejo
ReadWritePaths=/etc/forgejo
ReadWritePaths=/home/git

[Install]
WantedBy=multi-user.target

MemoryHigh throttles and pushes to swap first, MemoryMax is the hard wall. The point is not to be stingy, it is that without a cgroup limit the kernel OOM killer picks its victim by badness score, and on a small box that is as likely to be sshd as the process that actually misbehaved. Bounding the service means a runaway repack dies alone and everything else survives.

Sized for 3.7 GB above. The 950 MiB box used GOMEMLIMIT=380MiB, MemoryHigh=520M, MemoryMax=760M, and correspondingly pack.threads = 1, pack.windowMemory = 96m, PULL_LIMIT = 2. Idle footprint was about 230 MB there and 110 MB here. Do not copy these numbers between machines in either direction.

If the box has no swap, add some, or MemorySwapMax has nowhere to spill:

sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

One firewall, not two

#!/usr/sbin/nft -f
# Single source of truth for inbound filtering on this host.
#
# Deliberately the ONLY input firewall here: multiple nftables base chains can
# attach to the same hook at the same priority and all of them run, so a second
# tool (ufw, firewalld) layered on top means every future port has to be opened
# in both places or it silently fails.
#
# Apply with: sudo nft -f /etc/nftables.conf   (nftables.service does this at boot)

table inet filter
delete table inet filter

table inet filter {
	chain input {
		type filter hook input priority filter; policy drop;

		iifname "lo" accept
		ct state established,related accept
		ct state invalid drop

		icmp type { echo-request, destination-unreachable, time-exceeded, parameter-problem } accept
		icmpv6 type { echo-request, destination-unreachable, time-exceeded, parameter-problem, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept

		tcp dport 22 accept   # host sshd
		tcp dport 80 accept   # nginx: ACME HTTP-01 + redirect to https
		tcp dport 443 accept  # nginx -> Forgejo
		tcp dport 2222 accept # Forgejo built-in SSH (git over ssh)

		udp dport 68 accept   # DHCP lease renewal (v4)
		udp dport 546 accept  # DHCP lease renewal (v6)
	}

	chain forward {
		type filter hook forward priority filter; policy drop;
	}

	chain output {
		type filter hook output priority filter; policy accept;
	}
}

sudo nft -f /etc/nftables.conf
sudo nft list chain inet filter input   # confirm the port 22 rule BEFORE logging out
sudo systemctl enable --now nftables

This is the part I got wrong, and it cost the most time, so it is worth being precise about why.

Multiple nftables base chains can attach to the same hook at the same priority, and every one of them runs. A packet has to be accepted by all of them, and a drop in any single chain is final. There is no first-match-wins across chains and no precedence to reason about.

So on a host that already had a policy drop table from an unrelated service, my freshly configured ufw reported 443 ALLOW IN and was completely powerless, because the other table only ever allowed udp dport 443 and never tcp. The service answered perfectly over loopback and was unreachable from the internet.

The counters say so plainly, if you look:

chain ufw-user-input {
  tcp dport 22   counter packets 40 bytes 2320 accept
  tcp dport 443  counter packets 0  bytes 0    accept
}

Zero packets on a rule that should be busy means that rule is not in the path. Read the whole ruleset with nft list ruleset, not just the table you wrote.

TLS

Get the certificate before writing a vhost that references it, since nginx will not start on a missing certificate file:

sudo certbot certonly --webroot -w /var/www/html -d git.example.com \
     --non-interactive --agree-tos -m [email protected]

# git.example.com -> Forgejo on 127.0.0.1:3000
#
# Works both DNS-only and behind the Cloudflare proxy. When the orange cloud
# is on, real client IPs arrive in CF-Connecting-IP and are restored by
# /etc/nginx/conf.d/cloudflare-realip.conf.

upstream forgejo {
    server 127.0.0.1:3000;
    keepalive 4;
}

server {
    listen 80;
    listen [::]:80;
    server_name git.example.com;

    # Leave room for certbot renewals over HTTP-01.
    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name git.example.com;

    ssl_certificate     /etc/letsencrypt/live/git.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/git.example.com/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/git.example.com/chain.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers off;
    ssl_session_cache shared:SSL:2m;
    ssl_session_timeout 1d;

    add_header Strict-Transport-Security "max-age=31536000" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    # Git pushes and LFS uploads can be large; do not cap them at the proxy.
    # (Note: the Cloudflare free plan enforces its own 100 MB request cap.)
    client_max_body_size 0;
    # Stream request bodies straight through instead of spooling to disk;
    # /tmp is a 476 MB tmpfs on this host, so buffering a big push would
    # consume RAM.
    proxy_request_buffering off;
    proxy_buffering off;
    proxy_http_version 1.1;

    # Large clones on 1 vCPU are slow. Be patient.
    proxy_connect_timeout 60s;
    proxy_send_timeout    900s;
    proxy_read_timeout    900s;

    location / {
        proxy_pass http://forgejo;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_set_header Connection        "";
    }
}

client_max_body_size 0 and proxy_request_buffering off are both about git. Unbuffered matters more than it looks: if /tmp is a tmpfs, which it is by default on recent Debian and Ubuntu, then buffering a large push spools it into RAM.

certonly installs no reload hook, so a renewal two months out would quietly keep serving the old certificate until something restarted nginx:

sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh >/dev/null <<'EOF'
#!/bin/sh
set -e
nginx -t && systemctl reload nginx
EOF
sudo chmod 755 /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
sudo certbot renew --dry-run

Delete any ssl_stapling lines from your boilerplate too. Let's Encrypt no longer runs OCSP responders, so they only produce a warning.

Behind Cloudflare

If the domain is already on Cloudflare, restore real client IPs at the origin or every log line and rate limit sees the edge instead of the client:

#!/bin/bash
# Refresh the nginx set_real_ip_from list from Cloudflare published ranges.
# Safe to run while DNS-only: set_real_ip_from only trusts CF-Connecting-IP
# when the request actually arrives from one of these ranges.
set -euo pipefail
OUT=/etc/nginx/conf.d/cloudflare-realip.conf
TMP=$(mktemp)
cleanup() { rm -f "$TMP"; }
trap cleanup EXIT
{
  echo "# Generated by cloudflare-realip-update on $(date -Is). Do not edit."
  for u in https://www.cloudflare.com/ips-v4 https://www.cloudflare.com/ips-v6; do
    curl -fsS --max-time 20 "$u" | sed "s/^/set_real_ip_from /; s/$/;/"
  done
  echo "real_ip_header CF-Connecting-IP;"
  echo "real_ip_recursive on;"
} > "$TMP"
grep -q "^set_real_ip_from" "$TMP" || { echo "no ranges fetched; keeping existing" >&2; exit 1; }
install -m 644 "$TMP" "$OUT"
nginx -t && systemctl reload nginx

Safe to install before enabling the proxy, because set_real_ip_from only trusts CF-Connecting-IP when the request actually arrives from one of those ranges, so a direct client cannot forge it. Refresh it monthly from cron.

Then orange-cloud the record and set SSL/TLS to Full (strict), which works because the origin has a real certificate. Three limits to know first:

• The free plan caps request bodies at 100 MB. That is a push limit, not a clone limit, and pull mirrors are unaffected because the server dials upstream directly rather than through the edge.
• SSH is not proxied without Spectrum, so git over SSH connects to the origin address directly, which also discloses the IP the orange cloud would otherwise conceal.
• A slow upload-pack can exceed the 100 s edge timeout and surface as a 524.

Confirm the edge actually reaches your origin by forcing a Cloudflare address rather than trusting DNS:

curl -sI --resolve git.example.com:443:104.21.12.54 https://git.example.com/ \
  | grep -iE 'HTTP|server|cf-ray'

A cf-ray header means you went through the edge. And once real-IP restoration is working, Cloudflare's own addresses stop appearing in the nginx access log, which is the feature working rather than a failure.

Admin user and token

FJ() { sudo -u git env GITEA_WORK_DIR=/var/lib/forgejo HOME=/home/git \
       /usr/local/bin/forgejo "$@" --config /etc/forgejo/app.ini; }

FJ admin user create --admin --username myuser --email [email protected] \
   --password "$PW" --must-change-password=false

FJ admin user generate-access-token --username myuser \
   --token-name mirror-bootstrap \
   --scopes write:repository,write:user,write:organization --raw

write:organization is required if any upstream repositories live under an organisation. write:repository does not imply it, and the failure arrives as token does not have at least one of required scope(s) only once you are already mid-run.

Creating the mirrors

Doing dozens of repositories by hand is not the intent. This enumerates each upstream account and creates a pull mirror per repository:

#!/usr/bin/env python3
"""Bulk-create Forgejo pull mirrors of a GitHub / GitLab / Codeberg account.

Reads credentials from /etc/forgejo/mirror-tokens.env (mode 600).

    forgejo-add-mirrors codeberg
    forgejo-add-mirrors github
    forgejo-add-mirrors gitlab
    forgejo-add-mirrors all
    DRY_RUN=1 forgejo-add-mirrors all     # show a plan, change nothing

Idempotent: a repo that already exists locally is skipped, so re-running after
creating new upstream repos only adds the new ones.

Two things this gets right that are easy to get wrong:

  * Upstream credentials are passed as `auth_token` ONLY. Supplying
    `auth_username` alongside it makes Forgejo present the *username* to the
    upstream as the token, which fails with a confusing auth cascade
    ("access token does not exist [sha: <username>]").

  * Upstream organisation structure is preserved. A local org is created for
    each upstream owner that is not you, so `myorg/thing` mirrors to
    `myorg/thing` rather than colliding with your own `thing`.
"""

import json
import os
import subprocess
import sys
import time
import urllib.error
import urllib.request

ENV_FILE = "/etc/forgejo/mirror-tokens.env"
FORGEJO_API = os.environ.get("FORGEJO_API", "http://127.0.0.1:3000/api/v1")
DRY_RUN = os.environ.get("DRY_RUN", "0") == "1"
# Seconds to pause between migrations. This box has 1 vCPU; each migration
# clones synchronously, so back-to-back requests just queue up behind git.
CREATE_DELAY = float(os.environ.get("CREATE_DELAY", "3"))
INTERVAL = os.environ.get("INTERVAL", "24h")
TIMEOUT = int(os.environ.get("HTTP_TIMEOUT", "1800"))


def log(*a):
    print(time.strftime("[%H:%M:%S]"), *a, flush=True)


def load_env(path):
    vals = {}
    try:
        text = open(path).read()
    except OSError as e:
        sys.exit("cannot read %s: %s" % (path, e))
    for line in text.splitlines():
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        k, v = line.split("=", 1)
        vals[k.strip()] = v.strip().strip("\"'")
    return vals


ENV = load_env(ENV_FILE)


def need(key):
    v = ENV.get(key)
    if not v:
        sys.exit("set %s in %s" % (key, ENV_FILE))
    return v


def request(url, token=None, method="GET", data=None, header="token"):
    req = urllib.request.Request(url, method=method)
    if token:
        if header == "token":
            req.add_header("Authorization", "token " + token)
        elif header == "bearer":
            req.add_header("Authorization", "Bearer " + token)
        else:
            req.add_header(header, token)
    req.add_header("Accept", "application/json")
    if data is not None:
        req.add_header("Content-Type", "application/json")
        data = json.dumps(data).encode()
    try:
        with urllib.request.urlopen(req, data, timeout=TIMEOUT) as r:
            body = r.read().decode()
            return r.status, (json.loads(body) if body.strip() else None)
    except urllib.error.HTTPError as e:
        body = e.read().decode()
        try:
            return e.code, json.loads(body)
        except Exception:
            return e.code, {"message": body[:400]}
    except Exception as e:
        return 0, {"message": str(e)}


# --------------------------------------------------------------------------
# Local Forgejo side
# --------------------------------------------------------------------------
FJ_TOKEN = need("FORGEJO_TOKEN")

status, me = request(FORGEJO_API + "/user", FJ_TOKEN)
if status != 200 or not me:
    sys.exit("cannot authenticate to Forgejo (%s): %s" % (status, me))
FJ_USER = me["login"]
log("authenticated to Forgejo as %s" % FJ_USER)

_known_orgs = set()


def ensure_org(name):
    """Make sure a local org exists to hold an upstream owner's repos."""
    if name.lower() == FJ_USER.lower() or name in _known_orgs:
        return
    status, _ = request("%s/orgs/%s" % (FORGEJO_API, name), FJ_TOKEN)
    if status == 200:
        _known_orgs.add(name)
        return
    if DRY_RUN:
        log("  DRY_RUN would create local org %s" % name)
        _known_orgs.add(name)
        return
    status, resp = request(FORGEJO_API + "/orgs", FJ_TOKEN, "POST", {
        "username": name,
        "visibility": "private",
        "description": "Mirrored organisation",
    })
    if status in (200, 201):
        log("  created local org %s" % name)
        _known_orgs.add(name)
    else:
        log("  WARNING could not create org %s (%s): %s"
            % (name, status, (resp or {}).get("message")))


def local_repo(owner, name):
    status, body = request("%s/repos/%s/%s" % (FORGEJO_API, owner, name), FJ_TOKEN)
    return body if status == 200 else None


def create_mirror(clone_url, owner, name, token, service, upstream_empty=False):
    """Create one pull mirror. `token` is the UPSTREAM credential."""
    existing = local_repo(owner, name)
    if existing:
        # A failed migration leaves a database record with no git data behind.
        # Skipping that forever would silently keep a broken mirror, so rebuild
        # it -- but only when upstream actually has commits to fetch.
        if existing.get("empty") and not upstream_empty:
            if DRY_RUN:
                log("  DRY_RUN would REBUILD %s/%s (exists but empty)" % (owner, name))
                return "planned"
            log("  rebuilding %s/%s (exists but empty)" % (owner, name))
            status, resp = request("%s/repos/%s/%s" % (FORGEJO_API, owner, name),
                                   FJ_TOKEN, "DELETE")
            if status not in (200, 204):
                log("  FAILED to delete broken %s/%s (%s): %s"
                    % (owner, name, status, (resp or {}).get("message")))
                return "failed"
        else:
            log("  skip %s/%s (exists)" % (owner, name))
            return "skipped"

    if DRY_RUN:
        log("  DRY_RUN would mirror %s -> %s/%s" % (clone_url, owner, name))
        return "planned"

    ensure_org(owner)
    payload = {
        "clone_addr": clone_url,
        "repo_name": name,
        "repo_owner": owner,
        "service": service,
        "mirror": True,
        "mirror_interval": INTERVAL,
        "private": True,
        "wiki": True,
        "labels": True,
        "issues": True,
        "milestones": True,
        "releases": True,
        "pull_requests": False,
        "lfs": False,
        "description": "Backup mirror of " + clone_url,
        # auth_token ONLY -- see module docstring.
        "auth_token": token,
    }
    status, resp = request(FORGEJO_API + "/repos/migrate", FJ_TOKEN, "POST", payload)
    if status in (200, 201):
        log("  mirrored %s/%s" % (owner, name))
        time.sleep(CREATE_DELAY)
        return "created"
    msg = (resp or {}).get("message", "")
    log("  FAILED %s/%s (%s): %s" % (owner, name, status, str(msg).replace("\n", " | ")[:300]))
    return "failed"


def local_owner_for(upstream_owner, upstream_user):
    """Own repos go under your account; everything else under a local org."""
    if upstream_owner.lower() == upstream_user.lower():
        return FJ_USER
    return upstream_owner.replace("/", "-")


# --------------------------------------------------------------------------
# Upstream enumeration
# --------------------------------------------------------------------------
def paged(url_tmpl, token, header, per_page=50, limit_key="limit"):
    page = 1
    while True:
        url = url_tmpl % {"page": page, "per": per_page, "limit": limit_key}
        status, body = request(url, token, header=header)
        if status != 200 or not body:
            if status != 200:
                log("  enumeration stopped at page %d (HTTP %s): %s"
                    % (page, status, (body or {}).get("message")))
            return
        if not isinstance(body, list) or not body:
            return
        for item in body:
            yield item
        if len(body) < per_page:
            return
        page += 1


def do_codeberg(stats):
    token = need("CODEBERG_TOKEN")
    user = need("CODEBERG_USER")
    log("enumerating Codeberg repos for %s" % user)
    url = "https://codeberg.org/api/v1/user/repos?limit=%(per)d&page=%(page)d"
    for r in paged(url, token, "token"):
        owner = local_owner_for(r["owner"]["login"], user)
        stats[create_mirror(r["clone_url"], owner, r["name"], token, "gitea",
                            bool(r.get("empty")))] += 1


def do_github(stats):
    token = need("GITHUB_TOKEN")
    user = need("GITHUB_USER")
    log("enumerating GitHub repos for %s" % user)
    url = ("https://api.github.com/user/repos?per_page=%(per)d&page=%(page)d"
           "&affiliation=owner,organization_member")
    for r in paged(url, token, "bearer"):
        owner = local_owner_for(r["owner"]["login"], user)
        stats[create_mirror(r["clone_url"], owner, r["name"], token, "github",
                            bool(r.get("size") == 0))] += 1


def do_gitlab(stats):
    token = need("GITLAB_TOKEN")
    log("enumerating GitLab projects")
    status, me_gl = request("https://gitlab.com/api/v4/user", token, header="PRIVATE-TOKEN")
    gl_user = (me_gl or {}).get("username", "")
    url = ("https://gitlab.com/api/v4/projects?membership=true&per_page=%(per)d"
           "&page=%(page)d")
    for r in paged(url, token, "PRIVATE-TOKEN"):
        ns = r.get("namespace", {}).get("full_path", "")
        owner = local_owner_for(ns, gl_user)
        stats[create_mirror(r["http_url_to_repo"], owner, r["path"], token, "gitlab",
                            bool(r.get("empty_repo")))] += 1


def disk_free():
    try:
        out = subprocess.run(["df", "-h", "--output=avail", "/"],
                             capture_output=True, text=True).stdout
        return out.strip().splitlines()[-1].strip()
    except Exception:
        return "?"


def main():
    if len(sys.argv) != 2 or sys.argv[1] not in ("github", "gitlab", "codeberg", "all"):
        sys.exit("usage: forgejo-add-mirrors {github|gitlab|codeberg|all}")
    which = sys.argv[1]
    if DRY_RUN:
        log("DRY_RUN=1, nothing will be created")
    log("disk free before: %s" % disk_free())

    stats = {"created": 0, "skipped": 0, "failed": 0, "planned": 0}
    for name, fn in (("codeberg", do_codeberg), ("github", do_github), ("gitlab", do_gitlab)):
        if which in (name, "all"):
            if which == "all" and not any(
                    ENV.get(k) for k in ("%s_TOKEN" % name.upper(),)):
                log("skipping %s (no token configured)" % name)
                continue
            fn(stats)

    log("done: %d created, %d skipped, %d failed, %d planned"
        % (stats["created"], stats["skipped"], stats["failed"], stats["planned"]))
    log("disk free after: %s" % disk_free())
    if stats["failed"]:
        log("check failures with: journalctl -u forgejo --since '1 hour ago' | grep -i migrat")
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

sudo DRY_RUN=1 forgejo-add-mirrors codeberg   # plan
sudo forgejo-add-mirrors codeberg             # execute
sudo forgejo-add-mirrors all

Three traps in there, each of which I hit:

Pass auth_token alone. Supplying auth_username next to it makes Forgejo present the username to the upstream as the credential. The upstream rejects it and its error comes back wrapped in your own API's error envelope, complete with a "url" pointing at your own swagger endpoint, so it reads like a local authentication failure:

token is malformed: token contains an invalid number of segments
user's password is invalid [uid: NNNNNN, name: myuser]
access token does not exist [sha: myuser]

That last line is the giveaway: the upstream is looking up a token whose value is your username. What makes it confusing is that both plain forms work fine against the upstream API directly, and only the mixed form fails.

Derive local names from the full upstream path, not basename. If some repositories live under an organisation, myorg/thing and myuser/thing both reduce to thing and the second silently collides with the first. Create a local organisation per upstream owner and mirror into it.

Treat an empty local repository as a failure, not as done. Forgejo inserts the repository row before it clones, so an interrupted migration leaves a record with empty: true and no git data behind it. An existence check alone skips that forever, and the backup that appears to be present is not there at all. Audit for it:

curl -sS -H "Authorization: token $TOKEN" \
  "$API/repos/$OWNER/$NAME" | jq '{empty, mirror, size}'

Backing up the backup

Metadata only: the database, app.ini, and the SSH host keys. Repository content is deliberately excluded, because every repository here is a mirror and the recovery path is re-cloning from upstream rather than restoring:

#!/bin/bash
# Back up the Forgejo *metadata* only: the SQLite DB, app.ini and SSH host keys.
#
# Repository content is deliberately excluded. These repos are themselves
# mirrors of Codeberg/GitLab/GitHub, so re-cloning from upstream is the
# recovery path. That keeps each backup a few MB instead of many GB, which
# matters on a 23 GB disk.
#
# Restore: stop forgejo, untar over /var/lib/forgejo/data + /etc/forgejo,
# chown -R git:git /var/lib/forgejo, start forgejo.

set -euo pipefail

DEST=/var/lib/forgejo/backups
KEEP=4
DB=/var/lib/forgejo/data/forgejo.db

mkdir -p "$DEST"
chmod 700 "$DEST"

STAMP=$(date +%Y%m%d-%H%M%S)
OUT="$DEST/forgejo-meta-$STAMP.tar.zst"
TMP=$(mktemp -d)
cleanup() { rm -rf "$TMP"; }
trap cleanup EXIT

# Consistent snapshot of a live, WAL-mode SQLite database.
if command -v sqlite3 >/dev/null 2>&1; then
    sqlite3 "file:${DB}?mode=ro" ".backup '$TMP/forgejo.db'"
else
    cp -a "$DB" "$TMP/" 2>/dev/null || true
    cp -a "${DB}-wal" "${DB}-shm" "$TMP/" 2>/dev/null || true
fi

cp -a /etc/forgejo/app.ini "$TMP/app.ini"
# Built-in SSH server host keys: preserving these avoids host-key warnings
# for anyone who has cloned over ssh://[email protected]:2222.
cp -a /var/lib/forgejo/data/ssh "$TMP/ssh" 2>/dev/null || true

tar -C "$TMP" -cf - . | zstd -q -19 -o "$OUT"
chmod 600 "$OUT"

# Prune all but the newest $KEEP archives.
ls -1t "$DEST"/forgejo-meta-*.tar.zst 2>/dev/null | tail -n +$((KEEP + 1)) | xargs -r rm -f

logger -t forgejo-backup "wrote $OUT ($(du -h "$OUT" | cut -f1))"

That produces tens of KB per archive instead of gigabytes. .backup rather than cp, because the database is live and in WAL mode, and a plain copy gives a torn snapshot with a stale -wal sibling. Keeping the host keys means nobody sees a host-key warning after a restore.

Restoring is the DB plus one script run, since the mirror list lives in the database and the rebuild-empty logic above re-clones the content:

sudo systemctl stop forgejo
# unpack, copy forgejo.db / app.ini / ssh into place, chown -R git:git
sudo systemctl start forgejo
sudo forgejo-add-mirrors all

Watching the disk

Mirroring several accounts onto a small volume is exactly the kind of thing that fills up quietly:

#!/bin/bash
# Disk and memory guard for the Forgejo backup forge.
#
# This box has 23 GB total / ~16 GB free and 950 MiB RAM, so mirroring three
# accounts can plausibly fill it. Run hourly from cron; findings go to
# journald (tag forgejo-diskcheck) and to the status file below.
#
#   journalctl -t forgejo-diskcheck --since today
#   cat /var/lib/forgejo/diskcheck.status

set -uo pipefail

WARN_PCT=80
CRIT_PCT=90
REPO_DIR=/var/lib/forgejo/data/forgejo-repositories
STATUS_FILE=/var/lib/forgejo/diskcheck.status

say() {
    local level="$1"; shift
    logger -t forgejo-diskcheck -p "daemon.${level}" -- "$*"
    printf '%s [%s] %s\n' "$(date -Is)" "$level" "$*"
}

pct=$(df --output=pcent / | tail -1 | tr -dc '0-9')
avail=$(df -h --output=avail / | tail -1 | tr -d ' ')
repo_size=$(du -sh "$REPO_DIR" 2>/dev/null | cut -f1)
repo_count=$(find "$REPO_DIR" -maxdepth 2 -name '*.git' -type d 2>/dev/null | wc -l)
mem_avail=$(free -m | awk '/^Mem:/ {print $7}')
swap_used=$(free -m | awk '/^Swap:/ {print $3}')

{
    say info "disk ${pct}% used, ${avail} free | repos ${repo_count} using ${repo_size:-0} | mem ${mem_avail}MiB avail, swap ${swap_used}MiB used"

    if (( pct >= CRIT_PCT )); then
        say crit "CRITICAL: root filesystem ${pct}% full (${avail} left). Mirror syncs will start failing. Free space or resize the volume."
        # Stop scheduling new mirror work so a half-written repo does not wedge
        # the disk completely. Existing data is untouched.
        if systemctl is-active --quiet forgejo; then
            say warning "leaving forgejo running; disable the update_mirrors cron in the admin UI if the disk does not recover"
        fi
    elif (( pct >= WARN_PCT )); then
        say warning "WARNING: root filesystem ${pct}% full (${avail} left)."
    fi

    if (( mem_avail < 80 )); then
        say warning "low memory: only ${mem_avail}MiB available, ${swap_used}MiB swap in use"
    fi

    # Largest repos, useful when deciding what to drop.
    if (( pct >= WARN_PCT )) && [[ -d "$REPO_DIR" ]]; then
        say info "largest mirrors: $(du -s "$REPO_DIR"/*/* 2>/dev/null | sort -rn | head -5 | awk '{printf "%s(%.0fMB) ", $2, $1/1024}')"
    fi
} > "$STATUS_FILE" 2>&1

exit 0

And the cron that ties it together:

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Disk/memory guard: journalctl -t forgejo-diskcheck | /var/lib/forgejo/diskcheck.status
17 *  * * * root /usr/local/bin/forgejo-diskcheck >/dev/null 2>&1
# Metadata-only backup (DB + app.ini + host keys), keeps 4
23 3  * * 0 root /usr/local/bin/forgejo-backup-config >/dev/null 2>&1
# Refresh Cloudflare IP ranges for real-IP restoration
43 4  3 * * root /usr/local/bin/cloudflare-realip-update >/dev/null 2>&1

Verify from somewhere else

This is the other mistake worth confessing, because it invalidated everything I thought I had checked. My first HTTPS test returned a clean 200 and meant nothing: my shell had https_proxy set to a local SOCKS listener whose exit node was that same server, so the request hairpinned internally and never crossed the public internet. The evidence was sitting in the access log the whole time, where the client address was the server's own.

If a reachability test shows the origin talking to itself, it has tested nothing. Test from a third host:

for p in 22 80 443 2222; do
  timeout 8 bash -c "exec 3<>/dev/tcp/$IP/$p" 2>/dev/null \
    && echo "port $p open" || echo "port $p blocked"
done

curl -s -o /dev/null -w '%{http_code} tlsverify=%{ssl_verify_result}\n' \
     https://git.example.com/
echo | openssl s_client -connect git.example.com:443 -servername git.example.com \
     2>/dev/null | openssl x509 -noout -subject -dates
ssh-keyscan -p 2222 -T 10 $IP

git clone https://USER:[email protected]/USER/repo.git

The last one is the only test that actually proves the thing works, so do not stop before it.

Summary of the traps