Metrics and a Homepage
Two additions to a small VPS that was already running a git forge, an RSS reader and a password vault: a metrics stack for actual history, and a homepage for the at-a-glance view. Combined footprint is around 250 MB resident, on a box with 3.7 GB total.
VictoriaMetrics instead of Prometheus
Same query language, same scrape config format, and Grafana treats it as a
Prometheus datasource. On this workload it sits at about 157 MB resident where
Prometheus would want considerably more, and retention is a single flag
(-retentionPeriod=6 for six months, which at a 30 second scrape interval of one
node is tens of megabytes).
Nothing about the setup is VictoriaMetrics-specific. If you already know Prometheus, you already know this.
Host networking, deliberately
All three components use network_mode: host and bind their own loopback
addresses:
victoriametrics:
image: victoriametrics/victoria-metrics:v1.112.0
network_mode: host
command:
- '-httpListenAddr=127.0.0.1:8428'
- '-promscrape.config=/etc/prometheus.yml'
- '-retentionPeriod=6'
node-exporter:
image: prom/node-exporter:v1.9.1
network_mode: host
pid: host
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- '--web.listen-address=127.0.0.1:9100'
- '--path.procfs=/host/proc'
- '--path.sysfs=/host/sys'
- '--path.rootfs=/rootfs'
grafana:
image: grafana/grafana:11.6.1
network_mode: host
environment:
- GF_SERVER_HTTP_ADDR=127.0.0.1
- GF_SERVER_HTTP_PORT=3001
- GF_SERVER_ROOT_URL=https://metrics.example.com
- GF_USERS_ALLOW_SIGN_UP=false
The alternative, bridge networking with published ports, means VictoriaMetrics
cannot reach node_exporter on 127.0.0.1:9100 and Grafana cannot reach
VictoriaMetrics. You end up putting them on a shared bridge and hardcoding
gateway addresses that change whenever a network is recreated. Host networking
with explicit loopback binds is less machinery and exposes nothing; nginx in
front is the only thing on a public port.
The catch is that you must set each listen address, because the defaults bind
0.0.0.0. Host networking means a default-bound service is immediately public.
Two smaller traps. grafana-data has to be owned by uid 472, the user Grafana
runs as in the official image, or it cannot create its own sqlite database. And
despite --path.rootfs=/rootfs, node_exporter still labels the root filesystem
mountpoint="/", so queries use that, not /rootfs. I wrote a dashboard panel
against /rootfs first and got a silent no-data panel.
Provisioned, not clicked
Datasource and dashboards as files, so the whole thing is reproducible:
# provisioning/datasources/vm.yml
apiVersion: 1
datasources:
- name: VictoriaMetrics
type: prometheus
access: proxy
uid: victoriametrics
url: http://127.0.0.1:8428
isDefault: true
Every panel in the dashboard JSON references
{"type": "prometheus", "uid": "victoriametrics"}. If that uid does not match
the provisioned datasource, panels render empty with no error at all, which is a
miserable thing to debug. It is the first thing to check when a provisioned
dashboard looks broken.
Change the default password before the cert exists
Grafana starts on admin/admin. Worth thinking about the ordering here: issuing a
TLS certificate publishes the hostname to public Certificate Transparency logs
within seconds, and scanners read those logs continuously. In my case automated
probes arrived at brand-new hostnames within minutes of the certificate being
issued.
So anything with a first-run setup state is exposed from the moment the cert exists, not from the moment you tell someone the URL. That applies to Grafana's default credentials and, more sharply, to any service where the first visitor becomes the administrator, which is how several self-hosted dashboards bootstrap. Set credentials before or immediately after issuing the certificate.
One footgun if you script it: GF_SECURITY_ADMIN_PASSWORD is applied on container
start, so a stale value left in your compose file silently resets the password
back to it on the next recreate. Either manage the password entirely through that
variable, or do not set it at all.
The homepage layer
Separately from Grafana, a Glance instance for the things you want on one page: clock, service up/down monitors, RSS, repository releases, bookmarks, host stats. About 8 MB resident, which makes it hard to argue with.
Two things I would tell anyone setting it up.
Set server.host to 0.0.0.0 inside the container. The default of
localhost binds the container's own loopback, which makes a published port
unreachable. Docker does the loopback restriction from the outside; the app should
not.
Validate the config before restarting. It refuses to start on a bad config, so a typo takes the dashboard down. Validating against the config directory without touching the running container costs nothing:
docker run --rm -v /path/to/config:/app/config:ro glanceapp/glance config:validate
Silence means valid. It reports one error at a time with a line number. This caught two mistakes for me before they mattered: an indentation problem, and two widget types that appear in the current documentation but do not exist in the released binary. When a widget name is rejected, trust the binary over the docs:
docker run --rm --entrypoint sh glanceapp/glance -c \
'strings "$(command -v glance)" | grep -oE "\b(rss|monitor|todo|calendar-legacy)\b" | sort -u'
Also set slug: explicitly on every page. Auto-derived slugs from titles
containing punctuation do not produce the URL you would guess, and the page 404s
while the root still works.
Almost every widget needs outbound network
Obvious in hindsight, and worth a sentence because it made the dashboard look broken for reasons unrelated to the dashboard. Glance is mostly a feed aggregator: RSS, releases, monitors, weather, all of it is outbound HTTP from the container. On this host, container egress was silently blackholed by an unrelated firewall problem, so the dashboard rendered clocks and bookmarks and nothing else.
If widgets are uniformly empty, test the container's network before reading any config:
docker exec glance sh -c 'wget -q -T 8 -O /dev/null https://api.github.com && echo OK || echo FAIL'