Home Blog

Metrics and a Homepage

Two additions to a small VPS that was already running a git forge, an RSS reader and a password vault: a metrics stack for actual history, and a homepage for the at-a-glance view. Combined footprint is around 250 MB resident, on a box with 3.7 GB total.

VictoriaMetrics instead of Prometheus

Same query language, same scrape config format, and Grafana treats it as a Prometheus datasource. On this workload it sits at about 157 MB resident where Prometheus would want considerably more, and retention is a single flag (-retentionPeriod=6 for six months, which at a 30 second scrape interval of one node is tens of megabytes).

Nothing about the setup is VictoriaMetrics-specific. If you already know Prometheus, you already know this.

Host networking, deliberately

All three components use network_mode: host and bind their own loopback addresses:

  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.112.0
    network_mode: host
    command:
      - '-httpListenAddr=127.0.0.1:8428'
      - '-promscrape.config=/etc/prometheus.yml'
      - '-retentionPeriod=6'

  node-exporter:
    image: prom/node-exporter:v1.9.1
    network_mode: host
    pid: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--web.listen-address=127.0.0.1:9100'
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'

  grafana:
    image: grafana/grafana:11.6.1
    network_mode: host
    environment:
      - GF_SERVER_HTTP_ADDR=127.0.0.1
      - GF_SERVER_HTTP_PORT=3001
      - GF_SERVER_ROOT_URL=https://metrics.example.com
      - GF_USERS_ALLOW_SIGN_UP=false

The alternative, bridge networking with published ports, means VictoriaMetrics cannot reach node_exporter on 127.0.0.1:9100 and Grafana cannot reach VictoriaMetrics. You end up putting them on a shared bridge and hardcoding gateway addresses that change whenever a network is recreated. Host networking with explicit loopback binds is less machinery and exposes nothing; nginx in front is the only thing on a public port.

The catch is that you must set each listen address, because the defaults bind 0.0.0.0. Host networking means a default-bound service is immediately public.

Two smaller traps. grafana-data has to be owned by uid 472, the user Grafana runs as in the official image, or it cannot create its own sqlite database. And despite --path.rootfs=/rootfs, node_exporter still labels the root filesystem mountpoint="/", so queries use that, not /rootfs. I wrote a dashboard panel against /rootfs first and got a silent no-data panel.

Provisioned, not clicked

Datasource and dashboards as files, so the whole thing is reproducible:

# provisioning/datasources/vm.yml
apiVersion: 1
datasources:
  - name: VictoriaMetrics
    type: prometheus
    access: proxy
    uid: victoriametrics
    url: http://127.0.0.1:8428
    isDefault: true

Every panel in the dashboard JSON references {"type": "prometheus", "uid": "victoriametrics"}. If that uid does not match the provisioned datasource, panels render empty with no error at all, which is a miserable thing to debug. It is the first thing to check when a provisioned dashboard looks broken.

Change the default password before the cert exists

Grafana starts on admin/admin. Worth thinking about the ordering here: issuing a TLS certificate publishes the hostname to public Certificate Transparency logs within seconds, and scanners read those logs continuously. In my case automated probes arrived at brand-new hostnames within minutes of the certificate being issued.

So anything with a first-run setup state is exposed from the moment the cert exists, not from the moment you tell someone the URL. That applies to Grafana's default credentials and, more sharply, to any service where the first visitor becomes the administrator, which is how several self-hosted dashboards bootstrap. Set credentials before or immediately after issuing the certificate.

One footgun if you script it: GF_SECURITY_ADMIN_PASSWORD is applied on container start, so a stale value left in your compose file silently resets the password back to it on the next recreate. Either manage the password entirely through that variable, or do not set it at all.

The homepage layer

Separately from Grafana, a Glance instance for the things you want on one page: clock, service up/down monitors, RSS, repository releases, bookmarks, host stats. About 8 MB resident, which makes it hard to argue with.

Two things I would tell anyone setting it up.

Set server.host to 0.0.0.0 inside the container. The default of localhost binds the container's own loopback, which makes a published port unreachable. Docker does the loopback restriction from the outside; the app should not.

Validate the config before restarting. It refuses to start on a bad config, so a typo takes the dashboard down. Validating against the config directory without touching the running container costs nothing:

docker run --rm -v /path/to/config:/app/config:ro glanceapp/glance config:validate

Silence means valid. It reports one error at a time with a line number. This caught two mistakes for me before they mattered: an indentation problem, and two widget types that appear in the current documentation but do not exist in the released binary. When a widget name is rejected, trust the binary over the docs:

docker run --rm --entrypoint sh glanceapp/glance -c \
  'strings "$(command -v glance)" | grep -oE "\b(rss|monitor|todo|calendar-legacy)\b" | sort -u'

Also set slug: explicitly on every page. Auto-derived slugs from titles containing punctuation do not produce the URL you would guess, and the page 404s while the root still works.

Almost every widget needs outbound network

Obvious in hindsight, and worth a sentence because it made the dashboard look broken for reasons unrelated to the dashboard. Glance is mostly a feed aggregator: RSS, releases, monitors, weather, all of it is outbound HTTP from the container. On this host, container egress was silently blackholed by an unrelated firewall problem, so the dashboard rendered clocks and bookmarks and nothing else.

If widgets are uniformly empty, test the container's network before reading any config:

docker exec glance sh -c 'wget -q -T 8 -O /dev/null https://api.github.com && echo OK || echo FAIL'