Three small follow-ups that turn the fleet management system from 'all
the pieces ship green' into 'actually produces telemetry':
cmd/worker/main.go: go workerService.RunHealth(ctx, 30s) alongside
Heartbeat. The sampler snapshots rolling 1m counters into a WorkerHealth
event via the existing event bus + codec path.
cmd/backend/main.go: background goroutine refreshes
worker_capacity_view every minute via REFRESH MATERIALIZED VIEW
CONCURRENTLY. The assignment loop, Rebalancer, Scaler, and
QuarantineEvaluator all read from the view, so it's the freshness gate
for the whole system.
internal/app/worker/event_send_email.go + health_record.go: classify
every wmail.SendResult into the right counter (auth / rate-limit /
bounce-hard / bounce-soft / success) and record SMTP latency. Falls
back to free-text message classification when the error code is
generic, so signal stays useful as new error paths are added.
End-to-end: a worker that bounces 10% of sends now lands in the
'quarantined' band within 5min of the QuarantineEvaluator tick,
auto-drains via Rebalancer, and triggers a Scaler alert if its
removal drops fleet capacity below the warning threshold.
Hetzner CX32 + 16 Primary IPs becomes 16 sending identities with one
install command, without expanding ops complexity.
cmd/worker/main.go: WORKER_ID now resolves via 4-tier precedence:
1. WORKER_ID env (explicit UUID)
2. WORKER_BIND_IP env (derive UUIDv5 from the bound IP)
3. hostname-as-UUID (legacy single-IP VPS)
4. generated UUID (local dev fallback)
Boot also constructs the chosen Codec + EventBus + EncryptedKeyStore
via the FromEnv factories from earlier commits, so a worker process is
fully configured by its envelope env file plus the runtime config it
pulls from the backend on first boot.
scripts/install-worker.sh gains --ips <ipv4,ipv4,...> which:
- writes a warmbly-worker@.service systemd template
- drops a per-instance env file at /etc/warmbly/instances/<dashed-ip>.env
with WORKER_BIND_IP and WORKER_ID
- shares one /etc/warmbly/worker.env for the common config
- --status, --update, --uninstall now multi-IP aware
- single-IP mode preserved when --ips is absent
5 worker tests pin the UUIDv5 derivation against the installer's
uuidgen --sha1 output so the two never drift.
docs/MULTI_IP_WORKERS.md is the operator runbook with the Hetzner
recipe, OS-level IP attachment, rDNS automation, day-2 ops, and the
25%-of-fleet blast-radius rule.
Matthew Meszaros