The bind-mounted .git is owned by the host user but the dev
container runs as root, so git aborts with "dubious ownership"
and Go's automatic VCS stamping fails the build with:
error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping.
Production Dockerfiles are unaffected because they do not
bind-mount .git; they COPY source into the image and let Go
stamp normally.
Until now, only the web service hot-reloaded (Vite HMR via the
node:22-alpine container + ./web mount). The Go services (backend,
consumer, worker) used their production multi-stage Dockerfiles, so
every code change meant `docker compose build <svc> && docker compose
up -d <svc>` — ~30s per service.
Switched all Go services to a shared dev image (go.dev.Dockerfile)
that ships:
- full Go 1.25 toolchain on alpine
- CGO deps for librdkafka (gcc, musl-dev, librdkafka-dev, pkgconf)
- air v1.61.7 (the source watcher / hot-recompile tool)
docker-compose mounts the repo at /app and runs `air -c <config>`.
Each Go service has its own air.SERVICE.toml (build target +
exclusions). Named volumes for the Go module cache and build cache
so the first build is slow (~60s for module download) but subsequent
rebuilds after a save are ~2s.
Per-service compose changes:
- backend, consumer: dockerfile, volumes, and command updated
- worker-base (the YAML anchor used by all 3 workers): same
Production Dockerfiles in deploy/docker/{backend,consumer,worker}.
Dockerfile are unchanged and still used by release CI. The seed
one-shot in compose continues to use backend.Dockerfile (it's a
short-lived job, no benefit from the dev image).
Rust (tracking) and Elixir (realtime) still build-on-change. They
change far less often; documenting the workaround in
resources/local-development.md for now.
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Hoist the dev/sim stack to a single docker-compose.yml at the repo root.
Adds profiles (default / sim / seed / tools) so you can opt into heavier
setups, and bundles dependencies that were previously missing:
- LocalStack (KMS + DynamoDB + S3) with a localstack-init one-shot that
idempotently creates alias/master-key-dev, the UserEncryptedKeys and
EmailMessageData tables, and the main S3 bucket. Backend and workers
wait on it via service_completed_successfully.
- stripe-mock for billing flows
- kafka-ui under the tools profile
Three workers with deterministic UUIDv5 hostnames (shared / premium /
dedicated) so assignment, rebalancing, and per-pool routing all have
real targets to exercise.
Richer seed (cmd/seed/main.go) loads 3 orgs across tiers, 6 mailboxes
joined to free/premium warmup pools, a Beta campaign with a 2-step
sequence, and 10 contacts (2 unsubscribed) so suppression behaviour is
visible in the UI. Idempotent — safe to re-run.
Makefile targets:
make dev — infra + app + one worker
make sim — adds premium + dedicated workers
make seed — rich fixtures
make tools — kafka-ui at :18090
make reset — nuke volumes
Matthew Meszaros