Refresh developer docs, deployment notes, and public-site copy to describe the Postgres-backed encrypted key and message-map model after DynamoDB removal.
Add passkey enrollment and login wiring, including a Safari-safe explicit login path that prefetches the WebAuthn challenge before the click and calls the credential ceremony immediately from the user gesture.
Every docker compose invocation in the Makefile now pins
-p warmbly so all git worktrees target the same compose
project. This means infra (postgres, redis, kafka, mailpit,
localstack, stripe-mock, cloud-tasks-emulator, zookeeper,
schema-registry) is brought up once and stays running across
worktree switches. App services (backend, consumer, worker,
tracking, realtime, web) recreate in place per worktree against
the bind-mounted source.
Removed targets:
- dev, dev-down, dev-logs (and the DEV_SVCS / SVCS vars)
Added targets:
- infra, infra-down
- app, app-down, app-logs
Daily flow becomes:
make infra # once, from any worktree
cd /path/to/worktree-a
make app # bring up app code for branch A
cd /path/to/worktree-b
make app # recreates app against branch B;
# infra untouched, caches warm
The named cache volumes already shared their content across
worktrees (warmbly_gomodcache, warmbly_gocache, warmbly_cargo_home,
warmbly_cargo_target, warmbly_mix_deps, warmbly_mix_build); pinning
the project name additionally makes container ownership shared,
which is what eliminates the per-worktree cold start.
README.md, resources/local-development.md, resources/deployment-guide.md,
deploy/README.md, and docker-compose.dev.yml all updated to reflect
the new targets.
The bind-mounted .git is owned by the host user but the dev
container runs as root, so git aborts with "dubious ownership"
and Go's automatic VCS stamping fails the build with:
error obtaining VCS status: exit status 128
Use -buildvcs=false to disable VCS stamping.
Production Dockerfiles are unaffected because they do not
bind-mount .git; they COPY source into the image and let Go
stamp normally.
Most container ports go back to their natural defaults — the offsets
that existed weren't justified, they just made URLs harder to remember.
Now standard:
backend 8080 (was always 8080)
tracking 3000 (was 13000)
realtime 4000 (was 14000)
web 5173 (was 15173 — already changed)
kafka 9092 (was 19092)
schema-registry 8081 (was 18081)
localstack 4566 (was 14566)
cloud-tasks 8123 (was 18123)
stripe-mock 12111 (always was)
Kept offset (the defaults conflict too often on real dev machines):
postgres 15432 (system postgres / sibling project)
redis 16379 (sibling docker projects with redis)
mailpit ui 18025 (sibling docker projects with mailpit)
mailpit smtp 11025 (same)
kafka-ui 18090 (8080 already used by backend)
Touched: docker-compose.yml, Makefile (test-seed SEED_TEST_DB), READMEs
(root + deploy), local-development.md + deployment-guide.md. Internal
docker-network refs (kafka:29092, mailpit:1025, etc.) unchanged — only
host-port mappings moved. Compose validated, all default-profile
services come up healthy on the new ports.
Until now, only the web service hot-reloaded (Vite HMR via the
node:22-alpine container + ./web mount). The Go services (backend,
consumer, worker) used their production multi-stage Dockerfiles, so
every code change meant `docker compose build <svc> && docker compose
up -d <svc>` — ~30s per service.
Switched all Go services to a shared dev image (go.dev.Dockerfile)
that ships:
- full Go 1.25 toolchain on alpine
- CGO deps for librdkafka (gcc, musl-dev, librdkafka-dev, pkgconf)
- air v1.61.7 (the source watcher / hot-recompile tool)
docker-compose mounts the repo at /app and runs `air -c <config>`.
Each Go service has its own air.SERVICE.toml (build target +
exclusions). Named volumes for the Go module cache and build cache
so the first build is slow (~60s for module download) but subsequent
rebuilds after a save are ~2s.
Per-service compose changes:
- backend, consumer: dockerfile, volumes, and command updated
- worker-base (the YAML anchor used by all 3 workers): same
Production Dockerfiles in deploy/docker/{backend,consumer,worker}.
Dockerfile are unchanged and still used by release CI. The seed
one-shot in compose continues to use backend.Dockerfile (it's a
short-lived job, no benefit from the dev image).
Rust (tracking) and Elixir (realtime) still build-on-change. They
change far less often; documenting the workaround in
resources/local-development.md for now.
Old docs described a k8s/ArgoCD/Terraform deployment that no longer
exists, with ASCII-art system diagrams that hadn't aged well. Rewritten
to match how the project actually ships:
- README: control plane (Railway) + execution plane (per-VPS workers)
split, dashboard-driven worker management, credentials/profiles,
auto-update from GitHub releases, OS package updates, self-hosting
knobs. Removed all ASCII art.
- resources/architecture.md: control vs execution plane, encryption
model (worker SSH keys + platform secrets under the same KMS-envelope
cipher as user secrets), worker identity from public IPv4, credentials
model, push-driven release flow, anti-abuse layers, source anchors.
- resources/deployment-guide.md: end-to-end from "provision a VPS" to
"auto-update on release". No more k8s, ArgoCD, kubectl, or Terraform.
Step-by-step backend env, webhook setup, worker add flow, day-2 ops,
rollback per plane.
- resources/local-development.md: the five make targets (dev / sim /
seed / tools / reset), what each profile runs, LocalStack bootstrap,
rich seed contents, native dev against containerized infra, the
offset-port URL table.
- resources/cicd.md: the two-plane build/release flow, image tag scheme
({sha} / dev / vX.Y.Z / vX.Y / vX / prod), webhook setup, release
process, security notes around HMAC and least-privilege worker AWS
keys.
- deploy/README.md: tight version of the same.
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Hoist the dev/sim stack to a single docker-compose.yml at the repo root.
Adds profiles (default / sim / seed / tools) so you can opt into heavier
setups, and bundles dependencies that were previously missing:
- LocalStack (KMS + DynamoDB + S3) with a localstack-init one-shot that
idempotently creates alias/master-key-dev, the UserEncryptedKeys and
EmailMessageData tables, and the main S3 bucket. Backend and workers
wait on it via service_completed_successfully.
- stripe-mock for billing flows
- kafka-ui under the tools profile
Three workers with deterministic UUIDv5 hostnames (shared / premium /
dedicated) so assignment, rebalancing, and per-pool routing all have
real targets to exercise.
Richer seed (cmd/seed/main.go) loads 3 orgs across tiers, 6 mailboxes
joined to free/premium warmup pools, a Beta campaign with a 2-step
sequence, and 10 contacts (2 unsubscribed) so suppression behaviour is
visible in the UI. Idempotent — safe to re-run.
Makefile targets:
make dev — infra + app + one worker
make sim — adds premium + dedicated workers
make seed — rich fixtures
make tools — kafka-ui at :18090
make reset — nuke volumes
Workers need IP diversity, but k8s nodes typically NAT all pods through a
small set of egress IPs — defeating the point of a DaemonSet for cold mail.
Plus, the control plane is moving to Railway and workers will be managed
per-VPS, so the kustomize tree no longer reflects how anything actually
ships.
Matthew Meszaros