warmbly/resources/deployment-guide.md

Deployment Guide

End-to-end guide for deploying Warmbly. The control plane runs on a container host (Railway in production); workers run on as many VPSes as you want and are managed from the admin dashboard.

Overview

Plane	What runs	Where
Control	backend, consumer, tracking, realtime, web	One container host with stable region pinning (Railway, ECS Fargate, Fly, Kubernetes — any of them works). Postgres + Redis + Kafka + Schema Registry + S3 + KMS + DynamoDB are external dependencies.
Execution	worker	One process per VPS, anywhere in the world, supervised by systemd, configured from the admin dashboard over SSH.

The control plane never needs to scale beyond a few replicas; the worker fleet is what fans out for IP diversity.

What you need

For the control plane:

• A Postgres 16 instance (RDS, Neon, Supabase, self-hosted — anything)
• A Redis 7 instance
• A Kafka broker with a Schema Registry (Confluent Cloud, Aiven, Redpanda, self-hosted)
• An S3-compatible bucket
• AWS KMS for envelope-encryption root, or a KMS-compatible service via LocalStack for local dev
• DynamoDB (or compatible) for per-user encrypted DEKs
• A Stripe account (or stripe-mock for non-prod)

For the workers:

• A VPS per worker (Hetzner / OVH / Vultr / DigitalOcean / Linode — anywhere with a public IPv4). Spread providers for IP diversity.
• Root SSH access on the VPS at install time.

Step 1: Provision the control-plane dependencies

Whatever your stack, get the seven external services above running and collect the connection details. Local dev can substitute everything with make sim from the repo root (see local-development.md) — that runs LocalStack for KMS/Dynamo/S3 and includes everything else as containers.

Step 2: Configure the backend

The backend reads from environment variables. Minimum set:

APP_ENV=prod
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

PRIMARY_DB=postgres://user:pass@host:5432/warmbly?sslmode=require
REDIS=redis://default:pass@host:6379

KAFKA_BOOTSTRAP_SERVERS=broker1:9092,broker2:9092
KAFKA_SASL_USERNAME=...
KAFKA_SASL_PASSWORD=...
SCHEMA_REGISTRY_URL=https://schema.example.com
SCHEMA_REGISTRY_KEY=...
SCHEMA_REGISTRY_SECRET=...

AUTH_SECRET=<32+ char random>
STRIPE_SECRET_KEY=sk_live_...
STRIPE_WEBHOOK_SECRET=whsec_...
TURNSTILE_SECRET=...
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
GOOGLE_REDIRECT_URI=https://api.example.com/auth/google/callback

Worker-related env (for the dashboard's worker orchestration):

WORKER_IMAGE=ghcr.io/youruser/yourfork/worker:prod
WORKER_INSTALLER_PATH=/app/scripts/install-worker.sh
WORKER_AWS_ACCESS_KEY_ID=...
WORKER_AWS_SECRET_ACCESS_KEY=...

Release auto-update (optional; safe to omit on a fresh install):

RELEASES_ENABLED=true
RELEASES_GITHUB_REPO=warmbly/warmbly             # or your fork
RELEASES_WORKER_IMAGE_REPO=ghcr.io/warmbly/warmbly/worker
RELEASES_WEBHOOK_SECRET=<shared with GitHub>
RELEASES_GITHUB_TOKEN=<optional PAT for higher API limits>

A full env reference is deploy/config/env.example.

Step 3: Build and push images

Tagging a release on GitHub triggers release.yml, which builds and pushes:

ghcr.io/<owner>/warmbly/worker:vX.Y.Z
ghcr.io/<owner>/warmbly/worker:vX.Y
ghcr.io/<owner>/warmbly/worker:vX
ghcr.io/<owner>/warmbly/worker:prod

And the same set of tags for backend, consumer, tracking, realtime. The build-push.yml workflow on main publishes :{sha} and :dev.

For self-hosters: this works out of the box on any fork as long as you have packages: write permission enabled in repo settings.

Step 4: Deploy the control plane

Pick one. The Dockerfiles in deploy/docker/ are the deployment unit:

• Railway (what we use in production): one project per environment, one service per Dockerfile, env vars set in the Railway UI. Pushes to main auto-deploy.
• ECS Fargate: a task definition per service, ALB in front, secrets via AWS Secrets Manager. ~200 lines of Terraform.
• Fly.io: fly launch per service; nice if you want Elixir distributed clustering for realtime.
• Single VPS: each service as a systemd unit fronted by Caddy. Works for small deployments.

The control plane needs database migration to be run before the backend starts. Backend runs migrations automatically on boot (internal/infrastructure/db/migrate.go), so no separate migration step is required.

Step 5: Set up the GitHub Release webhook

If RELEASES_ENABLED=true:

1. In your GitHub repo → Settings → Webhooks → Add webhook
2. Payload URL: https://api.example.com/webhooks/github/releases
3. Content type: application/json
4. Secret: same value as RELEASES_WEBHOOK_SECRET on the backend
5. Events: select Releases only
6. Save

From now on, publishing a GitHub release fires the webhook and the backend resolves each profile's channel.

Step 6: Add a worker

Two ways.

From the dashboard (recommended)

1. Provision a VPS (Hetzner et al). Note its public IPv4 and root user.
2. In Warmbly admin → Workers → Add Worker. Fill in host, port, user, name. Pick a worker profile (or leave blank to use backend env defaults). Click Create.
3. Copy the generated SSH public key. Paste it into the VPS's ~/.ssh/authorized_keys:

   ssh root@<vps> 'mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys'
   # paste the key, then Ctrl-D
   

4. In the dashboard → worker detail page → Test connection. Should succeed and pin the host fingerprint.
5. Click Install. Backend uploads scripts/install-worker.sh, runs it, and the worker is live.

From the VPS itself

curl -fsSL https://get.example.com/worker | sudo bash -s -- \
  --kafka kafka.example.com:9092 \
  --schema-registry https://schema.example.com \
  --redis redis://cache.example.com:6379 \
  --aws-region us-east-1 --aws-key AKIA... --aws-secret ...

You can later "adopt" this worker into the dashboard by adding the SSH key to it and pointing a profile at it.

Step 7: Wire up the worker profile

In the admin dashboard → Credentials → AWS Credentials → Add. Name it, region, access key, secret. The secret is encrypted via KMS before storage.

Then → Worker Profiles → Add. Pick the AWS Credentials row. Fill in Kafka bootstrap, SASL username/password, Schema Registry URL/key/secret, Redis URL. Pick a release channel (pinned, stable, dev) and whether auto_update is on.

Assign workers to this profile on each worker's detail page. Click Apply to re-write worker.env on the VPS and restart.

Day-2 operations

Everything happens from the worker detail page:

Action	Effect
Test connection	Open an SSH session, run true
Install	Upload installer + env, run it. Idempotent.
Restart	systemctl restart warmbly-worker.service
Pull latest & restart	Pull the new image, regenerate unit, restart
Apply config & restart	Rewrite /etc/warmbly/worker.env, restart (no docker pull)
Update OS packages	apt/dnf/pacman/apk upgrade noninteractively
Reboot VPS	shutdown -r +1
Rotate SSH keys	Generate new ed25519 keypair, install on target
Uninstall	Stop + remove systemd unit and container
Delete	Uninstall + remove row from workers table

When auto_update=true is set on a profile and a new release fires the webhook, every assigned worker gets the new image automatically. Otherwise the dashboard shows an "update available" hint and waits for you to click Apply.

Releasing a new version

git tag -a v1.2.4 -m "Release v1.2.4"
git push origin v1.2.4

release.yml builds and pushes images. If a release webhook is configured, the backend picks up the new version, updates any profile on stable or dev channels, and rolls profiles with auto_update=true. Otherwise the dashboard shows the new version and you Apply manually.

Rolling back

• Per-profile: edit the profile, switch channel to pinned, set worker_image to a specific older tag (e.g. ghcr.io/.../worker:v1.2.3), save, then Apply to all workers on that profile.
• Per-worker: same idea, but assign the worker to a different profile (or detach and run the installer manually with the old image).

Troubleshooting

Worker shows install_state: error Open the worker detail page, click Test. If that fails, the SSH key isn't in authorized_keys on the VPS. If Test passes but Install fails, check the last_error shown on the detail page and the output captured under Logs.

Auto-update didn't fire Check GitHub repo → Settings → Webhooks → Recent Deliveries. If the webhook delivery shows 401, the secret is wrong. If it shows 200 but nothing happened, check Releases panel → Last error in the dashboard, or journalctl -u backend for the boot check.

Worker stuck on old image "Apply config & restart" only rewrites env, not image. Use "Pull latest & restart" instead, or the release auto-update path.

Heartbeat shows offline

• Worker can reach Kafka? (firewall, SASL creds)
• Worker container actually running? (worker detail → Live status)
• VPS's clock skewed? (Kafka SASL auth is time-sensitive)

Quick reference

# control plane: locally
make infra      # postgres / redis / kafka / etc. (leave running)
make app        # backend / consumer / worker / tracking / realtime / web

# load fixtures
make seed

# full simulation with extra workers + localstack + stripe-mock
make sim

# tear down everything
make reset

# manual worker install on a VPS
curl -fsSL https://your-host/install-worker.sh | sudo bash -s -- --help

# trigger a release manually (build+push happens via GH Actions)
git tag -a v1.2.4 -m "Release v1.2.4" && git push origin v1.2.4