warmbly/docs/content/docs/development/deployment-guide.mdx

---
title: Deployment guide
description: Deploy the control plane and a distributed worker fleet to production.
---

End-to-end guide for deploying Warmbly. The default is a no-cloud stack you can
run on a single machine; every cloud service is opt-in.

## No-cloud quick start

On one machine with Docker:

```bash
git clone https://github.com/warmbly/warmbly && cd warmbly
make gen-key          # prints a KMS_LOCAL_MASTER_KEY — put it in .env
make up               # or: docker compose up --build
```

That runs the whole platform — Postgres, Redis, NATS, backend, consumer, worker,
tracking, realtime, dashboard, admin — with **no AWS, GCP, Stripe, or Kafka**.
Before exposing it, set your own secrets in a `.env` next to `docker-compose.yml`:
`AUTH_SECRET`, `KMS_LOCAL_MASTER_KEY`, `CREDENTIALS_ENCRYPTION_KEY`,
`INTERNAL_API_TOKEN` (and `BOX_GOOGLE_CLIENT_ID`/`SECRET` to connect Gmail). Full
list: [`deploy/config/env.example`](https://github.com/warmbly/warmbly/blob/main/deploy/config/env.example).

**Back up `KMS_LOCAL_MASTER_KEY` and `CREDENTIALS_ENCRYPTION_KEY`** — they seal
every stored mailbox credential and losing them is unrecoverable.

## Providers

Each subsystem is an env-var switch. No-cloud defaults on the left:

| Subsystem     | Default (no cloud)         | Opt-in                       |
|---------------|----------------------------|------------------------------|
| Database      | PostgreSQL 16              | RDS / Cloud SQL / any Postgres |
| Cache         | Redis 7                   | ElastiCache                  |
| Event bus     | NATS JetStream (`EVENTBUS_PROVIDER=nats`) | Kafka (`=kafka`, build with `GO_TAGS=kafka`) |
| Blob storage  | Filesystem (`BLOB_PROVIDER=filesystem`) | S3/MinIO/R2 (`=s3`)      |
| Root key      | Local AES (`KMS_PROVIDER=local`) | AWS KMS (`=aws`)          |
| Task scheduler| In-process poller (`TASKS_PROVIDER=local`) | GCP Cloud Tasks (`=gcloud`) |
| Codec         | JSON (`CODEC_PROVIDER=json`) | Avro + Schema Registry (`=avro`) |
| Billing       | Off (`BILLING_PROVIDER=none`) | Stripe (`=stripe`)        |
| Captcha       | Off                       | Cloudflare Turnstile (`CAPTCHA_PROVIDER=turnstile`) |

## Reaching it from your network

Every service already binds to `0.0.0.0` via the Docker port mappings, so it
listens on all interfaces. The app's URLs default to `localhost`, so to use it
from another machine set one variable in `.env`, and the API URL, CORS origins,
websocket URL, and tracking domain all derive from it:

```bash
PUBLIC_HOST=192.168.1.50        # your machine's LAN IP, or a domain
```

Then open `http://192.168.1.50:5173`. The dashboard (`web`) and admin panel
(`admin`) run as production static builds served by nginx, not Vite dev servers.
Their URLs are injected at container start from `WARMBLY_*` env (rendered into a
`/config.js`), so the published image is deployment-agnostic: compose derives
`WARMBLY_API_URL` / `WARMBLY_APP_URL` / `WARMBLY_TRACKING_DOMAIN` from
`PUBLIC_HOST` and `API_PUBLIC_URL`, and the same image runs anywhere. For a
public domain over HTTPS, run a reverse proxy (Caddy or nginx) in front
terminating TLS and point `PUBLIC_HOST` (or the individual `APP_URL` /
`CORS_ALLOW_ORIGINS` / `WEBSOCKET_URL` / `API_PUBLIC_URL` / `TRACKING_DOMAIN`
vars) at your domain.

## Scaling

Outbound mail leaves through each mailbox's own provider (Gmail API, Microsoft
Graph, or the mailbox's SMTP relay), so the source IP is the provider's, not the
worker's. Add capacity by connecting more mailboxes and running more workers
(`docker compose up --scale worker=3`, or attach a machine you own via the admin
panel's SSH enrollment). Workers are interchangeable executors.

## Cloud-scale deployment (optional)

The rest of this page covers running the control plane on a managed host with
cloud services. You only need this if you are deliberately opting into Kafka, S3,
KMS, or Stripe.

**For the control plane**: Postgres 16, Redis 7, and — only if you opt in — a
Kafka broker + Schema Registry, an S3-compatible bucket, AWS KMS, and a Stripe
account.

**For the workers**: a VPS per worker with root SSH access at install time.

## Step 2: Configure the backend

The backend reads from environment variables. Minimum set:

```
APP_ENV=prod
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

PRIMARY_DB=postgres://user:pass@host:5432/warmbly?sslmode=require
REDIS=redis://default:pass@host:6379

KAFKA_BOOTSTRAP_SERVERS=broker1:9092,broker2:9092
KAFKA_SASL_USERNAME=...
KAFKA_SASL_PASSWORD=...
SCHEMA_REGISTRY_URL=https://schema.example.com
SCHEMA_REGISTRY_KEY=...
SCHEMA_REGISTRY_SECRET=...

AUTH_SECRET=<32+ char random>
STRIPE_SECRET_KEY=sk_live_...
STRIPE_WEBHOOK_SECRET=whsec_...
TURNSTILE_SECRET=...
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
GOOGLE_REDIRECT_URI=https://api.example.com/auth/google/callback
```

Worker-related env (for the dashboard's worker orchestration):

```
WORKER_IMAGE=ghcr.io/youruser/yourfork/worker:prod
WORKER_INSTALLER_PATH=/app/scripts/install-worker.sh
WORKER_AWS_ACCESS_KEY_ID=...
WORKER_AWS_SECRET_ACCESS_KEY=...
```

Mobile push via APNs (optional; omit to disable push delivery while device registration keeps working). Set on the backend and the consumer; the digest window controls how push bursts are batched (first event immediate, the rest summarized when the window closes):

```
APNS_KEY_PATH=/etc/warmbly/AuthKey_ABC123.p8   # or APNS_KEY with the PEM contents inline
APNS_KEY_ID=ABC123DEFG
APNS_TEAM_ID=TEAM123456
APNS_TOPIC=com.warmbly.app
NOTIFICATION_PUSH_WINDOW=5h                    # optional, default 5h
```

Notification email delivery (optional; set on the backend and the consumer). Notification emails bundle on a per-user window (30 minutes minimum, up to daily; there is no per-event mode), and each user additionally has a rolling 24h budget for non-security notification emails:

```
NOTIFICATION_EMAIL_DAILY_CAP=0                 # optional, default 25; 0 disables the per-user daily budget
```

Release auto-update (optional; safe to omit on a fresh install):

```
RELEASES_ENABLED=true
RELEASES_GITHUB_REPO=warmbly/warmbly             # or your fork
RELEASES_WORKER_IMAGE_REPO=ghcr.io/warmbly/warmbly/worker
RELEASES_WEBHOOK_SECRET=<shared with GitHub>
RELEASES_GITHUB_TOKEN=<optional PAT for higher API limits>
```

AI provider (optional; omit all of it to run with the AI features off, in which case the assistant returns a clean `503`). Warmbly is OpenAI-first, but the provider is a plain OpenAI-compatible client. Pick a backend with `AI_PROVIDER` and supply a key plus a model; the preset fills in the base URL. Set these on the backend and the consumer:

```
AI_PROVIDER=openai       # openai | openrouter | groq | ollama | anthropic | custom
AI_API_KEY=sk-...        # the key for that provider (not needed for ollama)
AI_MODEL=                # model id; blank uses the provider's default
AI_BASE_URL=             # override the preset endpoint (required for custom)
AI_FREE=                 # true => free/un-metered model (skips credits); auto-true for ollama

SEARCH_PROVIDER=         # optional; enables the assistant's web-search tool
SEARCH_API_URL=
SEARCH_API_KEY=
```

The presets:

| `AI_PROVIDER` | Endpoint | Key | Notes |
|---|---|---|---|
| `openai` | api.openai.com | required | default `gpt-4o-mini` / `gpt-4o` |
| `openrouter` | openrouter.ai | required | one key fronts every vendor; switch models via `AI_MODEL` (e.g. `anthropic/claude-3.7-sonnet`, `deepseek/deepseek-chat`, `...:free`) |
| `groq` | api.groq.com | required (free tier) | fast; e.g. `AI_MODEL=openai/gpt-oss-20b` |
| `ollama` | localhost:11434 | none | free local model; `AI_FREE` defaults true |
| `anthropic` | api.anthropic.com | required | the Anthropic connector |
| `custom` | your `AI_BASE_URL` | as needed | any OpenAI-compatible endpoint (vLLM, LocalAI, LM Studio) |

Examples:

```
# OpenRouter: one key, switch models by changing AI_MODEL
AI_PROVIDER=openrouter
AI_API_KEY=sk-or-...
AI_MODEL=deepseek/deepseek-chat

# Free local model (Ollama): no key, un-metered
AI_PROVIDER=ollama
AI_MODEL=llama3.1        # or qwen2.5
```

A full env reference is `deploy/config/env.example`.

## Step 3: Build and push images

Tagging a release on GitHub triggers `release.yml`, which builds and pushes:

```
ghcr.io/<owner>/warmbly/worker:vX.Y.Z
ghcr.io/<owner>/warmbly/worker:vX.Y
ghcr.io/<owner>/warmbly/worker:vX
ghcr.io/<owner>/warmbly/worker:prod
```

And the same set of tags for `backend`, `consumer`, `tracking`, `realtime`, and the two frontends `web` (dashboard) and `admin`. The `build-push.yml` workflow on `main` publishes `:{sha}` and `:dev`.

For self-hosters: this works out of the box on any fork as long as you have `packages: write` permission enabled in repo settings.

## Step 4: Deploy the control plane

Pick one. The Dockerfiles in `deploy/docker/` are the deployment unit:

- **Railway** (what we use in production): one project per environment, one service per Dockerfile, env vars set in the Railway UI. Pushes to `main` auto-deploy.
- **ECS Fargate**: a task definition per service, ALB in front, secrets via AWS Secrets Manager. ~200 lines of Terraform.
- **Fly.io**: `fly launch` per service; nice if you want Elixir distributed clustering for realtime.
- **Single VPS**: each service as a systemd unit fronted by Caddy. Works for small deployments.

The control plane needs database migration to be run before the backend starts. Backend runs migrations automatically on boot (`internal/infrastructure/db/migrate.go`), so no separate migration step is required.

## Step 5: Set up the GitHub release webhook

If `RELEASES_ENABLED=true`:

1. In your GitHub repo → Settings → Webhooks → Add webhook
2. Payload URL: `https://api.example.com/webhooks/github/releases`
3. Content type: `application/json`
4. Secret: same value as `RELEASES_WEBHOOK_SECRET` on the backend
5. Events: select **Releases** only
6. Save

From now on, publishing a GitHub release fires the webhook and the backend resolves each profile's channel.

## Step 6: Add a worker

Two ways.

### From the dashboard (recommended)

1. Provision a VPS (Hetzner et al). Note its public IPv4 and root user.
2. In Warmbly admin → Workers → **Add Worker**. Fill in host, port, user, name. Pick a worker profile (or leave blank to use backend env defaults). Click Create.
3. Copy the generated SSH public key. Paste it into the VPS's `~/.ssh/authorized_keys`:
   ```bash
   ssh root@<vps> 'mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys'
   # paste the key, then Ctrl-D
   ```
4. In the dashboard → worker detail page → **Test connection**. Should succeed and pin the host fingerprint.
5. Click **Install**. Backend uploads `scripts/install-worker.sh`, runs it, and the worker is live.

### From the VPS itself

```bash
curl -fsSL https://get.example.com/worker | sudo bash -s -- \
  --kafka kafka.example.com:9092 \
  --schema-registry https://schema.example.com \
  --redis redis://cache.example.com:6379 \
  --aws-region us-east-1 --aws-key AKIA... --aws-secret ...
```

You can later "adopt" this worker into the dashboard by adding the SSH key to it and pointing a profile at it.

## Step 7: Wire up the worker profile

In the admin dashboard → Credentials → AWS Credentials → Add. Name it, region, access key, secret. The secret is encrypted via KMS before storage.

Then → Worker Profiles → Add. Pick the AWS Credentials row. Fill in Kafka bootstrap, SASL username/password, Schema Registry URL/key/secret, Redis URL. Pick a release channel (`pinned`, `stable`, `dev`) and whether `auto_update` is on.

Assign workers to this profile on each worker's detail page. Click Apply to re-write `worker.env` on the VPS and restart.

## Day-2 operations

Everything happens from the worker detail page:

| Action | Effect |
|--------|--------|
| Test connection | Open an SSH session, run `true` |
| Install | Upload installer + env, run it. Idempotent. |
| Restart | `systemctl restart warmbly-worker.service` |
| Pull latest & restart | Pull the new image, regenerate unit, restart |
| Apply config & restart | Rewrite `/etc/warmbly/worker.env`, restart (no docker pull) |
| Update OS packages | apt/dnf/pacman/apk upgrade noninteractively |
| Reboot VPS | `shutdown -r +1` |
| Rotate SSH keys | Generate new ed25519 keypair, install on target |
| Uninstall | Stop + remove systemd unit and container |
| Delete | Uninstall + remove row from `workers` table |

When `auto_update=true` is set on a profile and a new release fires the webhook, every assigned worker gets the new image automatically. Otherwise the dashboard shows an "update available" hint and waits for you to click Apply.

## Releasing a new version

```bash
git tag -a v1.2.4 -m "Release v1.2.4"
git push origin v1.2.4
```

`release.yml` builds and pushes images. If a release webhook is configured, the backend picks up the new version, updates any profile on `stable` or `dev` channels, and rolls profiles with `auto_update=true`. Otherwise the dashboard shows the new version and you Apply manually.

## Rolling back

- **Per-profile**: edit the profile, switch channel to `pinned`, set `worker_image` to a specific older tag (e.g. `ghcr.io/.../worker:v1.2.3`), save, then Apply to all workers on that profile.
- **Per-worker**: same idea, but assign the worker to a different profile (or detach and run the installer manually with the old image).

## Troubleshooting

**Worker shows `install_state: error`**
Open the worker detail page, click Test. If that fails, the SSH key isn't in `authorized_keys` on the VPS. If Test passes but Install fails, check the `last_error` shown on the detail page and the output captured under Logs.

**Auto-update didn't fire**
Check GitHub repo → Settings → Webhooks → Recent Deliveries. If the webhook delivery shows 401, the secret is wrong. If it shows 200 but nothing happened, check `Releases panel → Last error` in the dashboard, or `journalctl -u backend` for the boot check.

**Worker stuck on old image**
"Apply config & restart" only rewrites env, not image. Use "Pull latest & restart" instead, or the release auto-update path.

**Heartbeat shows offline**
- Worker can reach Kafka? (firewall, SASL creds)
- Worker container actually running? (worker detail → Live status)
- VPS's clock skewed? (Kafka SASL auth is time-sensitive)

## Quick reference

```bash
# control plane: locally
make infra      # postgres / redis / kafka / etc. (leave running)
make app        # backend / consumer / worker / tracking / realtime / web

# load fixtures
make seed

# full simulation with extra workers + localstack + stripe-mock
make sim

# tear down everything
make reset

# manual worker install on a VPS
curl -fsSL https://your-host/install-worker.sh | sudo bash -s -- --help

# trigger a release manually (build+push happens via GH Actions)
git tag -a v1.2.4 -m "Release v1.2.4" && git push origin v1.2.4
```