---
title: Architecture
description: How Warmbly's control plane and execution plane fit together, with the encryption and worker model.
---

Warmbly is split into two planes:

- **Control plane**: backend API, consumer, tracking, realtime, web. Runs in one region on a container host (Railway in production).
- **Execution plane**: a fleet of worker processes, one per VPS, spread across many providers and IPs. Each VPS runs the worker as a systemd-supervised Docker container.

The boundary exists for two reasons. First, cold-mail deliverability lives at the IP level, so workers must be spread across distinct machine-level network identities. Second, the control plane owns relational state and the worker fleet should remain disposable.

<Mermaid
  chart={`
flowchart LR
  subgraph CP["Control plane"]
    API["Backend"] --> PG[("Postgres")]
    API --> BUS{{"Event bus"}}
    BUS --> CONS["Consumer"] --> PG
    TRK["Tracking"] --> BUS
    API -. "Redis pub/sub" .-> RT["Realtime"]
  end
  subgraph EP["Execution plane"]
    W1["Worker VPS"]
    W2["Worker VPS"]
  end
  BUS <--> W1
  BUS <--> W2
  W1 --> P["Gmail · Microsoft · SMTP"]
  W2 --> P
`}
/>

## Services

| Service | Language | Plane | Notes |
|---------|----------|-------|-------|
| Backend | Go (Gin) | Control | REST API, auth, business logic, worker orchestration |
| Consumer | Go | Control | Event bus processor → Postgres |
| Tracking | Rust (Axum) | Control | Open/click pixels and redirects → event bus |
| Forms | Go (Gin) + React (TanStack) | Control | Hosted form pages (`forms/` app), embeds and public submissions → backend internal API |
| Realtime | Elixir (Phoenix) | Control | WebSocket fanout |
| Worker | Go | Execution | One per VPS; subscribes to a per-worker event bus topic; never opens a Postgres connection |
| Web | React (Vite) | n/a | Dashboard frontend |
| Admin | React (Vite) | n/a | Platform admin panel |

## Data flow

The frontend talks to the backend over REST + JWT, and to the realtime service over WebSocket. Backend writes business state to Postgres. Backend, tracking, and workers publish to the event bus: NATS JetStream with JSON encoding by default, Kafka with Avro and Schema Registry as an opt-in build. The consumer reads the bus and updates Postgres (analytics, suppression, deliverability). Workers subscribe to a topic named for their worker UUID (`w.<uuid>`) and publish results to `jobs.worker-events`; see the [event system](/development/events/) for the full topic map.

Realtime fanout is a separate channel: backend and consumer publish JSON events over Redis pub/sub (or Google Cloud Pub/Sub when `PUBSUB_ENABLED=true`); the Elixir realtime service subscribes and pushes to connected WebSocket clients.

Object storage: encrypted email bodies (EMSG format) live in the blob store (filesystem by default, S3-compatible opt-in).

## Data stores

| Store | Purpose |
|-------|---------|
| Postgres | Users, organizations, campaigns, mailboxes, workers, credentials, warmup state, per-organization encrypted DEKs, message-ID maps, Gmail history IDs |
| Redis | Caching (including decrypted DEKs), rate limiting, realtime bridge, ephemeral state |
| Blob store | Email body blobs (EMSG); filesystem or any S3-compatible bucket |
| Master key | Root of trust for envelope encryption; local AES key or AWS KMS |

## Encryption model

Warmbly uses envelope encryption end-to-end for sensitive data.

KMS holds the master key. Each organization gets a 32-byte data encryption key (DEK), generated by KMS and stored encrypted in the `organization_encrypted_keys` Postgres table keyed by organization ID (workers reach it over the backend's internal API rather than touching Postgres). The DEK is decrypted only at the moment of use; cached in Redis with a TTL to amortize cost.

Application-layer secrets are sealed with AES-256-GCM under the DEK and base64-encoded. This applies to:

- email account credentials (IMAP/SMTP passwords, OAuth tokens)
- email body content stored in S3 (EMSG format)
- **worker SSH private keys** (since admins drive workers over SSH)
- **AWS credential rows and worker-profile secrets** (Kafka SASL passwords, Schema Registry secrets, Redis URLs) used to configure remote workers

Worker-related secrets are encrypted under a platform DEK (key ID = `uuid.Nil`): the same envelope as organization secrets, the same trust boundary, a different identity. See `internal/app/cipher/` and `internal/app/worker_orchestrator/orchestrator.go`.

## Worker model

Workers are added and managed from the admin dashboard. The flow:

1. Admin fills out host/port/user. Backend generates an ed25519 keypair, encrypts the private key, stores the row in `pending` state.
2. Admin pastes the generated public key into the VPS's `~/.ssh/authorized_keys`.
3. Admin clicks Test. Backend opens an SSH session and runs `true`. First success pins the host SHA256 fingerprint (trust-on-first-use).
4. Admin clicks Install. Backend uploads `scripts/install-worker.sh` and a per-worker env file, runs the installer, which installs Docker if missing, generates a deterministic UUID from the VPS's public IPv4 (UUIDv5, URL namespace), writes a systemd unit, and starts the worker container with `--hostname <uuid>`.
5. The worker reads its identity from `os.Hostname()`, subscribes to the event bus topic named for that UUID, and heartbeats to the backend's internal API every 90 seconds.

From then on, every lifecycle operation (restart, update image, uninstall, rotate keys, system updates, reboot, tail logs, fetch live status) happens via SSH from the dashboard.

### Why per-VPS instead of a Kubernetes DaemonSet

A k8s `DaemonSet` was the previous shape. It was removed because k8s nodes typically NAT all pods through a small set of egress IPs, defeating the IP-diversity goal. Pods churn but IPs accumulate reputation, so the unit of identity needs to be the IP, not the pod. The worker also has no Postgres dependency, so cluster-level service discovery and RBAC buy us nothing.

### Identity from IP

Worker UUID is `UUIDv5(URL_namespace, public_ipv4)`. Properties:

- same IP → same worker (reputation persists across reinstalls)
- new IP → new worker (fresh identity, no inherited history)
- deterministic, no state needed at the control plane to recover it

## Credentials and profiles

Workers never carry hardcoded credentials. Two reusable entities in the admin dashboard:

- **AWS credentials**: a named keypair; the secret access key is stored as ciphertext from the cipher service.
- **Worker profile**: a named bundle of event bus settings (NATS URL or Kafka bootstrap + SASL + Schema Registry), Redis URL, backend internal API URL and token, worker image, and release channel, with an optional foreign key to one AWS credentials row.

Workers reference one profile. When the admin edits the profile, the backend marks any assigned worker whose `config_applied_at` is older than `profile.updated_at` as having stale config. The dashboard shows a warning; one click rewrites `/etc/warmbly/worker.env` over SSH and restarts the worker.

Schema: the `workers`, `worker_profiles`, and `aws_credentials` tables in `internal/infrastructure/db/migrations/000001_baseline.up.sql`.

## Auto-update from GitHub releases

Each profile picks a release channel:

- `pinned`: admin sets the image tag manually
- `stable`: latest non-prerelease GitHub Release
- `dev`: latest release (including prereleases)

Trigger model is push-driven, not poll-driven:

- One-shot check on backend boot, populates the dashboard.
- GitHub webhook (`POST /webhooks/github/releases`, HMAC-validated) on every release event.
- Admin "Check now" button as a manual fallback.

When auto_update is on and a new tag is resolved, the backend records the new image on the profile, then rolls each assigned worker by SSHing in, re-running the installer with `--update --image <new>`, which regenerates the systemd unit, pulls the image, and restarts. Workers' running version is recorded in `workers.image_version` so the dashboard can show a `v1.2.3 → v1.2.4` diff.

All configuration is env-driven (`RELEASES_GITHUB_REPO`, `RELEASES_WORKER_IMAGE_REPO`, `RELEASES_WEBHOOK_SECRET`, `RELEASES_ENABLED`) so self-hosters can point at their own fork and registry.

See `internal/app/releases/service.go`.

## Worker safety policy

Cold-email throughput is mailbox-first, not worker-first. A worker's safe outbound volume is the sum of its assigned mailbox budgets, not a flat global cap. Defaults in `internal/config/constants.go`:

- default cold campaign cap per mailbox: 50/day
- default minimum gap per mailbox: 600s
- default warmup start per mailbox: 10/day
- default warmup ceiling per mailbox: 40/day
- default warmup ramp: +1/day

Pool model: warmup traffic is segregated into `free` and `premium` pools (the `warmup_pools` tables in `internal/infrastructure/db/migrations/000001_baseline.up.sql`). Dedicated-worker customers still pick a pool explicitly; tier and pool are orthogonal.

Warmup content generation is an offline control-plane workload. The autonomous controller runs every six hours, derives a shared-bank target from seven-day send demand, maintains a 200-thread floor, scales up to 5,000 threads, and submits at most 250 replacements per batch within a 1,000-thread daily cap. It deliberately does not split generation by customer mailbox tags, which would create unbounded queues and smaller, more repetitive content cohorts. Scheduled top-ups use `gpt-5-mini` through durable provider batch jobs rather than synchronous model calls, with a database uniqueness guard that prevents duplicate in-flight scheduled batches when several backend replicas are running. Content with at least 20 sampled sends is automatically archived when it has at least 3 spam placements and a placement rate of 15% or more. The send path atomically selects and increments a least-used active conversation through an indexed query, then falls back to the static reviewed library if no generated content is available. Model availability therefore never gates a warmup send.

## Anti-abuse layers

There is no single ML fraud engine. Layered controls instead:

- CAPTCHA (Cloudflare Turnstile) on auth-sensitive flows
- Per-user, per-category API rate limiting (Redis-backed)
- Per-WebSocket join/message/event rate limiting (realtime service)
- Warmup-token verification with invalid-attempt counting and auto-blocking
- Tracking event deduplication (in-memory + persistent)
- Idempotent deliverability event processing
- Worker-side mailbox-sync fair use: the sync governor (`internal/app/worker/wmail/governor.go`) with priority, live and backfill lanes, deferral instead of dropping, and flood or chronic-overage escalation
- Suppression lists for bounced/complained/unsubscribed recipients
- Admin ban + manual override surface

See the [event system reference](/development/events/) for the Kafka events, and the codebase's `internal/app/consumer/` for the event handlers.

## Source anchors

These files are the fastest way to rebuild context:

- `README.md`
- `docs/content/docs/development/deployment-guide.mdx`
- `cmd/worker/main.go`
- `internal/app/worker/assignment.go`
- `internal/app/worker_orchestrator/orchestrator.go`
- `internal/app/releases/service.go`
- `internal/app/cipher/cipher.go`
- `internal/tasks/email_task.go`
- `internal/repository/pg_worker.go`
- `internal/repository/pg_worker_ssh.go`
- `internal/repository/pg_credentials.go`
- `internal/repository/pg_warmup.go`
