warmbly/docs/content/docs/development/architecture.mdx

---
title: Architecture
description: How Warmbly's control plane and execution plane fit together, with the encryption and worker model.
---

Warmbly is split into two planes:

- **Control plane**: backend API, consumer, tracking, realtime, web. Runs in one region on a container host (Railway in production).
- **Execution plane**: a fleet of worker processes, one per VPS, spread across many providers and IPs. Each VPS runs the worker as a systemd-supervised Docker container.

The boundary exists for two reasons. First, cold-mail deliverability lives at the IP level, so workers must be spread across distinct machine-level network identities. Second, the control plane owns relational state and the worker fleet should remain disposable.

<Mermaid
  chart={`
flowchart LR
  subgraph CP["Control plane"]
    API["Backend"] --> PG[("Postgres")]
    API --> BUS{{"Event bus"}}
    BUS --> CONS["Consumer"] --> PG
    TRK["Tracking"] --> BUS
    API -. "Redis pub/sub" .-> RT["Realtime"]
  end
  subgraph EP["Execution plane"]
    W1["Worker VPS"]
    W2["Worker VPS"]
  end
  BUS <--> W1
  BUS <--> W2
  W1 --> P["Gmail · Microsoft · SMTP"]
  W2 --> P
`}
/>

## Services

| Service | Language | Plane | Notes |
|---------|----------|-------|-------|
| Backend | Go (Gin) | Control | REST API, auth, business logic, worker orchestration |
| Consumer | Go | Control | Event bus processor → Postgres |
| Tracking | Rust (Axum) | Control | Open/click pixels and redirects → event bus |
| Forms | Go (Gin) + React (TanStack) | Control | Hosted form pages (`forms/` app), embeds and public submissions → backend internal API |
| Realtime | Elixir (Phoenix) | Control | WebSocket fanout |
| Worker | Go | Execution | One per VPS; subscribes to a per-worker event bus topic; never opens a Postgres connection |
| Web | React (Vite) | n/a | Dashboard frontend |
| Admin | React (Vite) | n/a | [Platform admin panel](/development/admin-panel/) |

## Data flow

The frontend talks to the backend over REST + JWT, and to the realtime service over WebSocket. Backend writes business state to Postgres. Backend, tracking, and workers publish to the event bus: NATS JetStream with JSON encoding by default, Kafka with Avro and Schema Registry as an opt-in build. The consumer reads the bus and updates Postgres (analytics, suppression, deliverability). Workers subscribe to a topic named for their worker UUID (`w.<uuid>`) and publish results to `jobs.worker-events`; see the [event system](/development/events/) for the full topic map.

Realtime fanout is a separate channel: backend and consumer publish JSON events over Redis pub/sub (or Google Cloud Pub/Sub when `PUBSUB_ENABLED=true`); the Elixir realtime service subscribes and pushes to connected WebSocket clients.

Object storage: encrypted email bodies (EMSG format) live in the blob store (filesystem by default, S3-compatible opt-in).

## Data stores

| Store | Purpose |
|-------|---------|
| Postgres | Users, organizations, campaigns, mailboxes, workers, credentials, warmup state, per-organization encrypted DEKs, message-ID maps, Gmail history IDs |
| Redis | Caching (including decrypted DEKs), rate limiting, realtime bridge, ephemeral state |
| Blob store | Email body blobs (EMSG); filesystem or any S3-compatible bucket |
| Master key | Root of trust for envelope encryption; local AES key or AWS KMS |

## Encryption model

Warmbly uses envelope encryption end-to-end for sensitive data.

KMS holds the master key. Each organization gets a 32-byte data encryption key (DEK), generated by KMS and stored encrypted in the `organization_encrypted_keys` Postgres table keyed by organization ID (workers reach it over the backend's internal API rather than touching Postgres). The DEK is decrypted only at the moment of use; cached in Redis with a TTL to amortize cost.

Application-layer secrets are sealed with AES-256-GCM under the DEK and base64-encoded. This applies to:

- email account credentials (IMAP/SMTP passwords, OAuth tokens)
- email body content stored in S3 (EMSG format)
- **worker SSH private keys** (since admins drive workers over SSH)
- **AWS credential rows and worker-profile secrets** (Kafka SASL passwords, Schema Registry secrets, Redis URLs) used to configure remote workers

Worker-related secrets are encrypted under a platform DEK (key ID = `uuid.Nil`): the same envelope as organization secrets, the same trust boundary, a different identity. See `internal/app/cipher/` and `internal/app/worker_orchestrator/orchestrator.go`.

## Worker model

Workers are added and managed from the admin dashboard. The flow:

1. Admin fills out host/port/user. Backend generates an ed25519 keypair, encrypts the private key, stores the row in `pending` state.
2. Admin pastes the generated public key into the VPS's `~/.ssh/authorized_keys`.
3. Admin clicks Test. Backend opens an SSH session and runs `true`. First success pins the host SHA256 fingerprint (trust-on-first-use).
4. Admin clicks Install. Backend uploads `scripts/install-worker.sh` and a per-worker env file, runs the installer, which installs Docker if missing, generates a deterministic UUID from the VPS's public IPv4 (UUIDv5, URL namespace), writes a systemd unit, and starts the worker container with `--hostname <uuid>`.
5. The worker reads its identity from `os.Hostname()`, subscribes to the event bus topic named for that UUID, and heartbeats to the backend's internal API every 90 seconds.

From then on, every lifecycle operation (restart, update image, uninstall, rotate keys, system updates, reboot, tail logs, fetch live status) happens via SSH from the dashboard.

### Why per-VPS instead of a Kubernetes DaemonSet

A k8s `DaemonSet` was the previous shape. It was removed because k8s nodes typically NAT all pods through a small set of egress IPs, and pods churn while IPs do not. What a mailbox provider remembers is the address an account signs in from, so the unit of identity has to be the IP, not the pod: a mailbox whose client address changes every deploy collects sign-in risk challenges for no reason. The worker also has no Postgres dependency, so cluster-level service discovery and RBAC buy us nothing.

### Identity from IP

Worker UUID is `UUIDv5(URL_namespace, public_ipv4)`. Properties:

- same IP → same worker (reputation persists across reinstalls)
- new IP → new worker (fresh identity, no inherited history)
- deterministic, no state needed at the control plane to recover it

## Worker placement

There is one kind of worker. You install it, it heartbeats, and the control plane decides what runs on it. Workers carry no tier, type, risk pool or egress category, and nothing about a mailbox has to match anything about a machine.

That follows from where the sending identity actually lives. A worker never talks to a recipient's MX. It authenticates to the customer's own mailbox provider (Gmail, Microsoft Graph, or a customer SMTP host) and that provider delivers the message from its own outbound pool. Two consequences shape the whole model:

- **The worker's IP is invisible to recipient spam filtering.** Google strips the submitting client's IP from outgoing messages and Microsoft dropped `X-Originating-IP` years ago. So co-locating a spam-prone mailbox next to a healthy one cannot contaminate the healthy one's sending reputation, and segregating workers by customer tier buys nothing.
- **The worker's IP is very visible to the mailbox provider.** It is what drives sign-in risk challenges, per-IP authentication throttles (`454 4.7.0`) and per-IP rate limits (`421 4.7.28`).

So the levers invert: IP *stability* per mailbox beats IP diversity, and a migration is a cost rather than a win.

### Choosing a worker

Placement scores every live worker and takes the best (`internal/app/worker/placement.go`). Hard constraints are only about whether the work can be done at all: the worker has to be heartbeating, in `healthy` or `watch`, and have capacity headroom for the mailbox's weight. Everything else is a preference:

| Term | Why |
|---|---|
| Capacity headroom | Fill the fleet evenly |
| Incumbency | Staying put keeps one client IP in front of the provider. Weighted highest |
| Region match | Sign-ins from where the provider expects them raise fewer challenges |
| Tenant blast radius | Spread one customer across workers so a single failure does not stop their sending |
| Provider crowding | Many accounts of one provider signing in from one address is what earns a per-IP throttle |
| Foreign tenants | Only for organizations entitled to isolated egress |

Capacity is one number for every worker, in cold-mailbox equivalents, because each mailbox already declares its own cost: an `smtp_imap` mailbox weighs `1.0`, a Gmail or Outlook API mailbox `0.05`, and a warmup-only assignment `0.4`.

### Moving a mailbox

Rotation (`internal/app/worker/rotation.go`) is deliberately reluctant, and a fleet where nothing moves is a healthy one. Only mailboxes whose worker is dead, degraded, over capacity, or on the wrong side of an isolated-egress reservation are considered at all:

| Urgency | Trigger | Residency floor | Destination bar |
|---|---|---|---|
| Immediate | Worker inactive, not heartbeating, blocked or quarantined | none | anything eligible |
| Elevated | Worker throttled | 6 hours | anything eligible |
| Opportunistic | Worker over 85% utilization, or isolated-egress drift | 72 hours | must score materially better |

### Isolated egress

Plans that reserve egress bind an organization to a worker (`dedicated_worker_assignments`). It is a strong placement preference, not a pin: the worker carries no marking, so if it dies the organization's mailboxes place normally instead of stranding, and the rotation loop pulls them onto a replacement once one exists. Mailboxes belonging to other tenants drift off a reserved worker on the same loop.

## Credentials and profiles

Workers never carry hardcoded credentials. Two reusable entities in the admin dashboard:

- **AWS credentials**: a named keypair; the secret access key is stored as ciphertext from the cipher service.
- **Worker profile**: a named bundle of event bus settings (NATS URL or Kafka bootstrap + SASL + Schema Registry), Redis URL, backend internal API URL and token, worker image, and release channel, with an optional foreign key to one AWS credentials row.

Workers reference one profile. When the admin edits the profile, the backend marks any assigned worker whose `config_applied_at` is older than `profile.updated_at` as having stale config. The dashboard shows a warning; one click rewrites `/etc/warmbly/worker.env` over SSH and restarts the worker.

Schema: the `workers`, `worker_profiles`, and `aws_credentials` tables in `internal/infrastructure/db/migrations/000001_baseline.up.sql`.

## Auto-update from GitHub releases

Each profile picks a release channel:

- `pinned`: admin sets the image tag manually
- `stable`: latest non-prerelease GitHub Release
- `dev`: latest release (including prereleases)

Trigger model is push-driven, not poll-driven:

- One-shot check on backend boot, populates the dashboard.
- GitHub webhook (`POST /webhooks/github/releases`, HMAC-validated) on every release event.
- Admin "Check now" button as a manual fallback.

When auto_update is on and a new tag is resolved, the backend records the new image on the profile, then rolls each assigned worker by SSHing in, re-running the installer with `--update --image <new>`, which regenerates the systemd unit, pulls the image, and restarts. Workers' running version is recorded in `workers.image_version` so the dashboard can show a `v1.2.3 → v1.2.4` diff.

All configuration is env-driven (`RELEASES_GITHUB_REPO`, `RELEASES_WORKER_IMAGE_REPO`, `RELEASES_WEBHOOK_SECRET`, `RELEASES_ENABLED`) so self-hosters can point at their own fork and registry.

See `internal/app/releases/service.go`.

## Worker safety policy

Cold-email throughput is mailbox-first, not worker-first. A worker's safe outbound volume is the sum of its assigned mailbox budgets, not a flat global cap. Defaults in `internal/config/constants.go`:

- default cold campaign cap per mailbox: 50/day
- default minimum gap per mailbox: 600s
- default warmup start per mailbox: 10/day
- default warmup ceiling per mailbox: 40/day
- default warmup ramp: +1/day

Pool model: warmup traffic is segregated into `free` and `premium` pools (the `warmup_pools` tables in `internal/infrastructure/db/migrations/000001_baseline.up.sql`). Pool membership is a property of the mailbox, independent of which worker happens to be sending for it.

Warmup content generation is an offline control-plane workload. The autonomous controller runs every six hours, derives a shared-bank target from seven-day send demand, maintains a 200-thread floor, scales up to 5,000 threads, and submits at most 250 replacements per batch within a 1,000-thread daily cap. It deliberately does not split generation by customer mailbox tags, which would create unbounded queues and smaller, more repetitive content cohorts. Scheduled top-ups use `gpt-5-mini` through durable provider batch jobs rather than synchronous model calls, with a database uniqueness guard that prevents duplicate in-flight scheduled batches when several backend replicas are running. Content with at least 20 sampled sends is automatically archived when it has at least 3 spam placements and a placement rate of 15% or more. The send path atomically selects and increments a least-used active conversation through an indexed query, then falls back to the static reviewed library if no generated content is available. Model availability therefore never gates a warmup send.

## Anti-abuse layers

There is no single ML fraud engine. Layered controls instead:

- CAPTCHA (Cloudflare Turnstile) on auth-sensitive flows
- Per-user, per-category API rate limiting (Redis-backed)
- Per-WebSocket join/message/event rate limiting (realtime service)
- Warmup-token verification with invalid-attempt counting and auto-blocking
- Tracking event deduplication (in-memory + persistent)
- Idempotent deliverability event processing
- Worker-side mailbox-sync fair use: the sync governor (`internal/app/worker/wmail/governor.go`) with priority, live and backfill lanes, deferral instead of dropping, and flood or chronic-overage escalation
- Suppression lists for bounced/complained/unsubscribed recipients
- Admin ban + manual override surface

See the [event system reference](/development/events/) for the Kafka events, and the codebase's `internal/app/consumer/` for the event handlers.

## Source anchors

These files are the fastest way to rebuild context:

- `README.md`
- `docs/content/docs/development/deployment-guide.mdx`
- `cmd/worker/main.go`
- `internal/app/worker/assignment.go`
- `internal/app/worker_orchestrator/orchestrator.go`
- `internal/app/releases/service.go`
- `internal/app/cipher/cipher.go`
- `internal/tasks/email_task.go`
- `internal/repository/pg_worker.go`
- `internal/repository/pg_worker_ssh.go`
- `internal/repository/pg_credentials.go`
- `internal/repository/pg_warmup.go`