warmbly/resources/architecture.md

Architecture

Warmbly is split into two planes:

• Control plane — backend API, consumer, tracking, realtime, web. Runs in one region on a container host (Railway in production).
• Execution plane — a fleet of worker processes, one per VPS, spread across many providers and IPs. Each VPS runs the worker as a systemd-supervised Docker container.

The boundary exists for two reasons. First, cold-mail deliverability lives at the IP level, so workers must be spread across distinct machine-level network identities. Second, the control plane owns relational state and the worker fleet should remain disposable.

Services

Service	Language	Plane	Notes
Backend	Go (Gin)	Control	REST API, auth, business logic, worker orchestration
Consumer	Go	Control	Kafka event processor → Postgres
Tracking	Rust (Axum)	Control	Open/click pixels and redirects → Kafka
Realtime	Elixir (Phoenix)	Control	WebSocket fanout
Worker	Go	Execution	One per VPS; subscribes to a per-worker Kafka topic; never opens a Postgres connection
Web	React (Vite)	–	Frontend

Data flow

The frontend talks to the backend over REST + JWT, and to the realtime service over WebSocket. Backend writes business state to Postgres. Backend, tracking, and consumer all publish to Kafka with Avro encoding (Schema Registry). The consumer reads Kafka events and updates Postgres (analytics, suppression, deliverability). Workers subscribe to a Kafka topic named for their worker UUID and publish results back to Kafka.

Realtime fanout: backend publishes events to a pub/sub fanout; the Elixir realtime service subscribes and pushes to connected WebSocket clients.

Object storage: encrypted email bodies (EMSG format — see EMSG.md) live in S3.

Data stores

Store	Purpose
Postgres	Users, organizations, campaigns, mailboxes, workers, credentials, warmup state
Redis	Caching (including decrypted DEKs), rate limiting, ephemeral state
DynamoDB	Per-user encrypted DEKs, message-ID maps, Gmail history IDs
S3	Email body blobs (EMSG)
KMS	Root of trust for envelope encryption

Encryption model

Warmbly uses envelope encryption end-to-end for sensitive data.

KMS holds the master key. Each user gets a 32-byte data encryption key (DEK), generated by KMS and stored encrypted in DynamoDB keyed by user ID. The DEK is decrypted only at the moment of use; cached in Redis with a TTL to amortize cost.

Application-layer secrets are sealed with AES-256-GCM under the DEK and base64-encoded. This applies to:

• email account credentials (IMAP/SMTP passwords, OAuth tokens)
• email body content stored in S3 (EMSG format)
• worker SSH private keys (since admins drive workers over SSH)
• AWS credential rows and worker-profile secrets (Kafka SASL passwords, Schema Registry secrets, Redis URLs) used to configure remote workers

Worker-related secrets are encrypted under a platform DEK (user ID = uuid.Nil). Same envelope as user secrets — same trust boundary, different identity. See internal/app/cipher/ and internal/app/worker_orchestrator/orchestrator.go.

Worker model

Workers are added and managed from the admin dashboard. The flow:

1. Admin fills out host/port/user. Backend generates an ed25519 keypair, encrypts the private key, stores the row in pending state.
2. Admin pastes the generated public key into the VPS's ~/.ssh/authorized_keys.
3. Admin clicks Test. Backend opens an SSH session and runs true. First success pins the host SHA256 fingerprint (trust-on-first-use).
4. Admin clicks Install. Backend uploads scripts/install-worker.sh and a per-worker env file, runs the installer, which installs Docker if missing, generates a deterministic UUID from the VPS's public IPv4 (UUIDv5, URL namespace), writes a systemd unit, and starts the worker container with --hostname <uuid>.
5. The worker reads its identity from os.Hostname(), subscribes to a Kafka topic named for that UUID, sends a periodic heartbeat.

From then on, every lifecycle operation — restart, update image, uninstall, rotate keys, system updates, reboot, tail logs, fetch live status — happens via SSH from the dashboard.

Why per-VPS instead of a Kubernetes DaemonSet

A k8s DaemonSet was the previous shape. It was removed because k8s nodes typically NAT all pods through a small set of egress IPs, defeating the IP-diversity goal. Pods churn but IPs accumulate reputation, so the unit of identity needs to be the IP, not the pod. The worker also has no Postgres dependency, so cluster-level service discovery and RBAC buy us nothing.

Identity from IP

Worker UUID is UUIDv5(URL_namespace, public_ipv4). Properties:

• same IP → same worker (reputation persists across reinstalls)
• new IP → new worker (fresh identity, no inherited history)
• deterministic, no state needed at the control plane to recover it

Credentials and profiles

Workers never carry hardcoded credentials. Two reusable entities in the admin dashboard:

• AWS credentials — named keypair; the secret access key is stored as ciphertext from the cipher service.
• Worker profile — a named bundle of Kafka bootstrap + SASL + Schema Registry + Redis URL + worker image + release channel, with a foreign key to one AWS credentials row.

Workers reference one profile. When the admin edits the profile, the backend marks any assigned worker whose config_applied_at is older than profile.updated_at as having stale config. The dashboard shows a warning; one click rewrites /etc/warmbly/worker.env over SSH and restarts the worker.

Migrations: internal/infrastructure/db/migrations/000028_worker_ssh.up.sql, 000029_worker_credentials.up.sql, 000030_worker_releases.up.sql.

Auto-update from GitHub Releases

Each profile picks a release channel:

• pinned — admin sets the image tag manually
• stable — latest non-prerelease GitHub Release
• dev — latest release (including prereleases)

Trigger model is push-driven, not poll-driven:

• One-shot check on backend boot, populates the dashboard.
• GitHub webhook (POST /webhooks/github/releases, HMAC-validated) on every release event.
• Admin "Check now" button as a manual fallback.

When auto_update is on and a new tag is resolved, the backend records the new image on the profile, then rolls each assigned worker by SSHing in, re-running the installer with --update --image <new>, which regenerates the systemd unit, pulls the image, and restarts. Workers' running version is recorded in workers.image_version so the dashboard can show a v1.2.3 → v1.2.4 diff.

All configuration is env-driven (RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO, RELEASES_WEBHOOK_SECRET, RELEASES_ENABLED) so self-hosters can point at their own fork and registry.

See internal/app/releases/service.go.

Worker safety policy

Cold-email throughput is mailbox-first, not worker-first. A worker's safe outbound volume is the sum of its assigned mailbox budgets, not a flat global cap. Defaults in internal/config/constants.go:

• default cold campaign cap per mailbox: 50/day
• default minimum gap per mailbox: 600s
• default warmup start per mailbox: 10/day
• default warmup ceiling per mailbox: 40/day
• default warmup ramp: +1/day

Pool model: warmup traffic is segregated into free and premium pools (internal/infrastructure/db/migrations/000010_warmup_pools.up.sql). Dedicated-worker customers still pick a pool explicitly; tier and pool are orthogonal.

Anti-abuse layers

There is no single ML fraud engine. Layered controls instead:

• CAPTCHA (Cloudflare Turnstile) on auth-sensitive flows
• Per-user, per-category API rate limiting (Redis-backed)
• Per-WebSocket join/message/event rate limiting (realtime service)
• Warmup-token verification with invalid-attempt counting and auto-blocking
• Tracking event deduplication (in-memory + persistent)
• Idempotent deliverability event processing
• Worker-side mailbox-sync rate limiting (internal/app/worker/wmail/ratelimit.go)
• Suppression lists for bounced/complained/unsubscribed recipients
• Admin ban + manual override surface

See Events.md for the Kafka event reference, and the codebase's internal/app/consumer/ for the event handlers.

Source anchors

These files are the fastest way to rebuild context:

• README.md
• resources/deployment-guide.md
• cmd/worker/main.go
• internal/app/worker/assignment.go
• internal/app/worker_orchestrator/orchestrator.go
• internal/app/releases/service.go
• internal/app/cipher/cipher.go
• internal/tasks/email_task.go
• internal/repository/pg_worker.go
• internal/repository/pg_worker_ssh.go
• internal/repository/pg_credentials.go
• internal/repository/pg_warmup.go