Matthew Meszaros
a7518a8558
* feat: rewrite the self-hosting docs against repo ground truth: turn the deployment guide into a full self-host guide (quick start with first-admin bootstrap via make grant-admin, .env secrets with exact key formats, PUBLIC_HOST derivation and HTTPS reverse-proxy vars, provider switches with build-tag caveats, mailbox OAuth, remote worker enrollment via SSH or wmenroll tokens, real CI image tags, upgrades and backups), rewrite the events page around the real NATS/Kafka bus topics and {type,body} envelopes, fix Kafka-era and make-target claims in architecture/local-development/deploy README, add API_PUBLIC_URL and drop the dead LOG_DISCORD_WEBHOOK_URL in env.example, and remove the docker-compose.kafka.yml comment pointing at a file that does not exist
* feat: make the self-hosting docs visual and skimmable by adding a Mermaid MDX component (client-rendered, theme-aware) to the docs site, condensing the self-host guide around a control-plane topology diagram, a worker enrollment sequence diagram, a dashboard screenshot, and symptom/check troubleshooting + optional-subsystem tables, and adding an execution-plane flowchart to the architecture page
* feat: stop the docs root flashing a 'Continue to the Warmbly docs' link before redirecting by navigating with an inline location.replace that runs during HTML parse, and demoting the visible link and meta refresh to no-JS fallbacks inside noscript
* feat: cut docs bulk and duplication by deleting three orphaned API pages that were stale forks of the reference section and were unreachable from the sidebar (porting their unique social sign-in, promo-code, and referral endpoints into api/reference/account-org.mdx as compact tables), condensing the deliverability and warmup guides to roughly half their length around tables instead of prose, replacing prose em dashes across the guides and MCP pages, and adding the required trailing slashes to internal links in 24 files
* feat: condense the sequences guide by about 40 percent, folding the switch-step deciders and branch conditions into tables and cutting restated prose while keeping every rule about threading, instant branches, reply matching, and stop on reply
* feat: condense the automations, unibox, advisor, and expressions guides by roughly 40 percent each, folding trigger lists, action catalogs, sending controls, and advisor checks into tables, adding a trigger-condition-action flow diagram to automations, and cutting restated prose while preserving every threshold, permission boundary, and rule
* feat: condense the mailboxes, campaigns, analytics, and team-roles guides by roughly 45 percent each, replacing prose walks through providers, rotation modes, lead statuses, counting rules, A/B confidence, and the permission matrix with compact tables and collapsing the four-way role grid into one capability table plus a one-line mapping
* feat: condense the AI-steps, security, and contacts-CRM guides by roughly 40 percent, turning sign-in methods, AI step modes, switch deciders, credit and failure behavior, import field mappings, and deal views into tables while keeping every safety boundary and dedupe rule
* feat: condense the meetings, notifications, AI-credits, and AI-assistant guides by roughly 40 percent, merging notification categories and their defaults into one table, collapsing credit costs, spend controls, and plan allowances into tables, and tightening the assistant page around its approval and permission boundaries
* feat: condense the integrations, collaboration, zapier, and make guides by roughly 35 percent, grouping the thirty-row Zapier and Make action lists into eight labelled areas, folding CRM default field mappings and presence indicators into tables, and promoting the destructive-action and unattended-delete warnings into callouts
* fix: correct three factual errors in the development docs: NOTIFICATION_EMAIL_DAILY_CAP=0 means uncapped rather than disabled (overEmailBudget returns false at limit<=0, so documenting it as a kill switch inverted the behavior), and the worker-SSH and warmup-pool migration citations in architecture.mdx pointed at pre-squash filenames that no longer exist or now belong to unrelated migrations, so both now cite the tables in 000001_baseline.up.sql
* feat: add the missing docs SEO primitives: a build-time sitemap.xml covering all 64 pages, a robots.txt that points at it and keeps the llms.mdx and og mirrors out of the index as duplicate content, and per-page canonical plus richer OpenGraph URL/title/description metadata
* fix: use the single real team@warmbly.com address everywhere a human is told to write in, replacing the invented hello/sales/legal/support inboxes across the marketing site, the transactional email footer, and the admin outreach composer default Reply-To (which pointed replies at a mailbox that does not exist), and collapse the contact page's two-inbox framing into one inbox with one published response time
7.3 KiB
Warmbly Deployment
Two distinct planes, deployed differently.
| Plane | Services | How |
|---|---|---|
| Control | backend, consumer, tracking, realtime, web | Container hosting in one region (Railway in production). Stable region-pinning so KMS/S3 calls stay local. |
| Execution | worker | One process per VPS, anywhere with a public IPv4. Managed from the admin dashboard over SSH. |
Directory layout
deploy/
├── docker/
│ ├── backend.Dockerfile # also builds the seed + migrate binaries
│ ├── consumer.Dockerfile
│ ├── worker.Dockerfile
│ ├── realtime.Dockerfile
│ ├── go-dev.Dockerfile # hot-reload dev images (make app)
│ ├── rust-dev.Dockerfile
│ ├── elixir-dev.Dockerfile
│ └── air.toml
└── config/
└── env.example
The tracking Dockerfile lives at tracking/Dockerfile, and the frontends build from web/Dockerfile and admin/Dockerfile (nginx static builds with runtime config injection). The self-host compose is docker-compose.yml at the repo root.
Building images
docker build -f deploy/docker/backend.Dockerfile -t warmbly/backend .
docker build -f deploy/docker/consumer.Dockerfile -t warmbly/consumer .
docker build -f deploy/docker/worker.Dockerfile -t warmbly/worker .
docker build -f deploy/docker/realtime.Dockerfile -t warmbly/realtime .
docker build -f tracking/Dockerfile -t warmbly/tracking tracking/
docker build -f web/Dockerfile -t warmbly/web web/
docker build -f admin/Dockerfile -t warmbly/admin admin/
The default builds have no Kafka/Avro support; add --build-arg GO_TAGS=kafka (Go images) or --build-arg CARGO_FEATURES=kafka (tracking) to opt in.
GitHub Actions publishes these to GHCR automatically. See the self-hosting guide.
Local development
make dev # one-command native dev stack (infra + migrations + seed + app)
make infra # postgres, redis, nats, mailpit (leave running, shared across worktrees)
make app # backend, consumer, worker, tracking, realtime, web, admin (hot reload, in Docker)
make seed # rich fixtures
make reset # nuke volumes
Full reference: local development.
Deploying the control plane
The Dockerfiles in deploy/docker/ are the deployment unit. Production runs on Railway. Other valid targets: Fly.io, ECS Fargate, single-VPS systemd. Migrations run automatically on backend boot.
Configuration is env-driven — see deploy/config/env.example for the full env reference, or the self-hosting guide for a step-by-step.
Realtime transport
Backend, consumer, and the Elixir realtime service all pick their event transport from one flag, PUBSUB_ENABLED, so they cannot disagree:
PUBSUB_ENABLED=false(default): events bridge over Redis (REDIS_URL). No GCP needed. This is the local-dev and simple self-host path.PUBSUB_ENABLED=true: events flow through Google Pub/Sub. Also setGCP_PROJECT_IDandGOOGLE_APPLICATION_CREDENTIALS_JSONon every service. The backend and consumer auto-provision the realtime topics and their<topic>-subpull subscriptions on boot (idempotent), so there is no manualgcloudstep. The service account needsroles/pubsub.editor.
Set the flag the same on all three services. A publisher on Pub/Sub with a subscriber on Redis silently drops every realtime event.
Worker deployment
Workers run on per-VPS machines so cold-mail traffic spreads across many IPs. Worker identity is a deterministic UUIDv5 derived from the VPS's public IPv4 — same IP, same worker.
Add a worker from the admin dashboard:
- Provision a VPS, note its public IP + root user
- Admin → Workers → Add Worker
- Copy the generated enrollment command
- Run it on the VPS as root
The installer is served by the backend at /worker-install.sh. It exchanges the one-time token for worker config, writes /etc/warmbly/worker.env, configures systemd, enables a daily randomized self-update timer, and starts the worker container. The worker then heartbeats back to the backend and marks itself installed.
The older SSH-managed path is still supported: paste the generated SSH public key into the VPS's ~/.ssh/authorized_keys, then click Test and Install. From then on, lifecycle operations (restart, update, system updates, reboot, rotate keys, logs, uninstall) can happen from the dashboard.
Manual install on the VPS is also supported:
curl -fsSL https://api.example.com/worker-install.sh | sudo bash -s -- \
--enroll wmenroll_... \
--api-base https://api.example.com
# or fully manual, passing a prepared env file:
sudo bash scripts/install-worker.sh \
--image ghcr.io/<owner>/warmbly/worker:prod \
--env-file worker.env
scripts/install-worker.sh --help lists every flag (--ips for multi-IP machines, --update, --uninstall, --purge, --status, --no-auto-update, plus the legacy Kafka/AWS prompts).
Why per-VPS instead of Kubernetes DaemonSet
Cold-mail reputation lives at the IP level. K8s nodes typically NAT pods through a small set of egress IPs, so a per-node DaemonSet does not deliver IP diversity. Workers don't depend on Postgres, so cluster-level service discovery isn't needed. Spreading across VPS providers and regions is the only thing that actually moves the deliverability needle.
Worker env reference
Workers in production should be assigned to a worker profile in the dashboard. The profile bundles all of these:
| Env var | Source | Notes |
|---|---|---|
APP_ENV |
profile | |
EVENTBUS_PROVIDER / NATS_URL |
profile | nats on the default stack |
CODEC_PROVIDER |
profile | json on the default stack |
REDIS |
profile | full URL with embedded password; encrypted at rest |
ENCRYPTED_KEYS_BACKEND_URL |
profile | the backend's public/internal URL |
ENCRYPTED_KEYS_WORKER_TOKEN |
profile | must equal the backend's INTERNAL_API_TOKEN |
BOX_GOOGLE_* / BOX_OUTLOOK_* |
profile | mailbox OAuth clients; needed for token refresh |
KAFKA_* / SCHEMA_REGISTRY_* |
profile | Kafka path only; secrets encrypted at rest |
AWS_REGION / AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY |
profile (via AWS credentials row) | only when using AWS KMS/S3; secret encrypted at rest |
WORKER_TIER |
(worker row) | shared or dedicated |
The worker does not open a Postgres connection. Do not add one.
Auto-update
Each worker profile picks a release channel (pinned / stable / dev) and an auto_update toggle. When a GitHub release fires the webhook, the backend resolves the channel and (if auto_update=true) rolls every assigned worker. See the self-hosting guide.
Health checks
curl http://localhost:8080/health # backend
curl http://localhost:3000/health # tracking
curl http://localhost:4000/health # realtime