Added a small set of layout primitives every page can pull from:
<Page> outer container with width=default|wide|full + padding
<PageHeader> title + optional subtitle/eyebrow + right-side actions
<PageSection> labeled child block with optional actions
<StatCard> icon + label + big number tile
<EmptyState> icon + headline + supporting line + optional CTA
Every page that opted in now reads with the same vocabulary — the
title sits at the top-left of the white content panel with consistent
breathing room, primary actions float to the right of the header,
empty states use the same dashed-card pattern.
Pages refactored:
Stubs (10): billing, settings, team, templates, api-keys, all of
crm (deals/pipelines/tasks). Each was an h1 + description + card
with slightly different paddings; now they're 1:1 PageHeader +
EmptyState. Switched the "primary" colour from zinc-900 to sky-600
to match the new chrome.
Real pages (3): emails, campaigns list, analytics.
- emails: stat strip reduced to <StatCard>×4, header moved to
<PageHeader>, table colours pulled from zinc to slate, accent
switched to sky-600.
- campaigns: same treatment; card hover tint sky-200 instead of
zinc-300; status dot colours kept (emerald/amber/slate).
- analytics: stat row + chart card + side panel + warmup empty
state all reflowed onto the same primitives.
Skipped on purpose:
- contacts (just delegates to ContactsTable; the table itself can
take a tonal pass later)
- unibox (custom chat-style split layout; PageHeader doesn't fit)
- campaigns/[id]/* (sub-pages live inside CampaignLayout which has
its own header; cleaner to redesign that layout once than each
sub-page individually)
Replaces the flat zinc sidebar + thin header with one continuous
sky-coloured chrome wrapping a clean white work surface. Reads as
"work happens inside a room with sky outside the window frame."
Layout shape (the L):
┌──────────────────────────────────────────────────────┐
│ [logo] > [org] > [section] [⌘K ●] │ AppHeader
├──────────┬───────────────────────────────────────────┤
│ │ ╭─── content (white, rounded-tl) ──────╮ │
│ AppNav │ │ │ │
│ │ │ │ │
└──────────┴───────────────────────────────────────────┘
Header and sidebar share one sky-gradient backdrop (SkyChrome). The
content panel tucks into the inner corner with rounded-tl-2xl and a
4px margin on right/bottom so a sliver of sky stays visible at every
edge except the seam — the inner corner is the only "merged" edge.
New components in components/layout/:
- SkyChrome: gradient + two soft blurred glows. No animation; the
auth page is where theatrical clouds live, the dashboard is
intentionally quieter.
- AppHeader: one-row breadcrumb spanning the full width. Warmbly
logo lives in the sidebar-width left zone, then an org-picker
button, then the URL-derived section/subpages. Connection
indicator + ⌘K search on the far right.
- AppNav: sidebar list, styled for the dark sky bg. Same section
grammar as before (Email, CRM, Resources) plus a primary
"New Campaign" action up top, settings + user menu pinned at
the bottom. Hover/active states use translucent white pills.
- AppShell: composes the three above, swaps in the keyboard
shortcuts modal + command palette, takes over from AppLayout.
OrgSwitcher and UserNav restyled for the dark chrome (white text,
faint white borders/hover states) and lost their dependency on
@/components/ui/sidebar — there's no SidebarProvider any more, the
new shell doesn't need one.
Removed components/layout/AppSidebar.tsx (the old zinc sidebar).
Existing per-page layouts (e.g. admin's tab bar) keep working — they
render inside the white content panel exactly as before.
Most container ports go back to their natural defaults — the offsets
that existed weren't justified, they just made URLs harder to remember.
Now standard:
backend 8080 (was always 8080)
tracking 3000 (was 13000)
realtime 4000 (was 14000)
web 5173 (was 15173 — already changed)
kafka 9092 (was 19092)
schema-registry 8081 (was 18081)
localstack 4566 (was 14566)
cloud-tasks 8123 (was 18123)
stripe-mock 12111 (always was)
Kept offset (the defaults conflict too often on real dev machines):
postgres 15432 (system postgres / sibling project)
redis 16379 (sibling docker projects with redis)
mailpit ui 18025 (sibling docker projects with mailpit)
mailpit smtp 11025 (same)
kafka-ui 18090 (8080 already used by backend)
Touched: docker-compose.yml, Makefile (test-seed SEED_TEST_DB), READMEs
(root + deploy), local-development.md + deployment-guide.md. Internal
docker-network refs (kafka:29092, mailpit:1025, etc.) unchanged — only
host-port mappings moved. Compose validated, all default-profile
services come up healthy on the new ports.
The backend's CORS allow-list resolution falls back through:
1. CORS_ALLOW_ORIGINS
2. APP_URL
3. origin derived from WEBSOCKET_URL
In compose only WEBSOCKET_URL was set, so allow-list landed at
http://localhost:14000 — the realtime port, not where the browser is
loaded from. The dev fallback that includes localhost:5173 only fires
when the list is empty, not when it's wrong-by-derivation. Result:
every POST to /auth/login etc. failed preflight with 403, which axios
surfaces as a generic "NetworkError" — login appeared to silently fail
after entering the password.
Setting APP_URL=http://localhost:5173 on the backend service makes the
allow-list match where the Vite dev server actually serves from. Sanity
checked with curl -X OPTIONS — preflight now returns 204.
The new admin API clients (audit, credentials, workers) imported Request
with four '..' segments instead of three. Vite's import-analysis failed
with "Failed to resolve import ../../../../Request" because that path
resolves to api/Request, not client/Request. tsc didn't catch it because
the resolver was permissive enough to keep going, but the runtime is
strict. Matched the existing pattern from roles/getRoles.ts (three dots
for Request, four for models).
Separately: web was on host port 15173, offset from the canonical 5173
to avoid colliding with a locally-running Vite outside Docker. Nobody
actually runs Vite locally in this setup, and the offset makes the URL
non-obvious. Moved back to 5173:5173 and updated VITE_APP_URL plus the
docs.
If a developer one day wants to run a host-side Vite alongside the
container, change the mapping back to "15173:5173" — the offset is the
escape hatch, not the default.
Four interlocking problems were causing the web container to spin in
its retry loop forever:
1. web/package.json aliased vite to "npm:rolldown-vite@7.1.14".
rolldown-vite is being deprecated (its own warning told us to use
7.3.1 for migration, or move to vite 8). vitest 4.x has a transitive
`vite` dep that pnpm tried to resolve against the public registry,
where vite@7.1.14 doesn't exist as a release (only 7.3.3 and 8.x).
Result: ERR_PNPM_NO_MATCHING_VERSION on every retry.
2. The "resolutions" block was meant to force the alias on transitive
deps. resolutions is Yarn syntax; pnpm doesn't read it. So the alias
wasn't propagating, which is exactly why (1) blew up.
3. pnpm 11 stopped reading the "pnpm" field in package.json. Settings
moved to pnpm-workspace.yaml. New file added with allowBuilds.esbuild
set so pnpm doesn't refuse to compile esbuild's native binary at
install time (it's transitively pulled in by vite + vitest).
4. The docker-compose web service had `until pnpm install; do echo
"pnpm install retry..."; sleep 3; done` which spins forever on
permanent dep-resolution errors and buries the actual message under
thousands of retries. Replaced with fail-fast that prints a hint
directing the admin to fix package.json and `make restart web`.
Also deleted the stale pnpm-lock.yaml so pnpm regenerates against the
fresh dep tree. Verified vite v7.3.3 boots, esbuild postinstall runs,
and Vite serves on http://localhost:15173 cleanly.
vite.config.ts has no rolldown-specific config, so the move from
rolldown-vite to plain vite is a no-op behaviourally.
make logs # everything, --tail=200 + follow
make logs backend # one service
make logs backend consumer # several
Same positional-args trick as `make restart`, reused. Ctrl-C to exit
the follow.
Two names for the same action was just clutter. `restart` is enough.
If you ever need to genuinely restart without rebuilding (container
restart that preserves the binary), `docker compose restart <svc>`
works directly — that's a rare enough case to not need a wrapper.
Previous attempt distinguished restart (no rebuild) from rebuild
(rebuild + restart). That distinction was useless in practice because
'docker compose restart' alone keeps the old binary — your code
change never appears. So every iteration was actually 'make rebuild',
and 'make restart' was a trap.
Collapsed both names into one behaviour. `restart` and `rebuild` are
aliases now; both do rebuild + restart, both take the service name
positionally:
make restart backend # was: make rebuild SVC=backend
make rebuild backend # same thing
make restart-go # all Go services
make restart-all # + Rust + Elixir
Positional argument plumbing via the standard Makefile trick:
captures non-target words after `restart`/`rebuild`, turns them into
no-op rules so make doesn't error.
If anyone genuinely needs the old container-restart-without-rebuild
behaviour (env var change, re-applying a migration the backend
already has), `docker compose restart <svc>` still works directly.
Documented that escape hatch.
Simpler than full hot reload for the Go side. The web service already
runs in dev mode (Vite HMR via the node container + ./web mount), so
frontend iteration was never the problem — only Go required a manual
docker rebuild + restart, which is a sequence everyone forgets.
Three new targets:
make restart SVC=backend restart without rebuild (config/env
changes, re-applying migrations)
make rebuild SVC=backend rebuild + restart one service
make rebuild-go rebuild + restart all Go services
(backend + consumer + worker)
make rebuild-all same plus tracking (Rust) + realtime
(Elixir) — the safe one when you've
touched things across stacks
local-development.md updated with an "Iterating on code" block so
this is discoverable.
Until now, only the web service hot-reloaded (Vite HMR via the
node:22-alpine container + ./web mount). The Go services (backend,
consumer, worker) used their production multi-stage Dockerfiles, so
every code change meant `docker compose build <svc> && docker compose
up -d <svc>` — ~30s per service.
Switched all Go services to a shared dev image (go.dev.Dockerfile)
that ships:
- full Go 1.25 toolchain on alpine
- CGO deps for librdkafka (gcc, musl-dev, librdkafka-dev, pkgconf)
- air v1.61.7 (the source watcher / hot-recompile tool)
docker-compose mounts the repo at /app and runs `air -c <config>`.
Each Go service has its own air.SERVICE.toml (build target +
exclusions). Named volumes for the Go module cache and build cache
so the first build is slow (~60s for module download) but subsequent
rebuilds after a save are ~2s.
Per-service compose changes:
- backend, consumer: dockerfile, volumes, and command updated
- worker-base (the YAML anchor used by all 3 workers): same
Production Dockerfiles in deploy/docker/{backend,consumer,worker}.
Dockerfile are unchanged and still used by release CI. The seed
one-shot in compose continues to use backend.Dockerfile (it's a
short-lived job, no benefit from the dev image).
Rust (tracking) and Elixir (realtime) still build-on-change. They
change far less often; documenting the workaround in
resources/local-development.md for now.
Two complementary axes for organizing the fleet, on one shared
mechanism:
User tags (workers.tags)
Free-form lowercase strings the admin applies for whatever they
care about — region (eu-west, fra), provider (hetzner, ovh),
role (warmup-only, burst-capacity), customer cohort. Edited via
a chip-style input with autocomplete from existing tags. Saved
to the worker_tags table.
Smart labels (computed client-side)
Auto-derived from the worker row so they're always in sync:
type:shared / type:dedicated
tier:free / tier:premium (shared only)
pool:clean / pool:risky / pool:quarantine (shared only)
state:installed / state:error / ...
ver:v1.2.3 (if image_version set)
liveness:online / stale / offline
Rendered with tone-aware backgrounds (red for offline / error
/ quarantine, amber for risky / stale, green for online).
Workers list:
- new Tags column showing user tags + the high-signal smart labels
(offline, error, risky, quarantine) with a "+N" overflow
- "filter by tag" chip strip above the table built from the
frequency of every tag (user + smart) in the current result. One
click filters; click again to clear.
Worker detail:
- Tags section near the top showing all smart labels and a full
TagEditor (chip input + autocomplete + suggestions dropdown +
Save button). Saving propagates to the list via react-query
cache invalidation.
The smart labels are never written to the database — they're
recomputed every render. Means renaming an enum value (e.g. risk
pool name changes) doesn't require a backfill.
Migration 000032 + repo + endpoints for arbitrary string tags on
workers. The fixed attributes (worker_type, free_tier, risk_pool)
cover the dimensions assignment logic uses. Tags cover everything
else admins want to group by: region (eu-west, fra), provider
(hetzner, ovh), role (warmup-only, burst-capacity), customer cohort —
whatever.
Schema:
- worker_tags(worker_id, tag) composite PK
- tag VARCHAR(64), lowercase + dashed via CHECK constraint
- ON DELETE CASCADE so deleting a worker drops its tags
Endpoints:
- GET /admin/workers/tags list distinct tags (autocomplete)
- PUT /admin/workers/:id/tags replace tag set; normalizes input
Repo:
- GetWorkerTags / SetWorkerTags / ListAllWorkerTags
- HydrateWorkerTags batch-loads tags onto a slice of workers in one
round-trip so the dashboard list doesn't do N+1 queries
PUT is transactional (delete + bulk insert) so the list view never
catches a worker mid-tag-swap. Auto-derived "smart" labels
(tier:free, pool:risky, state:error) are NOT stored — those are
computed client-side from the worker row so they stay in sync with
the source attributes automatically. Next commit wires the UI.
Replaces the flat /workers/new form with a five-step wizard that asks
"what's this worker for?" first and uses the answer to default everything
else. The previous form put every decision (worker_type, free_tier,
risk_pool, profile, owner) on screen at once with no guidance — fine if
you already know what you're doing, miserable otherwise.
Steps (Owner step skipped unless purpose=dedicated):
1. Purpose — shared / dedicated / risky-pool, with explanatory
cards. This drives the rest: risky → risky pool,
dedicated → unlocks step 4.
2. Connection — host/port/user. "Test reachability" button hits
the new TCP preflight endpoint before any row is
created — typos and firewalls fail loudly here
instead of at the SSH test stage later.
3. Identity — name (auto-derived from host on focus), notes,
profile, tier, risk pool. Risk pool defaults from
purpose but the admin can override.
4. Owner (deds) — user search via /admin/users + subscription ID.
Wizard remembers these and uses them in step 5.
5. Activate — review summary, "Install immediately" toggle
(default on), big Create button.
Post-create panel runs the full pipeline inline without leaving the
page when auto-install is on:
- Show pubkey + copy button + ready-to-paste ssh one-liner
- "I've pasted the key" checkbox unlocks Install
- Install button chains: Test → Install → (if dedicated) convert with
the previously-collected user/sub IDs → redirect to detail page
- Each step's outcome streams into a progress log
Progress dots at the top so the admin sees where they are. Empty-state
hint on step 1 when no workers exist yet.
POST /admin/workers/preflight {host, port} runs a 5s TCP dial against
host:port and returns ok + latency, or an error. Used by the worker
creation wizard to catch typos / firewall problems while the form is
still open — much better UX than discovering an unreachable VPS at the
SSH test step after the row already exists.
Doesn't attempt an SSH handshake (no credentials at this stage). A green
preflight just means "something is listening there." The actual SSH test
runs later, after the admin pastes the generated pubkey.
Worker list grows a Pool column (clean=green, risky=amber,
quarantine=red badge). Dedicated workers render "n/a" — risk pools are
a shared-worker concept since dedicated workers don't share IPs across
customers.
Worker detail page (shared workers only) gets a "Risk pool" section
with three big buttons. Clicking a non-current pool confirms, then
calls PUT /admin/workers/:id/risk-pool. Action audited with the new
pool value.
Saving doesn't migrate accounts directly — the hourly rebalancer
notices the mismatch and moves mailboxes to a matching-pool worker
on its next tick. Documented in the section's helper text.
Endpoint accepts {risk_pool: "clean"|"risky"|"quarantine"} and is
gated by AdminPermManageWorkers. The worker detail row scan now
includes risk_pool so the column actually has data.
New background job in the consumer process:
1. Pulls up to 1000 mailbox candidates joined with their worst warmup
health state (across all pools they participate in) and their
current worker's risk_pool. Dedicated workers are excluded — single
tenant, segregation not applicable.
2. Recomputes risk_band from health state via RiskBandFromHealth.
If it changed, writes the new band.
3. If the band's matching pool doesn't equal the worker's pool, picks
a new worker via SelectSharedWorkerForBand and migrates the mailbox.
Increments/decrements account counts.
4. Logs each migration to admin_audit_log with action=
"risk_rebalance_migrate" so operators see what moved and why.
Boot-time run + hourly ticker. Rebalancing is intentionally batch, not
event-driven: warmup health states change on a slow rolling-window basis
(warmup_health_sweep is also hourly), so reacting in real time gains
nothing and would cause thundering-herd migrations.
JobsService gets an AssignmentService dep. Nil disables the job (lets
self-hosters opt out by simply not wiring it).
New method on WorkerAssignmentService picks the least-loaded shared
worker whose risk_pool matches the mailbox's risk band. Three-step
fallback chain:
1. Exact match: same pool, same tier
2. Fall back to clean pool of the same tier when no matching-pool
worker is available (better to land risky mailboxes on clean
workers than refuse; the rebalancer will move them later)
3. Last resort: any worker of the right tier (preserves legacy
behavior for installations that haven't provisioned risky/
quarantine pools)
Existing SelectSharedWorker is unchanged so call sites that don't
know about risk bands keep working. The next commit (background
rebalancer) is the first consumer of the new method.
Threat-level segregation, schema layer. Two new concepts:
workers.risk_pool ∈ {clean, risky, quarantine}
buckets shared workers by acceptable risk. Dedicated workers don't
use it (single tenant = no cross-contamination risk).
email_accounts.risk_band ∈ {clean, risky, quarantine}
per-mailbox classification, derived from warmup_health_state by the
rebalancer (next commit). Never set by user input.
The mapping is one-way and intentionally simple:
healthy → clean
watch, throttled → risky
quarantined, → quarantine
blocked
Rebalancer code lands in the next commit. This commit just adds:
- migration 000031 with enums + columns + filtered indexes
- WorkerRiskPool / EmailRiskBand types + RiskBandFromHealth helper
- WorkerRepository methods: SetWorkerRiskPool, SetEmailAccountRiskBand,
GetSharedWorkersByTierAndPool, ListRiskCandidates
- RiskCandidate result type joining email_accounts + warmup health
(picks WORST state across pools via CASE ranking) + worker columns
so the rebalancer can decide migrations in one scan
POST /admin/workers/:id/convert-to-dedicated does three things in
sequence:
1. Drain existing accounts to a supplied drain_to_worker_id (required
if the source has any accounts; we don't auto-pick per-account
targets because the right choice depends on each account's
owning org).
2. Flip workers.worker_type from "shared" to "dedicated".
3. Atomic create of dedicated_worker_assignments binding the worker
to a specific user/subscription (uses the existing
CreateDedicatedAssignmentIfNotExists so re-running is safe).
Refusal cases:
- already dedicated → 400
- has accounts but no drain target → 400 (admin must pick where they go)
- drain target equals source → 400
Worker detail UI gains a "Convert to dedicated" section, shown only
when the worker is currently shared. Inline form, no modal. The drain
dropdown excludes self, only lists shared+installed workers, sorts
least-loaded first.
Audit-logged with action="convert_to_dedicated" and the user_id,
subscription_id, drain target, and account count in details.
The free-tier-vs-paid separation in AssignWorkerToEmail is one of those
rules that's silently load-bearing: if a free org ever slips onto a
premium worker, the IPs of paying customers absorb the deliverability
hit. The code is correct today (strict isPaidOrg check at line 67, free/
premium pool sync at line 116), but nothing was guarding against a
regression.
Five table-thin tests, hand-rolled stub repos (embed the interface as a
nil field so unused methods panic loudly):
- free org → free shared worker → free warmup pool
- paid org → premium shared worker → premium warmup pool
- paid org with DedicatedWorkers > 0 + an assignment → dedicated worker
- paid org with DedicatedWorkers > 0 but no assignment → falls back to
premium shared (not free!)
- SelectSharedWorker with no workers → ErrNoAvailableWorkers
No code changes — this commit is documentation.
Worker detail page gains a "Move accounts to another worker" section
with a dropdown of eligible targets (same tier, currently installed,
sorted least-loaded first). One click moves every email account on the
current worker to the picked target via the existing AdminReassignEmails
endpoint.
Useful when:
- a worker is down and you want to shift its workload to a healthy
sibling while you investigate
- you want to drain a worker before uninstalling it
- a profile-level change forced too much onto one worker and you want
to rebalance manually
The endpoint emits the standard audit log row (admin_user_id + action
"reassign"), so every manual rewire is visible in the audit viewer.
WorkerHealthAlert lives in the admin layout and renders on every admin
page when any worker needs attention. Polls the workers list every 30s
in the background. Hidden on the workers page itself (which has its own
richer banner with filter chips).
Surfaces three categories:
- errored: install_state == "error"
- offline: installed + no heartbeat for 5+ minutes
- in progress: pending / provisioning / uninstalling
Tone goes red when there are errored or offline workers, amber otherwise.
A "review →" link deep-jumps to /app/admin/workers.
New /app/admin/audit page browses admin_audit_log with:
- Filter by action, target_type, target_id, admin_user_id, date range
- Action + target_type dropdowns are populated from a baseline set
AND from whatever appears in the current result (so new actions
surface automatically without code changes)
- Auto-refresh toggle (5s) for live tailing during operations
- Cursor-based pagination with Prev/Next + page counter
- Expandable rows show the details JSON + user-agent
- Color-coded actions (red for destructive, green for create/install,
amber for system auto-actions)
- Renders system actions (admin_user_id = uuid.Nil) as "system"
- Renders the admin's name + email when joined data is present
When the dead-worker job reassigns email accounts from a worker whose
heartbeat expired, write a row into admin_audit_log so the dashboard's
audit viewer surfaces these system actions alongside admin-driven ones.
admin_user_id is uuid.Nil (the platform identity), so admins searching
the log can distinguish "system did this" from "an admin did this" by
filtering on that ID. Details include the replacement worker, account
count, and reason.
JobsService gets an optional AdminRepo dep. Nil disables logging — keeps
the contract loose for any other call site that doesn't have one.
The audit calls added in the last commit went to AuditService.LogAction,
which writes to the general user-facing audit log (Cassandra). The admin
audit-log viewer at /admin/audit-logs queries the admin_audit_log table
in Postgres, so worker / credentials / release actions never showed up.
Add a public AdminService.LogAdminAction that wraps the existing private
logAction (writes to admin_audit_log with the same shape as ban_user /
update_worker / etc.). Repoint h.audit() at it.
Actions now visible in the audit viewer:
test, install, restart, upgrade, uninstall, rotate_keys, apply,
assign, system_update, reboot, check_releases (plus the existing
create/update/delete across workers, AWS creds, and profiles).
The seeder is one of the few things every developer runs on every new
checkout, but it had zero tests. With three migrations added in the last
few days and the rich-fixture path now creating 30+ rows, the chance of
silently breaking a schema migration without noticing was non-trivial.
cmd/seed/main_test.go connects to SEED_TEST_DB (skips otherwise — keeps
unit tests in CI fast and prevents accidentally clobbering a dev
database), runs migrations, wipes only the fixture rows, then:
1. Runs seedBaseline twice, verifies row count stays at 1.
2. Runs seedRich, asserts 9 different row counts match expectations
(users, orgs, workers, accounts, campaign, sequences, contacts,
unsubscribed contacts, campaign leads).
3. Re-runs seedRich, asserts every count is unchanged — the most
important guarantee the seeder makes.
4. Verifies warmup pool membership: 2 free, 4 premium, with the
correct accounts in each.
Plain testing package, table-driven, matches the existing style in
internal/app/warmup/service_test.go.
`make test-seed` brings up the docker-compose Postgres and runs the
suite against it.
Workers list now computes a health summary client-side from the existing
queries and shows a yellow banner with a count when any worker needs
attention (errored, offline, or mid-install). Filter chips below the
banner narrow the table by problem category so the admin can drill in
without opening each worker.
Categories surfaced:
- errored: install_state = "error"
- offline: installed, but no heartbeat in 5+ minutes
- in progress: pending / provisioning / uninstalling
- stale config: profile.updated_at > worker.config_applied_at
- update available: profile.resolved_image_tag != worker.image_version
Live polling stays at 15s, so a worker going offline shows up within
~75s of the last missed heartbeat. The list also fetches profiles so the
stale-config and update-available chips can compute without extra
round-trips.
The new admin endpoints for SSH worker management, AWS credentials,
worker profiles, releases, and system operations bypassed the existing
audit-logging pattern. Now every mutating action records a row in the
audit log with adminID, IP, user-agent, and operation-specific metadata
(never secret values — for credential updates we record which fields
were rotated, not what they became).
New action constants:
test, install, restart, upgrade, uninstall, rotate_keys, apply,
assign, system_update, reboot, check_releases
New entity types:
worker, aws_credentials, worker_profile, release
Read-only endpoints (list/get/status/logs) intentionally not audited —
they don't change state and would flood the log.
Each handler calls a small h.audit(c, action, entity, &id, metadata)
helper that pulls adminID from the admin middleware and fires the
existing AuditService.LogAction (fire-and-forget, never blocks the
response).
Workers heartbeat into Redis every 90s as RFC3339 timestamp values with a
3-min TTL. The dashboard surfaces liveness based on workers.last_seen_at,
but until now nothing populated that column — the "Live" badge was always
red.
New 60s job in the consumer reads each active worker's Redis heartbeat
value, parses the timestamp, and writes it to workers.last_seen_at. Runs
on its own interval (separate from the 5-min dead-worker detection job,
which does heavier reassignment work) so the UI sees fresh data within a
minute.
Old docs described a k8s/ArgoCD/Terraform deployment that no longer
exists, with ASCII-art system diagrams that hadn't aged well. Rewritten
to match how the project actually ships:
- README: control plane (Railway) + execution plane (per-VPS workers)
split, dashboard-driven worker management, credentials/profiles,
auto-update from GitHub releases, OS package updates, self-hosting
knobs. Removed all ASCII art.
- resources/architecture.md: control vs execution plane, encryption
model (worker SSH keys + platform secrets under the same KMS-envelope
cipher as user secrets), worker identity from public IPv4, credentials
model, push-driven release flow, anti-abuse layers, source anchors.
- resources/deployment-guide.md: end-to-end from "provision a VPS" to
"auto-update on release". No more k8s, ArgoCD, kubectl, or Terraform.
Step-by-step backend env, webhook setup, worker add flow, day-2 ops,
rollback per plane.
- resources/local-development.md: the five make targets (dev / sim /
seed / tools / reset), what each profile runs, LocalStack bootstrap,
rich seed contents, native dev against containerized infra, the
offset-port URL table.
- resources/cicd.md: the two-plane build/release flow, image tag scheme
({sha} / dev / vX.Y.Z / vX.Y / vX / prod), webhook setup, release
process, security notes around HMAC and least-privilege worker AWS
keys.
- deploy/README.md: tight version of the same.
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Hoist the dev/sim stack to a single docker-compose.yml at the repo root.
Adds profiles (default / sim / seed / tools) so you can opt into heavier
setups, and bundles dependencies that were previously missing:
- LocalStack (KMS + DynamoDB + S3) with a localstack-init one-shot that
idempotently creates alias/master-key-dev, the UserEncryptedKeys and
EmailMessageData tables, and the main S3 bucket. Backend and workers
wait on it via service_completed_successfully.
- stripe-mock for billing flows
- kafka-ui under the tools profile
Three workers with deterministic UUIDv5 hostnames (shared / premium /
dedicated) so assignment, rebalancing, and per-pool routing all have
real targets to exercise.
Richer seed (cmd/seed/main.go) loads 3 orgs across tiers, 6 mailboxes
joined to free/premium warmup pools, a Beta campaign with a 2-step
sequence, and 10 contacts (2 unsubscribed) so suppression behaviour is
visible in the UI. Idempotent — safe to re-run.
Makefile targets:
make dev — infra + app + one worker
make sim — adds premium + dedicated workers
make seed — rich fixtures
make tools — kafka-ui at :18090
make reset — nuke volumes
scripts/install-worker.sh is a single bash script any Debian/Ubuntu/RHEL/
Fedora/Arch/Alpine VPS can curl|sh to add a worker to the fleet.
Identity is bound to the VPS's public IPv4 via UUIDv5 (URL namespace):
same IP → same worker (reputation persists across reinstalls)
new IP → new worker (fresh identity, no inherited reputation)
The installer detects the public IP via api.ipify.org / ifconfig.me /
checkip.amazonaws.com, derives the deterministic UUID, installs Docker if
missing, writes /etc/warmbly/worker.env (0600) and /etc/warmbly/worker.id,
installs a systemd unit that runs the worker container with --hostname
<uuid>, and starts the service.
Supports --install/--update/--uninstall/--purge/--status, --env-file for
non-interactive config, --ip override, --image override, and a full set
of per-credential flags.
Worker reads its UUID from os.Hostname() at startup, so the systemd
hostname value becomes the worker identity — no separate registration
step needed.
Workers need IP diversity, but k8s nodes typically NAT all pods through a
small set of egress IPs — defeating the point of a DaemonSet for cold mail.
Plus, the control plane is moving to Railway and workers will be managed
per-VPS, so the kustomize tree no longer reflects how anything actually
ships.
Matthew Meszaros