The realtime channel handler `def join("user:" <> user_id, ...)` checks
`socket.assigns.user_id == user_id`, where socket.assigns.user_id is
the JWT `sub` claim (UUID). The frontend was building the topic from
user.email — every join was REFUSED.
Added `id: string` to the frontend User type (the backend already
serializes it as "id") and switched the channel topic in
RealtimeManager to use it.
After this + the previous round of WS fixes:
CONNECTED TO RealtimeWeb.UserSocket in 481µs
JOINED user:11111111-0000-0000-0000-000000000001 in 15µs
1) Contacts crash "c is null":
contactRepository.Search declared `var contacts []models.Contact` so
an empty result set returned a nil slice, which Go marshals as JSON
null. The frontend's flatMap((p) => p.data) over null yields [null],
and the page then accesses c.subscribed → throws. Initialize as
make([]models.Contact, 0, limit+1) so the wire format is always [].
Also defensive on the client: useSearchContacts + useCampaigns now
coerce p.data ?? [] and drop nulls before returning.
2) Campaigns panic on any non-empty result:
campaignRepository.Search allocated `make([]models.Campaign, 0, limit+1)`
(length 0) then did `campaigns[i] = campaign`. That's an
index-out-of-range on the first iteration. Switched to `append`.
Anyone with at least one campaign would see a 500 / blank screen.
3) Websocket "Token expired":
SocketTTL was 60s. The frontend reconnect backoff caps at 30s, so
after a rejected handshake the next attempt could fire 30-60s
later. Combined with rare back-pressure on /getaway the token was
already past exp by the time the realtime saw it. Bumped to 10 min
— short enough to keep the token low-impact, long enough to outlast
the backoff schedule.
4) Websocket "Connection limit exceeded":
Realtime.Connections only untracked on channel terminate, never on
socket disconnect. Sockets that connected and disconnected without
joining a channel leaked. Each reconnect loop bumped the counter
until the per-user limit (10) was hit, after which every legitimate
connect was rejected even after fixing #3.
Fix: GenServer Process.monitor's the socket pid on track, and
`:DOWN` handler calls do_untrack with the right (user_id, ip).
5) Phoenix protocol mismatch:
Frontend appended vsn=2.0.0 to the WS URL, but sendRaw + joinChannel
send the V1 object format. Realtime's Phoenix.Socket.V2.JSONSerializer
crashed with a badmatch on the first phx_join, killing the socket
right after connect. Switched to vsn=1.0.0 to match what the client
actually emits.
Backend root cause:
The frontend client omits ?limit= when it would equal the default
(DEFAULT_PAGINATION_LIMIT = 50). validate.Limit("") treated empty as
invalid and returned errx.ErrLimit → 400 on /contacts/search and
/campaigns. ContactsTable derived isLoading from `!contacts`, which
stays true forever when the query errors, so the page hung in the
skeleton state instead of surfacing the error.
Fix:
- validate.Limit now accepts "" and returns LimitDefault = 50, in
sync with the frontend constant. The frontend's omission semantics
("don't send the param when it's the default") was already correct;
it was the validator that was wrong.
Frontend:
- ContactsTable: use isPending/isError/refetch directly from react-query
instead of deriving from `contacts`. New explicit error block renders
inside the body with: red alert tile, server error message, Try-again
button (with spinner during refetch), and Reload-page fallback.
- Campaigns page: same error UI promoted from the old EmptyBlock CTA
to a prominent block — alert tile + message + retry + reload.
New ContactFilters sheet (was the legacy 800px poppins drawer):
- 420px right-side panel matching the rest of the theme.
- Sticky 48px header with "Filters · N active" eyebrow + close.
- Sticky 48px footer with Reset / Cancel / Apply (slate-900 primary).
- Hairline-divided SectionBars between groups: Search, Custom field
filters, Sort, Subscription, Campaign membership, Dates.
- Custom field rows pair TextInput + FILTER_TYPES popover + value
input + remove button — all 28px tall.
- Sort: SelectButton popover + asc/desc toggle.
- Subscription: 3-state pill toggle (Any / Subscribed / Unsubscribed).
- Min/max campaign rows: checkbox toggle + number input + suffix.
- Date rows: checkbox toggle + native date input.
- Draft state mirrors parent until Apply, so editing filters doesn't
trigger refetches mid-build.
Root cause for the 10-min auto-logout (confirmed via pg_stat_activity):
the postgres pool MaxConns was 4, and four repository functions opened
a tx without committing or rolling back. After four calls each leaked
a connection in "idle in transaction" state. Once all four were gone
the pool was permanently exhausted — every new request that needed a
connection blocked until the client gave up. The 10-min trigger is
because that's when the first /auth/refresh fires; refresh tries to
acquire a connection, hangs, eventually the browser aborts the request,
the frontend treats the failure as session expiry, kicks the user.
The four leaking sites:
- emailRepository.Search (drove the leak — Accounts page)
- campaignRepository.Search
- sequenceRepository.Create
- contactRepository.BulkUpdate
Each now has `defer tx.Rollback(ctx)` immediately after Begin, matching
the pattern used in the non-leaky sites in the same files. Rollback is
a no-op after Commit, so this is safe for both read-only tx (Search)
and read-write tx (Create / BulkUpdate).
Additional hardening so a future leak can't silently brick the backend:
- MaxConns 4 → 25. 4 was reckless even without leaks; one bursty
admin page would saturate. 25 is still well under postgres'
default max_connections=100.
- MinConns 0 → 2. Keep a couple of warm connections at idle so the
first request after a quiet period doesn't pay the connect cost.
- idle_in_transaction_session_timeout=300000 (5 min) as a session
RuntimeParam. If a code path forgets the defer, postgres aborts
the leaked tx after 5 min and reclaims the connection.
- statement_timeout=60000 (60 s) as a session RuntimeParam.
Statement runaway can't pin a connection forever.
Verified after backend restart:
SELECT count(*) FROM pg_stat_activity
WHERE datname='warmbly_dev' AND state='idle in transaction';
→ 0
Reliability:
- ErrorBoundary wraps every route. Silent white pages are gone — any
uncaught render error now surfaces inside the panel with name,
message, stack toggle, and back/retry buttons.
- Boundary keys on pathname so navigating away clears the error.
Campaigns blank fix:
- Drop the legacy HeadSelectMenu + Search components from the page
(suspected layout/click-outside collisions inside the slim SectionBar).
- Rewrite using the new dropdown + SearchInput primitives.
- Add StatStrip with clickable filters (All / Active / Paused / Draft).
- Loading shows skeleton rows. Empty splits between "no campaigns at
all" vs "no campaigns matching the current filter".
Dropdown primitive (web/src/components/ui/popover-menu.tsx):
- Brae-density popover menu — slim trigger, hairline border content
surface, h-7 items, mono kbd accents. Built from scratch rather than
via Radix so styles are authoritative and bundle stays small.
- Click-outside + Esc handling, controlled/uncontrolled open state,
side ("bottom"|"top") + align ("start"|"center"|"end") positioning
with viewport-aware clamping.
- SelectButton helper styled as a brae trigger pill.
Field primitive (web/src/components/ui/field.tsx):
- SearchInput + TextInput + Label. 28px tall, hairline border,
sky focus ring. Replaces the half-dozen ad-hoc inputs across pages.
- SearchInput supports Enter onSubmit and inline clear button.
Contacts browser (ContactsTable.tsx, rewritten in place):
- Standalone view: PageTopbar (Import / Export / New) + StatStrip
(All / Subscribed / Unsubscribed / In campaigns, clickable filters)
+ SectionBar (search + sort dropdown + filters).
- Embedded view: skips topbar/strip, drops into SectionBar — used
inside /app/campaigns/[id]/leads.
- Dense table with avatar + name + email-mono, optional company /
phone columns (hidden on smaller widths), subscribed/unsubscribed
pill, campaigns count, created date.
- Bulk selection floats a footer bar with Edit / Delete / Clear.
- Load-more button for infinite scroll (preserves the existing
useInfiniteQuery hook).
- Sort dropdown wired to the existing SearchContacts API params.
Unibox email browser:
- ConversationList: SectionBar header with count, SearchInput, all /
unread tabs with unread count badge, dense rows with avatar +
bold-when-unread sender + subject + preview + relative time.
Unread items get a thin sky rail on the left margin.
- ConversationItem: relative time formatter, name extraction from
"Name <email>" headers.
- ThreadView: 48px topbar (subject + mark-unread/archive/delete
actions) + section bar (n messages / k participants) + a divided
message stream + composer pinned to bottom.
- MessageBubble: no card chrome; just hairlines between messages.
Sender avatar + bold name + mono email + recipient line + mono
timestamp; prose-rendered body.
- ReplyComposer: edge-to-edge textarea with footer bar (Send,
Schedule popover with "in 1h / tomorrow 9 / next Mon 9", Discard,
char counter). ⌘+Enter to send.
Sidebar:
- Drop the generic "+ New Campaign" sky pill. Replaces it with a
LivePanel that reads ambient cold-email telemetry: status dot
(connected/idle/offline), mailbox count, active count, and a
daily sparkline placeholder. Clicks through to /app/analytics.
The sidebar now reflects what the system is doing rather than
nagging with a CTA.
- Nav rows shrink from h-8/13px to h-7/12.5px to match brae density.
- Section labels switch to small tracked-uppercase with a hairline
divider above each section instead of a margin gap.
Page primitive (web/src/components/layout/Page.tsx):
- New vocabulary: Page > PageTopbar (sticky h-12 with eyebrow +
subtitle + actions) + StatStrip (full-width stat row with vertical
rule dividers and Stat cells) + SectionBar (h-9 sub-header) +
PageBody (scrollable area) + Row + EmptyBlock + TopbarAction.
- Pages fill the entire content panel edge-to-edge. No max-w
ceilings, no centered narrow columns, no rounded card chrome,
no Georgia serif. Tracked-uppercase eyebrow at 10px replaces the
28px serif h1.
- Old PageHeader, StatCard, EmptyState, PageSection live on as
thin shims so any page not yet swept keeps compiling.
Pages swept:
- campaigns: list view becomes a hairline-divided row stream with
status dot + mono ID + status pill + relative date. Loading
shows skeleton rows, error and empty share the same EmptyBlock.
- emails: 4-card stat grid → StatStrip; table edge-to-edge with
sticky thead, hover row, uppercase status label.
- analytics: card grid → StatStrip + a 2-column body with
vertical-rule between the chart and the breakdown rail.
- templates, api-keys, billing, settings, team, crm/{tasks,
pipelines, deals}: PageTopbar + EmptyBlock, no more sky pills.
The Logo SVG was switched from hardcoded fill="white" to fill="currentColor"
so the dashboard could tint it gray. Auth-layout usages were never updated
and got rendered with the inherited default text color (near-black) on the
dark sky panel. Pass text-white explicitly at both logo sites.
Outer shell:
- AppShell: drop SkyChrome backdrop and the rounded-tl content tuck.
L-shape becomes side-by-side: white sidebar (220px) + white content
column with a hairline #e2e8f0 divider. No decorative background.
- SkyChrome: now a no-op stub. Kept as a named export so the import
path stays stable if we ever bring decoration back.
- AppNav: drop the "New Campaign" sky pill. Sidebar opens straight
into the nav tree — logo + org switcher in a slim 44px header row,
28px nav rows, sentence-case section labels, hairline borders top
and bottom. Active row = bg-slate-100 + slate-900 text. No shadows.
- AppHeader: 40px breadcrumb strip in the content column (was a
full-width 56px row spanning the shell). No logo here — it lives
in the sidebar header.
- OrgSwitcher: slate-900 initial tile instead of sky-600. Sized to
match the slimmer sidebar (28px tall, 12.5px text).
- UserNav: smaller 24px avatar, tighter row height, slate-100 hover.
Page primitive:
- Page padding: px-4 pt-3 pb-10 (was px-8 pt-8 pb-16).
- PageHeader: single 36px toolbar row — title + subtitle on the same
baseline, actions on the right. Drops the 28px serif title block
and the dedicated subtitle paragraph.
- PageSection: 28px header row with title (12.5px) and inline desc.
- StatCard: redesigned as a divided strip cell. Use inside StatRow
for a row of stats with vertical-rule dividers and a single outer
border, replacing the four floating shadow-tile cards.
- EmptyState: text-led, dashed-border block, 40px tall icon.
Page sweep:
- All /app pages: sky-600 buttons → slate-900 (h-7 px-2.5 rounded-md,
text-[12.5px]). Same height as the new toolbar so the page header
reads as one line.
- emails + analytics: switch StatCards into the new StatRow.
- emails: slate avatar tile instead of sky, slate accent on checkboxes.
- campaigns: slate card chrome instead of sky-tinted hover.
- analytics: slate bars instead of sky, smaller paddings, breakdown
list reads as a tight column instead of a card.
- AppNav: remove "Home" row that pointed at /app/emails (same as Accounts),
which is why two rows lit up together. Accounts is the de facto home.
- AppNav: active row goes from white pill (lighter than chrome) to bg-slate-200
pill (clearly darker than chrome) — a gray step down, not the dark pill from
the previous iteration.
- AppHeader: logo gets a blue lean (#8aa1c1 → #4e6285 on hover) so the brand
shows through without coloring the whole chrome.
- SkyChrome: base from #f4f7fb to #f5f6f8, drop the bottom sky vignette.
Clouds stay; the rest is neutral so the only color cue is intentional.
Reverting the AI-design-system flourishes in the chrome:
- Logo: text-slate-400 at rest, text-slate-700 on hover. Lighter
gray that warms slightly when you mouse it. Drops the sky-600.
- Sidebar nav: brought back the white-pill active state from
before (rounded card with hairline ring + 1px shadow). The
sky-50 + 2.5px rail experiment is gone — it read as a design
system showing off, not a workspace.
- Sidebar hover: bg-white/70 (almost invisible) → bg-slate-200/60.
Now hover actually registers — the row goes visibly grayer.
- Badge: sky-600 → slate-900. Same logic, no accent shouting in
the chrome.
Nothing outside components/layout/{AppHeader,AppNav}.tsx changed.
Four targeted upgrades to actually deliver on "have some taste":
1. Logo: the Logo SVG had fill="white" hardcoded. The moment the
shell flipped from dark sky to light, it disappeared. Switched
to fill="currentColor" so the className drives it (now sky-600
in the header, slate-600 elsewhere).
2. PageHeader: titles now render in Georgia serif (already declared
as --font-serif in global.css). 28px / 500 / -0.01em tracking
in slate-950. Subtitle moved to 13.5px with a max-width to keep
it readable. The serif voice is the same one the auth page
uses for "Your emails deserve the inbox." — gives the dashboard
the same character without explicit branding.
3. StatCard: hard border replaced with a ring-1 + two-layer shadow
(a 1px tight + a 24px diffuse). Cards now read as "sitting on"
the page rather than "drawn on" it — Linear-style elevation.
Number bumped 22→28px, weight 600→500, tracking tightened to
-0.02em. Label flipped to a UPPERCASE 12px caption.
4. EmptyState: rounded-2xl + ring instead of dashed border. Title
in serif to match PageHeader. Soft-tinted bg (#fafbfd) so the
block reads as a quiet placeholder rather than a "no data!"
shout. Icon tile gets a faint shadow.
5. Sidebar active rows: traded the white-pill chrome for a sky-50
tint + a 2.5px sky-500 rail flush against the left edge. Reads
as confident but quiet — the active state announces itself with
colour, not weight.
Also: bumped Page padding 6→8 horizontally, 12→16 bottom; widened
PageHeader bottom margin 6→8. Pages breathe.
After /auth/refresh, Postgres got the new access + refresh nonces but
the Redis cached session still held the OLD ones. The next request:
1. Frontend uses the new access token (new access_nonce in JWT)
2. Backend ValidateAccessToken → GetSession → hits Redis cache
3. Cached session has the OLD access_nonce
4. session.AccessNonce != t.Nonce → ErrToken (401)
5. Frontend tries to refresh with the new refresh token
6. RefreshToken → GetSession → again hits stale Redis
7. sess.RefreshNonce (old) != t.Nonce (new) → ErrToken
8. Frontend clears tokens and bounces to /auth/login
The access token's 10-minute TTL was the trigger window because that's
when the first refresh fires. After the first refresh, the stale cache
poisoned every subsequent request.
Fix: delete the cached session after a successful repository update,
mirroring what SwitchOrganization already does for the same reason
(it updates current_organization_id in Postgres and then drops the
Redis copy). Next GetSession misses, re-reads from Postgres, caches
the fresh nonces.
The deleteSession failure path is intentionally swallowed — the
refresh already succeeded and we returned the new tokens, so worst
case is the next request triggers another refresh, not a logout.
Scrapped the deep sky gradient. The dashboard is workspace, not
cinema — the user lives there for hours, the chrome shouldn't compete
with the content.
New shell aesthetic:
- Backdrop (SkyChrome): a near-white sky-tinted base (#f4f7fb),
one degree cooler than the white content panel so the panel
reads forward. Three blurred-white cloud blobs in the upper
half at very low opacity — atmosphere you feel rather than see.
A whisper-faint sky-tint vignette at the bottom. No animation.
- Content panel: pure white, only rounded at the inner corner
(top-left). Flush to the bottom and right edges of the viewport
— no margin band of visible chrome there. A single hairline
border on the top + left edges defines the panel without a
heavy shadow.
- Header: dark text on the light backdrop. Logo in sky-600 not
white. Wordmark slate-900. Chevrons slate-300, crumb text
slate-500/900. Connection indicator and ⌘K search drop their
translucent-white pills for slate hovers.
- Sidebar (AppNav): same light backdrop. Active rows go to a
white pill with hairline ring + 1px shadow (the row "lifts"
off the sidebar). Inactive rows slate-600. New Campaign
reverses to a confident sky-600 pill. Section labels
slate-400 uppercase 0.16em tracking, no longer fighting for
attention.
- OrgSwitcher + UserNav: light-theme triggers — slate-200 hover,
sky-600 + slate-900 avatar tiles, slate-900 names.
Also: campaigns page used to render a forever-skeleton when the
query errored. Switched to checking isLoading / isError explicitly
and added a retry button via EmptyState so the failure mode actually
shows up instead of looking like a slow load.
Six changes to the dashboard chrome, all small individually, together
the shell feels intentional instead of "okay first pass".
Sky gradient (SkyChrome):
- Switched to a directional 135deg gradient that's rich in the
upper-left (where the logo sits) and brightens toward the inner
corner near the content panel. Light pulls the eye to the work.
- Warm bloom moved from the upper-left to the inner corner with
cream/butter undertone — reads as late-sun reflecting off the
white panel back into the chrome.
- Added a cool slate wash over the dark zone to give it depth
without changing apparent hue.
- Hairline noise overlay (inline SVG feTurbulence, no asset) so
the gradient doesn't read as plastic at 4K.
Content panel (AppShell):
- Margin from 4px to 14px. Reads as a deliberate sky window-frame
rather than a near-miss border.
- rounded-tl-2xl → rounded-tl-3xl. Bigger radius = architectural,
not accidental.
- Soft 60px outer drop shadow + an inset top-edge highlight, so
the panel feels suspended in the sky chrome rather than glued
flush against it.
Header (AppHeader):
- h-12 → h-14, logo 28→32px, wordmark 15→16px. Brand has presence.
- Logo gets a 1px-y drop shadow so it sits above the gradient.
- Chevrons: w-3.5→w-4, white/30→white/45. Visible without being loud.
- Crumb text 13→13.5px, inactive segments white/55→white/65.
Sidebar (AppNav):
- "New Campaign" reverses contrast — white pill with sky-700 text
instead of a faint translucent button. Reads as a primary CTA,
not a quiet link.
- Nav rows: text 12.5→13px, inactive white/65→white/75, active
state gets a subtle inset highlight. Hover bg pop +33%.
- Section labels weight 500→600, opacity 40→50, tracking widened
from 0.16em to 0.18em. Quiet but legible.
- Badge gets a small shadow so it stands off the white pill.
Added a small set of layout primitives every page can pull from:
<Page> outer container with width=default|wide|full + padding
<PageHeader> title + optional subtitle/eyebrow + right-side actions
<PageSection> labeled child block with optional actions
<StatCard> icon + label + big number tile
<EmptyState> icon + headline + supporting line + optional CTA
Every page that opted in now reads with the same vocabulary — the
title sits at the top-left of the white content panel with consistent
breathing room, primary actions float to the right of the header,
empty states use the same dashed-card pattern.
Pages refactored:
Stubs (10): billing, settings, team, templates, api-keys, all of
crm (deals/pipelines/tasks). Each was an h1 + description + card
with slightly different paddings; now they're 1:1 PageHeader +
EmptyState. Switched the "primary" colour from zinc-900 to sky-600
to match the new chrome.
Real pages (3): emails, campaigns list, analytics.
- emails: stat strip reduced to <StatCard>×4, header moved to
<PageHeader>, table colours pulled from zinc to slate, accent
switched to sky-600.
- campaigns: same treatment; card hover tint sky-200 instead of
zinc-300; status dot colours kept (emerald/amber/slate).
- analytics: stat row + chart card + side panel + warmup empty
state all reflowed onto the same primitives.
Skipped on purpose:
- contacts (just delegates to ContactsTable; the table itself can
take a tonal pass later)
- unibox (custom chat-style split layout; PageHeader doesn't fit)
- campaigns/[id]/* (sub-pages live inside CampaignLayout which has
its own header; cleaner to redesign that layout once than each
sub-page individually)
Replaces the flat zinc sidebar + thin header with one continuous
sky-coloured chrome wrapping a clean white work surface. Reads as
"work happens inside a room with sky outside the window frame."
Layout shape (the L):
┌──────────────────────────────────────────────────────┐
│ [logo] > [org] > [section] [⌘K ●] │ AppHeader
├──────────┬───────────────────────────────────────────┤
│ │ ╭─── content (white, rounded-tl) ──────╮ │
│ AppNav │ │ │ │
│ │ │ │ │
└──────────┴───────────────────────────────────────────┘
Header and sidebar share one sky-gradient backdrop (SkyChrome). The
content panel tucks into the inner corner with rounded-tl-2xl and a
4px margin on right/bottom so a sliver of sky stays visible at every
edge except the seam — the inner corner is the only "merged" edge.
New components in components/layout/:
- SkyChrome: gradient + two soft blurred glows. No animation; the
auth page is where theatrical clouds live, the dashboard is
intentionally quieter.
- AppHeader: one-row breadcrumb spanning the full width. Warmbly
logo lives in the sidebar-width left zone, then an org-picker
button, then the URL-derived section/subpages. Connection
indicator + ⌘K search on the far right.
- AppNav: sidebar list, styled for the dark sky bg. Same section
grammar as before (Email, CRM, Resources) plus a primary
"New Campaign" action up top, settings + user menu pinned at
the bottom. Hover/active states use translucent white pills.
- AppShell: composes the three above, swaps in the keyboard
shortcuts modal + command palette, takes over from AppLayout.
OrgSwitcher and UserNav restyled for the dark chrome (white text,
faint white borders/hover states) and lost their dependency on
@/components/ui/sidebar — there's no SidebarProvider any more, the
new shell doesn't need one.
Removed components/layout/AppSidebar.tsx (the old zinc sidebar).
Existing per-page layouts (e.g. admin's tab bar) keep working — they
render inside the white content panel exactly as before.
Most container ports go back to their natural defaults — the offsets
that existed weren't justified, they just made URLs harder to remember.
Now standard:
backend 8080 (was always 8080)
tracking 3000 (was 13000)
realtime 4000 (was 14000)
web 5173 (was 15173 — already changed)
kafka 9092 (was 19092)
schema-registry 8081 (was 18081)
localstack 4566 (was 14566)
cloud-tasks 8123 (was 18123)
stripe-mock 12111 (always was)
Kept offset (the defaults conflict too often on real dev machines):
postgres 15432 (system postgres / sibling project)
redis 16379 (sibling docker projects with redis)
mailpit ui 18025 (sibling docker projects with mailpit)
mailpit smtp 11025 (same)
kafka-ui 18090 (8080 already used by backend)
Touched: docker-compose.yml, Makefile (test-seed SEED_TEST_DB), READMEs
(root + deploy), local-development.md + deployment-guide.md. Internal
docker-network refs (kafka:29092, mailpit:1025, etc.) unchanged — only
host-port mappings moved. Compose validated, all default-profile
services come up healthy on the new ports.
The backend's CORS allow-list resolution falls back through:
1. CORS_ALLOW_ORIGINS
2. APP_URL
3. origin derived from WEBSOCKET_URL
In compose only WEBSOCKET_URL was set, so allow-list landed at
http://localhost:14000 — the realtime port, not where the browser is
loaded from. The dev fallback that includes localhost:5173 only fires
when the list is empty, not when it's wrong-by-derivation. Result:
every POST to /auth/login etc. failed preflight with 403, which axios
surfaces as a generic "NetworkError" — login appeared to silently fail
after entering the password.
Setting APP_URL=http://localhost:5173 on the backend service makes the
allow-list match where the Vite dev server actually serves from. Sanity
checked with curl -X OPTIONS — preflight now returns 204.
The new admin API clients (audit, credentials, workers) imported Request
with four '..' segments instead of three. Vite's import-analysis failed
with "Failed to resolve import ../../../../Request" because that path
resolves to api/Request, not client/Request. tsc didn't catch it because
the resolver was permissive enough to keep going, but the runtime is
strict. Matched the existing pattern from roles/getRoles.ts (three dots
for Request, four for models).
Separately: web was on host port 15173, offset from the canonical 5173
to avoid colliding with a locally-running Vite outside Docker. Nobody
actually runs Vite locally in this setup, and the offset makes the URL
non-obvious. Moved back to 5173:5173 and updated VITE_APP_URL plus the
docs.
If a developer one day wants to run a host-side Vite alongside the
container, change the mapping back to "15173:5173" — the offset is the
escape hatch, not the default.
Four interlocking problems were causing the web container to spin in
its retry loop forever:
1. web/package.json aliased vite to "npm:rolldown-vite@7.1.14".
rolldown-vite is being deprecated (its own warning told us to use
7.3.1 for migration, or move to vite 8). vitest 4.x has a transitive
`vite` dep that pnpm tried to resolve against the public registry,
where vite@7.1.14 doesn't exist as a release (only 7.3.3 and 8.x).
Result: ERR_PNPM_NO_MATCHING_VERSION on every retry.
2. The "resolutions" block was meant to force the alias on transitive
deps. resolutions is Yarn syntax; pnpm doesn't read it. So the alias
wasn't propagating, which is exactly why (1) blew up.
3. pnpm 11 stopped reading the "pnpm" field in package.json. Settings
moved to pnpm-workspace.yaml. New file added with allowBuilds.esbuild
set so pnpm doesn't refuse to compile esbuild's native binary at
install time (it's transitively pulled in by vite + vitest).
4. The docker-compose web service had `until pnpm install; do echo
"pnpm install retry..."; sleep 3; done` which spins forever on
permanent dep-resolution errors and buries the actual message under
thousands of retries. Replaced with fail-fast that prints a hint
directing the admin to fix package.json and `make restart web`.
Also deleted the stale pnpm-lock.yaml so pnpm regenerates against the
fresh dep tree. Verified vite v7.3.3 boots, esbuild postinstall runs,
and Vite serves on http://localhost:15173 cleanly.
vite.config.ts has no rolldown-specific config, so the move from
rolldown-vite to plain vite is a no-op behaviourally.
make logs # everything, --tail=200 + follow
make logs backend # one service
make logs backend consumer # several
Same positional-args trick as `make restart`, reused. Ctrl-C to exit
the follow.
Two names for the same action was just clutter. `restart` is enough.
If you ever need to genuinely restart without rebuilding (container
restart that preserves the binary), `docker compose restart <svc>`
works directly — that's a rare enough case to not need a wrapper.
Previous attempt distinguished restart (no rebuild) from rebuild
(rebuild + restart). That distinction was useless in practice because
'docker compose restart' alone keeps the old binary — your code
change never appears. So every iteration was actually 'make rebuild',
and 'make restart' was a trap.
Collapsed both names into one behaviour. `restart` and `rebuild` are
aliases now; both do rebuild + restart, both take the service name
positionally:
make restart backend # was: make rebuild SVC=backend
make rebuild backend # same thing
make restart-go # all Go services
make restart-all # + Rust + Elixir
Positional argument plumbing via the standard Makefile trick:
captures non-target words after `restart`/`rebuild`, turns them into
no-op rules so make doesn't error.
If anyone genuinely needs the old container-restart-without-rebuild
behaviour (env var change, re-applying a migration the backend
already has), `docker compose restart <svc>` still works directly.
Documented that escape hatch.
Simpler than full hot reload for the Go side. The web service already
runs in dev mode (Vite HMR via the node container + ./web mount), so
frontend iteration was never the problem — only Go required a manual
docker rebuild + restart, which is a sequence everyone forgets.
Three new targets:
make restart SVC=backend restart without rebuild (config/env
changes, re-applying migrations)
make rebuild SVC=backend rebuild + restart one service
make rebuild-go rebuild + restart all Go services
(backend + consumer + worker)
make rebuild-all same plus tracking (Rust) + realtime
(Elixir) — the safe one when you've
touched things across stacks
local-development.md updated with an "Iterating on code" block so
this is discoverable.
Until now, only the web service hot-reloaded (Vite HMR via the
node:22-alpine container + ./web mount). The Go services (backend,
consumer, worker) used their production multi-stage Dockerfiles, so
every code change meant `docker compose build <svc> && docker compose
up -d <svc>` — ~30s per service.
Switched all Go services to a shared dev image (go.dev.Dockerfile)
that ships:
- full Go 1.25 toolchain on alpine
- CGO deps for librdkafka (gcc, musl-dev, librdkafka-dev, pkgconf)
- air v1.61.7 (the source watcher / hot-recompile tool)
docker-compose mounts the repo at /app and runs `air -c <config>`.
Each Go service has its own air.SERVICE.toml (build target +
exclusions). Named volumes for the Go module cache and build cache
so the first build is slow (~60s for module download) but subsequent
rebuilds after a save are ~2s.
Per-service compose changes:
- backend, consumer: dockerfile, volumes, and command updated
- worker-base (the YAML anchor used by all 3 workers): same
Production Dockerfiles in deploy/docker/{backend,consumer,worker}.
Dockerfile are unchanged and still used by release CI. The seed
one-shot in compose continues to use backend.Dockerfile (it's a
short-lived job, no benefit from the dev image).
Rust (tracking) and Elixir (realtime) still build-on-change. They
change far less often; documenting the workaround in
resources/local-development.md for now.
Two complementary axes for organizing the fleet, on one shared
mechanism:
User tags (workers.tags)
Free-form lowercase strings the admin applies for whatever they
care about — region (eu-west, fra), provider (hetzner, ovh),
role (warmup-only, burst-capacity), customer cohort. Edited via
a chip-style input with autocomplete from existing tags. Saved
to the worker_tags table.
Smart labels (computed client-side)
Auto-derived from the worker row so they're always in sync:
type:shared / type:dedicated
tier:free / tier:premium (shared only)
pool:clean / pool:risky / pool:quarantine (shared only)
state:installed / state:error / ...
ver:v1.2.3 (if image_version set)
liveness:online / stale / offline
Rendered with tone-aware backgrounds (red for offline / error
/ quarantine, amber for risky / stale, green for online).
Workers list:
- new Tags column showing user tags + the high-signal smart labels
(offline, error, risky, quarantine) with a "+N" overflow
- "filter by tag" chip strip above the table built from the
frequency of every tag (user + smart) in the current result. One
click filters; click again to clear.
Worker detail:
- Tags section near the top showing all smart labels and a full
TagEditor (chip input + autocomplete + suggestions dropdown +
Save button). Saving propagates to the list via react-query
cache invalidation.
The smart labels are never written to the database — they're
recomputed every render. Means renaming an enum value (e.g. risk
pool name changes) doesn't require a backfill.
Migration 000032 + repo + endpoints for arbitrary string tags on
workers. The fixed attributes (worker_type, free_tier, risk_pool)
cover the dimensions assignment logic uses. Tags cover everything
else admins want to group by: region (eu-west, fra), provider
(hetzner, ovh), role (warmup-only, burst-capacity), customer cohort —
whatever.
Schema:
- worker_tags(worker_id, tag) composite PK
- tag VARCHAR(64), lowercase + dashed via CHECK constraint
- ON DELETE CASCADE so deleting a worker drops its tags
Endpoints:
- GET /admin/workers/tags list distinct tags (autocomplete)
- PUT /admin/workers/:id/tags replace tag set; normalizes input
Repo:
- GetWorkerTags / SetWorkerTags / ListAllWorkerTags
- HydrateWorkerTags batch-loads tags onto a slice of workers in one
round-trip so the dashboard list doesn't do N+1 queries
PUT is transactional (delete + bulk insert) so the list view never
catches a worker mid-tag-swap. Auto-derived "smart" labels
(tier:free, pool:risky, state:error) are NOT stored — those are
computed client-side from the worker row so they stay in sync with
the source attributes automatically. Next commit wires the UI.
Replaces the flat /workers/new form with a five-step wizard that asks
"what's this worker for?" first and uses the answer to default everything
else. The previous form put every decision (worker_type, free_tier,
risk_pool, profile, owner) on screen at once with no guidance — fine if
you already know what you're doing, miserable otherwise.
Steps (Owner step skipped unless purpose=dedicated):
1. Purpose — shared / dedicated / risky-pool, with explanatory
cards. This drives the rest: risky → risky pool,
dedicated → unlocks step 4.
2. Connection — host/port/user. "Test reachability" button hits
the new TCP preflight endpoint before any row is
created — typos and firewalls fail loudly here
instead of at the SSH test stage later.
3. Identity — name (auto-derived from host on focus), notes,
profile, tier, risk pool. Risk pool defaults from
purpose but the admin can override.
4. Owner (deds) — user search via /admin/users + subscription ID.
Wizard remembers these and uses them in step 5.
5. Activate — review summary, "Install immediately" toggle
(default on), big Create button.
Post-create panel runs the full pipeline inline without leaving the
page when auto-install is on:
- Show pubkey + copy button + ready-to-paste ssh one-liner
- "I've pasted the key" checkbox unlocks Install
- Install button chains: Test → Install → (if dedicated) convert with
the previously-collected user/sub IDs → redirect to detail page
- Each step's outcome streams into a progress log
Progress dots at the top so the admin sees where they are. Empty-state
hint on step 1 when no workers exist yet.
POST /admin/workers/preflight {host, port} runs a 5s TCP dial against
host:port and returns ok + latency, or an error. Used by the worker
creation wizard to catch typos / firewall problems while the form is
still open — much better UX than discovering an unreachable VPS at the
SSH test step after the row already exists.
Doesn't attempt an SSH handshake (no credentials at this stage). A green
preflight just means "something is listening there." The actual SSH test
runs later, after the admin pastes the generated pubkey.
Worker list grows a Pool column (clean=green, risky=amber,
quarantine=red badge). Dedicated workers render "n/a" — risk pools are
a shared-worker concept since dedicated workers don't share IPs across
customers.
Worker detail page (shared workers only) gets a "Risk pool" section
with three big buttons. Clicking a non-current pool confirms, then
calls PUT /admin/workers/:id/risk-pool. Action audited with the new
pool value.
Saving doesn't migrate accounts directly — the hourly rebalancer
notices the mismatch and moves mailboxes to a matching-pool worker
on its next tick. Documented in the section's helper text.
Endpoint accepts {risk_pool: "clean"|"risky"|"quarantine"} and is
gated by AdminPermManageWorkers. The worker detail row scan now
includes risk_pool so the column actually has data.
New background job in the consumer process:
1. Pulls up to 1000 mailbox candidates joined with their worst warmup
health state (across all pools they participate in) and their
current worker's risk_pool. Dedicated workers are excluded — single
tenant, segregation not applicable.
2. Recomputes risk_band from health state via RiskBandFromHealth.
If it changed, writes the new band.
3. If the band's matching pool doesn't equal the worker's pool, picks
a new worker via SelectSharedWorkerForBand and migrates the mailbox.
Increments/decrements account counts.
4. Logs each migration to admin_audit_log with action=
"risk_rebalance_migrate" so operators see what moved and why.
Boot-time run + hourly ticker. Rebalancing is intentionally batch, not
event-driven: warmup health states change on a slow rolling-window basis
(warmup_health_sweep is also hourly), so reacting in real time gains
nothing and would cause thundering-herd migrations.
JobsService gets an AssignmentService dep. Nil disables the job (lets
self-hosters opt out by simply not wiring it).
New method on WorkerAssignmentService picks the least-loaded shared
worker whose risk_pool matches the mailbox's risk band. Three-step
fallback chain:
1. Exact match: same pool, same tier
2. Fall back to clean pool of the same tier when no matching-pool
worker is available (better to land risky mailboxes on clean
workers than refuse; the rebalancer will move them later)
3. Last resort: any worker of the right tier (preserves legacy
behavior for installations that haven't provisioned risky/
quarantine pools)
Existing SelectSharedWorker is unchanged so call sites that don't
know about risk bands keep working. The next commit (background
rebalancer) is the first consumer of the new method.
Threat-level segregation, schema layer. Two new concepts:
workers.risk_pool ∈ {clean, risky, quarantine}
buckets shared workers by acceptable risk. Dedicated workers don't
use it (single tenant = no cross-contamination risk).
email_accounts.risk_band ∈ {clean, risky, quarantine}
per-mailbox classification, derived from warmup_health_state by the
rebalancer (next commit). Never set by user input.
The mapping is one-way and intentionally simple:
healthy → clean
watch, throttled → risky
quarantined, → quarantine
blocked
Rebalancer code lands in the next commit. This commit just adds:
- migration 000031 with enums + columns + filtered indexes
- WorkerRiskPool / EmailRiskBand types + RiskBandFromHealth helper
- WorkerRepository methods: SetWorkerRiskPool, SetEmailAccountRiskBand,
GetSharedWorkersByTierAndPool, ListRiskCandidates
- RiskCandidate result type joining email_accounts + warmup health
(picks WORST state across pools via CASE ranking) + worker columns
so the rebalancer can decide migrations in one scan
POST /admin/workers/:id/convert-to-dedicated does three things in
sequence:
1. Drain existing accounts to a supplied drain_to_worker_id (required
if the source has any accounts; we don't auto-pick per-account
targets because the right choice depends on each account's
owning org).
2. Flip workers.worker_type from "shared" to "dedicated".
3. Atomic create of dedicated_worker_assignments binding the worker
to a specific user/subscription (uses the existing
CreateDedicatedAssignmentIfNotExists so re-running is safe).
Refusal cases:
- already dedicated → 400
- has accounts but no drain target → 400 (admin must pick where they go)
- drain target equals source → 400
Worker detail UI gains a "Convert to dedicated" section, shown only
when the worker is currently shared. Inline form, no modal. The drain
dropdown excludes self, only lists shared+installed workers, sorts
least-loaded first.
Audit-logged with action="convert_to_dedicated" and the user_id,
subscription_id, drain target, and account count in details.
The free-tier-vs-paid separation in AssignWorkerToEmail is one of those
rules that's silently load-bearing: if a free org ever slips onto a
premium worker, the IPs of paying customers absorb the deliverability
hit. The code is correct today (strict isPaidOrg check at line 67, free/
premium pool sync at line 116), but nothing was guarding against a
regression.
Five table-thin tests, hand-rolled stub repos (embed the interface as a
nil field so unused methods panic loudly):
- free org → free shared worker → free warmup pool
- paid org → premium shared worker → premium warmup pool
- paid org with DedicatedWorkers > 0 + an assignment → dedicated worker
- paid org with DedicatedWorkers > 0 but no assignment → falls back to
premium shared (not free!)
- SelectSharedWorker with no workers → ErrNoAvailableWorkers
No code changes — this commit is documentation.
Worker detail page gains a "Move accounts to another worker" section
with a dropdown of eligible targets (same tier, currently installed,
sorted least-loaded first). One click moves every email account on the
current worker to the picked target via the existing AdminReassignEmails
endpoint.
Useful when:
- a worker is down and you want to shift its workload to a healthy
sibling while you investigate
- you want to drain a worker before uninstalling it
- a profile-level change forced too much onto one worker and you want
to rebalance manually
The endpoint emits the standard audit log row (admin_user_id + action
"reassign"), so every manual rewire is visible in the audit viewer.
WorkerHealthAlert lives in the admin layout and renders on every admin
page when any worker needs attention. Polls the workers list every 30s
in the background. Hidden on the workers page itself (which has its own
richer banner with filter chips).
Surfaces three categories:
- errored: install_state == "error"
- offline: installed + no heartbeat for 5+ minutes
- in progress: pending / provisioning / uninstalling
Tone goes red when there are errored or offline workers, amber otherwise.
A "review →" link deep-jumps to /app/admin/workers.
New /app/admin/audit page browses admin_audit_log with:
- Filter by action, target_type, target_id, admin_user_id, date range
- Action + target_type dropdowns are populated from a baseline set
AND from whatever appears in the current result (so new actions
surface automatically without code changes)
- Auto-refresh toggle (5s) for live tailing during operations
- Cursor-based pagination with Prev/Next + page counter
- Expandable rows show the details JSON + user-agent
- Color-coded actions (red for destructive, green for create/install,
amber for system auto-actions)
- Renders system actions (admin_user_id = uuid.Nil) as "system"
- Renders the admin's name + email when joined data is present
When the dead-worker job reassigns email accounts from a worker whose
heartbeat expired, write a row into admin_audit_log so the dashboard's
audit viewer surfaces these system actions alongside admin-driven ones.
admin_user_id is uuid.Nil (the platform identity), so admins searching
the log can distinguish "system did this" from "an admin did this" by
filtering on that ID. Details include the replacement worker, account
count, and reason.
JobsService gets an optional AdminRepo dep. Nil disables logging — keeps
the contract loose for any other call site that doesn't have one.
The audit calls added in the last commit went to AuditService.LogAction,
which writes to the general user-facing audit log (Cassandra). The admin
audit-log viewer at /admin/audit-logs queries the admin_audit_log table
in Postgres, so worker / credentials / release actions never showed up.
Add a public AdminService.LogAdminAction that wraps the existing private
logAction (writes to admin_audit_log with the same shape as ban_user /
update_worker / etc.). Repoint h.audit() at it.
Actions now visible in the audit viewer:
test, install, restart, upgrade, uninstall, rotate_keys, apply,
assign, system_update, reboot, check_releases (plus the existing
create/update/delete across workers, AWS creds, and profiles).
The seeder is one of the few things every developer runs on every new
checkout, but it had zero tests. With three migrations added in the last
few days and the rich-fixture path now creating 30+ rows, the chance of
silently breaking a schema migration without noticing was non-trivial.
cmd/seed/main_test.go connects to SEED_TEST_DB (skips otherwise — keeps
unit tests in CI fast and prevents accidentally clobbering a dev
database), runs migrations, wipes only the fixture rows, then:
1. Runs seedBaseline twice, verifies row count stays at 1.
2. Runs seedRich, asserts 9 different row counts match expectations
(users, orgs, workers, accounts, campaign, sequences, contacts,
unsubscribed contacts, campaign leads).
3. Re-runs seedRich, asserts every count is unchanged — the most
important guarantee the seeder makes.
4. Verifies warmup pool membership: 2 free, 4 premium, with the
correct accounts in each.
Plain testing package, table-driven, matches the existing style in
internal/app/warmup/service_test.go.
`make test-seed` brings up the docker-compose Postgres and runs the
suite against it.
Workers list now computes a health summary client-side from the existing
queries and shows a yellow banner with a count when any worker needs
attention (errored, offline, or mid-install). Filter chips below the
banner narrow the table by problem category so the admin can drill in
without opening each worker.
Categories surfaced:
- errored: install_state = "error"
- offline: installed, but no heartbeat in 5+ minutes
- in progress: pending / provisioning / uninstalling
- stale config: profile.updated_at > worker.config_applied_at
- update available: profile.resolved_image_tag != worker.image_version
Live polling stays at 15s, so a worker going offline shows up within
~75s of the last missed heartbeat. The list also fetches profiles so the
stale-config and update-available chips can compute without extra
round-trips.
The new admin endpoints for SSH worker management, AWS credentials,
worker profiles, releases, and system operations bypassed the existing
audit-logging pattern. Now every mutating action records a row in the
audit log with adminID, IP, user-agent, and operation-specific metadata
(never secret values — for credential updates we record which fields
were rotated, not what they became).
New action constants:
test, install, restart, upgrade, uninstall, rotate_keys, apply,
assign, system_update, reboot, check_releases
New entity types:
worker, aws_credentials, worker_profile, release
Read-only endpoints (list/get/status/logs) intentionally not audited —
they don't change state and would flood the log.
Each handler calls a small h.audit(c, action, entity, &id, metadata)
helper that pulls adminID from the admin middleware and fires the
existing AuditService.LogAction (fire-and-forget, never blocks the
response).
Workers heartbeat into Redis every 90s as RFC3339 timestamp values with a
3-min TTL. The dashboard surfaces liveness based on workers.last_seen_at,
but until now nothing populated that column — the "Live" badge was always
red.
New 60s job in the consumer reads each active worker's Redis heartbeat
value, parses the timestamp, and writes it to workers.last_seen_at. Runs
on its own interval (separate from the 5-min dead-worker detection job,
which does heavier reassignment work) so the UI sees fresh data within a
minute.
Old docs described a k8s/ArgoCD/Terraform deployment that no longer
exists, with ASCII-art system diagrams that hadn't aged well. Rewritten
to match how the project actually ships:
- README: control plane (Railway) + execution plane (per-VPS workers)
split, dashboard-driven worker management, credentials/profiles,
auto-update from GitHub releases, OS package updates, self-hosting
knobs. Removed all ASCII art.
- resources/architecture.md: control vs execution plane, encryption
model (worker SSH keys + platform secrets under the same KMS-envelope
cipher as user secrets), worker identity from public IPv4, credentials
model, push-driven release flow, anti-abuse layers, source anchors.
- resources/deployment-guide.md: end-to-end from "provision a VPS" to
"auto-update on release". No more k8s, ArgoCD, kubectl, or Terraform.
Step-by-step backend env, webhook setup, worker add flow, day-2 ops,
rollback per plane.
- resources/local-development.md: the five make targets (dev / sim /
seed / tools / reset), what each profile runs, LocalStack bootstrap,
rich seed contents, native dev against containerized infra, the
offset-port URL table.
- resources/cicd.md: the two-plane build/release flow, image tag scheme
({sha} / dev / vX.Y.Z / vX.Y / vX / prod), webhook setup, release
process, security notes around HMAC and least-privilege worker AWS
keys.
- deploy/README.md: tight version of the same.
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Hoist the dev/sim stack to a single docker-compose.yml at the repo root.
Adds profiles (default / sim / seed / tools) so you can opt into heavier
setups, and bundles dependencies that were previously missing:
- LocalStack (KMS + DynamoDB + S3) with a localstack-init one-shot that
idempotently creates alias/master-key-dev, the UserEncryptedKeys and
EmailMessageData tables, and the main S3 bucket. Backend and workers
wait on it via service_completed_successfully.
- stripe-mock for billing flows
- kafka-ui under the tools profile
Three workers with deterministic UUIDv5 hostnames (shared / premium /
dedicated) so assignment, rebalancing, and per-pool routing all have
real targets to exercise.
Richer seed (cmd/seed/main.go) loads 3 orgs across tiers, 6 mailboxes
joined to free/premium warmup pools, a Beta campaign with a 2-step
sequence, and 10 contacts (2 unsubscribed) so suppression behaviour is
visible in the UI. Idempotent — safe to re-run.
Makefile targets:
make dev — infra + app + one worker
make sim — adds premium + dedicated workers
make seed — rich fixtures
make tools — kafka-ui at :18090
make reset — nuke volumes
scripts/install-worker.sh is a single bash script any Debian/Ubuntu/RHEL/
Fedora/Arch/Alpine VPS can curl|sh to add a worker to the fleet.
Identity is bound to the VPS's public IPv4 via UUIDv5 (URL namespace):
same IP → same worker (reputation persists across reinstalls)
new IP → new worker (fresh identity, no inherited reputation)
The installer detects the public IP via api.ipify.org / ifconfig.me /
checkip.amazonaws.com, derives the deterministic UUID, installs Docker if
missing, writes /etc/warmbly/worker.env (0600) and /etc/warmbly/worker.id,
installs a systemd unit that runs the worker container with --hostname
<uuid>, and starts the service.
Supports --install/--update/--uninstall/--purge/--status, --env-file for
non-interactive config, --ip override, --image override, and a full set
of per-credential flags.
Worker reads its UUID from os.Hostname() at startup, so the systemd
hostname value becomes the worker identity — no separate registration
step needed.
Matthew Meszaros