User flagged the old folders popup as "fucking bad" and out of theme.
It was the legacy ModalBase + ModalSplit + ModalDnd + ModalBox stack
with poppins serif, blue accents, big illustration columns and long
description paragraphs — nothing matched the rest of the chrome.
Replaced with a single LabelListModal primitive that both Folders
and Tags use:
- Center modal 480px wide, max-h-80vh, brae chrome.
- 48px header band: eyebrow + subtitle + close.
- Hairline row per item; hover reveals edit + delete.
- Inline edit swaps the row with a color popover (8-swatch
palette) + title input + Save/Cancel inline.
- "+ New folder" / "+ New tag" footer that toggles into an
add-row inline (Enter to submit, Esc to cancel).
- Slate-900 primary on Save and Done; red on delete confirm.
- Confirm.show() prompt before destructive delete.
FoldersModal + TagsModal both shrank from ~160 lines of legacy modal
plumbing each to ~45 lines that hand the LabelListModal callbacks
hitting the existing folder/tag client functions and updating the
cached User in react-query directly. No more id-scoped hooks per
row.
Emails (Accounts) page filter dropdown:
- Was using the legacy HeadSelectMenu + SelectOption stack
(animated scale popover, blue check, off-theme typography).
- Was also looking up the selected tag from `user.folders`
instead of `user.tags` — a stale bug from before, where the
folder list rendered in the tags dropdown.
- Replaced with PopoverMenu / SelectButton (same primitive used
on campaigns + contacts). Now pulls from `user.tags`,
"Manage tags" entry opens TagsModal as expected.
- Search field switched to the standard SearchInput so it
matches the 28px hairline-border styling everywhere else.
Other dead clicks:
- InboxDetails "Cancel" button was onClick={() => {}}. Now
closes the panel (setView("")).
Audited every visible button across the dashboard. Most were rendered
with no onClick — clicking them did nothing and there was no signal
that the action was unreached. Fixed in two passes:
Real wiring (already had hooks behind them):
- Campaigns:
* New campaign → opens NewCampaignDialog (useCreateCampaign,
navigates to the new campaign on success).
* Folders → setFoldersEdit(true) (the existing FoldersModal).
* Sort dropdown → backs by sort state (newest / oldest / name);
list re-orders client-side from useMemo so we don't pay another
fetch.
* Row pause/play→ useStartCampaign / useStopCampaign behind a
confirm.show() prompt; toast.promise surfaces status.
* Empty-state "New campaign" → same dialog.
- Contacts:
* New contact → NewContactDialog (useAddContacts, single-row).
* Export → client-side CSV from the loaded page, downloads
a contacts-YYYY-MM-DD.csv with the standard columns.
* Embedded "Add lead" inside campaign leads view → same dialog.
- Emails:
* Fire (warmup) row icon → now opens the inbox detail panel; was
a no-op button.
New brae-density dialogs:
- NewCampaignDialog: center-aligned modal, 48px header band,
hairline footer, slate-900 primary. Name + description fields.
- NewContactDialog: same chrome, email (required) + first/last/
company/phone. Toast.promise feedback.
Placeholder wiring for surfaces whose backend or flow isn't built yet:
- Templates / API keys / CRM (deals, pipelines, tasks) / Team /
Billing upgrade / Settings save → all surface a clear "X is
coming soon." toast (icon 🚧) via the new comingSoon() helper.
Clear signal that the click registered, no more silent dead
buttons.
- Contacts "Import CSV" surfaces the same coming-soon notice
(export ships, import is the harder path).
Refactor:
- web/src/lib/helper/comingSoon.ts — tiny shared toast helper so
each placeholder doesn't reinvent the wording.
- Settings page now displays the actual user email instead of a
placeholder string.
- Billing "View all plans" anchor is now a real Link to /#pricing.
The campaigns route children only had {path: ":id"} — no index entry —
so /app/campaigns matched the parent but rendered <Outlet/> with no
child, producing react-router's:
Matched leaf route at location "/app/campaigns" does not have an
element or Component. This means it will render an <Outlet /> with
a null value by default resulting in an "empty" page.
That's why the page appeared blank no matter what the backend returned.
Imported the list page and added it as the index child.
The realtime channel handler `def join("user:" <> user_id, ...)` checks
`socket.assigns.user_id == user_id`, where socket.assigns.user_id is
the JWT `sub` claim (UUID). The frontend was building the topic from
user.email — every join was REFUSED.
Added `id: string` to the frontend User type (the backend already
serializes it as "id") and switched the channel topic in
RealtimeManager to use it.
After this + the previous round of WS fixes:
CONNECTED TO RealtimeWeb.UserSocket in 481µs
JOINED user:11111111-0000-0000-0000-000000000001 in 15µs
1) Contacts crash "c is null":
contactRepository.Search declared `var contacts []models.Contact` so
an empty result set returned a nil slice, which Go marshals as JSON
null. The frontend's flatMap((p) => p.data) over null yields [null],
and the page then accesses c.subscribed → throws. Initialize as
make([]models.Contact, 0, limit+1) so the wire format is always [].
Also defensive on the client: useSearchContacts + useCampaigns now
coerce p.data ?? [] and drop nulls before returning.
2) Campaigns panic on any non-empty result:
campaignRepository.Search allocated `make([]models.Campaign, 0, limit+1)`
(length 0) then did `campaigns[i] = campaign`. That's an
index-out-of-range on the first iteration. Switched to `append`.
Anyone with at least one campaign would see a 500 / blank screen.
3) Websocket "Token expired":
SocketTTL was 60s. The frontend reconnect backoff caps at 30s, so
after a rejected handshake the next attempt could fire 30-60s
later. Combined with rare back-pressure on /getaway the token was
already past exp by the time the realtime saw it. Bumped to 10 min
— short enough to keep the token low-impact, long enough to outlast
the backoff schedule.
4) Websocket "Connection limit exceeded":
Realtime.Connections only untracked on channel terminate, never on
socket disconnect. Sockets that connected and disconnected without
joining a channel leaked. Each reconnect loop bumped the counter
until the per-user limit (10) was hit, after which every legitimate
connect was rejected even after fixing #3.
Fix: GenServer Process.monitor's the socket pid on track, and
`:DOWN` handler calls do_untrack with the right (user_id, ip).
5) Phoenix protocol mismatch:
Frontend appended vsn=2.0.0 to the WS URL, but sendRaw + joinChannel
send the V1 object format. Realtime's Phoenix.Socket.V2.JSONSerializer
crashed with a badmatch on the first phx_join, killing the socket
right after connect. Switched to vsn=1.0.0 to match what the client
actually emits.
Backend root cause:
The frontend client omits ?limit= when it would equal the default
(DEFAULT_PAGINATION_LIMIT = 50). validate.Limit("") treated empty as
invalid and returned errx.ErrLimit → 400 on /contacts/search and
/campaigns. ContactsTable derived isLoading from `!contacts`, which
stays true forever when the query errors, so the page hung in the
skeleton state instead of surfacing the error.
Fix:
- validate.Limit now accepts "" and returns LimitDefault = 50, in
sync with the frontend constant. The frontend's omission semantics
("don't send the param when it's the default") was already correct;
it was the validator that was wrong.
Frontend:
- ContactsTable: use isPending/isError/refetch directly from react-query
instead of deriving from `contacts`. New explicit error block renders
inside the body with: red alert tile, server error message, Try-again
button (with spinner during refetch), and Reload-page fallback.
- Campaigns page: same error UI promoted from the old EmptyBlock CTA
to a prominent block — alert tile + message + retry + reload.
New ContactFilters sheet (was the legacy 800px poppins drawer):
- 420px right-side panel matching the rest of the theme.
- Sticky 48px header with "Filters · N active" eyebrow + close.
- Sticky 48px footer with Reset / Cancel / Apply (slate-900 primary).
- Hairline-divided SectionBars between groups: Search, Custom field
filters, Sort, Subscription, Campaign membership, Dates.
- Custom field rows pair TextInput + FILTER_TYPES popover + value
input + remove button — all 28px tall.
- Sort: SelectButton popover + asc/desc toggle.
- Subscription: 3-state pill toggle (Any / Subscribed / Unsubscribed).
- Min/max campaign rows: checkbox toggle + number input + suffix.
- Date rows: checkbox toggle + native date input.
- Draft state mirrors parent until Apply, so editing filters doesn't
trigger refetches mid-build.
Reliability:
- ErrorBoundary wraps every route. Silent white pages are gone — any
uncaught render error now surfaces inside the panel with name,
message, stack toggle, and back/retry buttons.
- Boundary keys on pathname so navigating away clears the error.
Campaigns blank fix:
- Drop the legacy HeadSelectMenu + Search components from the page
(suspected layout/click-outside collisions inside the slim SectionBar).
- Rewrite using the new dropdown + SearchInput primitives.
- Add StatStrip with clickable filters (All / Active / Paused / Draft).
- Loading shows skeleton rows. Empty splits between "no campaigns at
all" vs "no campaigns matching the current filter".
Dropdown primitive (web/src/components/ui/popover-menu.tsx):
- Brae-density popover menu — slim trigger, hairline border content
surface, h-7 items, mono kbd accents. Built from scratch rather than
via Radix so styles are authoritative and bundle stays small.
- Click-outside + Esc handling, controlled/uncontrolled open state,
side ("bottom"|"top") + align ("start"|"center"|"end") positioning
with viewport-aware clamping.
- SelectButton helper styled as a brae trigger pill.
Field primitive (web/src/components/ui/field.tsx):
- SearchInput + TextInput + Label. 28px tall, hairline border,
sky focus ring. Replaces the half-dozen ad-hoc inputs across pages.
- SearchInput supports Enter onSubmit and inline clear button.
Contacts browser (ContactsTable.tsx, rewritten in place):
- Standalone view: PageTopbar (Import / Export / New) + StatStrip
(All / Subscribed / Unsubscribed / In campaigns, clickable filters)
+ SectionBar (search + sort dropdown + filters).
- Embedded view: skips topbar/strip, drops into SectionBar — used
inside /app/campaigns/[id]/leads.
- Dense table with avatar + name + email-mono, optional company /
phone columns (hidden on smaller widths), subscribed/unsubscribed
pill, campaigns count, created date.
- Bulk selection floats a footer bar with Edit / Delete / Clear.
- Load-more button for infinite scroll (preserves the existing
useInfiniteQuery hook).
- Sort dropdown wired to the existing SearchContacts API params.
Unibox email browser:
- ConversationList: SectionBar header with count, SearchInput, all /
unread tabs with unread count badge, dense rows with avatar +
bold-when-unread sender + subject + preview + relative time.
Unread items get a thin sky rail on the left margin.
- ConversationItem: relative time formatter, name extraction from
"Name <email>" headers.
- ThreadView: 48px topbar (subject + mark-unread/archive/delete
actions) + section bar (n messages / k participants) + a divided
message stream + composer pinned to bottom.
- MessageBubble: no card chrome; just hairlines between messages.
Sender avatar + bold name + mono email + recipient line + mono
timestamp; prose-rendered body.
- ReplyComposer: edge-to-edge textarea with footer bar (Send,
Schedule popover with "in 1h / tomorrow 9 / next Mon 9", Discard,
char counter). ⌘+Enter to send.
Sidebar:
- Drop the generic "+ New Campaign" sky pill. Replaces it with a
LivePanel that reads ambient cold-email telemetry: status dot
(connected/idle/offline), mailbox count, active count, and a
daily sparkline placeholder. Clicks through to /app/analytics.
The sidebar now reflects what the system is doing rather than
nagging with a CTA.
- Nav rows shrink from h-8/13px to h-7/12.5px to match brae density.
- Section labels switch to small tracked-uppercase with a hairline
divider above each section instead of a margin gap.
Page primitive (web/src/components/layout/Page.tsx):
- New vocabulary: Page > PageTopbar (sticky h-12 with eyebrow +
subtitle + actions) + StatStrip (full-width stat row with vertical
rule dividers and Stat cells) + SectionBar (h-9 sub-header) +
PageBody (scrollable area) + Row + EmptyBlock + TopbarAction.
- Pages fill the entire content panel edge-to-edge. No max-w
ceilings, no centered narrow columns, no rounded card chrome,
no Georgia serif. Tracked-uppercase eyebrow at 10px replaces the
28px serif h1.
- Old PageHeader, StatCard, EmptyState, PageSection live on as
thin shims so any page not yet swept keeps compiling.
Pages swept:
- campaigns: list view becomes a hairline-divided row stream with
status dot + mono ID + status pill + relative date. Loading
shows skeleton rows, error and empty share the same EmptyBlock.
- emails: 4-card stat grid → StatStrip; table edge-to-edge with
sticky thead, hover row, uppercase status label.
- analytics: card grid → StatStrip + a 2-column body with
vertical-rule between the chart and the breakdown rail.
- templates, api-keys, billing, settings, team, crm/{tasks,
pipelines, deals}: PageTopbar + EmptyBlock, no more sky pills.
The Logo SVG was switched from hardcoded fill="white" to fill="currentColor"
so the dashboard could tint it gray. Auth-layout usages were never updated
and got rendered with the inherited default text color (near-black) on the
dark sky panel. Pass text-white explicitly at both logo sites.
Outer shell:
- AppShell: drop SkyChrome backdrop and the rounded-tl content tuck.
L-shape becomes side-by-side: white sidebar (220px) + white content
column with a hairline #e2e8f0 divider. No decorative background.
- SkyChrome: now a no-op stub. Kept as a named export so the import
path stays stable if we ever bring decoration back.
- AppNav: drop the "New Campaign" sky pill. Sidebar opens straight
into the nav tree — logo + org switcher in a slim 44px header row,
28px nav rows, sentence-case section labels, hairline borders top
and bottom. Active row = bg-slate-100 + slate-900 text. No shadows.
- AppHeader: 40px breadcrumb strip in the content column (was a
full-width 56px row spanning the shell). No logo here — it lives
in the sidebar header.
- OrgSwitcher: slate-900 initial tile instead of sky-600. Sized to
match the slimmer sidebar (28px tall, 12.5px text).
- UserNav: smaller 24px avatar, tighter row height, slate-100 hover.
Page primitive:
- Page padding: px-4 pt-3 pb-10 (was px-8 pt-8 pb-16).
- PageHeader: single 36px toolbar row — title + subtitle on the same
baseline, actions on the right. Drops the 28px serif title block
and the dedicated subtitle paragraph.
- PageSection: 28px header row with title (12.5px) and inline desc.
- StatCard: redesigned as a divided strip cell. Use inside StatRow
for a row of stats with vertical-rule dividers and a single outer
border, replacing the four floating shadow-tile cards.
- EmptyState: text-led, dashed-border block, 40px tall icon.
Page sweep:
- All /app pages: sky-600 buttons → slate-900 (h-7 px-2.5 rounded-md,
text-[12.5px]). Same height as the new toolbar so the page header
reads as one line.
- emails + analytics: switch StatCards into the new StatRow.
- emails: slate avatar tile instead of sky, slate accent on checkboxes.
- campaigns: slate card chrome instead of sky-tinted hover.
- analytics: slate bars instead of sky, smaller paddings, breakdown
list reads as a tight column instead of a card.
- AppNav: remove "Home" row that pointed at /app/emails (same as Accounts),
which is why two rows lit up together. Accounts is the de facto home.
- AppNav: active row goes from white pill (lighter than chrome) to bg-slate-200
pill (clearly darker than chrome) — a gray step down, not the dark pill from
the previous iteration.
- AppHeader: logo gets a blue lean (#8aa1c1 → #4e6285 on hover) so the brand
shows through without coloring the whole chrome.
- SkyChrome: base from #f4f7fb to #f5f6f8, drop the bottom sky vignette.
Clouds stay; the rest is neutral so the only color cue is intentional.
Reverting the AI-design-system flourishes in the chrome:
- Logo: text-slate-400 at rest, text-slate-700 on hover. Lighter
gray that warms slightly when you mouse it. Drops the sky-600.
- Sidebar nav: brought back the white-pill active state from
before (rounded card with hairline ring + 1px shadow). The
sky-50 + 2.5px rail experiment is gone — it read as a design
system showing off, not a workspace.
- Sidebar hover: bg-white/70 (almost invisible) → bg-slate-200/60.
Now hover actually registers — the row goes visibly grayer.
- Badge: sky-600 → slate-900. Same logic, no accent shouting in
the chrome.
Nothing outside components/layout/{AppHeader,AppNav}.tsx changed.
Four targeted upgrades to actually deliver on "have some taste":
1. Logo: the Logo SVG had fill="white" hardcoded. The moment the
shell flipped from dark sky to light, it disappeared. Switched
to fill="currentColor" so the className drives it (now sky-600
in the header, slate-600 elsewhere).
2. PageHeader: titles now render in Georgia serif (already declared
as --font-serif in global.css). 28px / 500 / -0.01em tracking
in slate-950. Subtitle moved to 13.5px with a max-width to keep
it readable. The serif voice is the same one the auth page
uses for "Your emails deserve the inbox." — gives the dashboard
the same character without explicit branding.
3. StatCard: hard border replaced with a ring-1 + two-layer shadow
(a 1px tight + a 24px diffuse). Cards now read as "sitting on"
the page rather than "drawn on" it — Linear-style elevation.
Number bumped 22→28px, weight 600→500, tracking tightened to
-0.02em. Label flipped to a UPPERCASE 12px caption.
4. EmptyState: rounded-2xl + ring instead of dashed border. Title
in serif to match PageHeader. Soft-tinted bg (#fafbfd) so the
block reads as a quiet placeholder rather than a "no data!"
shout. Icon tile gets a faint shadow.
5. Sidebar active rows: traded the white-pill chrome for a sky-50
tint + a 2.5px sky-500 rail flush against the left edge. Reads
as confident but quiet — the active state announces itself with
colour, not weight.
Also: bumped Page padding 6→8 horizontally, 12→16 bottom; widened
PageHeader bottom margin 6→8. Pages breathe.
Scrapped the deep sky gradient. The dashboard is workspace, not
cinema — the user lives there for hours, the chrome shouldn't compete
with the content.
New shell aesthetic:
- Backdrop (SkyChrome): a near-white sky-tinted base (#f4f7fb),
one degree cooler than the white content panel so the panel
reads forward. Three blurred-white cloud blobs in the upper
half at very low opacity — atmosphere you feel rather than see.
A whisper-faint sky-tint vignette at the bottom. No animation.
- Content panel: pure white, only rounded at the inner corner
(top-left). Flush to the bottom and right edges of the viewport
— no margin band of visible chrome there. A single hairline
border on the top + left edges defines the panel without a
heavy shadow.
- Header: dark text on the light backdrop. Logo in sky-600 not
white. Wordmark slate-900. Chevrons slate-300, crumb text
slate-500/900. Connection indicator and ⌘K search drop their
translucent-white pills for slate hovers.
- Sidebar (AppNav): same light backdrop. Active rows go to a
white pill with hairline ring + 1px shadow (the row "lifts"
off the sidebar). Inactive rows slate-600. New Campaign
reverses to a confident sky-600 pill. Section labels
slate-400 uppercase 0.16em tracking, no longer fighting for
attention.
- OrgSwitcher + UserNav: light-theme triggers — slate-200 hover,
sky-600 + slate-900 avatar tiles, slate-900 names.
Also: campaigns page used to render a forever-skeleton when the
query errored. Switched to checking isLoading / isError explicitly
and added a retry button via EmptyState so the failure mode actually
shows up instead of looking like a slow load.
Six changes to the dashboard chrome, all small individually, together
the shell feels intentional instead of "okay first pass".
Sky gradient (SkyChrome):
- Switched to a directional 135deg gradient that's rich in the
upper-left (where the logo sits) and brightens toward the inner
corner near the content panel. Light pulls the eye to the work.
- Warm bloom moved from the upper-left to the inner corner with
cream/butter undertone — reads as late-sun reflecting off the
white panel back into the chrome.
- Added a cool slate wash over the dark zone to give it depth
without changing apparent hue.
- Hairline noise overlay (inline SVG feTurbulence, no asset) so
the gradient doesn't read as plastic at 4K.
Content panel (AppShell):
- Margin from 4px to 14px. Reads as a deliberate sky window-frame
rather than a near-miss border.
- rounded-tl-2xl → rounded-tl-3xl. Bigger radius = architectural,
not accidental.
- Soft 60px outer drop shadow + an inset top-edge highlight, so
the panel feels suspended in the sky chrome rather than glued
flush against it.
Header (AppHeader):
- h-12 → h-14, logo 28→32px, wordmark 15→16px. Brand has presence.
- Logo gets a 1px-y drop shadow so it sits above the gradient.
- Chevrons: w-3.5→w-4, white/30→white/45. Visible without being loud.
- Crumb text 13→13.5px, inactive segments white/55→white/65.
Sidebar (AppNav):
- "New Campaign" reverses contrast — white pill with sky-700 text
instead of a faint translucent button. Reads as a primary CTA,
not a quiet link.
- Nav rows: text 12.5→13px, inactive white/65→white/75, active
state gets a subtle inset highlight. Hover bg pop +33%.
- Section labels weight 500→600, opacity 40→50, tracking widened
from 0.16em to 0.18em. Quiet but legible.
- Badge gets a small shadow so it stands off the white pill.
Added a small set of layout primitives every page can pull from:
<Page> outer container with width=default|wide|full + padding
<PageHeader> title + optional subtitle/eyebrow + right-side actions
<PageSection> labeled child block with optional actions
<StatCard> icon + label + big number tile
<EmptyState> icon + headline + supporting line + optional CTA
Every page that opted in now reads with the same vocabulary — the
title sits at the top-left of the white content panel with consistent
breathing room, primary actions float to the right of the header,
empty states use the same dashed-card pattern.
Pages refactored:
Stubs (10): billing, settings, team, templates, api-keys, all of
crm (deals/pipelines/tasks). Each was an h1 + description + card
with slightly different paddings; now they're 1:1 PageHeader +
EmptyState. Switched the "primary" colour from zinc-900 to sky-600
to match the new chrome.
Real pages (3): emails, campaigns list, analytics.
- emails: stat strip reduced to <StatCard>×4, header moved to
<PageHeader>, table colours pulled from zinc to slate, accent
switched to sky-600.
- campaigns: same treatment; card hover tint sky-200 instead of
zinc-300; status dot colours kept (emerald/amber/slate).
- analytics: stat row + chart card + side panel + warmup empty
state all reflowed onto the same primitives.
Skipped on purpose:
- contacts (just delegates to ContactsTable; the table itself can
take a tonal pass later)
- unibox (custom chat-style split layout; PageHeader doesn't fit)
- campaigns/[id]/* (sub-pages live inside CampaignLayout which has
its own header; cleaner to redesign that layout once than each
sub-page individually)
Replaces the flat zinc sidebar + thin header with one continuous
sky-coloured chrome wrapping a clean white work surface. Reads as
"work happens inside a room with sky outside the window frame."
Layout shape (the L):
┌──────────────────────────────────────────────────────┐
│ [logo] > [org] > [section] [⌘K ●] │ AppHeader
├──────────┬───────────────────────────────────────────┤
│ │ ╭─── content (white, rounded-tl) ──────╮ │
│ AppNav │ │ │ │
│ │ │ │ │
└──────────┴───────────────────────────────────────────┘
Header and sidebar share one sky-gradient backdrop (SkyChrome). The
content panel tucks into the inner corner with rounded-tl-2xl and a
4px margin on right/bottom so a sliver of sky stays visible at every
edge except the seam — the inner corner is the only "merged" edge.
New components in components/layout/:
- SkyChrome: gradient + two soft blurred glows. No animation; the
auth page is where theatrical clouds live, the dashboard is
intentionally quieter.
- AppHeader: one-row breadcrumb spanning the full width. Warmbly
logo lives in the sidebar-width left zone, then an org-picker
button, then the URL-derived section/subpages. Connection
indicator + ⌘K search on the far right.
- AppNav: sidebar list, styled for the dark sky bg. Same section
grammar as before (Email, CRM, Resources) plus a primary
"New Campaign" action up top, settings + user menu pinned at
the bottom. Hover/active states use translucent white pills.
- AppShell: composes the three above, swaps in the keyboard
shortcuts modal + command palette, takes over from AppLayout.
OrgSwitcher and UserNav restyled for the dark chrome (white text,
faint white borders/hover states) and lost their dependency on
@/components/ui/sidebar — there's no SidebarProvider any more, the
new shell doesn't need one.
Removed components/layout/AppSidebar.tsx (the old zinc sidebar).
Existing per-page layouts (e.g. admin's tab bar) keep working — they
render inside the white content panel exactly as before.
The new admin API clients (audit, credentials, workers) imported Request
with four '..' segments instead of three. Vite's import-analysis failed
with "Failed to resolve import ../../../../Request" because that path
resolves to api/Request, not client/Request. tsc didn't catch it because
the resolver was permissive enough to keep going, but the runtime is
strict. Matched the existing pattern from roles/getRoles.ts (three dots
for Request, four for models).
Separately: web was on host port 15173, offset from the canonical 5173
to avoid colliding with a locally-running Vite outside Docker. Nobody
actually runs Vite locally in this setup, and the offset makes the URL
non-obvious. Moved back to 5173:5173 and updated VITE_APP_URL plus the
docs.
If a developer one day wants to run a host-side Vite alongside the
container, change the mapping back to "15173:5173" — the offset is the
escape hatch, not the default.
Four interlocking problems were causing the web container to spin in
its retry loop forever:
1. web/package.json aliased vite to "npm:rolldown-vite@7.1.14".
rolldown-vite is being deprecated (its own warning told us to use
7.3.1 for migration, or move to vite 8). vitest 4.x has a transitive
`vite` dep that pnpm tried to resolve against the public registry,
where vite@7.1.14 doesn't exist as a release (only 7.3.3 and 8.x).
Result: ERR_PNPM_NO_MATCHING_VERSION on every retry.
2. The "resolutions" block was meant to force the alias on transitive
deps. resolutions is Yarn syntax; pnpm doesn't read it. So the alias
wasn't propagating, which is exactly why (1) blew up.
3. pnpm 11 stopped reading the "pnpm" field in package.json. Settings
moved to pnpm-workspace.yaml. New file added with allowBuilds.esbuild
set so pnpm doesn't refuse to compile esbuild's native binary at
install time (it's transitively pulled in by vite + vitest).
4. The docker-compose web service had `until pnpm install; do echo
"pnpm install retry..."; sleep 3; done` which spins forever on
permanent dep-resolution errors and buries the actual message under
thousands of retries. Replaced with fail-fast that prints a hint
directing the admin to fix package.json and `make restart web`.
Also deleted the stale pnpm-lock.yaml so pnpm regenerates against the
fresh dep tree. Verified vite v7.3.3 boots, esbuild postinstall runs,
and Vite serves on http://localhost:15173 cleanly.
vite.config.ts has no rolldown-specific config, so the move from
rolldown-vite to plain vite is a no-op behaviourally.
Two complementary axes for organizing the fleet, on one shared
mechanism:
User tags (workers.tags)
Free-form lowercase strings the admin applies for whatever they
care about — region (eu-west, fra), provider (hetzner, ovh),
role (warmup-only, burst-capacity), customer cohort. Edited via
a chip-style input with autocomplete from existing tags. Saved
to the worker_tags table.
Smart labels (computed client-side)
Auto-derived from the worker row so they're always in sync:
type:shared / type:dedicated
tier:free / tier:premium (shared only)
pool:clean / pool:risky / pool:quarantine (shared only)
state:installed / state:error / ...
ver:v1.2.3 (if image_version set)
liveness:online / stale / offline
Rendered with tone-aware backgrounds (red for offline / error
/ quarantine, amber for risky / stale, green for online).
Workers list:
- new Tags column showing user tags + the high-signal smart labels
(offline, error, risky, quarantine) with a "+N" overflow
- "filter by tag" chip strip above the table built from the
frequency of every tag (user + smart) in the current result. One
click filters; click again to clear.
Worker detail:
- Tags section near the top showing all smart labels and a full
TagEditor (chip input + autocomplete + suggestions dropdown +
Save button). Saving propagates to the list via react-query
cache invalidation.
The smart labels are never written to the database — they're
recomputed every render. Means renaming an enum value (e.g. risk
pool name changes) doesn't require a backfill.
Replaces the flat /workers/new form with a five-step wizard that asks
"what's this worker for?" first and uses the answer to default everything
else. The previous form put every decision (worker_type, free_tier,
risk_pool, profile, owner) on screen at once with no guidance — fine if
you already know what you're doing, miserable otherwise.
Steps (Owner step skipped unless purpose=dedicated):
1. Purpose — shared / dedicated / risky-pool, with explanatory
cards. This drives the rest: risky → risky pool,
dedicated → unlocks step 4.
2. Connection — host/port/user. "Test reachability" button hits
the new TCP preflight endpoint before any row is
created — typos and firewalls fail loudly here
instead of at the SSH test stage later.
3. Identity — name (auto-derived from host on focus), notes,
profile, tier, risk pool. Risk pool defaults from
purpose but the admin can override.
4. Owner (deds) — user search via /admin/users + subscription ID.
Wizard remembers these and uses them in step 5.
5. Activate — review summary, "Install immediately" toggle
(default on), big Create button.
Post-create panel runs the full pipeline inline without leaving the
page when auto-install is on:
- Show pubkey + copy button + ready-to-paste ssh one-liner
- "I've pasted the key" checkbox unlocks Install
- Install button chains: Test → Install → (if dedicated) convert with
the previously-collected user/sub IDs → redirect to detail page
- Each step's outcome streams into a progress log
Progress dots at the top so the admin sees where they are. Empty-state
hint on step 1 when no workers exist yet.
Worker list grows a Pool column (clean=green, risky=amber,
quarantine=red badge). Dedicated workers render "n/a" — risk pools are
a shared-worker concept since dedicated workers don't share IPs across
customers.
Worker detail page (shared workers only) gets a "Risk pool" section
with three big buttons. Clicking a non-current pool confirms, then
calls PUT /admin/workers/:id/risk-pool. Action audited with the new
pool value.
Saving doesn't migrate accounts directly — the hourly rebalancer
notices the mismatch and moves mailboxes to a matching-pool worker
on its next tick. Documented in the section's helper text.
Endpoint accepts {risk_pool: "clean"|"risky"|"quarantine"} and is
gated by AdminPermManageWorkers. The worker detail row scan now
includes risk_pool so the column actually has data.
POST /admin/workers/:id/convert-to-dedicated does three things in
sequence:
1. Drain existing accounts to a supplied drain_to_worker_id (required
if the source has any accounts; we don't auto-pick per-account
targets because the right choice depends on each account's
owning org).
2. Flip workers.worker_type from "shared" to "dedicated".
3. Atomic create of dedicated_worker_assignments binding the worker
to a specific user/subscription (uses the existing
CreateDedicatedAssignmentIfNotExists so re-running is safe).
Refusal cases:
- already dedicated → 400
- has accounts but no drain target → 400 (admin must pick where they go)
- drain target equals source → 400
Worker detail UI gains a "Convert to dedicated" section, shown only
when the worker is currently shared. Inline form, no modal. The drain
dropdown excludes self, only lists shared+installed workers, sorts
least-loaded first.
Audit-logged with action="convert_to_dedicated" and the user_id,
subscription_id, drain target, and account count in details.
Worker detail page gains a "Move accounts to another worker" section
with a dropdown of eligible targets (same tier, currently installed,
sorted least-loaded first). One click moves every email account on the
current worker to the picked target via the existing AdminReassignEmails
endpoint.
Useful when:
- a worker is down and you want to shift its workload to a healthy
sibling while you investigate
- you want to drain a worker before uninstalling it
- a profile-level change forced too much onto one worker and you want
to rebalance manually
The endpoint emits the standard audit log row (admin_user_id + action
"reassign"), so every manual rewire is visible in the audit viewer.
WorkerHealthAlert lives in the admin layout and renders on every admin
page when any worker needs attention. Polls the workers list every 30s
in the background. Hidden on the workers page itself (which has its own
richer banner with filter chips).
Surfaces three categories:
- errored: install_state == "error"
- offline: installed + no heartbeat for 5+ minutes
- in progress: pending / provisioning / uninstalling
Tone goes red when there are errored or offline workers, amber otherwise.
A "review →" link deep-jumps to /app/admin/workers.
New /app/admin/audit page browses admin_audit_log with:
- Filter by action, target_type, target_id, admin_user_id, date range
- Action + target_type dropdowns are populated from a baseline set
AND from whatever appears in the current result (so new actions
surface automatically without code changes)
- Auto-refresh toggle (5s) for live tailing during operations
- Cursor-based pagination with Prev/Next + page counter
- Expandable rows show the details JSON + user-agent
- Color-coded actions (red for destructive, green for create/install,
amber for system auto-actions)
- Renders system actions (admin_user_id = uuid.Nil) as "system"
- Renders the admin's name + email when joined data is present
Workers list now computes a health summary client-side from the existing
queries and shows a yellow banner with a count when any worker needs
attention (errored, offline, or mid-install). Filter chips below the
banner narrow the table by problem category so the admin can drill in
without opening each worker.
Categories surfaced:
- errored: install_state = "error"
- offline: installed, but no heartbeat in 5+ minutes
- in progress: pending / provisioning / uninstalling
- stale config: profile.updated_at > worker.config_applied_at
- update available: profile.resolved_image_tag != worker.image_version
Live polling stays at 15s, so a worker going offline shows up within
~75s of the last missed heartbeat. The list also fetches profiles so the
stale-config and update-available chips can compute without extra
round-trips.
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Matthew Meszaros