The new admin API clients (audit, credentials, workers) imported Request
with four '..' segments instead of three. Vite's import-analysis failed
with "Failed to resolve import ../../../../Request" because that path
resolves to api/Request, not client/Request. tsc didn't catch it because
the resolver was permissive enough to keep going, but the runtime is
strict. Matched the existing pattern from roles/getRoles.ts (three dots
for Request, four for models).
Separately: web was on host port 15173, offset from the canonical 5173
to avoid colliding with a locally-running Vite outside Docker. Nobody
actually runs Vite locally in this setup, and the offset makes the URL
non-obvious. Moved back to 5173:5173 and updated VITE_APP_URL plus the
docs.
If a developer one day wants to run a host-side Vite alongside the
container, change the mapping back to "15173:5173" — the offset is the
escape hatch, not the default.
Two complementary axes for organizing the fleet, on one shared
mechanism:
User tags (workers.tags)
Free-form lowercase strings the admin applies for whatever they
care about — region (eu-west, fra), provider (hetzner, ovh),
role (warmup-only, burst-capacity), customer cohort. Edited via
a chip-style input with autocomplete from existing tags. Saved
to the worker_tags table.
Smart labels (computed client-side)
Auto-derived from the worker row so they're always in sync:
type:shared / type:dedicated
tier:free / tier:premium (shared only)
pool:clean / pool:risky / pool:quarantine (shared only)
state:installed / state:error / ...
ver:v1.2.3 (if image_version set)
liveness:online / stale / offline
Rendered with tone-aware backgrounds (red for offline / error
/ quarantine, amber for risky / stale, green for online).
Workers list:
- new Tags column showing user tags + the high-signal smart labels
(offline, error, risky, quarantine) with a "+N" overflow
- "filter by tag" chip strip above the table built from the
frequency of every tag (user + smart) in the current result. One
click filters; click again to clear.
Worker detail:
- Tags section near the top showing all smart labels and a full
TagEditor (chip input + autocomplete + suggestions dropdown +
Save button). Saving propagates to the list via react-query
cache invalidation.
The smart labels are never written to the database — they're
recomputed every render. Means renaming an enum value (e.g. risk
pool name changes) doesn't require a backfill.
Replaces the flat /workers/new form with a five-step wizard that asks
"what's this worker for?" first and uses the answer to default everything
else. The previous form put every decision (worker_type, free_tier,
risk_pool, profile, owner) on screen at once with no guidance — fine if
you already know what you're doing, miserable otherwise.
Steps (Owner step skipped unless purpose=dedicated):
1. Purpose — shared / dedicated / risky-pool, with explanatory
cards. This drives the rest: risky → risky pool,
dedicated → unlocks step 4.
2. Connection — host/port/user. "Test reachability" button hits
the new TCP preflight endpoint before any row is
created — typos and firewalls fail loudly here
instead of at the SSH test stage later.
3. Identity — name (auto-derived from host on focus), notes,
profile, tier, risk pool. Risk pool defaults from
purpose but the admin can override.
4. Owner (deds) — user search via /admin/users + subscription ID.
Wizard remembers these and uses them in step 5.
5. Activate — review summary, "Install immediately" toggle
(default on), big Create button.
Post-create panel runs the full pipeline inline without leaving the
page when auto-install is on:
- Show pubkey + copy button + ready-to-paste ssh one-liner
- "I've pasted the key" checkbox unlocks Install
- Install button chains: Test → Install → (if dedicated) convert with
the previously-collected user/sub IDs → redirect to detail page
- Each step's outcome streams into a progress log
Progress dots at the top so the admin sees where they are. Empty-state
hint on step 1 when no workers exist yet.
Worker list grows a Pool column (clean=green, risky=amber,
quarantine=red badge). Dedicated workers render "n/a" — risk pools are
a shared-worker concept since dedicated workers don't share IPs across
customers.
Worker detail page (shared workers only) gets a "Risk pool" section
with three big buttons. Clicking a non-current pool confirms, then
calls PUT /admin/workers/:id/risk-pool. Action audited with the new
pool value.
Saving doesn't migrate accounts directly — the hourly rebalancer
notices the mismatch and moves mailboxes to a matching-pool worker
on its next tick. Documented in the section's helper text.
Endpoint accepts {risk_pool: "clean"|"risky"|"quarantine"} and is
gated by AdminPermManageWorkers. The worker detail row scan now
includes risk_pool so the column actually has data.
POST /admin/workers/:id/convert-to-dedicated does three things in
sequence:
1. Drain existing accounts to a supplied drain_to_worker_id (required
if the source has any accounts; we don't auto-pick per-account
targets because the right choice depends on each account's
owning org).
2. Flip workers.worker_type from "shared" to "dedicated".
3. Atomic create of dedicated_worker_assignments binding the worker
to a specific user/subscription (uses the existing
CreateDedicatedAssignmentIfNotExists so re-running is safe).
Refusal cases:
- already dedicated → 400
- has accounts but no drain target → 400 (admin must pick where they go)
- drain target equals source → 400
Worker detail UI gains a "Convert to dedicated" section, shown only
when the worker is currently shared. Inline form, no modal. The drain
dropdown excludes self, only lists shared+installed workers, sorts
least-loaded first.
Audit-logged with action="convert_to_dedicated" and the user_id,
subscription_id, drain target, and account count in details.
Worker detail page gains a "Move accounts to another worker" section
with a dropdown of eligible targets (same tier, currently installed,
sorted least-loaded first). One click moves every email account on the
current worker to the picked target via the existing AdminReassignEmails
endpoint.
Useful when:
- a worker is down and you want to shift its workload to a healthy
sibling while you investigate
- you want to drain a worker before uninstalling it
- a profile-level change forced too much onto one worker and you want
to rebalance manually
The endpoint emits the standard audit log row (admin_user_id + action
"reassign"), so every manual rewire is visible in the audit viewer.
New /app/admin/audit page browses admin_audit_log with:
- Filter by action, target_type, target_id, admin_user_id, date range
- Action + target_type dropdowns are populated from a baseline set
AND from whatever appears in the current result (so new actions
surface automatically without code changes)
- Auto-refresh toggle (5s) for live tailing during operations
- Cursor-based pagination with Prev/Next + page counter
- Expandable rows show the details JSON + user-agent
- Color-coded actions (red for destructive, green for create/install,
amber for system auto-actions)
- Renders system actions (admin_user_id = uuid.Nil) as "system"
- Renders the admin's name + email when joined data is present
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Matthew Meszaros