Worker detail page gains a "Move accounts to another worker" section
with a dropdown of eligible targets (same tier, currently installed,
sorted least-loaded first). One click moves every email account on the
current worker to the picked target via the existing AdminReassignEmails
endpoint.
Useful when:
- a worker is down and you want to shift its workload to a healthy
sibling while you investigate
- you want to drain a worker before uninstalling it
- a profile-level change forced too much onto one worker and you want
to rebalance manually
The endpoint emits the standard audit log row (admin_user_id + action
"reassign"), so every manual rewire is visible in the audit viewer.
WorkerHealthAlert lives in the admin layout and renders on every admin
page when any worker needs attention. Polls the workers list every 30s
in the background. Hidden on the workers page itself (which has its own
richer banner with filter chips).
Surfaces three categories:
- errored: install_state == "error"
- offline: installed + no heartbeat for 5+ minutes
- in progress: pending / provisioning / uninstalling
Tone goes red when there are errored or offline workers, amber otherwise.
A "review →" link deep-jumps to /app/admin/workers.
New /app/admin/audit page browses admin_audit_log with:
- Filter by action, target_type, target_id, admin_user_id, date range
- Action + target_type dropdowns are populated from a baseline set
AND from whatever appears in the current result (so new actions
surface automatically without code changes)
- Auto-refresh toggle (5s) for live tailing during operations
- Cursor-based pagination with Prev/Next + page counter
- Expandable rows show the details JSON + user-agent
- Color-coded actions (red for destructive, green for create/install,
amber for system auto-actions)
- Renders system actions (admin_user_id = uuid.Nil) as "system"
- Renders the admin's name + email when joined data is present
Workers list now computes a health summary client-side from the existing
queries and shows a yellow banner with a count when any worker needs
attention (errored, offline, or mid-install). Filter chips below the
banner narrow the table by problem category so the admin can drill in
without opening each worker.
Categories surfaced:
- errored: install_state = "error"
- offline: installed, but no heartbeat in 5+ minutes
- in progress: pending / provisioning / uninstalling
- stale config: profile.updated_at > worker.config_applied_at
- update available: profile.resolved_image_tag != worker.image_version
Live polling stays at 15s, so a worker going offline shows up within
~75s of the last missed heartbeat. The list also fetches profiles so the
stale-config and update-available chips can compute without extra
round-trips.
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Matthew Meszaros