Reserve dedicated worker allocation for the control plane, auto-promote spare capacity when needed, and keep risky or quarantined mailboxes off clean shared workers.
Disable the linters that fire on legacy code without flagging real
bugs: `unused` (orphan repos kept for future feature flags),
`unconvert` (defensive type conversions), `gosimple` (style
suggestions in code we don't want to touch).
govet: disable `shadow` (idiomatic `err :=` re-decls in transaction
patterns) and `nilness` (legitimate defensive nil checks that look
tautological to the analyzer).
Ran `gofmt -w internal/ cmd/` — every Go file now passes
gofmt -l with no output.
Kept: govet, staticcheck, ineffassign, typecheck, bodyclose, noctx,
sqlclosecheck, gofmt, goimports, misspell — the real-bug checks.
Migration 000032 + repo + endpoints for arbitrary string tags on
workers. The fixed attributes (worker_type, free_tier, risk_pool)
cover the dimensions assignment logic uses. Tags cover everything
else admins want to group by: region (eu-west, fra), provider
(hetzner, ovh), role (warmup-only, burst-capacity), customer cohort —
whatever.
Schema:
- worker_tags(worker_id, tag) composite PK
- tag VARCHAR(64), lowercase + dashed via CHECK constraint
- ON DELETE CASCADE so deleting a worker drops its tags
Endpoints:
- GET /admin/workers/tags list distinct tags (autocomplete)
- PUT /admin/workers/:id/tags replace tag set; normalizes input
Repo:
- GetWorkerTags / SetWorkerTags / ListAllWorkerTags
- HydrateWorkerTags batch-loads tags onto a slice of workers in one
round-trip so the dashboard list doesn't do N+1 queries
PUT is transactional (delete + bulk insert) so the list view never
catches a worker mid-tag-swap. Auto-derived "smart" labels
(tier:free, pool:risky, state:error) are NOT stored — those are
computed client-side from the worker row so they stay in sync with
the source attributes automatically. Next commit wires the UI.
POST /admin/workers/preflight {host, port} runs a 5s TCP dial against
host:port and returns ok + latency, or an error. Used by the worker
creation wizard to catch typos / firewall problems while the form is
still open — much better UX than discovering an unreachable VPS at the
SSH test step after the row already exists.
Doesn't attempt an SSH handshake (no credentials at this stage). A green
preflight just means "something is listening there." The actual SSH test
runs later, after the admin pastes the generated pubkey.
Worker list grows a Pool column (clean=green, risky=amber,
quarantine=red badge). Dedicated workers render "n/a" — risk pools are
a shared-worker concept since dedicated workers don't share IPs across
customers.
Worker detail page (shared workers only) gets a "Risk pool" section
with three big buttons. Clicking a non-current pool confirms, then
calls PUT /admin/workers/:id/risk-pool. Action audited with the new
pool value.
Saving doesn't migrate accounts directly — the hourly rebalancer
notices the mismatch and moves mailboxes to a matching-pool worker
on its next tick. Documented in the section's helper text.
Endpoint accepts {risk_pool: "clean"|"risky"|"quarantine"} and is
gated by AdminPermManageWorkers. The worker detail row scan now
includes risk_pool so the column actually has data.
POST /admin/workers/:id/convert-to-dedicated does three things in
sequence:
1. Drain existing accounts to a supplied drain_to_worker_id (required
if the source has any accounts; we don't auto-pick per-account
targets because the right choice depends on each account's
owning org).
2. Flip workers.worker_type from "shared" to "dedicated".
3. Atomic create of dedicated_worker_assignments binding the worker
to a specific user/subscription (uses the existing
CreateDedicatedAssignmentIfNotExists so re-running is safe).
Refusal cases:
- already dedicated → 400
- has accounts but no drain target → 400 (admin must pick where they go)
- drain target equals source → 400
Worker detail UI gains a "Convert to dedicated" section, shown only
when the worker is currently shared. Inline form, no modal. The drain
dropdown excludes self, only lists shared+installed workers, sorts
least-loaded first.
Audit-logged with action="convert_to_dedicated" and the user_id,
subscription_id, drain target, and account count in details.
The audit calls added in the last commit went to AuditService.LogAction,
which writes to the general user-facing audit log (Cassandra). The admin
audit-log viewer at /admin/audit-logs queries the admin_audit_log table
in Postgres, so worker / credentials / release actions never showed up.
Add a public AdminService.LogAdminAction that wraps the existing private
logAction (writes to admin_audit_log with the same shape as ban_user /
update_worker / etc.). Repoint h.audit() at it.
Actions now visible in the audit viewer:
test, install, restart, upgrade, uninstall, rotate_keys, apply,
assign, system_update, reboot, check_releases (plus the existing
create/update/delete across workers, AWS creds, and profiles).
The new admin endpoints for SSH worker management, AWS credentials,
worker profiles, releases, and system operations bypassed the existing
audit-logging pattern. Now every mutating action records a row in the
audit log with adminID, IP, user-agent, and operation-specific metadata
(never secret values — for credential updates we record which fields
were rotated, not what they became).
New action constants:
test, install, restart, upgrade, uninstall, rotate_keys, apply,
assign, system_update, reboot, check_releases
New entity types:
worker, aws_credentials, worker_profile, release
Read-only endpoints (list/get/status/logs) intentionally not audited —
they don't change state and would flood the log.
Each handler calls a small h.audit(c, action, entity, &id, metadata)
helper that pulls adminID from the admin middleware and fires the
existing AuditService.LogAction (fire-and-forget, never blocks the
response).
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Matthew Meszaros