The new admin endpoints for SSH worker management, AWS credentials,
worker profiles, releases, and system operations bypassed the existing
audit-logging pattern. Now every mutating action records a row in the
audit log with adminID, IP, user-agent, and operation-specific metadata
(never secret values — for credential updates we record which fields
were rotated, not what they became).
New action constants:
test, install, restart, upgrade, uninstall, rotate_keys, apply,
assign, system_update, reboot, check_releases
New entity types:
worker, aws_credentials, worker_profile, release
Read-only endpoints (list/get/status/logs) intentionally not audited —
they don't change state and would flood the log.
Each handler calls a small h.audit(c, action, entity, &id, metadata)
helper that pulls adminID from the admin middleware and fires the
existing AuditService.LogAction (fire-and-forget, never blocks the
response).
Workers heartbeat into Redis every 90s as RFC3339 timestamp values with a
3-min TTL. The dashboard surfaces liveness based on workers.last_seen_at,
but until now nothing populated that column — the "Live" badge was always
red.
New 60s job in the consumer reads each active worker's Redis heartbeat
value, parses the timestamp, and writes it to workers.last_seen_at. Runs
on its own interval (separate from the 5-min dead-worker detection job,
which does heavier reassignment work) so the UI sees fresh data within a
minute.
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
- cipher: use the already-available ctx parameter for DynamoDB Put
- goog: use context.Background for OAuth token refresh callback since
it runs asynchronously outside any request lifecycle
The Tries counter on login and registration sessions was checked but
never incremented, making the brute-force protection dead code. An
attacker could retry verification codes indefinitely within the session
TTL. Now each failed attempt increments and persists the counter.
- Guard checkoutSession.Customer access with nil checks to prevent panic
when Stripe sends incomplete checkout session data
- Switch worker migration goroutines from request ctx to context.Background()
since these operations outlive the webhook HTTP request and would be
cancelled prematurely when the response completes
Increase warmup subjects from 5 to 20 and conversations from 3 to 12
to reduce repetition patterns that spam filters can detect. New themes
cover industry, tools, networking, feedback, planning, reading, travel,
wellness, and events.
- Replace all fmt.Printf calls in Kafka consumer/producer, tracking consumer,
and user email task with structured zerolog (log.Warn/Error/Info)
- Fix hashURL using SHA-256 instead of naive first-8-chars+length approach
which was collision-prone for deduplication
- Return actual captcha verification errors instead of generic
internal errors in login, registration, and reset password flows
- Fix Sentry capturing wrong variable (err -> xerr) in
RegistrationConfirm
- Pass AuthSecret as []byte to JWT SignedString/keyfunc
- Send form data in request body instead of PostForm for Turnstile
- Improve Turnstile non-200 error handling with response body logging
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Matthew Meszaros