PR #10 added 000035_api_key_suffix and PR #11 separately added
000035_api_key_smart_limits, both targeting the api_keys table.
golang-migrate refuses to load when two source files share a
version number, so the backend container failed health checks
in fresh stacks.
Bumping api_key_smart_limits to 000037 keeps PR #10 in its
original slot (it landed first) and matches the next free
version after 000036_contact_categories. The two migrations
touch different columns so apply order does not matter.
Backend:
- Fix contact-create 500 (nil custom_fields, doubled slice, bad RETURNING SQL)
- Avatar upload: migration 000033, S3 public-read, PNG/JPG only,
client-resized to 512px + server dimension cap (1024px)
- Pull avatar_url through user + organization repo queries
Frontend:
- Settings restructured into nested routes with a rail layout
(/app/settings/{profile,notifications,security,members,roles,
workspace,billing,danger}); flat Section/Row primitives replace
the per-card rectangles; Save buttons only render when dirty
- Standalone /app/billing and /app/team removed; legacy URLs
redirect to the settings sections; UserNav trimmed accordingly
- CRM rebuilt: Pipelines CRUD + stage editor, Deals kanban with
HTML5 drag/drop, Tasks bucketed by due-date with inline toggle.
Frontend models realigned with backend (Deal.name, CRMTask.status
enum, paginated list shapes)
- Avatars: AvatarUploader component, client-side canvas resize,
wired into Profile + Workspace settings; UserNav + OrgSwitcher
render the uploaded image with initials fallback
- RBAC: lib/permissions.ts mirrors organization_permission.go;
inline role picker in Members; Roles & access section shows the
permission matrix and per-role member counts
- Audit log page at /app/audit, gated to owner+admin via canManage
- Plans aligned with warmbly-web pricing: Starter/Grow/Business/
Enterprise via lib/plans.ts; PlanPill, billing page, sidebar
badges and LockedSurface all read from the same catalogue
- Header PlanPill shows current plan with status-aware coloring;
sidebar locked rows show the required-plan badge instead of a
generic lock icon
Perf:
- QueryClient defaults (staleTime: 30s, refetchOnWindowFocus: false,
retry: 1) — kills 3-5 round-trip storm on every navigation
- useSubscription, usePlans → staleTime: Infinity (only invalidate
on plan-change mutations); useUser/Timezones/Orgs get long stales
with refetchOnMount: false
- vite.config: optimizeDeps for heavy libs + server.warmup for the
most-mounted entry pages
Root cause for the 10-min auto-logout (confirmed via pg_stat_activity):
the postgres pool MaxConns was 4, and four repository functions opened
a tx without committing or rolling back. After four calls each leaked
a connection in "idle in transaction" state. Once all four were gone
the pool was permanently exhausted — every new request that needed a
connection blocked until the client gave up. The 10-min trigger is
because that's when the first /auth/refresh fires; refresh tries to
acquire a connection, hangs, eventually the browser aborts the request,
the frontend treats the failure as session expiry, kicks the user.
The four leaking sites:
- emailRepository.Search (drove the leak — Accounts page)
- campaignRepository.Search
- sequenceRepository.Create
- contactRepository.BulkUpdate
Each now has `defer tx.Rollback(ctx)` immediately after Begin, matching
the pattern used in the non-leaky sites in the same files. Rollback is
a no-op after Commit, so this is safe for both read-only tx (Search)
and read-write tx (Create / BulkUpdate).
Additional hardening so a future leak can't silently brick the backend:
- MaxConns 4 → 25. 4 was reckless even without leaks; one bursty
admin page would saturate. 25 is still well under postgres'
default max_connections=100.
- MinConns 0 → 2. Keep a couple of warm connections at idle so the
first request after a quiet period doesn't pay the connect cost.
- idle_in_transaction_session_timeout=300000 (5 min) as a session
RuntimeParam. If a code path forgets the defer, postgres aborts
the leaked tx after 5 min and reclaims the connection.
- statement_timeout=60000 (60 s) as a session RuntimeParam.
Statement runaway can't pin a connection forever.
Verified after backend restart:
SELECT count(*) FROM pg_stat_activity
WHERE datname='warmbly_dev' AND state='idle in transaction';
→ 0
Migration 000032 + repo + endpoints for arbitrary string tags on
workers. The fixed attributes (worker_type, free_tier, risk_pool)
cover the dimensions assignment logic uses. Tags cover everything
else admins want to group by: region (eu-west, fra), provider
(hetzner, ovh), role (warmup-only, burst-capacity), customer cohort —
whatever.
Schema:
- worker_tags(worker_id, tag) composite PK
- tag VARCHAR(64), lowercase + dashed via CHECK constraint
- ON DELETE CASCADE so deleting a worker drops its tags
Endpoints:
- GET /admin/workers/tags list distinct tags (autocomplete)
- PUT /admin/workers/:id/tags replace tag set; normalizes input
Repo:
- GetWorkerTags / SetWorkerTags / ListAllWorkerTags
- HydrateWorkerTags batch-loads tags onto a slice of workers in one
round-trip so the dashboard list doesn't do N+1 queries
PUT is transactional (delete + bulk insert) so the list view never
catches a worker mid-tag-swap. Auto-derived "smart" labels
(tier:free, pool:risky, state:error) are NOT stored — those are
computed client-side from the worker row so they stay in sync with
the source attributes automatically. Next commit wires the UI.
Threat-level segregation, schema layer. Two new concepts:
workers.risk_pool ∈ {clean, risky, quarantine}
buckets shared workers by acceptable risk. Dedicated workers don't
use it (single tenant = no cross-contamination risk).
email_accounts.risk_band ∈ {clean, risky, quarantine}
per-mailbox classification, derived from warmup_health_state by the
rebalancer (next commit). Never set by user input.
The mapping is one-way and intentionally simple:
healthy → clean
watch, throttled → risky
quarantined, → quarantine
blocked
Rebalancer code lands in the next commit. This commit just adds:
- migration 000031 with enums + columns + filtered indexes
- WorkerRiskPool / EmailRiskBand types + RiskBandFromHealth helper
- WorkerRepository methods: SetWorkerRiskPool, SetEmailAccountRiskBand,
GetSharedWorkersByTierAndPool, ListRiskCandidates
- RiskCandidate result type joining email_accounts + warmup health
(picks WORST state across pools via CASE ranking) + worker columns
so the rebalancer can decide migrations in one scan
Workers are no longer curl|sh-only. Admins add and manage them from the
dashboard over SSH, with all runtime config (Kafka, Schema Registry,
Redis, AWS keys) stored encrypted via the existing KMS-envelope cipher
service.
Worker lifecycle:
1. Admin POSTs host/port/user. Backend generates an ed25519 keypair,
encrypts the private key under uuid.Nil (platform identity), and
stores the row in 'pending' state.
2. Admin pastes the returned public key into the VPS's authorized_keys.
3. Test connection — runs `true` over SSH, pins the host SHA256
fingerprint on first success (TOFU).
4. Install — backend scp's install-worker.sh + a per-worker env file
and runs it. State moves pending → provisioning → installed.
5. From then on: restart, update image, apply config, uninstall,
rotate keys, tail logs, live status, OS package update, reboot —
all dashboard buttons backed by SSH operations.
Credentials are reusable entities:
- aws_credentials: named keypair, secret encrypted at rest
- worker_profiles: bundles Kafka + Schema Registry + Redis + image +
release channel, references one AWS credentials row
- workers.profile_id links a worker to a profile; many workers can
share one profile
Saving a profile doesn't restart anything. The dashboard compares
profile.updated_at to each worker's config_applied_at and shows a
"stale config" badge; Apply rewrites /etc/warmbly/worker.env over SSH
and restarts the unit.
Auto-update on GitHub release:
- profile.release_channel ∈ {pinned, stable, dev}
- profile.auto_update toggles automatic rollout
- Trigger model is push, not poll: one check on backend boot, then
the /webhooks/github/releases endpoint (HMAC-validated with
RELEASES_WEBHOOK_SECRET) on every release event. Manual "Check now"
button as fallback.
- When a new tag resolves, the orchestrator SSHes into each assigned
worker, runs install-worker.sh --update --image <new>, which now
rewrites the systemd unit (not just `docker pull`) so the image
actually changes. workers.image_version captures the running tag
for the UI's "v1.2.3 → v1.2.4" diff.
Self-hostable: every release knob is env-driven —
RELEASES_GITHUB_REPO, RELEASES_WORKER_IMAGE_REPO,
RELEASES_WEBHOOK_SECRET, RELEASES_GITHUB_TOKEN, RELEASES_ENABLED. Set
RELEASES_ENABLED=false to disable the feature entirely.
OS-level updates and reboot are also exposed: detect apt / dnf / yum /
pacman / apk, run the right upgrade noninteractively, return the full
output and a reboot-required flag. Reboots are never automatic.
Migrations:
000028_worker_ssh — ssh fields, install_state enum, last_seen,
host fingerprint
000029_worker_credentials — aws_credentials + worker_profiles +
workers.profile_id + workers.config_applied_at
000030_worker_releases — release_channel enum, auto_update,
resolved_image_tag, workers.image_version
Endpoints added:
POST /admin/workers (create + keypair)
GET /admin/workers/managed
GET /admin/workers/:id/managed
POST /admin/workers/:id/{test,install,restart,upgrade,uninstall,rotate-keys,apply,system-update,reboot}
PUT /admin/workers/:id/profile
GET /admin/workers/:id/{live-status,logs}
DELETE /admin/workers/:id
GET /admin/aws-credentials CRUD
GET /admin/worker-profiles CRUD + /workers + /apply + /release
GET /admin/releases/state
POST /admin/releases/check
POST /webhooks/github/releases public, HMAC-validated
Admin UI:
/app/admin/workers list with status + version columns
/app/admin/workers/new add form with profile dropdown
/app/admin/workers/:id detail with all actions + logs + system update
/app/admin/credentials tabs: AWS credentials + worker profiles,
Releases panel, channel selector +
auto-update toggle in profile form
Matthew Meszaros