starred/warmbly
1
0
Fork 0
mirror of https://github.com/warmbly/warmbly.git synced 2026-10-05 16:02:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,785 Commits 3 Branches 79 Tags
00a0a67e0b763be35e78377255b3bcb4f679dafe
Commit Graph
14 Commits
Author SHA1 Message Date
Matthew Meszaros a900f1e04c feat: make worker moves atomic and preserve safe concentration and health state 2026-09-17 07:45:23 -07:00
Matthew Meszaros bab9f86727 feat: correct worker capacity, mailbox distribution, observed IPv4, fleet pagination, and premium pool promotion 2026-09-17 04:15:05 -07:00
Matthew Meszaros fa2b5330d7 feat: drop the auth-pressure placement term because worker_capacity_view aggregates auth_errors (per-mailbox credential failures) and not rate_limit_errors (the 454/421 per-IP throttles it claimed to measure), measure projected utilization against an age-free Capacity.Target so a freshly joined node can relieve a full fleet instead of scoring as 200% loaded after one mailbox, bound the isolated-egress override with an explicit OverTarget check now that Eligible no longer caps it, and cap rotation moves per destination since a tick scores every mailbox against one frozen materialized-view snapshot 2026-09-10 05:51:17 -07:00
Matthew Meszaros 4fc6d1db77 feat: put every docker mount on one line in the generated unit, because command substitution strips the trailing newline and my previous multi-line blob mount collapsed the continuation so the node restart-looped on any filesystem-blob install, and mount BLOB_FS_ROOT only when it already exists rather than creating and recursively chowning a directory a co-located bare-metal backend owns 2026-09-09 05:59:44 -07:00
Matthew Meszaros 0a08983158 feat: keep the node's write path reachable and the image reference root-owned, by granting traverse on the state directory without making it writable, re-owning image-ref on every join so a machine joined by the earlier chown-everything script converges, mounting BLOB_FS_ROOT separately when blobs live under the state directory the mount no longer covers, and renaming the reserve dialog's remaining convert-to-dedicated button 2026-09-09 05:53:28 -07:00
Matthew Meszaros 92470e1528 feat: drop the racy liveness re-read from the placement live test, which re-queried whichever worker placement chose and so raced with the email package's live tests creating and deleting their own worker in a parallel package; that placement only ever picks a live worker is already covered by the unit tests, which need no database 2026-09-09 05:44:25 -07:00
Matthew Meszaros 65fe82c472 feat: stop the node container being able to choose the image root runs, by giving it a dedicated uid-1000 subdirectory to write its update target into instead of making the whole state directory writable, validating the tag it asks for before the updater acts on it, and no longer recursively chowning a directory a bare-metal install also keeps its blobs in 2026-09-09 05:41:16 -07:00
Matthew Meszaros bfdbd77b04 feat: fix the eight defects the second review pass found, including three where the previous fix did not land: the bind-mounted state dir was root-owned so the node running as uid 1000 still could not write its update target, elevating the reserved-worker eviction did nothing because the rotation loop bailed on target-equals-current before the urgency was consulted, and the warmup-pool assertion was vacuous which hid that warmupPoolFor checked the subscription repo before the billing provider and answered free on a self-host install 2026-09-09 05:32:30 -07:00
Matthew Meszaros 8b83062eff feat: fix ten defects the review found, the worst being that every seeder still wrote the worker columns migration 000141 moved to fleet_nodes so make dev could not seed, that the generated systemd unit used a command substitution systemd never expands so a joined machine restart-looped while the script reported success, that the state directory was not bind-mounted so the node wrote its update target inside the container where the host timer never saw it, and that placement stopped assigning warmup pool membership so paid mailboxes silently warmed in the free pool 2026-09-09 05:21:17 -07:00
Matthew Meszaros de60bc3784 feat: let an isolated-egress reservation move a mailbox without clearing the score gain, because the reserved worker scores lower than the incumbent by construction (the incumbent holds the stickiness bonus) so the rotation loop refused the move on every tick and the organization never converged onto the worker it pays for 2026-09-09 05:02:53 -07:00
Matthew Meszaros 435dbb522f feat: replace the worker tier/type/risk-pool/egress categories with a scored placement model and make the fleet pull-based, so a machine joins with one command, workers and consumers share one node registry with usage and liveness, nodes self-update to the version the control plane resolves, and the Hetzner provisioning, worker profiles and SSH orchestrator are removed 2026-09-09 04:54:01 -07:00
Matthew Meszaros c26300ae5e feat: backend half of the admin panel upgrade: delete the unrouted provisioning, releases, plan, discount and enterprise-inquiry admin handlers with their service and repository methods, retire the six admin permission bits nothing gated as reserved placeholders so live bit positions and existing masks are unchanged and IsSuperAdmin checks the live set, add forty admin endpoints for mailbox sync governor state with clear-throttle and restart-backfill, in-flight send reservations, cross-workspace dead letters with replay, task failures, webhook delivery health with reclaim, fleet capacity, the control loops decision log, dedicated bindings with release and the routed convert-to-dedicated, operator-driven workspace export and import, per-organization API keys and webhooks, warmup invalid-token abuse and action history, and signups by acquisition channel, and add a scheduled_job_runs table (migration 000135) with a jobrun package that every backend and consumer loop now records through and a run-now request the owning process picks up within fifteen seconds 2026-09-07 21:40:38 -07:00
Matthew Meszaros d2414ad29f ci(go): gofmt all flagged files 
Go CI fails on golangci-lint's gofmt check. Ran gofmt -w against
every file the linter named plus a handful of others that drifted
during the autonomous-fleet work. No semantic changes — alignment
of struct field whitespace and one mis-indented import block.

gofmt -l ./... is now empty; go build + go vet are clean.
 2026-05-27 16:40:23 +00:00
Matthew Meszaros faaa3cded5 fleet: autonomous rebalance + scale + quarantine loops 
Three closed-loop background goroutines on the backend that manage the
worker fleet without operator intervention.

Rebalancer (default 5min): for each tier, drain hot workers (>80%
utilization) onto cold workers (<50%, healthy). Safety rails: per-
mailbox 24h cooldown to prevent thrashing, max 200 in-flight migrations,
destination must be healthy or watch.

Scaler (default 1h): compute fleet utilization per tier. At >=70%
sustained, emit warning. At >=85% sustained, emit critical alert. If
AUTO_PROVISION is allowed by provisioning_policy, snapshot the active
auto-template for the tier into a new provisioning_jobs row — the state
machine picks it up and provisions the box without admin click.

QuarantineEvaluator (default 5min): inspect rolling 1h bounce/complaint
rates, transition workers between health bands (healthy / watch /
throttled / quarantined / blocked) using CLAUDE.md thresholds.
Quarantined and blocked workers are auto-drained by the Rebalancer
because ListCapacityCandidates excludes them.

Every action is written to decision_log so the admin Decisions page can
answer 'why did the system do X'.
 2026-05-27 15:56:33 +00:00