Adds a dedicated admin path for sending platform email — distinct from
the campaign emailsend service (which sends through customer mailboxes)
so the two abuse surfaces never share code paths.
Schema (000047) adds admin_outreach_messages: every send is recorded
with sent_by, the resolved to_email, the optional reply_to, subject,
body, and a queued → sent/failed status. Failed sends keep their error
column populated for the audit log.
Extends notify.EmailNotificationService with SendOutreach so both
backends (SES + SMTP) support custom Reply-To: SES via the native
ReplyToAddresses field, SMTP via a forged Reply-To header. The
existing transactional Send() remains unchanged so no other caller is
affected.
Service (internal/app/adminoutreach) resolves recipients three ways:
to_email (raw address), to_user_id (sends to the user's account email),
or to_org_id (sends to the workspace owner). Persist-then-send-then-
mark ensures the audit row exists even if the mailer hangs, and
mark-failed captures the error string verbatim.
Routes:
POST /admin/outreach manage_organizations
GET /admin/outreach view_organizations
Admin UI: composer with recipient mode picker (email / user_id / org_id),
configurable Reply-To (defaults to support@warmbly.com so customers can
actually reply), subject + HTML body editor, and an outreach log below
showing the last 50 sends with status badges and error details. Sidebar
entry under Accounts (Send icon).
Three surfaces close the loop on the limit-increase workflow:
- admin/dashboard/LimitRequestsPage.tsx queues every pending request
with full context (org → users → field → current vs requested →
+delta) and one-click approve/reject. Both actions open a review
dialog; approve notes are optional, reject notes are required and
surface to the customer.
- web/settings/limits/page.tsx is the customer-facing form. Resource
selector, requested value, reason textarea, plus a list of every
past request with its status (pending/approved/rejected/cancelled)
and the reviewer's notes when present. Pending rows expose a
cancel link. Footer links to the ToS limits clause.
- site/terms.astro grows a new section 07 ("Usage limits and
increase requests"). Explicit: "unlimited" means no plan-tier cap
but a product-wide hard ceiling still applies, increases are at
Warmbly's sole discretion, and previously granted increases can be
revoked when reputation signals deteriorate. Bumps every existing
section heading and id from 07 onward.
Admin sidebar grows a "Limit requests" entry under Accounts (Gauge
icon). Web settings layout grows a "Limits" section under owner-only
sections.
Wire the customer self-serve path for asking "please give me more
mailboxes / campaigns / contacts." Migration 000046 adds
limit_increase_requests with a partial unique index ensuring only one
pending request per (org, field) so the queue can't be spammed, plus
a CHECK requiring requested > current_effective so no-op rows never
reach an admin.
Service layer:
- SubmitLimitIncreaseRequest validates membership, rejects unknown
fields, snapshots the user's current effective limit at submission
time so the queue row carries the context the admin needs.
- CancelLimitRequest lets the original submitter walk back a pending
request; approved/rejected rows are immutable as the audit record.
- ApproveLimitRequest stamps the row and writes the corresponding
column on organization_limit_overrides via SetLimitOverrides —
same write path direct admin overrides use, so granted_by and
notes carry through and the audit log treats both flows uniformly.
- RejectLimitRequest stamps the row with required review notes.
Routes:
POST /v1/organization/:orgId/limit-requests
GET /v1/organization/:orgId/limit-requests
DELETE /v1/limit-requests/:id (submitter only)
GET /admin/limit-requests?status=pending
POST /admin/limit-requests/:id/approve
POST /admin/limit-requests/:id/reject
Admin approval and rejection both fire admin audit log entries with
field + requested + notes so the decision history survives any future
reorg of the request table.
UI (admin queue page + dashboard request form) plus the ToS clause
giving Warmbly the right to refuse any increase land in the next commit.
Add users.ban_scope INT NOT NULL DEFAULT 0 in migration 000045 so admins
can describe what a ban concretely stops (login / workspace creation /
outbound send) instead of relying on a single boolean banned_at flag
that meant "everything".
Wire flags in the BanScope enum (kept in sync with the migration) plus
a CHECK constraint guaranteeing non-negative values. Existing bans
backfill to BanScopeLogin so the historical "you can't log in"
semantics is preserved exactly — no behaviour changes silently at
deployment.
BanUserRequest gains an optional scope field, BanUser threads it through
the service to the repo write, and the UserBanDialog grows a checkbox
group with one option per flag. Reason still required; at least one
scope must be picked. Audit details now include the scope bitmask.
Runtime enforcement (refusing login when BanScopeLogin is set, etc.) is
intentionally separate from this commit — the existing codebase doesn't
yet have an active ban check anywhere, so wiring that lives across the
auth middleware, org-create handler, and emailsend service. This slice
ships the schema, the audit story, and the UI vocabulary so the
enforcement PR can land without database churn.
Replace the PlansPage stub with the real plan catalog over /admin/plans.
The list view surfaces visibility (public/private with a colored badge),
price + discounted price, and the four limit columns most often
touched: mailboxes, campaigns, members, contacts, plus daily-email
budget.
Edit dialog covers every safely-editable field — name, price, the four
org limits, daily caps, account limit, dedicated worker count, and the
public flag. Stripe price/product IDs are intentionally read-only;
those must be managed in Stripe and flow back via webhook.
Add /enterprise page wired to /admin/enterprise/inquiries. Sales-style
pipeline: pending → contacted → converted | declined, with an inline
<select> on each row so triage is one click per inquiry rather than a
detail-page round-trip.
Each row shows company, contact, estimated volume, team size, and the
free-form notes from the marketing-site form. Pending is the default
filter so the queue surfaces first; "All" reveals historical decisions.
Added a sidebar entry under Accounts (Briefcase icon) so the inquiry
queue is one click away from the rest of the customer-facing admin.
Replace the CampaignsPage stub with a real list backed by
/admin/campaigns. Search by name, filter by status (active/paused/done/all),
inline engagement counters (contacts, sent, opens, reply %, bounce %).
Bounce rates >5% are tinted red so abuse review is one glance.
Force-stop opens a dialog requiring a reason; the reason is written to
the admin audit log. Stop is disabled on completed/draft campaigns to
prevent accidental clicks.
Org name in each row links into the workspace admin page so an
investigator can pivot from "this campaign looks bad" to "who is
sending it, on what plan, with what override history" without leaving
the admin surface.
No backend changes — every endpoint already existed.
Replace the WarmupPage stub with the real safety surface CLAUDE.md
treats as the platform's most critical. Three layers, all wired to
existing /admin/warmup/* endpoints:
- Health summary: four cards (total participants by state, at-risk
count, avg spam-folder placement rate with green/amber/red tone,
blocked count). Refetches every 30s so an investigator sees pool
drift in near-real time.
- Per-pool table: free + premium with total/active/blocked counts.
Premium gets the purple badge to make the policy-isolation point
obvious at a glance.
- Blocked mailboxes: list with one-click unblock action. Appeals
state surfaces inline as an amber badge when present.
- Appeals queue: pending only by default with an approve/reject pair
that opens a review dialog requiring notes (notes land in the
audit log and may be shown to the appealing user).
No backend changes — every endpoint already existed.
The animated fanout SVG was not landing. Removes the entire diagram
panel so the page goes hero, then How it works, then categories.
Tightens the hero bottom padding now that nothing overlaps it.
Drops the fake floating dashboard. The new hero panel shows a real
story: a single campaign.reply_received event lands on the Warmbly hub
and fans out to five concrete provider channels (Calendly meeting
booked, HubSpot activity logged, Slack #sales notified, Cal.com demo
booked, Pipedrive deal advanced). Each channel has a traveling SVG
packet animation. Subtle pulsing hub ring and per-card receive pulses.
The diagram tells the integrations value proposition directly instead
of showing generic dashboard rows.
Adds /users/:id with the full preview payload (profile, orgs the user
belongs to, connected mailboxes with provider/status/warmup state, and
ban history) plus two action surfaces:
- UserBanDialog: ban or unban from the same component, reason field
required so the audit trail always carries it. Banning is disabled
for admin accounts at the button level mirroring the backend's
"cannot ban admin users" guard.
- UserRateLimitsDialog: per-user override editor for daily emails,
max concurrent connections, and the three websocket per-minute
caps. Same "blank = no change, 0 = clear override, positive =
explicit cap" convention as the org limit overrides for consistency.
Organization rows on the detail page link straight into the
organizations admin so an ops investigator can pivot user → workspace
→ workspace owner without ever leaving the admin surface.
Replace the UsersPage stub with a real list view. Search covers name and
email, status toggle splits active / banned / all, and an "admins only"
checkbox filters down to accounts with admin_permissions > 0. Each row
shows orgs, mailboxes, and campaign counts inline so abuse review is
one glance; admin accounts get the amber ADMIN badge from the same
visual vocabulary as the app shell.
Backend endpoints already exist — this commit only adds the typed API
client (admin/users.ts) and the page itself. Drill-in to the detail
page lands in the next commit alongside ban/unban and rate-limit
override editors.
Replaces deliverability + DNS sections with CRM, Automation,
Notifications, Meetings, and Data. Mock dashboard reflects the new
provider mix. Number-strip and copy adjusted to match the
twelve-provider catalog.
Replace the two-column "Plan limit + Headroom" usage table with a
five-column view showing Used / Plan / Override / Effective / Headroom.
The override column highlights non-zero entries in the amber accent so
admin-set caps stand out from plan defaults. Headroom bars compute
against effective, not plan, so they reflect what the runtime actually
enforces.
Add OrganizationOverridesDialog — a modal launched from the detail page
that does a partial PUT against /admin/organizations/:id/overrides.
Each numeric field is blank by default (no change on submit), 0
explicitly removes that column's override, and a positive value sets a
new ceiling. The plan default and current effective value are shown
alongside each input so the admin can see what they're about to change
before saving. Notes field is required when the override is granted so
other admins can read why.
When a save succeeds the org detail query is invalidated so the
plan/override/effective columns refresh in place, and the timestamp +
notes footer below the usage table updates accordingly.
Removes DMARC reports, DNS verifier, Postmaster snapshots, and DNS
verifications from the dashboard. Adds connect-drawer fields for the
new catalog set (HubSpot, Salesforce, Pipedrive, Close, Zapier, Make,
n8n, Slack, Discord), keeps Calendly, Cal.com, and Google Sheets.
Category order is now CRM, Automation, Notifications, Meetings, Data.
Wire the write path for the override table from the previous migration:
GET /admin/organizations/:id/overrides view_organizations
PUT /admin/organizations/:id/overrides manage_organizations
PUT is a partial upsert — nil fields leave existing values untouched,
and 0 explicitly removes that column's override (back to plan default
or the product hard cap). Every write stamps granted_by/granted_at and
fires an admin audit log with the diff the admin asked for.
Introduce product-level hard caps in config/constants.go so plans that
advertise "unlimited" still have a real backstop: 200 mailboxes, 500
total campaigns, 100 active campaigns, 100 team members, 1M contacts,
1k daily campaign sends. GetEffectiveLimits resolves per-field as
override > 0 ?: plan ?: hard_cap and now never returns nil pointers,
so downstream limit checks compare against a concrete ceiling on every
plan tier. CanAddMember / CanAddCampaign / CanAddEmailAccount now call
GetEffectiveLimits instead of GetOrganizationLimits, so admin overrides
and the product hard cap both bite at runtime.
AdminOrgDetail surfaces three limit blocks side-by-side — plan, raw
override row (0 = inherit), and effective limits — so the UI can show
exactly where each enforced number came from. Slice 2 UI lands in the
next commit.
Daily creation throttles ("no 1000 new campaigns in one day even on an
unlimited plan") are explicitly out of scope; they need a per-day
counter, tracked as a TODO on the hard-cap block.
The deliverability-data and DNS-write integrations were over-engineered
for the cold-email segment. Postmaster and SNDS require sending volume
our base typically does not hit, and no comparable cold-email tool
exposes DMARC ingestion or native DNS writes. Replaces the catalog with
the standard set: HubSpot, Salesforce, Pipedrive, Close, Zapier, Make,
n8n, Slack, Discord, Calendly, Cal.com, Google Sheets.
Removes dmarc_reports, dmarc_record_rows, postmaster_snapshots, and
dns_verifications tables from the migration. Deletes dmarc.go, dns.go,
cloudflare.go, postmaster.go from the integration package. Prunes the
matching repository methods and HTTP handlers.
Add two new admin permission bits — view_organizations (bit 20) and
manage_organizations (bit 21) — and switch the existing read-only
/admin/organizations routes off the borrowed AdminPermViewUsers bit they
were using as a placeholder. Backfill the three predefined roles
(support, ops, analyst) with view_organizations so existing role
mappings still resolve cleanly; super continues to pick up everything
via AllAdminPermissions.
Migration 000044 adds organization_limit_overrides, the table the next
commit's write path will target. Schema follows the "0 = inherit from
plan" convention from the design discussion: each numeric column
defaults to 0 and a CHECK constraint enforces non-negative values, so
reverting an override is a write of 0 (preserving the granted_by audit
trail) rather than a DELETE.
Bumping the permission count changes every role's numeric bitmask, so
update the make grant-admin role table to match — super is now
4194303, support 1086401, ops 1062960, analyst 1055233.
Replace the OrganizationsPage stub with a real list view over
/admin/organizations: search by name/slug/owner email, status filter
(active / pending deletion / all), and inline counts (members, mailboxes,
campaigns + active) per row. Owner banned status surfaces as a red badge
on the row so abuse review is one glance.
Add OrganizationDetailPage at /organizations/:id that composes
/admin/organizations/:id with /admin/organizations/:id/members. Header
summarises owner / plan / lifecycle in three cards; body renders
usage-vs-plan-limits with green/amber/red bars (over-limit shown in red,
"no cap" for unlimited plans) and a members table tagged with role icons.
When the per-org override layer lands the usage table will gain a
"source" column (plan default vs admin override) so a 0 in an override
column reads as "no admin change" by design.
Keeps the page structure (hero, floating dashboard, three-step connect
diagram, four category sections, webhooks + API, security, at-a-glance,
FAQ, CTA). Rewrites every headline and body in straightforward
descriptive prose. Drops the editorial "Stance" critique section.
Removes em dashes throughout in favor of periods, commas, and
parentheses.
Wire three GET endpoints behind the existing admin middleware so the
admin app can browse workspaces alongside users:
GET /admin/organizations list with q/cursor/limit/sort
GET /admin/organizations/:id detail + plan/sub + limits + counts
GET /admin/organizations/:id/members full member list with joined users
The list query inlines member/email-account/campaign/active-campaign
counts via subqueries so the table can render usage without an extra
fetch per row. Detail layers GetOrganizationLimits + GetOrganizationCounts
on top of the list shape, ensuring admin sees the same numbers the in-app
limit checks enforce.
Gated on AdminPermViewUsers for now since orgs are tightly coupled to
user admin context today; a dedicated ViewOrganizations/ManageOrganizations
pair will land alongside the write paths (per-org overrides, ban scope)
in the next slice.
The admin and marketing site sit outside the compose stack, so `make app`
never started them. Add `make admin` and `make site` to launch each
workspace's dev server (Vite on 5174, Astro on 4321), and update the root
README and admin/README to point at them instead of the old "open
localhost:5174" line that implied `make app` was enough.
Also add `make grant-admin EMAIL=... [ROLE=super|support|ops|analyst]`
plus `make revoke-admin` so the first super-admin can be seeded without
hand-writing SQL. Role bitmasks mirror AdminRolePermissions in
internal/models/admin_permission.go.
Strips the stance section, mock dashboard, 3-step model, "by the numbers"
strip, and FAQ — leaves a clean catalog: hero, four category grids
(Deliverability, DNS, Meetings, Data), and a Webhooks + API block with
the real event types. Each integration card now shows its auth method
explicitly.
Full rebuild matching the deliverability/warmup/developers pages:
HeroAtmosphere, floating dashboard mock, opinionated stance section,
3-step connection model, per-category sections (deliverability, DNS,
meetings, data), webhook event surface with sample payload, security
strip, by-the-numbers footer, FAQ accordion, CTA.
Content frames the tier 1/2 prioritization decided in the planning
session — short catalog (nine providers, all directly load-bearing)
plus an open webhook stream that anyone can build on top of.
Mirrors the backend models in TypeScript and adds React-Query hooks for
catalog, connections, DMARC reports, meeting bookings, DNS verifications,
and the connect / disconnect / verify mutations.
Adds an integrations app module covering the providers from the tier 1/2
plan: Calendly, Cal.com, Google Sheets, Google Postmaster, Microsoft SNDS,
DMARC ingestion, and Cloudflare/GoDaddy/Namecheap DNS. One unified
migration provisions integration_connections, dmarc_reports + record
rows, postmaster_snapshots, dns_verifications, and meeting_bookings.
The service exposes a generic CRUD surface for connection state with
per-provider files for parsing (calendly.go, dmarc.go), HTTP clients
(cloudflare.go, postmaster.go, google_sheets.go), and DNS verification
(dns.go). Inbound webhook routes use per-org URL-embedded secrets so
Calendly/Cal.com/DMARC providers post directly without Warmbly auth.
DNS verifier resolves SPF/DKIM/DMARC + tracking CNAME and surfaces
fixes when a record is missing.
Go CI fails on golangci-lint's gofmt check. Ran gofmt -w against
every file the linter named plus a handful of others that drifted
during the autonomous-fleet work. No semantic changes — alignment
of struct field whitespace and one mis-indented import block.
gofmt -l ./... is now empty; go build + go vet are clean.
admin/pnpm-workspace.yaml carries an 'overrides: picomatch: ^4.0.4'
block (added to mirror the dashboard) that was never reflected in
pnpm-lock.yaml — the lockfile was generated before the overrides were
introduced. Admin CI's pnpm install --frozen-lockfile rejected the
mismatch:
ERR_PNPM_LOCKFILE_CONFIG_MISMATCH Cannot proceed with the frozen
installation. The current 'overrides' configuration doesn't match
the value found in the lockfile.
Ran pnpm install --no-frozen-lockfile to regenerate the lockfile with
the override applied; --frozen-lockfile now passes locally.
Brings in PR #15 (email warmup process 4) plus its preceding commits:
customer-defined warmup routing on premium pool, free-trial warmup +
1 inbox for 14 days, customer webhook subscriptions with HMAC signing
+ retry, bumped default API rate limits to 100 req/s with flat per-
user/per-plan caps, plus dev-fixture additions.
One real conflict: internal/client/smtpimap/imap/client.go added
distinct imports on each side (this branch added 'net' for the
*net.TCPAddr BindIP field; main added 'sync' for a Mutex). Kept both.
Everything else auto-merged additively:
cmd/backend/main.go - imports + handler fields + DI lines
internal/api/handler/handler.go - new fields next to existing ones
internal/api/routes.go - new route group next to existing ones
Full build + test suite pass (no regressions).
Existing CI only ran the changes filter on web/; admin/ (Vite admin app)
and site/ (Astro marketing site) had no coverage.
Adds:
changes.outputs.admin / changes.outputs.site path filters
admin-ci - pnpm install + lint + typecheck + build
site-ci - pnpm install + build (Astro doesn't have a separate
lint/typecheck script today; build catches type errors,
missing imports, broken assets)
ci-status now depends on admin-ci + site-ci so a failure blocks the
branch protection check.
Both new jobs use the same pnpm 10 + node 20 setup as web-ci, with the
per-app pnpm-lock.yaml as the cache-dependency-path. Smoke-tested both
locally before pushing.
Shorter, cleaner path. The 'web-' prefix was redundant given the dir
sits at the repo root next to web/ and is unambiguously the admin web
app. Git tracked the rename so blame + history follow through to the
new location.
Updated README.md and docs/VENDOR_LOCKIN.md references plus the package
README header. No code changes.
Three small follow-ups that turn the fleet management system from 'all
the pieces ship green' into 'actually produces telemetry':
cmd/worker/main.go: go workerService.RunHealth(ctx, 30s) alongside
Heartbeat. The sampler snapshots rolling 1m counters into a WorkerHealth
event via the existing event bus + codec path.
cmd/backend/main.go: background goroutine refreshes
worker_capacity_view every minute via REFRESH MATERIALIZED VIEW
CONCURRENTLY. The assignment loop, Rebalancer, Scaler, and
QuarantineEvaluator all read from the view, so it's the freshness gate
for the whole system.
internal/app/worker/event_send_email.go + health_record.go: classify
every wmail.SendResult into the right counter (auth / rate-limit /
bounce-hard / bounce-soft / success) and record SMTP latency. Falls
back to free-text message classification when the error code is
generic, so signal stays useful as new error paths are added.
End-to-end: a worker that bounces 10% of sends now lands in the
'quarantined' band within 5min of the QuarantineEvaluator tick,
auto-drains via Rebalancer, and triggers a Scaler alert if its
removal drops fleet capacity below the warning threshold.
Three new admin surfaces on the existing Vite/React app, fully wired
against /admin/cloud-credentials, /admin/provisioning-templates,
/admin/provisioning-jobs.
Settings -> Cloud Providers: paste Hetzner API token, masked display,
Test connection button + green/red status banner with account email and
quota when healthy.
Settings -> Provisioning Templates: CRUD with every Hetzner option
exposed (provider, location, datacenter, server_type with cores/RAM/
price, server count, IPv4/IPv6 per server, worker tier, profile,
egress kind, image, placement group, private network, firewall, key=
value labels). Tier-exclusive 'auto-provision' checkbox enforced
client-side. Live cost preview card.
Workers page: amber 'Provision new' button opens two-tab modal (From
template / Custom). After submit, swaps to live state-machine progress
panel with checkmarks per state, polling every 2s via TanStack Query's
refetchInterval (stops on terminal state).
/workers/provisioning-jobs page: in-flight jobs at top (5s refresh),
30d history below with state + provider filters. Row click opens
timeline detail with retry button on failed jobs.
Fallback catalog for Hetzner locations + server types so the template
form stays usable even when the backend catalog endpoints are down
(small amber 'using built-in fallback list' note).
pnpm install + typecheck + build all clean. ~570KB production bundle.
Constructs CloudCredentialRepository, ProvisioningTemplateRepository,
ProvisioningJobRepository, ProvisioningPolicyRepository alongside the
existing StorageBackendRepository and threads them onto Handler.
Spawns three goroutines on the root context for the fleet loops
(Rebalancer, Scaler, QuarantineEvaluator) so the autonomous management
starts immediately on backend boot. All three cancel cleanly on
shutdown via the root context.
/admin/cloud-credentials CRUD + /:id/test (Hetzner Verify)
/admin/cloud-providers/:p/locations
/admin/cloud-providers/:p/server-types catalog for admin form dropdowns
/admin/cloud-providers/:p/images
/admin/provisioning-templates CRUD
/admin/provisioning-jobs list, detail, create (from
template or inline custom config)
/admin/provisioning-policy list, update (per-provider budget
caps + AUTO_PROVISION toggle)
All gated by AdminPermManageSettings (jobs use AdminPermManageWorkers).
Creating a job snapshots the template into the row's config jsonb so
mutating the template later doesn't retroactively change in-flight or
historical jobs.
The catalog endpoints proxy directly to the Hetzner API client,
returning whatever Hetzner exposes — admin UI doesn't need to know
the provider-specific shape, dropdowns just render Locations/
ServerTypes/Images verbatim.
Three closed-loop background goroutines on the backend that manage the
worker fleet without operator intervention.
Rebalancer (default 5min): for each tier, drain hot workers (>80%
utilization) onto cold workers (<50%, healthy). Safety rails: per-
mailbox 24h cooldown to prevent thrashing, max 200 in-flight migrations,
destination must be healthy or watch.
Scaler (default 1h): compute fleet utilization per tier. At >=70%
sustained, emit warning. At >=85% sustained, emit critical alert. If
AUTO_PROVISION is allowed by provisioning_policy, snapshot the active
auto-template for the tier into a new provisioning_jobs row — the state
machine picks it up and provisions the box without admin click.
QuarantineEvaluator (default 5min): inspect rolling 1h bounce/complaint
rates, transition workers between health bands (healthy / watch /
throttled / quarantined / blocked) using CLAUDE.md thresholds.
Quarantined and blocked workers are auto-drained by the Rebalancer
because ListCapacityCandidates excludes them.
Every action is written to decision_log so the admin Decisions page can
answer 'why did the system do X'.
POST /api/v1/internal/worker/heartbeat now inserts a row into the
workers table the first time an unknown worker_id checks in. Tier and
egress_kind come from the heartbeat body; subsequent heartbeats just
keep ip_addr fresh.
Means provisioned workers self-register without admin clicks — the
state machine waits for all 16 expected UUIDv5(IP) workers to ping in
during the verify step, then marks the job completed.
tierToColumns collapses the higher-level (shared_free / shared_premium /
dedicated) name down into the existing (worker_type, free_tier) columns
so the rest of the assignment logic keeps working unchanged.
Workers emit a WorkerHealth event every 30s with assigned mailbox count,
IMAP IDLE connections, memory, goroutines, and rolling 1h send/bounce/
complaint/auth-error/rate-limit counters. Consumer writes them to
worker_health_samples.
Schema additions on workers: egress_kind (cold_smtp / oauth_api /
warmup_only), health_state (healthy / watch / throttled / quarantined /
blocked), load_score (weighted utilization).
worker_capacity_view aggregates the latest hour of samples into a
per-worker capacity row used by the assignment loop. Effective capacity
= base_ceiling(egress_kind) × health_multiplier × age_ramp_multiplier
so a fresh worker earns its way up to base capacity over 72h, and a
worker with rising bounces or complaints automatically gets less load.
MailboxWeight returns 1.0 for cold_smtp, 0.05 for Gmail/Graph API
(worker IP doesn't matter), 0.4 for warmup-only. AssignWorkerToEmail
resolves the mailbox's weight and SelectSharedWorker filters by
headroom + sorts by utilization, so a 200-mailbox OAuth worker and a
16-mailbox cold worker balance fairly.
UnassignWorkerFromEmail refunds the load_score symmetrically.
provisioning.Service.Run drives a provisioning_jobs row through the
full lifecycle (create_server, create_ips, assign_ips, set_rdns,
install, verify). Each step records progress to the DB before the
next transition so a backend crash mid-provision is resumable from
the current state.
JobConfig is the in-row snapshot of the template (or inline custom
config) at the time of submission — mutating the template later
doesn't retroactively change in-flight or historical jobs.
WorkerIDForIP is the canonical UUIDv5-from-IP helper, kept in sync
with cmd/worker and scripts/install-worker.sh so the state machine
can compute the expected worker_ids before they heartbeat.
Installer is a tiny SSH-driven interface (Install only) with a
StubInstaller for tests. The real impl plugs in worker_orchestrator
or any other transport (cloud-init, Ansible) without touching the
state machine.
On any step failure, rollback unassigns + deletes provider-created
IPs and deletes the server, leaving no orphaned resources at the
provider before marking the job failed.
pg_provisioning.go covers cloud_credentials, provisioning_templates,
provisioning_jobs, and provisioning_policy. Job repo exposes the
state machine helpers the orchestrator needs (UpdateState,
RecordServer, AppendIPs, AppendWorkerIDs, MarkFailed, MarkCompleted)
so the package doesn't reach through *db.DB itself.
pg_decision_log.go is a thin insert + recent-history reader for the
audit trail powering the admin Decisions page.
cloudprovider.Provider interface (Locations, ServerTypes, Images,
Verify, CreateServer/DeleteServer, CreatePrimaryIP/AssignPrimaryIP/
UnassignPrimaryIP/DeletePrimaryIP/SetReverseDNS). One impl today
(Hetzner Cloud); adding OVH or Vultr later means implementing the same
six surfaces.
hetzner.Client is a minimal idiomatic Go REST client over
https://api.hetzner.cloud/v1. Bearer-token auth. Returns provider-
native IDs as strings so the orchestration layer can persist them for
rollback.
9 tests against httptest.Server covering token transmission, error
surfacing, parsing, request-body shape, and interface conformance.
Foundation for autonomous fleet management.
cloud_credentials stores encrypted API tokens per cloud provider.
worker_profiles bundles the env vars that get rendered into
/etc/warmbly/worker.env at install time.
provisioning_templates is a customizable saved config — every Hetzner
option the admin form exposes lives here, so the cheapest-US-single-IP
setup is a one-click pick once you've saved it.
provisioning_jobs is the state machine (pending -> creating_server ->
creating_ips -> assigning_ips -> setting_rdns -> installing ->
verifying -> completed | failed -> rolling_back).
provisioning_policy is per-provider budget caps + the auto_provision
toggle the scale loop checks.
decision_log records every automated action so admins can audit what
the system did and why.
README is now a landing-page-style entry point: centered hero banner,
badges, tagline, feature grid, architecture diagram, self-hosting
table, stack rationale, quick start, project layout, testing summary,
docs index. Image placeholders reference docs/assets/{banner,
dashboard-preview,admin-preview}.png with sizing guidance in
docs/assets/README.md.
docs/INTERNAL_API_AUTH.md walks through the bearer-token model the
worker uses to call /api/v1/internal/* endpoints: where the token
lives on both sides, the constant-time compare, fail-closed semantics
when INTERNAL_API_TOKEN is unset, and the planned per-worker JWT
upgrade path.
docs/VENDOR_LOCKIN.md is the honest scorecard: each external
dependency (AWS KMS, DynamoDB, S3, Cloud Tasks, Stripe, Turnstile,
Sentry) gets a row covering self-host status, alternatives, and what
'free of this dependency' actually looks like. Includes the minimum-
viable self-host env-var set.
Matt
Matthew Meszaros