pg_provisioning.go covers cloud_credentials, provisioning_templates,
provisioning_jobs, and provisioning_policy. Job repo exposes the
state machine helpers the orchestrator needs (UpdateState,
RecordServer, AppendIPs, AppendWorkerIDs, MarkFailed, MarkCompleted)
so the package doesn't reach through *db.DB itself.
pg_decision_log.go is a thin insert + recent-history reader for the
audit trail powering the admin Decisions page.
cloudprovider.Provider interface (Locations, ServerTypes, Images,
Verify, CreateServer/DeleteServer, CreatePrimaryIP/AssignPrimaryIP/
UnassignPrimaryIP/DeletePrimaryIP/SetReverseDNS). One impl today
(Hetzner Cloud); adding OVH or Vultr later means implementing the same
six surfaces.
hetzner.Client is a minimal idiomatic Go REST client over
https://api.hetzner.cloud/v1. Bearer-token auth. Returns provider-
native IDs as strings so the orchestration layer can persist them for
rollback.
9 tests against httptest.Server covering token transmission, error
surfacing, parsing, request-body shape, and interface conformance.
Foundation for autonomous fleet management.
cloud_credentials stores encrypted API tokens per cloud provider.
worker_profiles bundles the env vars that get rendered into
/etc/warmbly/worker.env at install time.
provisioning_templates is a customizable saved config — every Hetzner
option the admin form exposes lives here, so the cheapest-US-single-IP
setup is a one-click pick once you've saved it.
provisioning_jobs is the state machine (pending -> creating_server ->
creating_ips -> assigning_ips -> setting_rdns -> installing ->
verifying -> completed | failed -> rolling_back).
provisioning_policy is per-provider budget caps + the auto_provision
toggle the scale loop checks.
decision_log records every automated action so admins can audit what
the system did and why.
New storage_backends table is the runtime inventory of pluggable
infrastructure choices (KMS, encrypted_keys, blob, eventbus, cache).
Each kind has exactly one active row, enforced via a partial unique
index. Read-only rows are env-var driven; UI-mutable rows can be
flipped via SetActive.
settings.Registrar reflects boot-time backend choices into the table
so the admin UI sees what's actually running.
New admin endpoints under /admin/settings/backends:
GET /settings/backends?kind=...
GET /settings/backends/active/:kind
POST /settings/backends/:id/activate
New internal endpoints under /api/v1/internal:
GET /worker/config - workers fetch runtime config on boot
POST /worker/heartbeat - liveness ping
(DEK endpoints added in the encryptedkeys commit.)
handler.Handler grows EncryptedKeys + StorageBackendRepo fields.
5 registrar tests cover create / update-and-activate / skip-when-active /
lookup-error propagation / RegisterAll stop-on-first-error using a
mock repository.
WorkerService now holds eventbus.EventBus + codec.Codec instead of
*kafka.Producer / *kafka.Consumer. Receive() satisfies the
eventbus.Handler signature; Produce() goes through Codec.Serialize +
Bus.Publish.
events.Publisher likewise switches to (bus, codec) and stops
serializing via *kafka.Avrov2 directly.
The kafka package and its Avrov2/Producer/Consumer types remain for
the few non-worker call sites (tracking consumer, validate_credentials)
that haven't been migrated yet; bundled with KafkaBus.Producer() so
existing Avro framing on Kafka is preserved.
Worker boot wiring is split into cmd/* entry-point commits.
Codec interface (Serialize / Deserialize / Name) lets payload encoding
decouple from the transport choice. Two implementations:
AvroCodec - wraps the existing kafka.Avrov2 Schema Registry client
via NewAvroFromClient. Preserves identical wire format
for production deployments already on Kafka + SR.
JSONCodec - encoding/json based. No external dependency, suitable
for self-hosters who don't want a Schema Registry.
Factory FromEnv selects via CODEC_PROVIDER (default avro).
Once codec.Codec is in the worker boot and publisher, self-hosters can
pick EVENTBUS_PROVIDER=nats CODEC_PROVIDER=json for a Schema-Registry-
free deployment.
11 tests cover JSON round-trip, nil guards, factory paths, and Avro
interface conformance.
Transport-agnostic EventBus (Publish / Subscribe / Close / Name) with
two implementations:
KafkaBus - wraps the existing internal/infrastructure/kafka producer
and consumer. Existing call sites that use the kafka
package directly keep working.
NATSBus - JetStream-backed. File storage, 7d max age, durable
consumers per group, manual ack with redelivery on
handler error.
Factory FromEnv selects via EVENTBUS_PROVIDER (default kafka). NATS
needs NATS_URL; Kafka reuses the existing bootstrap+SASL config.
Subject naming converts Kafka colons to NATS dots so the existing
w:<uuid> topic naming works on both backends.
16 tests including round-trip ack-redelivery against an embedded
nats-server.
New encryptedkeys.Store interface with three impls:
postgres - backend default, durable via PG
dynamodb - existing AWS path, also covers Scylla Alternator via
AWS_ENDPOINT_URL_DYNAMODB
http - worker-side adapter that talks to the backend's new
/api/v1/internal/dek/:userID endpoint, so workers never
connect directly to Postgres
The HTTP endpoint sits behind a new InternalAuthMiddleware that does
constant-time bearer-token compare against INTERNAL_API_TOKEN. Fail-
closed if the env var is unset.
cipher.Service now takes an encryptedkeys.Store instead of a Dynamo
repository. The old internal/repository/dynamo_user_encrypted_keys.go
is deleted (the file also had a pre-existing copy-paste bug using
EmailMessageMapTable in Get/Del that's gone with it).
New migration 38 adds user_encrypted_keys (user_id PK, encrypted_data_key,
created_at, updated_at).
20 tests cover HTTP round-trip, conflict semantics, factory selection,
middleware auth (fail-closed / wrong-scheme / timing-safe / happy path),
and DEK handler responses through gin's test harness.
Seven call sites stop reaching through the embedded *s3.Client and
instead use the high-level storage.Store methods. Same runtime behavior
on the AWS path; opens the door to the Filesystem backend for
self-hosters.
avatar.go retains an S3-specific path for public-ACL + cache-control
on uploaded avatars and falls back to ServiceUnavailable on non-S3
backends. A future PublicStore interface could clean that up.
unibox/storage.go GetBody now propagates the emsg.DecodeBinary error
that the original code dropped on the floor.
One Store interface (Get/Put/Delete/Has/PresignedGetURL/Name) covers
both the AWS S3 client (now exposes high-level methods alongside the
legacy embedded *s3.Client) and a new FilesystemStore for self-hosters.
FilesystemStore writes atomically via temp-file + rename, rejects '..'
in keys before path normalization, returns ErrUnsupported from
PresignedGetURL.
The S3 impl works against AWS S3, MinIO, Cloudflare R2, Backblaze B2,
and Hetzner Object Storage via standard AWS endpoint-URL config.
Factory NewFromEnv selects via BLOB_PROVIDER (defaults to s3).
14 tests cover round-trip, ErrNotFound, atomic-write cleanup, traversal
rejection, factory env-selection paths.
SMTP and IMAP clients now expose a BindIP *net.TCPAddr field. When set,
outbound TCP binds to that IP; otherwise the netbind helper falls back to
WORKER_BIND_IP env var, then default route.
IMAP switched from raw tls.Dial to netbind.TLSDialer so the source IP is
configurable; behavior identical when BindIP is nil.
No call-site changes required: backwards compatible.
Tiny helper that builds a *net.Dialer (and tls.Dialer wrapper) with an
optional LocalAddr. Used by the SMTP and IMAP clients so a worker on a
multi-IP box can bind outbound TCP to a specific source IP.
Bind IP comes from an explicit *net.TCPAddr on the client, or falls back
to the WORKER_BIND_IP env var, or finally to the OS default route. The
env-var path is cached once via sync.Once.
4 tests cover explicit bind, nil-passthrough, tls.Dialer composition,
and timeout sanity.
Define kms.Provider so the cipher service can swap between AWS KMS and
a self-hostable local-key path. Local impl uses AES-256-GCM with the
master key sourced from KMS_LOCAL_MASTER_KEY (base64) or KMS_LOCAL_MASTER_KEY_FILE.
Factory FromEnv selects at boot via KMS_PROVIDER; defaults to aws for
backwards compatibility.
Ciphertext blob format is opaque: switching providers requires a DEK
migration because each provider can only decrypt its own blobs.
10 tests cover round-trip, tamper detection, nil-key rejection, and
factory env-selection paths.
the multiplier was a no-op since the default already sat at 1.0, and
plan tiers / admin overrides express the actual ceiling more clearly
as a fixed limit_*_pm value. enterprise customers who need more
throughput now get a direct bump on user_rate_limits.limit_*_pm
instead of an indirect multiplier.
migration 43 drops the column from both tables. service.go uses the
base limit as the ceiling. UpdateUserRateLimits no longer accepts a
burst_multiplier field.
BurstMultiplier was 2.0, which let clients briefly hit 200 req/s before
throttling. roll it back to 1.0 — predictable ceiling beats a peak that
can mask real traffic patterns.
- webhook_endpoints / webhook_deliveries schema (migration 42)
- service: dispatch + endpoint crud + hmac-sha256 signing
- delivery worker drains queue using FOR UPDATE SKIP LOCKED so multiple
api replicas can run safely without duplicate dispatch
- exponential backoff (30s → 1h cap, 8 attempts then abandoned)
- REST API under /webhooks: list/create/update/delete/rotate-secret/
list-deliveries. secret only returned at create + rotate
- header convention matches stripe-style: X-Warmbly-Signature: t=<unix>,v1=<hex>
- legacy header X-Warmbly-Token already renamed; new outbound webhook
headers are X-Warmbly-Signature / X-Warmbly-Event / X-Warmbly-Event-Id
- wired into email account connect/remove and warmup health transitions;
campaign/tracking/deliverability call sites will reuse the same
webhookService.Dispatch interface
DefaultRateLimits now permits 6000 read + 6000 write per minute = 100 req/s
sustained, with burst multiplier 2.0 (200 req/s peak). bulk operations
stay tighter at 600/min since each one is expensive. migration 41 bumps
existing rows that still hold the previous defaults — admin-customized
limits are left untouched.
new warmup_routing_rules table + repo + REST endpoints under /warmup/
routing. each rule matches a (sender, recipient) pair by domain, TLD,
provider bucket, or any wildcard, with a weight multiplier on the
selector. weight > 1 prefers the pairing, < 1 discourages, 0 excludes.
rules are evaluated in priority order (ascending) and combined with the
existing domain-diversity weighting. example use case: a customer can
say 'send Gmail-recipient warmup only from Google-classified senders'
with one rule; or 'never send to acme.com from this org's mailboxes'
with weight=0. premium pool only — free pool ignores rules.
introduce two distinct warmup_spam_reports report types:
- spam_placement: provider classifier put the message in Junk on arrival
- user_complaint: recipient explicitly flagged the warmup message
detect placement at warmup arrival via the incoming message flags and
record it through a new WarmupService.RecordSpamPlacement. evaluate the
two signals independently in the health sweep — user complaints now
have their own watch/quarantine/block thresholds (0.5 / 1.5 / 3.0%)
since they are a stronger negative signal per event than placement.
open CanUseWarmup and CanUseUnibox to free-trial orgs during the 14-day
window. add CanAddInbox feature gate and a FreeTrialInboxLimit of 1 so
trial orgs cannot seed the warmup pool with disposable accounts. enforce
the cap on both OAuthStart (avoids wasting the round-trip) and
OnboardSMTPIMAP, with distinct error messages for cap-reached vs
trial-expired.
the two cipher.Encrypt calls on subject and body in HandleEmailTask
discarded their ciphertext and the plaintext was what got sent. warmup
mail is not stored at rest, so the only effect was warming the user's
DEK in cache via the preceding Cipher() call — also removed since
nothing else in the warmup path needs it.
rename outbound warmup verification header from X-Warmbly-Token to a
generic X-Mailtrace-Verify and accept both on receive during rollout.
add slot-based subject synthesis that fires ~40% of the time, yielding
thousands of unique strings on top of the static corpus so vendors
cannot trivially cluster on a fixed subject list.
extend recent-partner exclusion from 24h to 72h, weight selection by
inverse-frequency of recipient domain over the last 7 days so a single
provider does not absorb the bulk of warmup traffic, and log a warning
when a pool drops below the diversity threshold.
persist conversation_theme on the warmup_token at send time, and look
it back up on reply so the body comes from the same topical bucket.
previously the reply picked a random conversation, so 'Re: time-
blocking question' could come back as a paragraph about travel.
CalculateNextWarmupTime now consults the participant's health state and
scales target_volume + min_wait_time down for throttled (0.5x volume,
2x spacing) and watch (0.7x volume, 1.5x spacing). previously the
state was set by the sweep but never read by the scheduler, so
'throttled' was a label without behavior.
mark_read, mark_important, remove_from_spam, and move_to_warmbly now run
on outlook and custom smtp/imap mailboxes, not just gmail. routes by
provider in the worker handler. carries source mailbox UIDValidity on
the action so the worker can SELECT the right folder before mutating.
PR #10 added 000035_api_key_suffix and PR #11 separately added
000035_api_key_smart_limits, both targeting the api_keys table.
golang-migrate refuses to load when two source files share a
version number, so the backend container failed health checks
in fresh stacks.
Bumping api_key_smart_limits to 000037 keeps PR #10 in its
original slot (it landed first) and matches the next free
version after 000036_contact_categories. The two migrations
touch different columns so apply order does not matter.
- gofmt-align ContactEngagement fields and drop trailing blank line
in contact/export.go
- remove no-op self-assignment ac.CustomFields = ac.CustomFields
flagged by govet
Matthew Meszaros