Commit Graph
41 Commits
Author SHA1 Message Date
Matthew Meszaros 51dedc90ee feat: put every runtime behind one optional error-reporting story: a single internal/observability/errs wrapper that is now the only package importing sentry-go, InitSentry for cmd/forms, release and environment tags on every service from the existing build stamp, optional Sentry in the admin panel and the public forms app, the sentry crate in the Rust tracking service, release tagging in realtime, CI source-map upload that only runs when a Sentry token is configured, and docs covering the DSN for each service 2026-09-07 03:51:06 -07:00
Matthew Meszaros 3bdb0fb82d feat: address the Greptile review on the pool link: require https for the cloud URL (loopback exempt for local development) since the instance token and mailbox passwords travel on it, remove the cloud copy when the local enrollment row cannot be written so a mailbox never warms in both places, delete the local enrollment row before the cloud one and restore it if the cloud call fails so a mailbox is never left with no warmup anywhere, and trim the new multi-line comments to the one-line style 2026-08-29 07:17:10 -07:00
Matthew Meszaros b75fdcf86c Merge remote-tracking branch 'origin/main' into feature/self-hosted-warmup-pool-access
# Conflicts:
#	internal/scheduler/warmup_scheduler.go
2026-08-29 07:10:38 -07:00
Matthew Meszaros 37b60b59d3 feat: let a self-hosted instance warm its mailboxes in the hosted pool: device-code link approved at /connect, instance-token API that enrolls SMTP/IMAP mailboxes as warmup-only accounts (no history import, non-warmup mail dropped), free for 10 mailboxes and unlimited on the seeded $15 pool plan, tier fallback to proven healthy mailboxes when a pool runs thin, local warmup stands down for enrolled mailboxes, Settings > Warmbly Cloud step flow and linked-instances page, docs guide, marketing copy, and fix SetWarmupLifecycle re-reading the row with an org-scoped lookup so every warmup start/pause returned 404 2026-08-29 07:09:04 -07:00
Matthew Meszaros 230d80db64 feat: condense the new warmup pool comments in email_task.go, handler.go, service.go, pg_worker.go and warmup_scheduler.go to the one-line form the repo convention asks for, keeping the non-obvious constraints (the stored tier is never empty, the tier column records what is paid for rather than where the mailbox warms) and dropping the narration 2026-08-28 21:31:06 -07:00
Matthew Meszaros c28ef3e82a feat: keep a restricted or suspended workspace out of the premium warmup pool by checking the organization's risk posture before the stored warmup_pool_type in resolveWarmupPoolType (tasks and email services), which is never empty so the restricted branch below it was dead code, make the scheduler's recipient-capacity count risk-aware the same way, wire the org risk repository into the email service, and refuse the premium membership move in UpdateEmailAccountWarmupPoolType while the owning organization is restricted so a worker rebalance cannot readmit a risky tenant 2026-08-28 21:26:57 -07:00
Matthew Meszaros b019174910 feat: give an organization one fused abuse posture (#233)
* feat: give an organization one fused abuse posture, because every existing control watches a single subject and an actor slightly wrong on several axes sits under all of them: organizations gains risk_state, risk_score, risk_reason and an append-only risk_signals evidence blob, modelled on the warmup participant health machine that already works rather than a second vocabulary for the same idea; restricted cuts per-mailbox cold volume to a quarter and forces the free warmup pool so a risky tenant cannot spend the paid pool's shared reputation, suspended stops sending at the send gate, and watch deliberately changes nothing a customer can feel so evidence accumulates before anything is taken away; an operator's suspension outranks the derived band so a detector clearing cannot release a workspace a human suspended, transitions ride the audit spine to every teammate's dashboard, a banner says which limit is active and why rather than letting volume drop silently, and the posture never travels in a workspace archive since it is one platform's verdict reached from evidence the destination never saw

* feat: make the suspension actually stop sending, and emit the audit transitions the PR claimed: emailsend.SendEmail is only the manual and API path, so campaign and warmup sends went nowhere near the gate and a suspended workspace kept sending on its schedule, while the restricted multiplier floored every mailbox at one a day which quietly turned suspension into a trickle rather than a stop; the campaign scheduler now defers the whole campaign with a reason and the warmup task skips as org-suspended, since warmup is outbound mail from the same domains; separately the band change emitted no audit entry at all despite the entity type and the frontend spine entry both existing, so no banner moved for a teammate and there was no trail of who was restricted when, and only a real transition is logged so a detector re-recording the same finding cannot fill the feed; one of my own live tests also asserted how far out a slot lands, which depends on the hour the suite runs, and now asserts the property it was about

* chore: drop a test that belongs to the signup-risk change and reached this branch by mistake, where the type it exercises does not exist
2026-08-28 11:20:19 -07:00
Matthew Meszaros d47d31b7c4 feat: let a warmup recipient answer the mailbox that wrote to it (#230)
* feat: let a warmup recipient answer the mailbox that just wrote to it, so a thread reads as a conversation rather than two mailboxes monologuing on their own ramps: warmup_tasks.target_account_id was written as nil and never read by anything, so a reply only happened when the recipient's own ramp fired AND the draw happened to land on that partner; a verified receipt now sometimes re-points the recipient's pending warmup task at the sender 25 minutes to 5 hours later inside its own warmup hours, which is a re-pointing rather than new work because only one warmup task may be pending per mailbox, it can never delay a send the mailbox had planned sooner, and it stops before the thread cap so replies cannot answer replies forever; the clock parser also moves into models.ClockMinutes so a second copy of the HH:MM parsing that silently disabled every sending window cannot drift back in

* feat: stop the reply-back drawing the reply rate twice, and stop its jitter escaping a short warmup window: the scheduler drew the recipient's reply rate to decide whether to answer at all, then the task handler drew it again to decide reply-versus-new, so a 30 percent reply rate produced a 9 percent answer rate and a directed task could send a fresh message to the mailbox it was meant to be answering; a directed task now IS the reply, and the opening-time jitter is capped to the window width so a mailbox warming 09:00 to 09:20 is not scheduled past its own close
2026-08-28 10:40:25 -07:00
Matthew Meszaros bc9978dab3 feat: feed per-provider placement back into warmup partner routing (#228)
* feat: feed each sender's per-provider spam placement back into warmup partner selection, so a mailbox failing only at Microsoft stops being handed Microsoft partners instead of waiting for an aggregate band to trip: pickWeightedPartner weighted on inverse domain frequency and routing rules alone, while PoolSpamPlacementsByProvider's segmentation only ever reached the admin overview; the new per-sender query keys on who RUNS the recipient's mail via models.ClassifyProvider rather than email_accounts.provider, because that enum collapses every custom host into smtp_imap which is exactly the bucket the signal matters in, and the weight multiplies by 1/(1+4*rate) with a five-send floor so a failing provider is downweighted but never excluded, since a sender that stops mailing a provider can never discover it recovered

* feat: stop failed warmup sends flattering a provider's junk rate, and stop a re-admitted partner sidestepping the penalty: a warmup_tokens row is written before the send leaves, so counting every token put failed sends in the denominator and understated the rate exactly when the sender was doing worst; the send count now joins tasks on status = completed, and GetPoolParticipantProviders is read unfiltered because that map only resolves a candidate's provider and never decides eligibility, so a just-unblocked partner missing from it scored an unpenalized 1.0

* feat: move the campaign content-lint work off this branch, where it landed by mistake: the pre-send scoring, its preflight check and the warmlint image and attachment heuristics belong to the content-lint change and are unrelated to per-provider partner routing, so this branch now carries only the routing signal it is about
2026-08-28 10:20:35 -07:00
Matthew Meszaros 4b1ccf0bed feat: keep an indefinite warmup block indefinite when migration 000097 collapses a dual membership: a blocked row carries blocked_until NULL to mean 'until an appeal succeeds', so MAX(blocked_until) skipped it as a missing value and adopted a sibling row's finite expiry, handing a permanently blocked mailbox (tampering) a release date and making it re-evaluatable by UpdateParticipantHealth, which refuses to touch a block only while blocked_until IS NULL; the merge now treats any indefinitely blocked row in the partition as the longest block there is, with a live test that an indefinite block survives a pool move, and the Go comments added by this branch are trimmed to the constraint they state 2026-08-27 03:21:08 -07:00
Matthew Meszaros cbb0b8f662 feat: keep a mailbox in exactly one warmup pool so a plan change moves it instead of leaving it in both: EnsurePoolMembership joined whichever pool the org is entitled to right now while RemovePoolMembership only ever deleted that same pool's row, so a downgraded mailbox kept its premium membership and went on being handed to paying customers as a warmup partner while every spam-score increment was counted once per row by the health evaluator; membership is now a move (one upsert keyed on the account that carries health state, block and spam score across, so changing pool cannot launder a penalty), removal is never pool-scoped, a tier change moves the participant row in the same transaction as email_accounts.warmup_pool_type, a new reconcile pass in the backend's warmup sweep evicts participants whose workspace lost warmup access and repairs pool drift for the mailboxes no warmup task ever revisits, an unreadable subscription no longer counts as "not entitled" anywhere that evicts, GetSpamScore takes the max rather than the sum and IncrementSpamScore clamps to the column ceiling instead of silently failing its CHECK, and migration 000097 merges any existing dual membership worst-wins before making one-pool-per-mailbox a unique index 2026-08-27 03:02:56 -07:00
Matthew Meszaros b77664cd09 feat: verify warmup mail that arrives without its verify header, because Microsoft Graph strips custom headers in transit and re-stamps the Message-ID, so every warmup email sent from an Outlook or Microsoft 365 mailbox reached its recipient unmarked, consumed no token, earned no engagement and was filed in the recipient's unibox as ordinary mail; the Graph client now creates the message as a draft and sends that so it can read the internetMessageId Exchange assigned, the consumer records that delivered id on the task and its warmup token alongside the sent subject, and the recipient resolves an unmarked warmup email by delivered Message-ID or by the pending sender/recipient/subject pair 2026-08-26 04:31:19 -07:00
Matthew Meszaros 1018068942 feat: stop recipient suppression and the entitlement gate being skipped when a campaign has no organization: the send path now fails closed (an orgless campaign is paused with the reason in its activity feed instead of mailing an unsubscribed, bounced or complained address, since routing's own suppression filter joins on the campaign's organization_id and matches nothing when it is NULL), the same tenancy gate covers warmup and unibox sends and an orgless mailbox resolves to the free pool rather than defaulting into the paid one, the state is no longer creatable (sequenceRepository.Create inherits organization_id from its campaign instead of inserting NULL, campaign creation and mailbox onboarding refuse without a workspace via errx.ErrNoOrganization, guardInboxLimit / guardMailboxThrottle / the campaign creation throttle no longer treat a missing org as exempt, and a new session resolves the user's default workspace so the orgless session that produced these rows cannot exist), and migration 000092 backfills then sets organization_id NOT NULL on campaigns, contacts, email_accounts and sequences plus sessions.current_organization_id, provisioning a recovery workspace for any user with none so no row is deleted to satisfy the constraint (live-tested in TestLiveOrglessCampaignDoesNotSendToSuppressedRecipient, TestLiveSuppressedRecipientIsSkipped, TestLiveCampaignRequiresAnOrganization and TestLiveHealthyCampaignStillSends) 2026-08-24 08:47:08 -07:00
Matthew Meszaros 3739a36b67 feat: enforce the persisted SPF/DKIM/DMARC state as a real cold-send and warmup gate behind a 72h grace clock and an operator toggle, after first fixing the DMARC organizational-domain fallback in dnsauth so a dedicated sending subdomain covered by its parent's record stops reading as unauthenticated, adding auth_state to the four mailbox loaders that never selected it (which would have made the gate dead code), stamping auth_failing_since on entry to failing so a resolver hiccup can never stop a campaign, notifying the org on that transition, and reporting an all-gated pool as ErrDomainAuthFailing instead of a message about sending windows (#160) 2026-08-22 09:37:26 -07:00
Matthew Meszaros f768029ec0 feat: automate warmup conversation generation, coherent replies, adaptive rotation, and admin observability 2026-07-22 12:11:19 +02:00
Matthew Meszaros 173736a004 feat: add a Stripe-grade webhook platform with a typed event catalog, HMAC-signed delivery, retries with backoff, endpoint ownership verification, per-endpoint throttling, an audit-spine event bridge, and firehose emit sites (migration 000048) 2026-06-15 08:11:20 +02:00
Matthew Meszaros ac3c11bf9a feat: seal outbound email content with the organization DEK
The send pipeline (publisher subject/body encryption, the S3 emsg blob,
and the worker-side decrypt) now keys off models.SendEmail.OrgID
instead of UserID. EmailMessage loses its UserID field entirely:
emailSender.Send derives the cipher identity from the email account's
OrganizationID and refuses to send for an account without one.

This also fixes two latent bugs. Emails sent through user_email_task
never set UserID, so they were silently encrypted under the zero-UUID
platform key. campaign_task's discarded Encrypt() pair is replaced with
an explicit DEK warm that fails fast when KMS is unavailable.
2026-06-10 17:16:42 +02:00
Matthew Meszaros 52bcc163fb feat: fix warmup reply threading
Persist outbound warmup Message-IDs so later warmup replies can find thread parents.

Normalize reply headers and re-check partner health during recipient selection before using a mailbox as a warmup recipient.
2026-06-03 06:40:27 +02:00
Matthew Meszaros 47bdf66899 feat: improve warmup health analytics
Record warmup spam placements by recipient provider and surface provider breakdowns in the admin health summary.

Track warmup replies and use stable static conversation ids so warmup analytics can correlate content reliably.
2026-06-03 06:26:39 +02:00
Matthew Meszaros 15ef9d4994 feat: add warmup content controls
Add warmup content generation and admin review surfaces, plus mailbox warmup appeal/status APIs.

Track warmup engagement and tampering signals so unsafe mailboxes can be handled by the warmup flow.
2026-06-03 05:05:53 +02:00
Matthew Meszaros aa391b5e0c feat: fix warmup reconciler guards 2026-06-01 11:27:44 +02:00
Matthew Meszaros a04d7450bc feat: guard warmup reconciliation by access
Make the warmup task reconciler re-check account state and org warmup access before scheduling replacement tasks. Remove pool membership for accounts whose org can no longer use warmup to avoid repeated no-access task churn.
2026-06-01 11:19:44 +02:00
Matthew Meszaros 0cb53a25a6 feat: improve warmup pool scheduling
Add explicit warmup participant roles so connected accounts can provide recipient-only pool capacity without becoming senders. Cap warmup scheduling by eligible recipients, avoid same-day recipient reuse, reschedule exhausted pools, and reconcile missing warmup task chains.
2026-05-31 18:07:13 +02:00
Matthew Meszaros 946cd087e5 feat: cap active campaign schedulers
Limit organizations to 50 active campaigns before starting another campaign, make campaign wakeups idempotent per campaign, keep warmup wakeups idempotent per mailbox, and no-op stale non-pending campaign/warmup dispatches.
2026-05-31 04:48:39 +00:00
Matthew Meszaros 43d24bc3d7 feat: customer-defined warmup routing rules on premium pool
new warmup_routing_rules table + repo + REST endpoints under /warmup/
routing. each rule matches a (sender, recipient) pair by domain, TLD,
provider bucket, or any wildcard, with a weight multiplier on the
selector. weight > 1 prefers the pairing, < 1 discourages, 0 excludes.
rules are evaluated in priority order (ascending) and combined with the
existing domain-diversity weighting. example use case: a customer can
say 'send Gmail-recipient warmup only from Google-classified senders'
with one rule; or 'never send to acme.com from this org's mailboxes'
with weight=0. premium pool only — free pool ignores rules.
2026-05-25 15:42:50 +00:00
Matthew Meszaros b4ece91383 perf: drop dead cipher.Encrypt calls from warmup task
the two cipher.Encrypt calls on subject and body in HandleEmailTask
discarded their ciphertext and the plaintext was what got sent. warmup
mail is not stored at rest, so the only effect was warming the user's
DEK in cache via the preceding Cipher() call — also removed since
nothing else in the warmup path needs it.
2026-05-25 15:25:06 +00:00
Matthew Meszaros e8f741dc82 feat: reduce warmup content + header fingerprint
rename outbound warmup verification header from X-Warmbly-Token to a
generic X-Mailtrace-Verify and accept both on receive during rollout.
add slot-based subject synthesis that fires ~40% of the time, yielding
thousands of unique strings on top of the static corpus so vendors
cannot trivially cluster on a fixed subject list.
2026-05-25 15:23:58 +00:00
Matthew Meszaros bf0693c7f0 feat: weighted, diverse warmup partner selection
extend recent-partner exclusion from 24h to 72h, weight selection by
inverse-frequency of recipient domain over the last 7 days so a single
provider does not absorb the bulk of warmup traffic, and log a warning
when a pool drops below the diversity threshold.
2026-05-25 15:21:27 +00:00
Matthew Meszaros 7cfff22380 feat: warmup replies stay on-topic with original thread
persist conversation_theme on the warmup_token at send time, and look
it back up on reply so the body comes from the same topical bucket.
previously the reply picked a random conversation, so 'Re: time-
blocking question' could come back as a paragraph about travel.
2026-05-25 15:19:18 +00:00
Matthew Meszaros 6766696ac4 fix: add 2-minute timeout context to task handlers to prevent hangs 2026-04-10 06:57:17 +00:00
Matthew Meszaros 7f90323d55 chore: remove metrics endpoint, prometheus instrumentation, and IP rate limiter 2026-04-10 06:32:30 +00:00
Matthew Meszaros 0799020dac feat: add metrics, warmup content variety, tz-aware scheduling, org budget, admin stubs, and bug fixes 2026-04-09 14:33:45 +00:00
Matthew Meszaros 6517b06893 feat: expand warmup subject lines and conversation templates
Increase warmup subjects from 5 to 20 and conversations from 3 to 12
to reduce repetition patterns that spam filters can detect. New themes
cover industry, tools, networking, feedback, planning, reading, travel,
wellness, and events.
2026-04-09 12:06:30 +00:00
Matthew Meszaros 308f72b874 feat: add CORS origin config, encrypt email bodies in S3, replace printfs with structured logging 2026-04-03 06:24:20 +00:00
Matthew Meszaros d8d88c7f69 feat: add warmup health tracking, migrate repos to postgres, and overhaul web UI 2026-04-03 06:08:52 +00:00
Matthew Meszaros 21ffb6a748 feat: add advanced outreach controls with A/B testing, deliverability dashboard, and DLQ 2026-02-20 08:59:17 +00:00
Matthew Meszaros c564b3ac95 feat: implement unibox replies, warmup conversations, and daily email limits 2026-02-20 04:54:46 +00:00
Máté Mészáros (Laptop) 41624a6f79 Analytics & Tracking 2026-01-29 05:59:04 +01:00
Máté Mészáros (Laptop) 6adb4cdd5a Organization, Subscription, Inqueries, limits. 2026-01-27 05:55:48 +01:00
Máté Mészáros (Laptop) 81d5970ea8 Realtime, Task Handler, Worker & Consumer Setup 2026-01-26 04:42:19 +01:00
Máté Mészáros (Laptop) ea787554ae Oidc Middleware & Tasks Service 2026-01-19 15:54:57 +01:00