mirror of
https://github.com/warmbly/warmbly.git
synced 2026-09-05 16:02:48 +00:00
b9cce2fbdcf5213ff6cd69df67a28601d8939abd
235
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matthew Meszaros
|
b9cce2fbdc |
feat: ease a graduating mailbox into cold volume (#231)
* feat: ease a graduating mailbox into cold volume instead of handing it the full cap the day it joins a campaign: cold sending read warmup HEALTH but never whether the mailbox had actually warmed enough, so one at its 40/day warmup ceiling could send 50 cold the next morning, which is the post-warmup spike providers penalise; effectiveCap gains a min() term that starts the mailbox at 5, 10 or 20 a day by how long it warmed and adds 5 per clean day toward its own cap, freezing on a spam placement through the same union-of-freezes the warmup ramp uses, anchored by a new cold_ramp_started_at stamped idempotently on the first cold send and reset on org import because it raises a ceiling the destination never watched being earned; mailboxes that never warmed are not gated, since capping senders who never opted into warmup is a different decision from smoothing the transition out of it * feat: anchor the graduation ramp on a CONFIRMED send, and make the hold the drawer reports the hold the scheduler applies: stamping cold_ramp_started_at at dispatch started the clock on a send the worker then failed, so a mailbox climbed on days it had not proven anything, and the stamp moves to the worker's EMAIL_SENT; separately the drawer computed its hold over every placement while ColdCeiling only counts placements after the first cold send, so a placement predating the ramp read as paused while the scheduler kept climbing, and both now go through one ColdHeldUntil * fix: restore the Callout closing tag my conflict resolution dropped, which types:check and lint both pass and only pnpm build catches |
||
|
Matthew Meszaros
|
d47d31b7c4 |
feat: let a warmup recipient answer the mailbox that wrote to it (#230)
* feat: let a warmup recipient answer the mailbox that just wrote to it, so a thread reads as a conversation rather than two mailboxes monologuing on their own ramps: warmup_tasks.target_account_id was written as nil and never read by anything, so a reply only happened when the recipient's own ramp fired AND the draw happened to land on that partner; a verified receipt now sometimes re-points the recipient's pending warmup task at the sender 25 minutes to 5 hours later inside its own warmup hours, which is a re-pointing rather than new work because only one warmup task may be pending per mailbox, it can never delay a send the mailbox had planned sooner, and it stops before the thread cap so replies cannot answer replies forever; the clock parser also moves into models.ClockMinutes so a second copy of the HH:MM parsing that silently disabled every sending window cannot drift back in * feat: stop the reply-back drawing the reply rate twice, and stop its jitter escaping a short warmup window: the scheduler drew the recipient's reply rate to decide whether to answer at all, then the task handler drew it again to decide reply-versus-new, so a 30 percent reply rate produced a 9 percent answer rate and a directed task could send a fresh message to the mailbox it was meant to be answering; a directed task now IS the reply, and the opening-time jitter is capped to the window width so a mailbox warming 09:00 to 09:20 is not scheduled past its own close |
||
|
Matthew Meszaros
|
ca5acb5aaf |
feat: score campaign copy as sent, not just as written (#229)
* feat: score campaign copy where it actually matters, on the message the recipient receives rather than only the template in the editor: warmlint.Score already ran at save time through /templates/score, but nothing checked the copy after merge fields, spintax, A/B selection and AI blocks resolved, which is exactly where a clean template becomes Hi comma or picks the one spammy spintax branch; the send path now scores the rendered message and writes one campaign-feed warning per step per day rather than one per recipient, RunPreflight gains the same check across every step, warmlint gains image-heavy, many-images and attachment heuristics, and the whole thing is advisory behind Settings > Sending > Content checks; the launch dialog also finally calls POST /campaigns/:id/preflight, whose report had no UI at all and so never reached the person about to send * feat: fix three defects the review found in the content lint: a preflight run that could not read the campaign's steps left the worst score at 100 and reported the check as PASSED, which is the one outcome a check that did not run must never produce, so it now reports failed with the reason; the send-time path read organization settings while preflight read campaign-effective ones, so a campaign that turned the check off or moved its floor was still warned, and EffectiveSettings is now exposed and used by both; and the check-then-insert dedupe let concurrent recipients of the same low-scoring step each find nothing and each write, so it becomes one advisory-locked conditional insert proven by twelve racers producing exactly one feed entry |
||
|
Matthew Meszaros
|
bc9978dab3 |
feat: feed per-provider placement back into warmup partner routing (#228)
* feat: feed each sender's per-provider spam placement back into warmup partner selection, so a mailbox failing only at Microsoft stops being handed Microsoft partners instead of waiting for an aggregate band to trip: pickWeightedPartner weighted on inverse domain frequency and routing rules alone, while PoolSpamPlacementsByProvider's segmentation only ever reached the admin overview; the new per-sender query keys on who RUNS the recipient's mail via models.ClassifyProvider rather than email_accounts.provider, because that enum collapses every custom host into smtp_imap which is exactly the bucket the signal matters in, and the weight multiplies by 1/(1+4*rate) with a five-send floor so a failing provider is downweighted but never excluded, since a sender that stops mailing a provider can never discover it recovered * feat: stop failed warmup sends flattering a provider's junk rate, and stop a re-admitted partner sidestepping the penalty: a warmup_tokens row is written before the send leaves, so counting every token put failed sends in the denominator and understated the rate exactly when the sender was doing worst; the send count now joins tasks on status = completed, and GetPoolParticipantProviders is read unfiltered because that map only resolves a candidate's provider and never decides eligibility, so a just-unblocked partner missing from it scored an unpenalized 1.0 * feat: move the campaign content-lint work off this branch, where it landed by mistake: the pre-send scoring, its preflight check and the warmlint image and attachment heuristics belong to the content-lint change and are unrelated to per-provider partner routing, so this branch now carries only the routing signal it is about |
||
|
Matthew Meszaros
|
7a8406e35a |
feat: close the warmup ramp loop with an early-signal hold (#227)
* feat: close the warmup ramp loop so an early spam placement holds the mailbox instead of letting it climb into the wall: every band in evaluateMetrics needs a sample floor before it can trip (20 warmup sends in 7 days, 100 delivered in 30), which a mailbox in its first fortnight never reaches, so one landing in junk on day three kept adding an email a day until it had sent enough to be judged; a placement inside 48h now cuts the day about a quarter and holds the ramp where it stood, and the resume subtracts the frozen days rather than catching them up because climbing three steps in one morning is the spike the hold exists to prevent; the ramp arithmetic moves into internal/app/warmupramp so the mailbox drawer reports the target the scheduler will actually act on instead of its own private copy, and says which signal cut it and when it resumes * feat: fix three defects the review found in the warmup ramp hold: the ramp is now elapsed days minus frozen days rather than a level held at the newest placement, because MAX(created_at) meant a second placement arriving mid-hold RAISED the held level and a mailbox landing in junk repeatedly ramped up; the early-signal cut and the health band are resolved in one shared warmupramp.Resolve so the scheduler's health gate and the dashboard's ungated copy can no longer disagree about the same mailbox; and the hold is reported for the whole 72h freeze instead of only the 48h cut window, so a mailbox between the two no longer shows a ramp that has stopped climbing with nothing to explain it |
||
|
Matthew Meszaros
|
b83a1a3ce5 |
feat: make send-time optimization actually schedule sends (#226)
* feat: make send-time optimization actually schedule sends, instead of being a documented setting with no caller: advanced.OptimizeSendTime had zero call sites anywhere in the codebase, so an org that enabled send_time_optimization through PATCH /outreach/settings changed nothing about when its campaign mail left, and the API reference said so in a callout; the campaign scheduler now resolves the recipient's timezone from the contact's timezone custom field, then the country-code suffix of its email domain, then the org fallback, and holds the slot until that clock reaches a preferred hour, raising hardFloor so the task handler reschedules rather than sending immediately, the snap is forward-only and never crosses the campaign end date, the default flips to off so no existing workspace silently re-times its sends, and Settings > Sending gives the block its first UI * feat: stop the recipient-hour gate deferring a send forever, and fix the window parser that silently disabled every campaign schedule: recipientSlot now searches for a moment BOTH calendars accept and yields when they never meet, because raising a hard floor at an hour the sender cannot serve made every tick re-derive it, defer, wake in the sender's window and defer again; separately parseTimeOfDay accepted only the 15:04 layout while start_time, end_time, warmup_start_time and warmup_end_time are Postgres time columns pgx renders as 09:00:00.000000, so every read parsed to 0 and effectiveWindows read that as unconstrained, leaving both the campaign sending window and its day-of-week gate off for every campaign on the legacy fields and pinning warmup to its 08:00-20:00 fallbacks; Settings > Sending also gains the MANAGE_SETTINGS gate a direct visit needs |
||
|
Matthew Meszaros
|
029bc27bfd | feat: bind a browser sign-in to the browser that started it, so a handoff link cannot be forwarded: one-time state proves the callback answers a request this server made, not one THIS browser made, so anyone could run the flow against their own Google or OIDC account and send the resulting URL to someone else, whose browser would then hold the sender's session (RFC 9700 4.7.1); begin now mints a binding secret that never reaches the provider and never appears in a URL, the callback carries it into the handoff, and the exchange refuses a collection that cannot present it with sso_wrong_browser, while the comments this PR added are condensed to the constraint they exist to state | ||
|
Matthew Meszaros
|
9ab42cfd65 | feat: build the browser half of social sign-in, which was never wired: GOOGLE_CLIENT_ID was read at boot and made the login screen render a Google button, but the button opened a popup at /auth/google/login which no route served, and authService.GoogleAuth/AppleAuth had no caller anywhere in the codebase; internal/app/socialauth now runs Google and Apple through the flow generic OIDC already used (one-time state, PKCE, nonce, id_token verified against the provider JWKS, identity keyed on issuer and subject, JIT provisioning, the ban and 2FA gates), the redirect URI defaults to API_PUBLIC_URL/v1/auth/<provider>/callback and is logged at boot because registering the dashboard origin instead is the mistake that produces a valid OAuth client and a dead button, /auth/config advertises only providers the backend can actually complete, the SSO landing page no longer swallows a two_fa_required response, and OIDC_PROVIDER_NAME finally reaches the button it documents | ||
|
Matthew Meszaros
|
bd435a2687 | feat: tell the worker to drop a mailbox the customer disabled or disconnected, the two paths #217 left out: emailService.Update now publishes REMOVE_EMAIL when a status leaves active (and re-ships the mailbox when it returns to active instead of waiting on the reconciler), Delete publishes it before the row goes and refuses to delete when it cannot, since after the row there is no assignment left to read; delete also stopped failing outright, because tasks and warmup_admin_actions referenced email_accounts with no delete action so disconnecting anything that had ever warmed up or sent a step raised a foreign key violation (migration 000098), read the mailbox through an org-scoped query while passing a user id so the warmup pool cleanup, realtime event and webhook never ran, and never refunded the worker's account count or load score | ||
|
Matthew Meszaros
|
e189320d28 | Merge remote-tracking branch 'origin/main' into fix/issue-207-reported-defects | ||
|
Matthew Meszaros
|
0a79997228 | feat: document the subscribed field on POST /contacts along with the enrich-not-duplicate behaviour a repeated address gets, and state the import error cap in terms of entries returned rather than rows failed since the list also carries per-row notes, adding the invariant that imported, updated, skipped and failed always sum to total | ||
|
Matthew Meszaros
|
d1ac72679b | feat: fill the contact and lead-sync holes in the endpoint scope map, which claims to be the source of truth for every route an API key can reach but was missing GET /contacts/:id, its emails and timeline reads, the lookup, the export, both import steps and the whole /lead-sync group, and note that import commit takes the stricter BULK_CONTACTS scope because one call writes up to 50,000 rows while a sync source's column mapping is validated when it is saved | ||
|
Matthew Meszaros
|
6a70ee7431 | feat: update the contacts and integrations API reference for the import contract: the custom column target and its custom_key, the character set a custom-field name may use, that a bad name or a mapping with no email column is a 400 raised before any row is written while per-row errors stay reserved for the data itself, the 1,000-row errors cap with errors_truncated, and that a Google Sheets source's column_mapping is validated when it is saved | ||
|
Matthew Meszaros
|
8cae0a76ed | feat: document the real import behaviour on the Contacts & CRM guide: which custom-field names are allowed and that the wizard fills one in from the column header and flags a bad one inline, the values the subscribed column accepts, that the categories column creates names you do not have yet up to 100 per import, that skipping an existing contact still enrols it in the import's campaign and categories, that a repeated address in one file becomes one contact, that a blank cell never erases a stored value, and that a mapping problem is reported once before anything is written | ||
|
Matthew Meszaros
|
cbb0b8f662 | feat: keep a mailbox in exactly one warmup pool so a plan change moves it instead of leaving it in both: EnsurePoolMembership joined whichever pool the org is entitled to right now while RemovePoolMembership only ever deleted that same pool's row, so a downgraded mailbox kept its premium membership and went on being handed to paying customers as a warmup partner while every spam-score increment was counted once per row by the health evaluator; membership is now a move (one upsert keyed on the account that carries health state, block and spam score across, so changing pool cannot launder a penalty), removal is never pool-scoped, a tier change moves the participant row in the same transaction as email_accounts.warmup_pool_type, a new reconcile pass in the backend's warmup sweep evicts participants whose workspace lost warmup access and repairs pool drift for the mailboxes no warmup task ever revisits, an unreadable subscription no longer counts as "not entitled" anywhere that evicts, GetSpamScore takes the max rather than the sum and IncrementSpamScore clamps to the column ceiling instead of silently failing its CHECK, and migration 000097 merges any existing dual membership worst-wins before making one-pool-per-mailbox a unique index | ||
|
Matthew Meszaros
|
0bbfa4b2ac | feat: make warmup health evaluation actually run, which it never has: GetParticipantHealth compared pgx's not-found error with == sql.ErrNoRows but pgx.ErrNoRows is a proxy that wraps it rather than being it, so "this account is not in this pool" surfaced as a hard error and the premium-first probe failed before ever reaching a free-pool account's own row; UpdateParticipantHealth used a bare $1 as both a health_state assignment and an equality test, so Postgres deduced character varying from one and text from the other and refused the statement with 42P08, meaning no health state was ever persisted for anyone in any pool; both are now fixed along with the other 18 == sql.ErrNoRows comparisons in the repository layer and the same unpreparable-parameter defect in the contacts bulk custom-field writes, the worker install-state update and the referral earnings upsert, all found by a new live test that asks Postgres to PREPARE every parameterised statement in the package; evaluation failures now log their real cause instead of collapsing into errx.InternalError, the hourly sweep reports how many participants it could not evaluate instead of skipping them in silence, ApplyInvalidTokenAttempt no longer reports failure once the attempt is persisted so the caller cannot double-count it, and migration 000096 adds a per-participant signal floor so the first working evaluation cannot block a mailbox on history gathered while nothing was watching | ||
|
Matthew Meszaros
|
42ddb78e60 | feat: make the realtime service survive a client that spams channel joins, and stop both clients from being that client: ws_join is now spent per phx_join in a shared RealtimeWeb.ChannelGuard that every channel runs BEFORE its Auth lookup (it was only ever spent on the socket handshake, so an established socket could issue unlimited joins and each one cost a Postgres query), handshakes move to their own ws_connect bucket so a reconnect storm cannot eat the budget a client needs to rejoin with, the retry_after_ms hint now points at the next window instead of shrinking as overage grew, every channel's join rejection carries the numeric code the API reference already promised (plus 4005 for a malformed topic), BulkChannel only pushes an event to the user who owns the operation instead of to anyone who guesses the id, and the web and admin clients treat a rate_limited join as transient (wait out the hint, keep the socket) while backing off per-channel rejoins on a decaying schedule rather than reconnecting on the 120ms floor forever; adds an ExUnit suite for realtime and a vitest suite for admin, both wired into CI | ||
|
Matthew Meszaros
|
0a8c3314d0 | feat: stop one undeliverable address from wedging a whole campaign, and stop the pre-send verifier from inventing the verdict that caused it: campaign routing in FindNextRoutedPair now excludes contacts whose address verification refused them (invalid, or risky with the campaign's send-to-risky toggle off) because the pre-send gates in campaign_task.go skip those without recording progress, so the finder re-served the same lead every tick and the campaign never reached the healthy leads behind it; the SMTP prober no longer reads every 5xx reply to RCPT TO as a dead mailbox, since Postfix defers HELO/sender/policy rejections to RCPT time and the old default greeting of localhost made them arrive as 504 5.5.2 Helo command rejected against the recipient, so a 5xx now becomes invalid only when the reply names the recipient and degrades to unknown otherwise; the HELO name falls back to the APP_URL host and the probe is skipped outright without a public FQDN; migration 000095 resets the verdicts the corrected classifier would no longer reach so poisoned contacts are re-verified; and skipped leads now report as undeliverable in the Leads view and in the campaign completion log instead of sitting at Queued forever | ||
|
Matthew Meszaros
|
b77664cd09 | feat: verify warmup mail that arrives without its verify header, because Microsoft Graph strips custom headers in transit and re-stamps the Message-ID, so every warmup email sent from an Outlook or Microsoft 365 mailbox reached its recipient unmarked, consumed no token, earned no engagement and was filed in the recipient's unibox as ordinary mail; the Graph client now creates the message as a draft and sends that so it can read the internetMessageId Exchange assigned, the consumer records that delivered id on the task and its warmup token alongside the sent subject, and the recipient resolves an unmarked warmup email by delivered Message-ID or by the pending sender/recipient/subject pair | ||
|
Matthew Meszaros
|
6ebc925a93 | feat: fix the campaigns guide so the docs site builds again: the new Duplicate and delete section wrote the derived copy name as "<name> (copy)", which MDX parses as an unclosed JSX tag and fails the Cloudflare Pages build; it is now inline code | ||
|
Matthew Meszaros
|
d396f53fc4 | feat: add Delete and Duplicate campaign actions to the dashboard (issue #185): every campaign row and the detail header get a ⋯ menu (Edit, Duplicate, Start/Pause, Delete) plus a Delete card at the bottom of Settings, all permission-gated with a confirm that spells out what goes; DELETE /campaigns/:id is now organization-scoped instead of user_id-scoped so teammates can delete, runs in one transaction that also deletes the campaign's pending tasks and cancels a wakeup tick claimed at that moment (campaign_tasks only nulls its link, so those rows kept firing), removes attachment objects and publishes CAMPAIGN_DELETED so a teammate's open detail page is sent back to the list; new POST /campaigns/:id/duplicate copies the campaign row as a draft with steps and their branch graph rewired onto new step ids, tags, folders, senders with rotation reset, A/B variants, advanced settings and attachments (quota-checked, blobs undone if the copy fails) and none of the leads, progress, logs, counters, ramp level, guardrail trip or past dates, naming it (copy)/(copy N) inside the 50 byte cap without splitting runes; a claimed campaign tick whose campaign vanished now ends the chain instead of staying active forever; covered by TestLiveCampaignLifecycle* against real SQL, RemapBranchTargets and duplicateName unit tests and a react-query vitest for the list cache, with API reference, endpoint map and campaigns guide updated | ||
|
Matthew Meszaros
|
efa914025c | feat: stop an active campaign sitting at "Queued / Not started" with nothing sending: a campaign is one self-perpetuating task, so a tick that found nothing due parked its successor at the literal next-due moment (three days out for a "wait 3 days" step) and that parked task was also the next time anything re-read the campaign, so leads imported meanwhile stayed invisible until it fired and the reconciler never noticed because it only re-seeds chains with no pending task; deferral parks are now capped at config.CampaignMaxDeferMinutes via scheduler.DeferSlot at all three enqueue sites (a tick that actually sent still parks at its paced interval, so send spacing is untouched), the reconciler re-checks any active campaign parked beyond CampaignStaleParkHours and pulls its wakeup forward when the real next slot is CampaignReparkMarginMinutes sooner, attaching leads to a running campaign wakes it immediately through one CampaignWaker seam in the contact service that covers add/update/bulk-edit/import/Sheets-sync, even distribution now paces across the whole sender pool via poolRemainingOn instead of the one mailbox the tick picked (a three-mailbox campaign was sending at one mailbox's rate), the flat +/-20 minute jitter that was wider than the interval it perturbed is scaled to half the distance to the slot so it stops landing slots in the past where notBefore collapsed them onto the min-gap, and on the dashboard a full-day window renders "12am-midnight" instead of "12am-12pm", the campaign lead strip uses the server's campaign-wide lead_counts instead of counting the 50 loaded rows, and channel state moves out of a ref into React state so a live campaign's panel stops reading "Disconnected" forever | ||
|
Matthew Meszaros
|
222c9d2554 | feat: scope campaign sender resolution to the campaign's organization instead of its owner so a multi-org user can no longer send organization A's campaign from an organization B mailbox: GetByTags/GetAllActiveInScope/GetByCampaignSenders now take a repository.AccountScope keyed on organization_id where a scope with no organization resolves to no mailboxes rather than widening to the owner (tags carry no organization of their own, so one user's tag legitimately spans workspaces and the predicate is the only thing holding the boundary), the campaign scheduler and the preflight tracking-domain check build that scope from campaign.OrganizationID, unibox compose scores only the current workspace's mailboxes, the 'all' lanes of AccountHasActiveCampaign/CountActiveCampaignsForAccount join ea.organization_id = c.organization_id instead of ea.user_id = c.user_id, dead and broken PauseAllByUserID (which wrote the reason string into status and had no callers) is removed, the campaigns/unibox guides now state that senders resolve inside the campaign's workspace, and TestLiveOrglessCampaignDoesNotSendToSuppressedRecipient is updated because an orgless campaign now finds no senders before routing is consulted while the send gate it covers still refuses; live-tested in TestLiveSenderResolutionStaysInsideTheCampaignOrg, TestLiveSenderSchedulerNeverPicksAnotherOrgMailbox, TestLiveSenderSchedulerPicksTheCampaignOrgMailbox, TestLiveSenderScopeWithoutAnOrganizationReachesNothing and TestLiveActiveCampaignLookupIsOrgScoped | ||
|
Matthew Meszaros
|
15e139e15d | feat: stop a campaign email going out twice when the progress write after dispatch is lost: a step is now RESERVED before its SEND_EMAIL reaches the bus (migration 000093 adds campaign_contact_progress.dispatched_at + dispatch_task_id, and ReserveSend takes the claim and the day's counters in one transaction) and routing treats a step as attempted on sent_at OR dispatched_at, so a crash or a failed stamp in the dispatch window can no longer read as "never sent" and email the same person again; the ON CONFLICT claim is exactly-once so two ticks racing the same pair cannot both send (the loser ends skipped_duplicate), the stamp is retried and escalated to the campaign feed instead of warned and swallowed, HandleEmailSent repairs a lost stamp from the worker's own confirmation, ReleaseSend gives a reservation back only when the command provably never left (a publish failure is ambiguous via ErrSendDispatchUnknown and keeps it), and StartStuckSendReclaimer walks back a reservation nobody answered after 30 minutes so a worker that died mid-send cannot park a lead in flight forever; live-tested in TestLiveLostProgressWriteDoesNotResend, TestLiveDispatchedSendIsNeverOfferedTwice, TestLiveConcurrentTicksSendOnce, TestLiveStuckDispatchIsReclaimed, TestLiveReclaimBelievesADeliveredSend and TestLiveInFlightSendIsNotOfferedAgain | ||
|
Matthew Meszaros
|
cbf4190f42 | feat: make a custom tracking domain verifiable instead of permanently "Pending DNS": the CNAME target is now this install's TRACKING_DOMAIN rather than a hardcoded t.warmbly.com that resolves nowhere, matching is exact on the label boundary (or on shared addresses, so a provider-flattened CNAME stops reading as no record at all) instead of strings.Contains, and every outcome carries the reason plus what DNS actually returned, including when the tracking host the customer is told to point at has no record of its own; a pasted URL is normalized to its host and a malformed one is rejected up front instead of saved and left pending forever; only a VERIFIED mailbox domain is used at send time with the shared host as the fallback and a campaign-feed entry saying why; pixels and click tickets are built from the configured host, and with none configured mail ships untracked rather than carrying links to another deployment's tracking service; adds GET /emails/:id/track and POST /emails/:id/track/verify plus an hourly re-resolution sweep so a record that propagates later starts being used and one that breaks stops routing links; and scopes the tracking-domain write by organization like the read, which also fixes GET /emails/:id passing a user id to an org-scoped query and 404ing for every caller | ||
|
Matthew Meszaros
|
019b45fca7 | feat: unblock workspace import for any archive carrying a webhook endpoint, a notification, or a campaign-scoped suppression: ResetOnImport columns are now left out of the insert so the destination's own DEFAULT applies instead of writing NULL, which aborted the whole transaction on the four NOT NULL ones (webhook_endpoints.consecutive_failures, notifications.email_state and email_attempts, ai_mcp_servers.last_error) and made a webhook endpoint arrive carrying the source's failure streak and auto-disable state, and three tables that sat above something they reference are moved below it (webhook_endpoints under oauth_applications, suppressed_recipients under campaigns, reply_intents under tasks) because referencePlan only clears a reference the run does not write at all, so a forward reference to a table written later landed as a foreign-key violation; both classes are now guarded against the live schema in TestLiveResetColumnsHaveSomethingToFallBackOn and TestLiveTablesAreInDependencyOrder, and a cross-instance export/import round trip carrying all three previously fatal tables completes with failure counters back at their defaults and the suppression rows still attributed to their campaign | ||
|
Matthew Meszaros
|
1018068942 | feat: stop recipient suppression and the entitlement gate being skipped when a campaign has no organization: the send path now fails closed (an orgless campaign is paused with the reason in its activity feed instead of mailing an unsubscribed, bounced or complained address, since routing's own suppression filter joins on the campaign's organization_id and matches nothing when it is NULL), the same tenancy gate covers warmup and unibox sends and an orgless mailbox resolves to the free pool rather than defaulting into the paid one, the state is no longer creatable (sequenceRepository.Create inherits organization_id from its campaign instead of inserting NULL, campaign creation and mailbox onboarding refuse without a workspace via errx.ErrNoOrganization, guardInboxLimit / guardMailboxThrottle / the campaign creation throttle no longer treat a missing org as exempt, and a new session resolves the user's default workspace so the orgless session that produced these rows cannot exist), and migration 000092 backfills then sets organization_id NOT NULL on campaigns, contacts, email_accounts and sequences plus sessions.current_organization_id, provisioning a recovery workspace for any user with none so no row is deleted to satisfy the constraint (live-tested in TestLiveOrglessCampaignDoesNotSendToSuppressedRecipient, TestLiveSuppressedRecipientIsSkipped, TestLiveCampaignRequiresAnOrganization and TestLiveHealthyCampaignStillSends) | ||
|
Matthew Meszaros
|
6c17f109fd | feat: stop one waiting lead from parking a whole campaign: FindNextRoutedPair now returns only a DUE pair (new leads now, follow-ups at last sent + wait_after days, plus a wait node's minutes) and skips not-yet-due contacts so other leads' first emails and due follow-ups keep sending, handing back the soonest due moment when nothing is due so the scheduler defers exactly until then (min'd with the next-day new-lead-cap deferral); drop send-time optimization from the successor wakeup, which by default pushed the next lead to 09:00 UTC tomorrow after any send past 17:00 UTC (and to a past time when the hour matched, defeating pacing); document that send_time_optimization is stored but not applied and that waits are per contact; live-tested in TestLiveWaitingFollowUpDoesNotBlockOtherLeads and TestLiveWaitNodeGatesTheStepAfterIt | ||
|
Matthew Meszaros
|
c3066f9cc9 | feat: unbox campaign start dates and make follow-up pacing real: accept today as "start now" and let an explicit null clear start/end dates on PATCH /campaigns (models.NullableTime distinguishes absent from null, which used to silently no-op while the error message told users to send null), reschedule an active campaign's parked wakeup when any schedule field changes so clearing a future start date takes effect immediately instead of at the old slot, let a completed campaign be started again and turn the past-end-date start 500 into a clear 400, gate the campaign task on the step's hard-constraint floor (wait_after, start date, windows, day capacity, mailbox min-gap) via ErrCampaignDeferred so an early successor tick can no longer send a wait-3-days follow-up seconds after step one (live-tested in TestLiveFollowUpWaitIsHonored), disable past days in the schedule date picker, and fix the sandbox seed leaving worker 1a01 free-tier after make seed which unassigned the paid org's mailboxes and failed every send | ||
|
Matthew Meszaros
|
7c2d08f204 | feat: renumber the campaign_send_outcome migration to 000091 so it stops colliding with 000089_email_auth_enforcement, which made golang-migrate refuse to init its source driver and left the backend restart-looping at boot on main, and guard the class of bug with scripts/check-migrations.sh (duplicate versions, gaps, unpaired up/down, unparseable filenames) wired into make check-migrations, make lint, a dedicated Migrations CI job and a TestEmbeddedMigrationsLoad that boots the real iofs source over the embedded FS | ||
|
Matthew Meszaros
|
8e0043878c | feat: consume the worker's EMAIL_SENT and EMAIL_FAILED results in the consumer, which until now dropped them with a warning so a send the worker could not complete stayed stamped as sent and the lead sat at 'processing' forever: a failure now marks the task failed, walks the step back, gives the day's counters back, writes a red entry to the campaign activity feed, reopens a campaign that completed while the send was in flight, marks the lead failed after five attempts, and routes a recipient refused at RCPT into the bounce pipeline instead of retrying it; a one-off compose or reply failure is pushed to the mailbox owner; covered by an opt-in live test against Postgres and documented in the events page, the campaign guide, the contacts API reference and the agent notes | ||
|
Matthew Meszaros
|
bd8868f881 | feat: connect the steps a campaign is created with in position order, because routing follows connections only and the wizard's 'Follow-up 1, after 3 days' was written as a disconnected step that never sent; repair existing campaigns with two or more email steps and no connection anywhere in migration 000090, correct the stale comments that promised a linear fallback, and fix the campaigns API reference (the create field is steps, wait_after is days before the step, conditions are the only routing) | ||
|
Matthew Meszaros
|
3739a36b67 | feat: enforce the persisted SPF/DKIM/DMARC state as a real cold-send and warmup gate behind a 72h grace clock and an operator toggle, after first fixing the DMARC organizational-domain fallback in dnsauth so a dedicated sending subdomain covered by its parent's record stops reading as unauthenticated, adding auth_state to the four mailbox loaders that never selected it (which would have made the gate dead code), stamping auth_failing_since on entry to failing so a resolver hiccup can never stop a campaign, notifying the org on that transition, and reporting an all-gated pool as ErrDomainAuthFailing instead of a message about sending windows (#160) | ||
|
Matthew Meszaros
|
94cf21d95e |
Fix truncated, unformatted and mis-encoded email content in the unibox (#137)
* feat: add internal/pkg/mailhtml, a mail-oriented HTML sanitizer and text flattener, because rendering a received message body means rendering the sender's markup: Sanitize builds on bluemonday's UGC policy but keeps what real email is made of (table layout attributes, inline CSS through the property-allowlisted style sanitizer, legacy font/center, data: and https: images) while dropping script, iframe, object and the text content of style/head blocks so a marketing email's stylesheet cannot render as body copy, forcing target=_blank plus nofollow/noreferrer on links and allowing only http, https, mailto and tel; ToText flattens the same input for previews, turning block boundaries into newlines and decoding entities back to the characters they stand for so an already-escaped body does not surface as literal &; LooksLikeHTML reports whether a stored body is actually markup, which is how a body recorded as HTML by an older sync but containing no tag at all can be recognised as the plain text it really is * feat: add internal/pkg/mailhdr for RFC 5322 header values, since headers are ASCII on the wire and every transport was writing raw UTF-8 into Subject and display names: Subject and AddressList RFC 2047-encode non-ASCII (a no-op on plain ASCII, and a bare address stays bare rather than being wrapped in angle brackets), DecodeWords reverses encoded-words with a charset hook wired to go-message so legacy encodings Go does not handle natively still decode, and Bare/BareList strip a display name down to the routable address for SMTP envelope commands where 'Ana <a@b.com>' in RCPT TO is a syntax error, promoting go-message from an indirect to a direct dependency * feat: encode outbound Subject and address headers on all three transports, so a subject or sender name containing an accent, a currency sign or an emoji reaches the recipient as the characters the user typed instead of mojibake: SMTP and Graph were writing the raw string into Subject (only the Gmail transport encoded it) and Graph built its From by fmt.Sprintf rather than mail.Address, so a non-ASCII display name went out unencoded there too, and all three joined To/Cc/Bcc entries verbatim so an encoded display name never appeared even when the caller supplied one; additionally the SMTP envelope now takes bare addresses through mailhdr.BareList, because an API caller may pass 'Name <addr>' (the compose handler has a bareAddress helper precisely because that arrives) and passing that to RCPT TO gets the recipient rejected by the server * feat: rewrite the IMAP body reader, which was the reason received mail from SMTP/IMAP mailboxes came back corrupted: it built one FetchItemBodySection with a hardcoded Part []int{1} and a comment saying it would adjust when recursing, which it never did, so on a multipart/alternative the text/plain bytes were fetched twice and the second copy was stored as the HTML body (plain text rendered as markup loses every line break, shows & as an entity and swallows anything inside angle brackets), and decodeIfNeeded never reversed Content-Transfer-Encoding at all, leaving quoted-printable bodies full of =E2=80=99 runs and = soft breaks and base64 bodies unreadable, while its charset detection parsed params off a media-type string that never carried any and its mail.ReadMessage call could silently eat leading body lines as headers; the reader now walks the body structure for real part paths, fetches every text leaf in a single FETCH with a server-side Partial size cap, decodes quoted-printable and base64 (tolerating a tail cut mid-quantum by the cap) then converts the part's charset to UTF-8 with go-message, skips attachment-disposition parts so a .txt attachment cannot stand in for the body, takes one part per type inside a multipart/alternative but treats sibling inline parts in mixed/related as additive, and is bounded at five text parts per message; the stored body cap also goes from 200 KB to 512 KB because 200 KB cuts real HTML newsletters mid-document * feat: decode Gmail's raw headers and entity-escaped snippets, because the Gmail API hands header values back exactly as they arrived on the wire, so a message from a sender whose subject or display name was RFC 2047-encoded showed in the dashboard as =?utf-8?q?caf=C3=A9?= rather than as the text it stands for, and the API's own snippet field is HTML-escaped, so a preview containing an apostrophe surfaced as ' in the conversation list and, until the thread reader stopped rendering snippets as message bodies, inside the message itself; getSingleHeader now runs values through mailhdr.DecodeWords (a no-op unless the value actually contains an encoded-word, so Message-ID and the warmup token header are untouched), the comma-split fallback in getAddressList does the same for display names net/mail could not parse, and the snippet is unescaped once on the way in * feat: fix the conversation-list snippet, which collapsed whitespace before splitting on newlines so the quoted-line and signature filters below it could never match a thing, stripped HTML with bluemonday's strict policy and then showed the escaped output verbatim so an ampersand in an HTML-only message read as & and a marketing email's stylesheet text rendered as body copy, and cut at 100 bytes with text[:100] so a multi-byte character or emoji at the boundary became a replacement glyph; it now flattens HTML through mailhtml.ToText (entities decoded, style and script content dropped) including when a sender puts markup in their text/plain part, filters quoted history and everything past the RFC 3676 signature delimiter while the text still has lines, collapses whitespace afterwards, and truncates on a rune boundary at 200 characters * feat: make GET /unibox/:id serve a display-safe body and stop it failing outright, sanitizing body_html through mailhtml before it leaves the API so every consumer gets markup that cannot execute rather than each call site having to defend itself, degrading a body blob that cannot be read to the message's preview text with a new body_truncated flag instead of returning 500 (which made a message with a missing blob unopenable, and hit every seed, sandbox and dev-history fixture row since only the '<seed-' prefix was recognised while the sandbox uses '<sbx-' and dev history '<dev-'), and treating a stored HTML body that contains no tag at all as the plain text it really is, because mail synced before the IMAP reader addressed parts individually recorded the plain part under both bodies and serving that as HTML is exactly what collapsed a ten-line message onto one line * feat: escape composer text before turning it into the HTML part of an outgoing email, replacing body_html: trimmedBody.replace(/\n/g, '<br />') in both the compose window and the reply composer with a shared plainToHtml that escapes the five markup characters first, so an email containing 'Terms & conditions' no longer ships a broken entity and one containing anything in angle brackets ('<see attached>', 'a < b', a pasted tag) no longer has the rest of the paragraph swallowed by the recipient's mail client as an unclosed tag, while runs of spaces survive as non-breaking spaces and bare URLs become links without eating the sentence punctuation after them; the same unescaped plain-to-HTML pattern in the campaign step editor's applyTemplate now goes through promptToHtml, which escapes as it paragraph-wraps * feat: render the real message body in the unibox thread reader instead of the list preview, which is the whole of the reported bug: ThreadView mapped each thread row to a UniboxEmail whose body was '<p>' + escapeHtml(m.snippet) + '</p>' and MessageBubble rendered that as the message, but a snippet is a preview capped at 100 characters with every run of whitespace collapsed to one space, so a ten-line email displayed as roughly two lines on a single continuous line, and Gmail's already-escaped snippet was escaped a second time so an apostrophe read as '; each expanded message now loads its own body from GET /unibox/:id (the newest message and anything unread open on mount, older messages collapse to their preview line so a long thread does not fetch every body at once) and renders it in a sandboxed iframe carrying no allow-scripts, which keeps a sender's stylesheet from restyling the dashboard and means nothing in the message can run even though the API already sanitized it, sizing itself from the inner document as images load, with the preview kept as the fallback when a body cannot be fetched and a notice when only a preview is stored * feat: document how a message body is read and returned, adding a 'Reading a message' section to the unibox guide covering the expand-on-open behaviour, that formatting and special characters are preserved as sent, that the conversation list preview is a summary and not the message, and that HTML mail renders in an isolated frame with links opening in a new tab, plus a paragraph in the API endpoint reference stating that GET /unibox and GET /unibox/thread return previews carrying snippet while GET /unibox/:id returns body_plain and a sanitized body_html, and what body_truncated means * feat: add email_accounts.save_to_sent, the per-mailbox switch for filing a copy of outbound mail in the Sent folder, defaulting on because plain SMTP submission leaves nothing behind in the sender's account while Gmail and Outlook file their own copy through their APIs, making it a per-mailbox choice rather than a global one since a submission server that files the copy itself (Gmail's SMTP, Fastmail, Zoho) would otherwise end up with two of everything, which is exactly why every desktop mail client ships the same switch, and wiring the column through the Email model, the mailbox read paths and UpdateEmail so it is readable and writable from the dashboard and the API * feat: teach the IMAP client to APPEND a sent message and the SMTP client to hand back the exact bytes it submitted, the two transport pieces the Sent-folder copy needs: AppendToSent resolves the folder from the RFC 6154 \\Sent special-use attribute first (requesting it only when the server advertises SPECIAL-USE) and falls back to matching the known names against both the full mailbox name and its leaf, since servers namespace as INBOX.Sent and localize the label, caches the result for the life of the connection, files the message flagged \\Seen and dated when it was sent, and returns a sentinel rather than an error when the account has no Sent folder at all; APPEND addresses its mailbox by argument and never touches the selected mailbox, so unlike the warmup MOVE/STORE actions it is safe to run while the sync loop is mid-fetch on the same connection * feat: file a copy of every SMTP send in the mailbox's Sent folder, closing the gap where a message sent from Warmbly through an SMTP/IMAP mailbox existed only in the recipient's inbox: nothing appeared in the customer's own mail client, and nothing appeared in the unibox either, whose thread reader can only show messages the sync found in a folder, so a user who sent from the dashboard and then went looking for what they sent found no record of it at all; the worker now APPENDs the exact bytes the SMTP client submitted after a successful send, best effort so a failed append never turns a delivered message into a failed task, skipping warmup traffic because filing dozens of machine-generated messages a day would bury the customer's real sent mail, and skipping Gmail and Graph mailboxes entirely since their APIs file their own copy; the per-mailbox setting rides along on the add-email worker payload as a pointer so an older control plane that does not send the field is read as unset and takes the default rather than as an explicit no * feat: expose the Sent folder copy as a mailbox setting in the dashboard, adding a 'Keep a copy of sent mail' toggle to the Settings tab of the mailbox drawer that only renders for SMTP/IMAP mailboxes (Gmail and Outlook file their own copy, so the control would be a lie there), tracked by the drawer's save bar alongside the other editable fields, and worded so the one case where it should be turned off is obvious: a provider that already saves its own copy, where leaving it on means seeing every sent message twice * feat: document the Sent folder copy in the mailboxes guide and the API reference, explaining why the toggle exists at all (SMTP submission leaves nothing in the sender's own account, so without it a sent message shows in neither the customer's mail client nor the unibox thread), when to turn it off (a provider such as Gmail, Fastmail or Zoho that already files its own copy of anything submitted over SMTP, where leaving it on doubles every message), that OAuth Gmail and Outlook mailboxes never show the control because their APIs file the copy themselves, that warmup traffic is deliberately excluded, and that PATCH /emails/:id takes save_to_sent * feat: add unibox_emails.body_text and its search index, because unibox search ran against search_tsv, a generated column built from subject and snippet, and a snippet is a truncated one-line preview, so searching for a phrase that appears in the third paragraph of an email returned nothing at all and read as broken search rather than as search that only covers the first line; message bodies stay in object storage where they belong, and what lands in Postgres is a bounded 16 KB plain-text rendering carried on the new-email worker event, indexed with a GIN expression index rather than a second stored generated column since adding one of those rewrites the whole table while this builds against a column that is empty on every existing row * feat: index what a message actually says, adding mailhtml.SearchText (HTML flattened, entities decoded, whitespace collapsed, quoted history deliberately kept because a phrase someone quoted back at you should still find the conversation, truncated on a rune boundary) and computing it on all three sync paths so IMAP, Gmail and Graph mail all arrive with searchable text, writing it on insert, and widening the unibox search filter to match either the existing subject-and-preview vector or the body expression, written exactly as the new index declares it so the index is actually used * feat: backfill the searchable text of messages that were synced before bodies were indexed, so search covers the archive a customer already has instead of only mail that arrives from now on, which would have made the feature useless on day one for exactly the people who need it; the sweep pages through unibox_emails by id, reads each body from object storage under the mailbox owner's key, renders it with the same helper the sync path uses and writes it back, at 100 rows per 30 seconds because nothing waits on it, and returns for good once a pass finds nothing left to visit, with rows whose stored body really is empty simply revisited after the next restart rather than needing a tried-and-failed marker in the schema * feat: document that unibox search now covers message text and not just subjects and previews, in the search paragraph of the unibox guide where the old wording only promised that search stays inside the current scope * feat: add generation.RenderThread, the shared way to put a conversation in front of a model, because every AI surface was grounding on preview snippets and a draft written from the first hundred characters of each email answers the greeting rather than the question; it strips quoted history and signatures (the earlier messages are already in the prompt on their own, so quoting them again spends the budget twice, though a reply written underneath the quote is kept rather than thrown away when there is nothing meaningful above the attribution line), spends a bounded character budget newest-message-first since the message being replied to matters most, degrades older messages to their preview line instead of dropping them once the budget runs low, and renders oldest-first so the transcript reads in order * feat: add grounding reads to the unibox service and repository, returning message text (the stored body, falling back to the preview for mail synced before bodies were indexed) for a thread or for all correspondence with one address, kept deliberately separate from the preview queries and given their own result type so a 16 KB body can never leak into a list response by accident, capped at twenty messages whatever a caller asks for, and paired with a RenderGrounding helper so every AI surface formats a conversation the same way instead of each one rolling its own transcript loop * feat: ground every AI writing surface in what the messages actually say, switching the unibox reply draft, the compose draft's correspondence history, the inbox agent's thread history and the assistant's read-thread tool from preview snippets to real message text through the new grounding reads, which is what makes a drafted reply answer the question that was asked rather than the first sentence of the email; the inbox agent's triviality gate also reads the reply's full text now, since a preview line cannot tell a one-word ack apart from a long message that happens to open with one, and the assistant tool returns a bounded body per message with quoted history stripped instead of a snippet field * feat: say in the docs that AI drafting reads the messages and not their previews, in both the unibox reply-draft section (adding that quoted history is stripped and the newest messages get the most room, so a draft answers what was asked rather than the opening sentence) and the inbox agent's grounding section, where 'the full thread so far' was true of the message list but not of how much of each message the model actually saw * feat: renumber the two new migrations to 000087 and 000088 after rebasing onto main, which landed its own 000085 (org data transfer) and 000086 (email sync state) in the meantime, so the sequence has no duplicate versions * feat: add the two new API fields to the OpenAPI spec that landed on main while this branch was open, documenting save_to_sent on the Mailbox and MailboxUpdate schemas and body_truncated on UniboxEmail, and saying on body_html that what the API returns is already sanitized so a client can render it directly |
||
|
Matthew Meszaros
|
e143cb0628 |
feat: give warmblyctl an API-key half so agents and scripts can operate any Warmbly instance, ship in-repo agent skills that teach it, and cut the README quick start and self-hosting sections down to commands plus docs links (#135)
* feat: cut the README quick start and self-hosting sections down to the install command, one paragraph of what it does, and links out to the local development, first-run, deployment and warmblyctl docs pages, dropping the recovery if/then table, the MAIL_TRANSPORT invitation note and the dependency matrix that all duplicate those pages * feat: give warmblyctl an API-key half so agents and scripts can operate any Warmbly instance including the hosted one, adding a raw passthrough (warmblyctl api get/post/patch/put/delete <path> with --data taking a literal, - or @file, and --idempotency-key) plus eleven typed families (me, campaign, contact, mailbox, inbox, analytics, settings, webhook, apikey, template, crm) driven by one spec table that generates dispatch, flags, per-command help and the request, covering list/get/create/update/delete, sequence steps, sender pools, preflight/start/stop/test-email, contact notes/timeline/import/export, mailbox behavior/verify/send and the six warmup controls, unibox threads/reply/compose/seen/agent-drafts/scheduled sends, outreach settings, webhook secrets and deliveries, and API key self-service, authenticated with Bearer wmbly_ keys from WARMBLY_API_KEY against WARMBLY_API_URL falling back to API_PUBLIC_URL then the hosted service, printing the API's JSON untouched and surfacing the error envelope's code and request_id with Retry-After on 429, while the DB-direct operator commands and their trust model stay exactly as they were * feat: ship two in-repo agent skills so AI assistants working against Warmbly discover the right warmblyctl half on their own, .claude/skills/warmbly-api teaching product operation over the API commands (key and URL setup including the seeded local dev key, the eleven command families, pagination and error-code and Idempotency-Key conventions, and a sending-safety section that names the six commands that put real mail on the wire and holds agents to preflight before start and the 50/day default cap) and .claude/skills/warmbly-ops teaching instance administration over the DB-direct commands (status --json as the contract to parse, the recovery command table, TTY versus -T piping, Redis-down behaviour, and org export/import handling including --dry-run first and the sensitivity of credential archives), each pointing at the other for what it does not cover, narrowing the .gitignore .claude/ rule to .claude/* with !.claude/skills/ so personal agent state stays local while the skills ship * feat: document warmblyctl's new API half on the warmblyctl reference page, reframing the intro around the two halves and their two trust models and replacing the 'no HTTP surface and never will' line with the accurate claim that the CLI never serves HTTP while the API commands are a client of the already-gated public API, adding WARMBLY_API_KEY and WARMBLY_API_URL to the environment table with the API_PUBLIC_URL-then-hosted fallback, renaming The commands to The operator commands, and adding an API commands section covering key setup, the eleven command families, the raw /v1 passthrough with curl-style --data forms, the pagination, idempotency-key and Retry-After conventions, a warning callout naming the six commands that put real mail on the wire with preflight-before-start guidance, and a pointer to the shipped .claude/skills agent skills, plus API authentication and permissions links in See also * feat: move the shipped agent skills from .claude/skills/ to a top-level skills/ directory so they follow the convention other repos use for distributable agent skills rather than living inside Claude Code's personal state directory, restoring the .gitignore .claude/ rule to its original form since nothing tracked lives under it anymore, and updating the warmblyctl reference's For AI agents section to name skills/ and show installing a skill by copying it into the agent's own skills directory or pointing the agent at SKILL.md directly * feat: clear the Security Scan failure by lifting the two flagged indirect Go modules past their fixed versions, github.com/moby/go-archive from v0.2.0 to v0.3.0 for the CVE-2026-17106 tar path traversal and golang.org/x/mod from v0.37.0 to v0.40.0 for the CVE-2026-56864 and CVE-2026-56865 GOSUMDB and GOPROXY forgery pair, with the x/sys, x/text and x/tools bumps go mod tidy pulls along * feat: take the Trivy dependency scan off the PR gate and restructure CI the way larger projects do, because a full-repo CVE scan on every pull request goes red the morning any dependency gets a new advisory regardless of what the PR touches, which is exactly how this branch failed on two indirect Go modules it never went near, moving the scan to its own security.yml running weekly, on demand, and on main pushes that change a dependency manifest, pinned to trivy-action 0.36.0 instead of @master, extracting the pnpm+Node+frozen-install boilerplate repeated across the web, admin and site jobs into a .github/actions/setup-pnpm composite action with the store cached per lockfile, and collapsing the CI Status rollup's ten hand-enumerated result checks that had to be edited in two places per new job into a single contains(needs.*.result, ...) expression over failure and cancelled, all validated with actionlint |
||
|
Matthew Meszaros
|
a75ea012a0 | feat: import a mailbox's recent history on connect and govern sync by fair use: a backfill on every provider (newest first, inside an operator-editable window and cap, resumable through a durable per-provider cursor relayed as SYNC_STATE), a per-mailbox sync governor with priority, live and backfill lanes on shared Redis windows that defers over-budget mail with the cursor held instead of dropping it and only deactivates a mailbox for a flood or chronic daily overage, sync.* budgets on the admin instance settings shipped inside ADD_EMAIL, saved IMAP folder cursors and last_synced_at finally written, a Sync card in the mailbox drawer fed by GET /emails/:id/sync, and docs | ||
|
Matthew Meszaros
|
93e8451738 |
feat: organization data export and import for moving a workspace between instances (#132)
* feat: add the org_export_jobs and org_import_jobs tables plus the models behind them, so a whole organization can be written to a portable archive and read back on another instance, keeping the option columns typed (a text[] of data groups, an include_secrets boolean, a conflict_strategy check constraint) rather than a settings blob because the option set is small and fixed, and reserving jsonb only for the genuinely free-form parts that are read back for display alone (the source archive's manifest, per-table row counts, the import warning list), with partial indexes on the in-flight and expiring rows so the maintenance sweep stays cheap however much transfer history accumulates, an OrgDataGroup catalog that names the twelve slices of a workspace and carries the dependencies between them, and an org_archive audit entity so an export or import rides the existing audit spine into every teammate's dashboard * feat: add the schema-generic repository behind workspace archives, which reads and writes tables by name rather than through typed structs because that is the only way an archive stays correct as the schema grows, moving rows as jsonb in both directions via to_jsonb on the way out and jsonb_populate_recordset on the way in so Postgres performs every type conversion and no hand-written Go column mapping can drift from arrays, jsonb, tsvector, inet or enums, lifting the pool's 60s statement_timeout inside the export transaction because a full inbox read legitimately runs longer than that, introspecting generated, identity and not-null columns plus primary keys and foreign keys from the catalog rather than trusting a compiled list, and treating identifier safety as structural: table names come from the compiled registry and column names are always intersected against the destination catalog before reaching a query, so nothing out of an uploaded archive is ever interpolated * feat: add the workspace archive registry and on-disk format, covering all 110 organization-owned relations with their scope SQL, dependency order and per-table policy, plus 10 explicitly excluded ones each carrying the reason it must never travel (the KMS-wrapped org data key, in-flight OAuth handshakes, the websocket outbox, live sessions, a pending deletion that would otherwise schedule the destination workspace for destruction), naming the two key domains separately because Warmbly seals mailbox credentials under the instance CREDENTIALS_ENCRYPTION_KEY and everything else under the per-organization DEK and confusing them produces mailboxes that authenticate against nothing, defining the archive as a plain zip of newline-delimited JSON so an operator can unzip it and read the data in a text editor and so the manifest can be written last yet still be read first, and sealing archive secrets under an argon2id passphrase key with parameters deliberately heavier than the login hash since it is derived once per archive and guards every credential in the workspace against offline grinding * feat: implement the workspace export and import engines, streaming rows straight through untouched for the tables that have neither secrets nor blobs so a million-row inbox export stays cheap and only decoding the rows that must change, opening every sealed value against whichever key domain wrote it and re-sealing it under the archive passphrase on the way out then against the destination's own keys on the way in, blanking a credential rather than sinking the whole export when one mailbox cannot be read and clearing the guard flag alongside it so no row is left claiming ciphertext it no longer holds, applying an import inside a single transaction because a half-applied workspace is far worse than a long-running one, rewriting the organization id and matching members to destination accounts by email with unresolvable people blanked where the column is nullable and redirected to the importer where it is not, and running transfers in the accepting process rather than through a queue for the one reason that matters: the passphrase is then never written down anywhere * feat: make the per-organization DEK cache nil-safe in internal/app/cipher so a process built without Redis falls through to KMS on every call instead of dereferencing a nil cache handle, which is what lets warmblyctl run the workspace export and import commands at all: it deliberately attaches Redis as optional because the whole point of that CLI is working while the rest of the instance is down, and the decrypted-key cache was always an optimisation rather than a requirement * feat: add the hourly workspace-archive maintenance job that deletes finished archives past their seven-day retention window, since each one is a complete copy of a workspace sitting in object storage and must not accumulate, and closes out any export or import whose process died mid-run, which is the necessary counterpart to executing transfers in the accepting process so the passphrase is never persisted: without this sweep a restart would leave a job reporting running forever * feat: expose workspace export and import over the JWT-only organization routes and wire the service into the backend, gating every endpoint on workspace ownership through the existing requireOrgOwner check rather than a permission bit because an export with credentials is the single most sensitive artifact this product can produce and an import rewrites the workspace wholesale, so both belong at the same level as deleting it, spooling uploads to a temporary file since a zip needs random access and a length that a multi-gigabyte archive cannot supply from memory, handing that file's ownership to the background import so it outlives the request and is closed exactly when the job ends, streaming downloads with the archive's sha256 in a response header, and constructing the service with both key domains plus object storage so an archive can be opened, re-keyed and stored * feat: add warmblyctl org list, export and import so a self-hoster can move a workspace from the box without a browser, running the same engine in-process against Postgres and adding no HTTP surface to a CLI whose entire trust model is container or host access, resolving --org from whichever handle the operator has (id, slug, or the owner's email), streaming the archive to a file or to stdout so it can be piped straight into ssh with progress still readable on stderr, prompting for the credential passphrase twice through the existing password prompt so the terminal and pipe rules stay identical across every command, and defaulting the import path to a preflight report that names what already exists here and which members have no account before anything is written, with --dry-run to stop there * feat: add the dashboard API layer for workspace archives, fetching the data-group catalog from the server rather than restating it in the client so a new group appears the moment the backend knows about it, mirroring the server's group-dependency closure in expandGroups so the toggles a user sees always match what the archive actually gets, polling only while a transfer is in flight and dropping to no interval the moment none are active since a running job has no realtime event of its own, and downloading a finished archive as a blob through the authenticated client because the endpoint is bearer-authenticated and a plain anchor href cannot carry the token * feat: build the Settings and Data dashboard page for exporting and importing a workspace, following the settings section conventions and the in-app confirm rather than window.confirm, defaulting the export to every data group because a migration that quietly leaves data behind is worse than one that takes a while, marking the heavy groups so nobody exports a decade of inbox history unaware, requiring the credential passphrase twice behind a confirm that states plainly what the file will contain, and making the import a two-step flow where a preflight reads the archive and reports its origin, row counts, unsealable credentials, existing rows and unknown members before a single byte is written, so confirming is never a leap of faith * feat: register the Data settings section in the dashboard rail, route and realtime spine, placing it under Advanced beside the danger zone and gating it to the workspace owner so the nav matches what the endpoints actually allow, and mapping the new org_archive audit entity to the export and import query keys in useRealtimeEvents so an archive starting or landing refreshes the page for every teammate through the existing audit spine rather than a bespoke event * feat: document workspace export and import as a customer guide registered under Account and team, covering what each of the twelve data groups contains and which four dominate archive size, why credentials need a passphrase to travel at all and what happens to mailboxes when they do not, how members are matched to destination accounts by email and what becomes of anyone without one, the difference between keeping existing rows and replacing them, and a table of what deliberately does not import with the reason for each, because billing, plan overrides, worker placement, sync checkpoints and warmup pool membership belong to an instance rather than to a workspace * feat: document org list, export and import in the warmblyctl reference and point the deployment guide at them as the supported route between a self-hosted install and the hosted service in either direction, adding every flag with what it does, the two extra environment variables those commands read and the difference between them (a missing KMS provider stops the command because sealed values cannot be opened, while a missing CREDENTIALS_ENCRYPTION_KEY is only a warning that mailbox credentials will not move), the behaviour when Redis is down, and the warning that an archive carrying credentials is the most sensitive file this product produces * feat: record in AGENTS.md that a migration adding an organization-scoped table is not finished until that table is registered in internal/app/orgtransfer/spec.go, either in Tables with its group and scope or in ExcludedTables with the reason it must not travel, because data left out of the registry is silently absent from every archive and nobody discovers it until a customer's migration lands on the other side missing a feature's data, and spelling out the four things that are easy to get wrong when adding one: dependency order, the group boundary that needs a Requires entry only when a NOT NULL foreign key crosses it, which of the two key domains seals a ciphertext column, and which columns name something only the source instance knows |
||
|
Matthew Meszaros
|
c39c29ab6b | feat: rebuild the new-campaign wizard with animated step transitions, a numbered stepper, the shared Toggle instead of a broken hand-rolled switch, per-step validation that explains itself and a discard guard, register PopoverMenu's click-outside in the capture phase so dropdowns inside dialogs close on click-away, add a Campaigns back link and clickable breadcrumb crumbs, add a From contacts leads picker with category filter and select-all-matching backed by the bulk add_campaigns path whose SQL now scopes campaigns by organization instead of the caller, and stop self-hosted no-billing deployments presenting as a free trial or plan-metered by exposing billing_enabled on GET /auth/config, showing a Self-hosted badge, hiding Billing and Refer & earn, and reporting AI credits as unlimited with the header gauge and cost copy hidden | ||
|
Matthew Meszaros
|
7521575cef | feat: correct the documented order of the four blocks warmblyctl status prints, in both the new warmblyctl reference and the first-run sample output, after running the command against a live instance showed printStatus emits Instance, Platform admins and How to get in before runStatus calls printChecks, so the checks come last and not third as both pages claimed, which also means the first-run sample had its Checks and How to get in blocks transposed, and an operator scrolling for the recovery commands was being told to look past a findings list that is actually printed after them (#129) | ||
|
Matthew Meszaros
|
619f4fd9a1 | feat: add a warmblyctl reference page at docs/content/docs/development/warmblyctl.mdx documenting all nine commands with every flag, because the README and the recovery sections only ever showed 'warmblyctl user create --email ... --admin' without saying where the password comes from, leaving self-hosters with an account they could not sign in to and no page that answered it, covering that user create prompts for the password twice on a terminal and refuses on a non-TTY unless --password-stdin is passed, that docker compose exec allocates the TTY those prompts need unless -T is given and piped input needs -T precisely because it removes it, that the password rule is the dashboard's own 8 to 128 characters, and that signing in afterwards needs nothing else on a stock self-host since AUTH_LOGIN_CODE defaults to off and REQUIRE_EMAIL_VERIFICATION to false when self-hosted, captcha stays off without TURNSTILE_SECRET, and --admin opens the panel on ADMIN_URL rather than APP_URL, plus the status JSON contract and exit codes, the four environment variables each command reads, the per-command behaviour when Redis is down, the admin role masks read from AdminRolePermissions, and the make wrappers, registering it in the Development meta.json between accounts-and-access and configuration, linking it from the four pages that already print these commands, correcting the super-admin mask in the first-run sample output from 4294967295 to the 4194303 that AllAdminPermissions actually is since it is (1 << 22) - 1 and not the full uint32 range, and shortening the README self-hosting section by folding the three-bullet gotcha list into a four-row table that also names make doctor and dropping the duplicated make dev claim warning already stated above it, keeping every fact (#128) | ||
|
Matthew Meszaros
|
bd8545a1c5 | feat: stop a freshly connected mailbox being silently excluded from every campaign send, by making an unset mailbox timezone representable as the empty string the campaign scheduler already checks for, since email_accounts.timezone defaulted to 'UTC' while campaigns.timezone defaults to 'Europe/London' and nothing in the OAuth or SMTP onboarding paths ever set either, so a brand new mailbox looked deliberately placed in UTC, was compared against the differing campaign zone and dropped by the hardcoded 8am-8pm business-hours gate whenever the current UTC hour fell outside it, emptying the candidate pool and failing the campaign start, adding a migration that changes the column default and converts existing 'UTC' rows because until now no API field, dashboard control or onboarding path could set that column at all so every such row is the old default rather than a choice, adding the missing Timezone field to UpdateEmail with IANA validation so the setting the sending-behaviour UI already tells people to change is finally reachable and an unloadable zone is rejected instead of being silently coerced to UTC by the scheduler, and replacing the misleading 'no active email accounts found for campaign's email tags' response for a pool that exists but is entirely gated out with a distinct message naming the real cause, via an ErrNoEligibleMailbox that wraps ErrNoEmailAccounts so the three callers that pause a campaign on it are unaffected (#126) (#125) (#127) | ||
|
Matthew Meszaros
|
f0846eb034 | feat: stop a freshly connected mailbox being silently excluded from every campaign send, by making an unset mailbox timezone representable as the empty string the campaign scheduler already checks for, since email_accounts.timezone defaulted to 'UTC' while campaigns.timezone defaults to 'Europe/London' and nothing in the OAuth or SMTP onboarding paths ever set either, so a brand new mailbox looked deliberately placed in UTC, was compared against the differing campaign zone and dropped by the hardcoded 8am-8pm business-hours gate whenever the current UTC hour fell outside it, emptying the candidate pool and failing the campaign start, adding a migration that changes the column default and converts existing 'UTC' rows because until now no API field, dashboard control or onboarding path could set that column at all so every such row is the old default rather than a choice, adding the missing Timezone field to UpdateEmail with IANA validation so the setting the sending-behaviour UI already tells people to change is finally reachable and an unloadable zone is rejected instead of being silently coerced to UTC by the scheduler, and replacing the misleading 'no active email accounts found for campaign's email tags' response for a pool that exists but is entirely gated out with a distinct message naming the real cause, via an ErrNoEligibleMailbox that wraps ErrNoEmailAccounts so the three callers that pause a campaign on it are unaffected (#126) (#125) | ||
|
Matthew Meszaros
|
fe9a21a79f | feat: stop a freshly connected mailbox being silently excluded from every campaign send, by making an unset mailbox timezone representable as the empty string the campaign scheduler already checks for, since email_accounts.timezone defaulted to 'UTC' while campaigns.timezone defaults to 'Europe/London' and nothing in the OAuth or SMTP onboarding paths ever set either, so a brand new mailbox looked deliberately placed in UTC, was compared against the differing campaign zone and dropped by the hardcoded 8am-8pm business-hours gate whenever the current UTC hour fell outside it, emptying the candidate pool and failing the campaign start, adding a migration that changes the column default and converts existing 'UTC' rows because until now no API field, dashboard control or onboarding path could set that column at all so every such row is the old default rather than a choice, adding the missing Timezone field to UpdateEmail with IANA validation so the setting the sending-behaviour UI already tells people to change is finally reachable and an unloadable zone is rejected instead of being silently coerced to UTC by the scheduler, and replacing the misleading 'no active email accounts found for campaign's email tags' response for a pool that exists but is entirely gated out with a distinct message naming the real cause, via an ErrNoEligibleMailbox that wraps ErrNoEmailAccounts so the three callers that pause a campaign on it are unaffected (#126) | ||
|
Matthew Meszaros
|
ee61c19faf | feat: create /data/blobs in the backend, consumer and worker images owned by the non-root user they run as, because Docker seeds a fresh named volume from the image and the path did not exist there, so it created the mount point root:root while every service runs as uid 1000 and the very first attempt to store an outbound body failed with 'mkdir /data/blobs/emails: permission denied' and the task went to the dead-letter queue, which is invisible until a real send is attempted since the stack is otherwise healthy, and documenting the one-time chown that existing installs still need because Docker only applies image ownership when it first creates the volume (#124) | ||
|
Matthew Meszaros
|
ec4cd3160b | feat: thread Unibox dashboard replies into the conversation they answer, by carrying the composer's thread_id all the way from email_tasks to the provider (EmailMessage had no ThreadID field at all, so the column was read from the database and silently dropped in user_email_task, and Gmail only appends to an existing thread when threadId is set on the outbound message since a matching Subject and In-Reply-To do not do it), populating the models.SendEmail.Parent field that already existed with an avro tag and that the worker already read but nothing ever set, replacing the worker's gate that required InReplyTo to be non-empty before it would look at Parent (a dashboard reply never sets that header, so a perfectly valid ThreadID was discarded and the provider opened a new conversation) with a parentReference helper shared by the Gmail and Graph send paths that resolves the two genuinely independent handles separately, and backfilling the RFC In-Reply-To header server-side in UniboxReply from the newest Message-ID in the thread via a new org-scoped LatestMessageIDInThread query, because a provider thread id is meaningless outside the mailbox that issued it and the recipient's mail client can only thread on References and In-Reply-To (#122) | ||
|
Matthew Meszaros
|
cdf200191c | feat: accept the API's origin in the mailbox OAuth callback listener so connecting Gmail or Outlook completes on a split-domain deployment, where the bridge page is served by the backend (deliberately, so the registered redirect_uri survives front-end changes) and therefore arrives with event.origin equal to API_URL while the dashboard only ever compared it against APP_URL, silently discarding every callback and leaving the connect modal on 'Waiting for authorization' forever even though the provider exchange had already succeeded, normalising both configured bases through URL.origin so a trailing slash no longer breaks the comparison either, and separately deriving the bridge's postMessage target origin from APP_URL when APP_ORIGIN is unset instead of falling back to a wildcard that posts the authorization code to whatever origin the opener happens to have, since compose never set APP_ORIGIN despite the configuration table claiming it was derived, with the app_origin_wildcard health check and both docs pages updated to match the narrower condition that now triggers it (#117) | ||
|
Matthew Meszaros
|
8d2968eaa8 | feat: build the Gmail and Outlook mailbox-connect redirect_uri from API_PUBLIC_URL instead of API_HOST, which is the listener's bind address and stays 0.0.0.0:8080 in a container, so every self-hosted OAuth connect sent Google and Microsoft a redirect_uri that is not even an absolute URI and was rejected with invalid_request before the account picker appeared, adding an oauthPublicBaseURL helper next to the existing oidcRedirectURL that reads the same already-documented variable and falls back to a browsable http://localhost:PORT derived from the bind address (mapping the 0.0.0.0, :: and bare-port wildcard forms to localhost, and passing through a value that is already a URL) so a stock local install emits exactly the http://localhost:8080/addresses/google/callback the guide tells operators to register, plus a test pinning the resolution order and the root-registered callback path, and a deployment-guide note naming API_PUBLIC_URL as the variable that has to match the registered URI (#116) | ||
|
Matthew Meszaros
|
1a7cdc955f | feat: seal Gmail and Outlook OAuth access and refresh tokens at rest in email_accounts_oauth instead of storing the provider's raw tokens, which the connect UI already promised were encrypted and which the read path could never open because it unconditionally hex-decodes, adding sealCredential/openCredential helpers that fail closed when CREDENTIALS_ENCRYPTION_KEY is unset, encrypting on both write paths (NewOauthAccount at connect time and RefreshBoxToken on every worker token refresh, which would otherwise revert a sealed row to plaintext on first refresh), migrating pre-existing plaintext rows lazily on first read because the key lives in the application and no SQL-only migration can reach it, restoring the missing return on the OAuth insert failure that let a failed token write commit an account row with no credentials, keeping token parameters out of Sentry error reports, and correcting the two docs tables that scoped the key to SMTP and IMAP only (#115) | ||
|
Matthew Meszaros
|
734cb5fe08 | feat: make self-hosted onboarding survivable by fixing invite_only, which could not onboard anyone (the accept route is JWT-only, so redeeming the invitation that would create your account required already having one, making the self-host default silently identical to fully closed), threading the invitation token through registration so an invited person lands in the inviting organization instead of a stray workspace, gating SSO just-in-time provisioning behind DISABLE_REGISTRATION (it bypassed the gate entirely, so an instance set to true was still open to anyone the IdP would assert) with SSO_AUTO_PROVISION as the opt-out, correcting the OIDC redirect URL that pointed at /api/v1 against a route at /v1 and 404'd every SSO login, scoping the first-launch exemption so it no longer overrides an explicit lockdown, preserving the remaining TTL when restoring a losing setup token so a public endpoint cannot hold the claim window open forever, replacing a generic 403 with typed registration_invite_only, registration_closed, invitation_invalid, setup_token_invalid and setup_already_complete codes that name the next step, logging why no claim link was issued on an already-claimed instance instead of staying silent, adding a warmblyctl operator CLI (status with health checks and a non-zero exit, reissuable setup-link, user create/list/reset-password/grant-admin/revoke-admin/disable-2fa, hash-password) so a locked-out operator no longer needs hand-written psql, adding read-only instance configuration over 104 environment variables with structural secret redaction and fingerprints, 35 health checks, a database-backed settings tier for the three keys no environment variable owns, hiding the signup form when the config already says invite_only rather than failing the whole form with a toast, and documenting first run, accounts and access, configuration, instance health and troubleshooting alongside the root .env.example the README told operators to write but never shipped (#114) | ||
|
Matthew Meszaros
|
0ae4db2c41 | feat: make self-hosted auth work without a mail relay by rewriting the platform SMTP transport with real AUTH and TLS (it did neither, so SMTP_USERNAME/SMTP_PASSWORD were dead and every documented relay was unreachable), adding MAIL_TRANSPORT=smtp|log|ses with a log transport that prints codes so a fresh install can sign in with no relay, demoting the emailed login code to AUTH_LOGIN_CODE=always|new_device|off (off on self-host, per NIST SP 800-63B and OWASP ASVS), claiming the first owner through a single-use setup link or WARMBLY_BOOTSTRAP_* instead of register-then-psql, deriving every emailed URL from APP_URL rather than a hardcoded app.warmbly.com that leaked live reset tokens to the vendor, fixing the confirm hooks that read path params against paramless routes and broke login, register and reset confirmation in the dashboard everywhere, adding generic OIDC with PKCE, one-time state, verified nonce and (issuer,subject) identity binding, enforcing 2FA on the social paths that skipped it, adding a per-IP limiter and trusted-proxy handling to the unthrottled auth group, refusing boot on the published default secrets, and dropping mailpit from the default stack (#99) |