Commit Graph
316 Commits
Author SHA1 Message Date
Matthew Meszaros 5a967cc3ce feat: let an IMAP mailbox exclude folders from sync (email_accounts.sync_skip_folders, migration 000196) so a folder another tool fills never reaches the unified inbox: the worker drops skipped folders and their subfolders before the walk, retires an already-synced one with its stored mail, and removes mail that later moves into one only when its Message-ID is found there; PUT /emails/:id/sync and the drawer's Sync card set the list, GET reports it with the server's folder list, warmbly mailbox skip-folders mirrors it, with docs, OpenAPI and error-code invalid_sync_folder 2026-09-22 05:15:49 -07:00
Matthew Meszaros eac3f6d615 Merge remote-tracking branch 'origin/main' into fix/sso-link-existing-password-account
# Conflicts:
#	docs/content/docs/guides/security.mdx
2026-09-21 04:28:20 -07:00
Matthew Meszaros a1b7a8ad9b feat: attach a parked federated identity only after the ban check and, on a 2FA account, only once the second factor passes by carrying it through the 2FA pending record into twofa.VerifyLogin, charge each sso_link password attempt atomically before the check, treat an identity a parallel challenge already linked as a re-login instead of a refusal, ask for no password when the identity cannot be linked, end an exhausted or expired challenge with sso_link_expired so the dashboard returns to the email step, and document the code in error-codes, the API reference and OpenAPI 2026-09-21 03:35:17 -07:00
Matthew Meszaros 9426c0da51 feat: validate every person, workspace and company name through internal/pkg/displayname on each write path (profile, onboarding, setup, IdP sign-in, org create and rename, org import, enterprise inquiry, admin testers, warmblyctl) with a 400 invalid_name code, render stored names in platform email through the same rules, mirror them in the web forms, check the org slug format, and document the rules in error-codes, security and AGENTS.md 2026-09-21 03:34:03 -07:00
Matthew Meszaros f11df9268f Merge pull request #640 from warmbly/fix/admin-list-pagination
feat: make every admin panel list page past the first and filter by id, and report failed admin requests
2026-09-21 09:29:48 +00:00
Matthew Meszaros cc244e5159 feat: make every admin panel list page past the first and filter by id: bind query-string ids through models.ParamUUID since gin cannot set a uuid.UUID, page the explorers and secondary lists by an opaque offset cursor with an id tiebreak instead of an id keyset that disagreed with the sort, page the audit log on (created_at, id) with an inclusive YYYY-MM-DD end day and read next_cursor on its page, cast every before-date bound to timestamptz, coalesce nullable audit ip and user agent, and report failed admin queries and 5xx mutations to PostHog or Sentry with method, path, status, code and request id 2026-09-21 02:11:38 -07:00
Matthew Meszaros 1513419a2a feat: delete warmup mail from each mailbox once past a per-mailbox retention window (email_accounts.warmup_retention_days, else retention.warmup_mail_days, default 30) via a consumer sweep that retires the receipt and sender copy and a worker delete action that trashes on Gmail, deletes on Graph, expunges on IMAP and drops the stored body, prune per-message warmup records after retention.warmup_event_days, and count a warmup deletion as tampering only within 24 hours of arrival and never for a retired message, judging Gmail's Trash label on the same rule 2026-09-21 00:52:02 -07:00
Matthew Meszaros 6026168334 feat: stop blocking boot on the GeoIP download and swap the database in when it lands, retry a refused mirror with backoff honouring Retry-After, revalidate a database older than a week with If-Modified-Since instead of keeping the first copy forever, and add a twice-weekly job mirroring both MaxMind editions to a private bucket so the fleet stops spending a 30-a-day allowance per boot 2026-09-20 17:55:38 +02:00
Matthew Meszaros d1a4add567 feat: add the new CHECK constraints NOT VALID and validate them in migration 000189 so campaign_leads is not scanned under a write lock, keep a step's copy-judgment findings open when the Advisor's judge cap or an outage left it unjudged, validate every TypeSafe answer's type and bounds and version the copy-judgment cache key, mirror the reads-as-bulk rule in the editor's all-clear, and refuse inbox classification without a configured client 2026-09-19 23:59:19 -07:00
Matthew Meszaros addb956ad6 feat: shared TypeSafe client under internal/pkg/typesafe with inbox tagging phases 2 and 3 (hold, stop, task, suppress behind workspace switches, reversible ones on by default), labels seeded at workspace creation and by the follow-up sweep, premade inbox views (hot leads, needs a reply, follow up, declined, automated), typed reply classification and a reply_intent branch condition, an inbox agent draft gate, Advisor copy judgment with a cached editor re-check, warmup content lint, bounce cause classification that keeps a blocked address sendable, and per-form submission triage 2026-09-19 23:33:37 -07:00
Matthew Meszaros dee54417a3 Merge pull request #612 from warmbly/feature/campaign-daily-send-view
feat: add a Today's sending plan to the campaign overview and feed the sidebar meter and wizard estimate from the scheduler's own clamps (issue #606)
2026-09-19 17:04:19 +00:00
Matthew Meszaros 150dc7df6e feat: add a Today's sending plan to the campaign overview (GET /campaigns/:id/send-plan, derived through the scheduler's own gates: per-mailbox cap clamps, warmup graduation, health bands, other campaigns on the same mailbox, hours, spacing, window, plan allowance, new-lead cap and leads due, as a waterfall that adds up), feed the sidebar meter and the wizard estimate from the same clamps instead of summing configured caps, floor the campaign chain's next tick at the pool's spacing rather than one mailbox's whole gap, fold the compact Advisor strip to one line, add warmbly campaign plan and warmblyctl campaign plan, and document it (issue #606) 2026-09-19 09:31:46 -07:00
Matthew Meszaros 62201a2dd3 feat: let each member choose, reorder and persist the contact list's columns (custom fields included) and sort on any of them: a user_view_preferences table (migration 000187) behind GET/PUT/DELETE /v1/me/views/:view, a column registry that renders the contacts and campaign Leads tables from a saved layout, a Columns chooser with drag reorder and a Sort menu on both toolbars, click-to-sort headers, sort_by custom:<key> plus company and phone sorts in POST /contacts/search resolved once for Search and SearchIDs with a nullable keyset cursor, and the contacts guide, API reference, openapi, error codes and export-import docs updated 2026-09-19 09:24:17 -07:00
Matthew Meszaros e668a2a36b feat: complete the ADA CASA v2.1.1 AL1 control set across authentication, sessions, access control, cryptography, input validation and configuration, adding a breached-password denylist and per-account login throttling, enforced multi-factor authentication on the admin panel, step-up confirmation before an action that mints a lasting credential, purpose-scoped session tokens, single-use TOTP steps, tenant verification on every cross-referenced identifier, security headers on every surface, encrypted webhook signing secrets, per-organization idempotency, PKCE and a minimal two-scope Gmail consent on the mailbox OAuth flow, bounded spreadsheet and archive decoding, a patched Go toolchain with govulncheck in CI, and the evidence pack under compliance/casa 2026-09-19 08:18:35 +02:00
Matthew Meszaros 6aebfe7e63 feat: store IMAP-synced addresses as Name <addr> like the Gmail and Graph syncs instead of Name (addr), teach mailhdr.Bare, the reply path's sender and recipient checks and the warmup sender fallback to read the old form for existing rows and older workers, so a reply into an IONOS or any other IMAP mailbox is attributed to its lead again after the address checks added on 16 September refused every one of them, and add a consumer sweep that re-offers unclaimed inbound mail answering a campaign send or coming from a contact to reply processing at boot and daily so the replies missed that week are attributed without anyone touching the database 2026-09-18 14:37:35 +02:00
Matthew Meszaros cd964aafa8 feat: stop a deleted mailbox living on inside its worker, by returning the worker assignments a scheduled org or user deletion destroys so the erasure path can evict them the way the single-mailbox delete already does, and by treating an assignment lookup that finds no row as the mailbox being gone rather than a failed lookup, which tells every live worker to drop it; and clear the rest of error tracking by retrying a mailbox delete that loses a deadlock to its own cascade instead of failing the person who clicked Disconnect, answering a daily-usage read for a mailbox that no longer exists with 404 rather than a reported 500, and guarding the inner data field on three list reads plus serialising an empty pool-link list as [] rather than null 2026-09-18 12:42:43 +02:00
Matthew Meszaros e37c5053c2 feat: make warmup refunds atomic, count confirmed partner diversity in local and cloud mailbox views, and document cloud-safe mailbox deletion 2026-09-17 21:15:28 -07:00
Matthew Meszaros 8ee1c50a1b feat: make a failed SMTP send name the step and the cause behind it instead of one bare SERVER_UNREACHABLE sentinel, give a refused warmup send its day back so sent_today can no longer climb past the target while the cap frees the slot, revoke a mailbox's Warmbly Cloud enrollment when it is deleted so the pool stops holding its password, and prefer warmup partners outside the sender's own workspace while showing the partner diversity a mailbox is actually getting (#574, #575) 2026-09-17 20:02:37 -07:00
Matthew Meszaros 2a1e55354d feat: merge latest main before requeueing worker capacity fixes
# Conflicts:
#	internal/app/stripe/service_test.go
2026-09-17 06:57:20 -07:00
Matthew Meszaros c1e45b194a feat: merge latest main before worker capacity queueing 2026-09-17 06:26:07 -07:00
Matthew Meszaros 8c7827c199 feat: make Stripe credit auto-top-ups idempotent across retries 2026-09-17 15:25:35 +02:00
Matthew Meszaros d6025ea4c0 feat: address worker capacity review findings with safe migrations and recovery reporting 2026-09-17 06:24:14 -07:00
Matthew Meszaros b733d09f89 feat: make inbox follow-up labels safe for manual labels and automated mail 2026-09-17 04:57:52 -07:00
SUMAN JANA 3b8761d361 feat(inbox): explain every tag on hover, label live mail, and keep follow-ups working with no external service 2026-09-17 04:57:52 -07:00
SUMAN JANA dff6b42558 feat(inbox): follow-up labels for who owes whom a reply, computed in code and swept hourly 2026-09-17 04:57:52 -07:00
Matthew Meszaros dcf26a2361 feat: make automatic inbox tagging atomic live and reviewable in production 2026-09-17 04:57:24 -07:00
SUMAN JANA 7e556f49e3 feat(inbox): backfill historical mail through the tagger, and keep a confident kind when the intent cannot be read 2026-09-17 04:57:24 -07:00
SUMAN JANA 260898bf20 feat(inbox): automatic tagging and relevance scoring for inbound mail, optional and off by default 2026-09-17 04:57:24 -07:00
Matthew Meszaros 8745dd988d feat: correct direct-mail analytics attribution and publish live engagement updates 2026-09-17 04:27:00 -07:00
Matthew Meszaros bab9f86727 feat: correct worker capacity, mailbox distribution, observed IPv4, fleet pagination, and premium pool promotion 2026-09-17 04:15:05 -07:00
SUMAN JANA 2134c7a143 feat(analytics): report on mail written by hand, with opt-in open and click tracking per mailbox 2026-09-17 10:26:25 +00:00
Matthew Meszaros 561f8ec671 feat: keep cloud and local warmup mail out of Unibox with durable verification and automatic cleanup 2026-09-16 03:55:36 -07:00
Matthew Meszaros 4784ee7d39 feat: fetch the MaxMind databases instead of requiring a mounted file (#529)
* feat: let the backend, consumer and tracking service fetch their own MaxMind databases from GEODB_URL and TRACKING_SCANNER_ASN_DB_URL, reading the archive shape from the content so a permalink tar.gz, a gzipped mmdb and a bare mmdb all work, never replacing a file already at the path, opening the bytes before installing them so a licence-key error page cannot become the database forever, skipping the AppleDouble sidecars a macOS tar writes ahead of the real file, and treating both URLs as secrets because the permalink carries the licence key

* feat: drop the trailing blank line cargo fmt --check rejects at the end of tracking/src/asndb.rs

* feat: stream the downloaded ASN archive instead of decompressing it whole, sizing each buffer from the gzip footer and the tar header so the member is allocated exactly once, which drops the peak of unwrapping a permalink tar.gz from 38 MB to 11.9 MB, essentially the database itself

* feat: stop the MaxMind licence key reaching the logs through net/http's and reqwest's own error text, which both print the URL they were given and so defeated the redaction beside them, drop userinfo as well as the query when redacting, refuse plain http for a URL carrying a credential and refuse an https-to-http redirect, and apply the size cap to the decoded database rather than the compressed transfer so a gzip bomb cannot fill the disk

* feat: strip basic-auth userinfo as well as the query when the tracking service redacts its database URL, parsing it rather than cutting at the first question mark so where a credential sits is the URL library's problem and not a guess
2026-09-15 03:06:34 -07:00
Matthew Meszaros e67b13e57e feat: stop reporting a mailbox's DKIM as missing when its selector was simply never probed, by deriving candidate selectors from the sending domain's own SPF and MX records on top of a wider default set, reporting a miss as the tri-state dkim_status undetermined rather than a red Missing row in the drawer, dropping DKIM from the Advisor's missing-records finding entirely, refusing a revoked p= key, holding the summary back from accusing anything when DNS never answered, and fixing the CLI auth-check table whose columns read mailbox fields the endpoint does not return (#528) 2026-09-15 02:27:38 -07:00
Matthew Meszaros dd98187231 fix: shorten recipient unsubscribe links to 22-character, 128-bit stored tickets (#498) (#525)
* feat: shorten every recipient unsubscribe link from a 96-character signed token to a 22-character stored ticket carrying 128 bits from crypto/rand, minted once per recipient per campaign and reused by every step, so the address the text/plain half of a cold email prints in full fits on one line and cannot be guessed, keeping the signed form working for links already in inboxes and as the fallback when the store cannot be written, and answering a failed lookup with a retryable 'try again shortly' instead of telling the recipient their opt-out is invalid (issue #498)

* fix: restore the disabled-signer guard in URLOn, which factoring the URL builder moved behind a token mint that dereferences the signing key, so a nil or origin-less signer returns the empty string every caller reads as 'no link can be minted' instead of panicking (PR #525 review)
2026-09-15 00:42:45 -07:00
Matthew Meszaros 1ca3bd19b3 feat: carry a unibox read or unread change out to the mailbox itself through a new MESSAGE_SEEN worker command, so a conversation read in Warmbly stops showing bold in Gmail, Outlook and IMAP, relaying the state the row holds rather than the one the request asked for, only for messages that actually changed, dispatched detached from the request and never retried (#515) 2026-09-14 21:20:42 -07:00
Matthew Meszaros 92e298b6ec fix: clear every open issue in error tracking by fixing the bugs behind them rather than the reports: a document-level mouseleave handing RippleProvider the document itself, whose classList is undefined; the admin panel posting /getaway without the /v1 its baseURL omits, so its realtime socket 404d on every page; Gmail throttles classified from the 403 status alone and told to re-authorize instead of back off; consumer flag and folder events retrying forever on a message the unibox never stored; a lost token-refresh race answered 500 instead of the documented 401; a nil email_accounts slice crashing the admin user page; Turnstile mounted with an empty sitekey; conditional passkey autofill run after the user navigated away; a boot log filed as an issue; and one publish failure per message on a topic the broker refuses (#519) 2026-09-14 21:06:14 -07:00
Matthew Meszaros 8d790ede6c feat: send from any address Google has verified a Gmail mailbox to send as and import the signature its owner already wrote in Gmail, reading both through gmail.settings.basic at connect and on demand via GET/POST /emails/:id/identity, validating the choice against the provider's own list in the service and again inside the UPDATE, clearing it when the provider stops verifying it, and never applying it to warmup (#514) 2026-09-14 10:13:36 -07:00
Matthew Meszaros 13e9ce8e10 feat: hold a lead whose mailbox answers out of office until they are back, resuming at the return date it names, plus a manual per-contact pause in one campaign that unsubscribing and the suppression list were the only stand-ins for 2026-09-14 09:26:06 -07:00
Matthew Meszaros c28f915648 feat: erase everything a disconnected mailbox leaves behind, revoking its OAuth grant at Google and deleting its stored message bodies through a durable retried queue, cascade the nine mailbox foreign keys that had none so warmup receipts, tampering events and provider message maps stop outliving the mailbox, clear thread labels and snoozes on conversations the delete emptied, make workspace deletion possible at all by cascading the four organization foreign keys with no delete action, and put Disconnect in the mailbox row menu and a Settings danger zone since it was only reachable from the selection bar (#506) 2026-09-14 07:55:01 -07:00
Matthew Meszaros 40506c4f05 fix: seed the two warmup pools on every instance under fixed ids and make one pool per type structural, since the baseline squash dropped the insert and a fresh self-hosted instance never warmed; move memberships onto the canonical pools, scope the standing mirror trigger to the columns it mirrors so a pool move keeps a retention window, commit the runtime and every seeder to the ids through MoveToPool, assert the pools at boot and in a warmup_pools_missing health check, and drop the guide's claim of cross-tier borrowing the health gate rejects (#493) 2026-09-13 21:37:17 -07:00
Matthew Meszaros 7300b3b021 feat: full PostHog coverage: identify the signed-in user and workspace in the dashboard and admin panel with autocapture, heatmaps, dead and rage clicks, web vitals, network timing, console capture and session replay masking only password fields, send server-side signup, trial and subscription events under the user id with the organization as a group, keep the marketing site and form pages cookieless while capturing everything stateless plus a form funnel, upload the form app's source maps, and add WARMBLY_POSTHOG_SESSION_REPLAY 2026-09-13 20:58:22 -07:00
Matthew Meszaros 6b6efca865 fix: campaign follow-ups opened a new conversation instead of replying in the contact's thread, so carry In-Reply-To/References and the Gmail threadId from the previous send, give every step a reply-in-thread switch, and let a threading step inherit the conversation's subject (issue #472) (#489) 2026-09-13 20:51:41 -07:00
Matthew Meszaros 43dcbde06c feat: tester accounts, creatable from the admin panel (#483)
* feat: excuse one named account from the emailed login code, so a vendor reviewer who cannot read this instance's mail can sign in without turning codes off for everyone, with the reason recorded beside it and every run of warmblyctl status naming the accounts that hold one

* feat: create and manage tester accounts from the admin panel, so letting a reviewer in is a form rather than a shell, with the password shown once and every live exemption listed on one page because forgetting one is the way this goes wrong

* fix: give tester management its own permission bit rather than borrowing ban_users, create the account and its exemption in one transaction so no invisible orphan survives a failure, require an accountable operator on the CLI grant, stop a halted row scan reading as the whole exempt list, and show a failed query as an error instead of as no testers

* chore: re-run CI after the aggregator tripped on a cancelled job from the branch update, with every underlying job green

* chore: retrigger CI, the previous run sat queued indefinitely while other branches ran

* feat: roll back a half-created tester when its workspace step fails and backfill the manage-testers bit onto admins already holding every other permission, so the address is not left taken by an unusable account and the new routes are not 403 for the existing admin

* feat: make the 000151 manage-testers backfill one-way, because clearing bit 22 on the way down would also revoke it from an admin granted it explicitly afterwards and the up migration would not restore that
2026-09-13 03:07:10 -07:00
Matthew Meszaros 017f4cf60f feat: plans an operator can grant, visible in the admin panel (#467)
* feat: add operator-granted plans so a workspace can be paid without Stripe, surfaced in the admin panel as a badge, a filter and a card carrying who granted it and why, because the only alternative was writing a fake stripe subscription id into the database

* fix: hold a granted plan beside the paid one rather than over it so a Stripe workspace returns to the plan it pays for when the grant ends, route entitlement lookups through EffectivePlanID, separate a repository failure from an unknown plan, end a grant at local end of day, and drop an index that served no query
2026-09-12 09:45:31 -07:00
Matthew Meszaros d456bc48c6 feat: fix the Warmbly Cloud pool link across both roles (#262): take an enrolled mailbox out of this instance's own warmup pool so local partners stop writing to it and their unverifiable warmup stops landing in the owner's unibox, recognise the cloud's warmup mail whose verify header did not survive delivery through a new warmup-deliveries lookup that ignores consumed_at because instance and cloud read the same mailbox, move the managed-mailbox access token route behind NODE_BROKER_TOKEN so the internet-facing tracking and forms services can no longer mint a live provider token, scope pause and resume to the caller's workspace, keep an enrolled mailbox listed once it goes inactive, release the cloud copy when the local mirror row cannot be written, refuse the one-time handshake when CREDENTIALS_ENCRYPTION_KEY is missing, blank an expired code's plaintext instance token, and stop errx answering 200 for a status outside its table 2026-09-12 06:58:25 -07:00
Matthew Meszaros 47defafa09 feat: fix the six self-host defects reported in issue #439 (#456)
* feat: fix the six defects reported in issue #439 by mapping the IMAP UNAVAILABLE, INUSE and NONEXISTENT response codes to retry-level errors instead of a critical reconnect prompt, synthesising a stable no-msgid key so one message with no Message-ID header can no longer 400 the internal map endpoint and wedge every later sync pass with its cursors held, adding mailhtml.FromText and HasContent so an API or agent-created step with a plain body stops shipping the composer's empty div placeholder as its text/html part (derived on create and plain-only update, exposed as body_html on update_campaign_step, dropped at send and preview time, and refused at campaign start with empty_step_body), honouring sender_strategy='explicit' in ResolveCampaignSenderPool and ValidateCampaignReady so an emptied explicit pool parks the campaign instead of widening it to every mailbox in the workspace, making the paused_no_accounts auto-pause loud with an error log line, an error-level activity-feed entry and an org-scoped CAMPAIGN_PAUSED realtime pulse, gating the admin sign-in's Turnstile widget on GET /v1/auth/config so a self-host with CAPTCHA_PROVIDER=none is not locked out, and parsing NATS_URL down to its host:port so a credentialed bus URL no longer reports NATS down

* feat: act on the self-review of the issue #439 fixes by dropping the campaign wizard's own escapeHtml body_html builder, which entity-escaped the quotes in a conditional and made the template fail to parse at send time, and letting the backend's FromText render that part instead so wizard-written steps also get their bare URLs linked for click tracking, correcting the docs and openapi description that claimed an explicit sender pool never falls back when it still unions its tags as migration 000013 designed, extracting the duplicated blank-HTML-part guard into dropBlankHTMLPart shared by the send path and the preview, and recording why the no-msgid key keeps the folder name despite a RENAME changing it

* feat: address the CodeRabbit review on the issue #439 fixes by holding the admin sign-in's Turnstile widget unmounted until /v1/auth/config resolves so an instance with no route to Cloudflare cannot raise a widget error on a screen nobody submitted, failing StartCampaign closed when the sequence read errors rather than skipping both the malformed-template and empty-body refusals, giving TCPCheck the default port its protocol assumes so a portless NATS_URL is no longer reported down, leaving a URL that carries a merge field unanchored because the send path renders bodies with text/template and a quoted contact value would break out of the href, and correcting the sequences guide and the Campaign and CampaignUpdate openapi descriptions that named the wrong tag field
2026-09-12 03:13:38 -07:00
Matthew Meszaros 80c3c79a31 feat: add live coverage that the machine windows survive the settings document's jsonb round trip and reach the classifier, a guard that every shipped scanner CIDR is written as its own network address since the loader truncates host bits silently, and correct the comments that claimed an edit lands on the very next event when the consumer reads through a thirty second cache in its own process 2026-09-11 03:11:35 -07:00
Matthew Meszaros 184a3dc08e feat: make the automated-open and automated-click windows operator-editable under Instance settings and raise their defaults to 60s and 30s, because the ten-second window was anchored on dispatch to the worker rather than on delivery and routinely expired before the recipient-side gateway it was meant to catch had even seen the message, and add Barracuda's published Email Gateway Defense blocks to the scanner catalogue with Proofpoint, Mimecast and Cisco shipped commented out because browser isolation renders a clicked page from the vendor's own network 2026-09-11 02:53:34 -07:00
Matthew Meszaros a84ab48729 Merge branch 'main' into feature/posthog-error-tracking 2026-09-10 10:25:21 -07:00