Commit Graph
165 Commits
Author SHA1 Message Date
Matthew Meszaros 59ec622253 feat: classify an IMAP sign-in refusal through the login throttle check before the missing-mailbox override, and redact the longest request URLs first so a URL that prefixes another leaves none of its query in a reported error 2026-11-08 09:56:54 +01:00
Matthew Meszaros 05d61c7635 feat: strip the query from every failed request in a reported error tree, including joined errors, unparseable URLs and quoted queries, and keep an IMAP sign-in refusal worded as a missing mailbox classified as an unknown IMAP error rather than a missing folder 2026-11-08 09:46:09 +01:00
Matthew Meszaros 3ef0b59b8c feat: classify an older Dovecot's codeless NO Mailbox doesn't exist as a missing folder instead of an unknown IMAP error, strip request query strings from reported errors so internal API failures group by endpoint in PostHog, and narrow the privacy page and dashboard PostHog notes to replays recorded only for sessions that hit an error 2026-11-08 09:34:06 +01:00
Matthew Meszaros b22530605a feat: refuse a Gmail push topic with an empty project or topic name for whole-domain grant mailboxes, warn at boot when GMAIL_PUSH_TOPIC is malformed, and assert the status in a refused watch 2026-11-08 08:03:54 +01:00
Matthew Meszaros 0d2db6abf0 feat: keep Google's reason in a refused Gmail watch, log at boot when whole-domain grant mailboxes cannot watch the push topic, and test malformed topic and key settings 2026-11-08 07:53:10 +01:00
Matthew Meszaros c9ac306788 feat: merge main into the NetEase masked STATUS branch, keeping main's EXAMINE fallback from #1191 in folders.go so this branch adds its masked STATUS regression tests 2026-11-07 18:30:53 +01:00
Matthew Meszaros 475b3b662b feat: assert the exact UIDVALIDITY, UIDNEXT, message count and HIGHESTMODSEQ an IMAP folder recovers from EXAMINE when its STATUS reply is unpaired, with and without CONDSTORE 2026-11-07 18:01:52 +01:00
Matthew Meszaros ec9ea297b6 feat: read an IMAP folder's cursors from EXAMINE when the server answers its STATUS under another name (NetEase masks non-ASCII folder names as "???"), so one such folder no longer fails every sync pass of the mailbox, taking mu only for that fallback so listings on servers that pair every STATUS are unchanged 2026-11-07 17:54:33 +01:00
Liu Mingyang b8b69834e0 test: cover masked STATUS replies for non-ASCII folders
This test file verifies the behavior of folder listings when the server responds with masked STATUS replies, particularly for non-ASCII folder names. It includes tests to ensure that folder listings remain consistent and do not trigger unnecessary commands.
2026-11-08 00:47:34 +08:00
Liu Mingyang e32ed235f6 fix: read folder cursors by EXAMINE when a STATUS reply cannot be paired (NetEase Enterprise Mail)
A STATUS reply naming the folder differently (NetEase masks non-ASCII names as "???") is dropped by go-imap, which left zero cursors and failed the whole listing as server unreachable. For non-ASCII folders only, read UIDVALIDITY/UIDNEXT with a read-only SELECT instead. Fixes #1189.
2026-11-08 00:45:49 +08:00
Matthew Meszaros 747b788e10 feat: classify provider sending refusals into typed mailbox pauses (rate_limited, daily_limit, sender_blocked, domain_auth) with escalating cooldowns that charge the lead no attempt, re-probe and owner Resume sending for SMTP holds, short pauses keeping bound leads waiting, auto-resume of paused_no_accounts campaigns, and a mailbox drawer pause notice with list badge (migration 000366) 2026-11-07 11:01:55 +01:00
Matthew Meszaros b8806a5ed5 feat: suppress a bounced address only on clear invalid-recipient evidence by letting the enhanced status beat weak recipient wording in dsn.Blame, adding block markers, quoting the sanitized SMTP server reply in send failures, reading the quoted reply from status-less failure notices, ignoring caller-supplied bounce causes, keeping unclassified bounces off the suppression list, and counting only address bounces as hard bounces in warmup and worker health 2026-11-07 11:01:54 +01:00
Matthew Meszaros a2b5db3dc5 feat: test that FindUIDInFolder returns a refused SELECT as an error while FindUIDByMessageID reads it as no match, against the in-memory Gmail-shaped IMAP server 2026-11-07 08:38:32 +01:00
Matthew Meszaros 4ede5e0914 feat: look up Gmail warmup messages with FindUIDInFolder, which reports a failed SELECT as an error so the rescue hands the action to the Gmail API instead of recording it as already out of spam 2026-11-07 08:27:47 +01:00
Matthew Meszaros 1db1ea0835 feat: request https://mail.google.com/ for new Google sign-in mailboxes (reauth still accepts gmail.modify) and run Gmail warmup engagement (spam rescue, mark read, star, mark important) over imap.gmail.com with the mailbox's own token, including delegated mailboxes through an optional mail.google.com delegation scope minted via ?purpose=imap on the internal token route, falling back per mailbox to the Gmail API when the scope or IMAP access is missing, with docs for scopes, warmup and existing delegation entries 2026-11-07 05:20:23 +01:00
Matthew Meszaros 6c73120bc6 feat: classify Gmail's [ALERT] Invalid credentials as the credentials refusal for the mailbox's auth kind and an IMAP TRYCREATE on a sync command as a missing folder, and stop reporting GOOGLE_AUTHENTICATION_FAILED and INVALID_CREDENTIALS from the worker to PostHog since the relay already deactivates the mailbox and tells its owner 2026-11-06 18:40:25 +01:00
Matthew Meszaros 0135dc8d7d feat: classify Zoho's wrong-region [ALERT] ("The server details you are using seems incorrect") as an INVALID_CREDENTIALS wrong-server error in the IMAP classifyAlert, keeping Zoho's alert text in the message and titling it Wrong mail server with an Update credentials action, so the mailbox is disconnected for its owner once instead of reporting IMAP_UNKNOWN to PostHog on every sync pass 2026-11-06 18:38:18 +01:00
Matthew Meszaros 74c51bfc82 feat: document account deletion refusals in their own error-code section, cover every non-renewing subscription state in the billing refusal test, and hide the address of a migration connection dropped mid-run 2026-11-06 17:15:31 +01:00
Matthew Meszaros f64a916fb9 feat: refuse account deletion while the account holds mailboxes, contacts, campaigns, templates, API keys, apps or subscriptions in workspaces it does not own or an owned workspace renews, lock owned workspaces before erasing them, and keep decoder text and migration connection details out of logs and error reports 2026-11-06 16:48:20 +01:00
Matthew Meszaros 4ed928e79b feat: name an undecodable Graph success body as msgraph.ErrUnreadableResponse and keep it out of worker error reports, so the pass retries without fabricating a server-unreachable event 2026-11-06 16:24:28 +01:00
Matthew Meszaros cdb25778c9 feat: classify a Graph success response whose body is not JSON as a retryable server-unreachable mail error in doJSON and FetchMessage instead of an unclassified decode error 2026-11-06 16:07:24 +01:00
Matthew Meszaros a7721887a8 feat: wake a Gmail mailbox once its push channel is stored, resend a PUSH_STATE the bus refused and report a Graph subscription that was deleted or gone even when its replacement fails, follow Graph subscription list continuations, drain a wake the timed pass covers, bound SYNC_WAKE publishes by the caller's deadline, keep the Google service account out of the compose worker, and correct the mailbox push docs on the Gmail publisher principal, what GMAIL_PUSH_TOPIC alone does, poll intervals and who calls the push routes 2026-11-04 16:32:27 +01:00
Matthew Meszaros 095d842aa9 feat: merge main into the mailbox push branch and renumber its migration to 000341_mailbox_push_channels after main took 000339 and 000340 2026-11-04 11:40:14 +01:00
Matthew Meszaros 1e4bdcd113 feat: mark an implicit-TLS handshake the server closes as a dropped handshake where the handshake runs, cover it through sendRaw, and order the reserved-start cooldown test through a replaceable timer 2026-11-03 17:39:17 +01:00
Matthew Meszaros 14d11f9a31 feat: treat an implicit-TLS handshake the server closes as server pacing, exempt trailing-dot loopback relays from pacing, and make the reserved-start cooldown test wait for its reservation 2026-11-03 17:31:01 +01:00
Matthew Meszaros d1d43111d0 feat: key SMTP pacing on the normalized server host and worker address, skip loopback relays, hold reserved starts that land in a later cooldown, measure the spaced window from the cooldown's end, classify dropped STARTTLS by transport error, and cover a long server-pacing retry time live 2026-11-03 17:18:48 +01:00
Matthew Meszaros 03cd541c62 feat: pace SMTP sessions per server and worker address after a 421, a 454 sign-in refusal or a dropped STARTTLS, hold due sends without contacting the server, and walk those sends back without spending a lead attempt or raising a mailbox error 2026-11-03 16:59:57 +01:00
Matthew Meszaros d241a836f8 feat: sync mailboxes in realtime through Graph change notifications, Gmail Pub/Sub watches and IMAP IDLE that wake the owning worker with SYNC_WAKE, drain stored arrivals continuously, and add an opt-in autoscaling AWS worker fleet without public IPv4 scaled on a published fleet utilization metric 2026-11-03 15:11:24 +01:00
Matthew Meszaros bcef5d160f feat: enforce hosted SMTP authentication policy across mailbox negotiation and platform relays while preserving self-host compatibility 2026-10-28 09:49:27 +01:00
Matthew Meszaros 6df80cfc5f feat: verify the platform SMTP relay certificate on hosted production whatever SMTP_TLS_INSECURE_SKIP_VERIFY says, send the SMTP XOAUTH2 bearer token only over TLS or to a loopback host, and report an ignored relay knob in instance health 2026-10-26 06:25:13 +01:00
Matthew MeszarosandDevin AI 79baf46462 feat: keep SMTP mailboxes executable through retryable IMAP startup failures and enforce provider login cooldowns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-25 07:44:19 +00:00
Matthew Meszaros 3532039037 feat: explain each refused mailbox connection in plain words with what to do (a host on a private network address, sign-in offered only unencrypted, no STARTTLS, no supported AUTH mechanism, an unverified or unpinned certificate, a send whose content could not be decrypted) in the connection test, stored mailbox errors and import causes, add an optional reason to API error responses with certificate_untrusted on mailbox_tls_failed for the dashboard's trust option, and treat tls_unverified and tls_pin as post-fixture fields in the Avro envelope test 2026-10-23 19:52:54 +02:00
Matthew Meszaros 03fd30f9bc feat: merge decoded-byte image validation with download-safe serving, own-domain form and domain redirects, TLS-only SMTP AUTH negotiation and pinned per-server certificate opt-outs, with MAIL_TLS_INSECURE honoured on self-hosted and development instances and resolved by the control plane for joining nodes 2026-10-23 16:20:30 +02:00
Matthew Meszaros 573e79ce8e feat: store each SMTP/IMAP server's certificate opt-out and pinned SPKI SHA-256 on email_accounts_smtp_imap (migration 000326, expand-only), carry tls_unverified and tls_pin in ADD_EMAIL, Cloud enrolment and vendor reconnect, keep an opt-out only for a certificate that fails CA verification and pin it to what the connection test saw, keep it across a credential change for the same host, name the presented certificate in a TLS refusal, and add GET /emails/:id/certificates plus session-only check and confirm endpoints that pin the fingerprints the owner confirmed 2026-10-23 16:15:55 +02:00
Matthew Meszaros 4624cc4a74 feat: negotiate mailbox SMTP AUTH by link (LOGIN then PLAIN only over TLS or loopback, CRAM-MD5 last, never a password mechanism in cleartext) for the send path, the connection test and the platform relay, honour MAIL_TLS_INSECURE only under APP_ENV=dev, and trust a mailbox server whose owner opted out of CA verification only on its pinned SPKI SHA-256 via VerifyConnection, reporting TLS_CERTIFICATE_CHANGED or TLS_CERTIFICATE_UNCONFIRMED as retryable connection errors 2026-10-23 16:15:55 +02:00
Matthew Meszaros d7bacf6112 feat: check every mailbox SMTP and IMAP dial against its resolved peer address in netbind, refusing loopback, private, link-local, CGNAT, multicast and metadata addresses on hosted production and wherever MAIL_ALLOW_PRIVATE_HOSTS=false, render that resolved policy into each joining node's env, and honour WARMBLY_ALLOW_UNSAFE_WEBHOOK_URLS only on self-hosted or development instances 2026-10-23 16:02:52 +02:00
Matthew Meszaros 2f42afa78f feat: report each warmup spam rescue's outcome from the recipient worker into placement history and the mailbox placement tab, rescue only mail that arrived in spam, and follow bounded Graph continuations when resolving warmup messages by Message-ID 2026-10-22 19:11:28 +02:00
Devin AI a2e81c2ef5 feat: keep Graph complaint reports retryable across temporary raw message failures and bound MIME depth, parts and decoded content before admitting an arrival 2026-10-19 06:19:42 +00:00
Matthew Meszaros 46bd956229 feat: give worker SEND_EMAIL commands a 120s deadline that covers the SMTP client's 90s budget, run the worker command subscription under a 195s bus budget (NATS ack wait and Kafka handler context via eventbus.WithHandlerTimeout), and add ordered per-mailbox command lanes on NATS with in-progress heartbeats 2026-10-16 06:57:14 +02:00
Matthew Meszaros 4ee78aebb2 feat: let a verified warmup arrival inside the arrival grace clear a recipient's unreachable result, and refuse an IMAP bare-envelope body fetch whose BODY[] section is missing 2026-10-15 19:30:27 +02:00
Matthew Meszaros b5891f3365 feat: keep IMAP sync running past RFC 6533 message/global bounces by stepping unreadable envelope fetches down to bare, single and skipped rungs with locally parsed MIME, and read global delivery-status parts as bounce reports on IMAP and Gmail 2026-10-15 19:08:54 +02:00
Matthew Meszaros 853318692a feat: retain privacy-safe IMAP connection evidence only from current mailbox generation and publish persisted warning text 2026-10-11 16:55:13 +00:00
devin-ai-integration[bot] 0e3ed53ede Merge pull request #934 from warmbly/devin/1791643401-graph-delete-lookup-authority
feat: preserve Graph mail and local bodies when warmup deletion lookup is uncertain
2026-10-10 15:30:30 +00:00
Devin AI 8641905138 feat: require complete Graph JSON response before confirming warmup message absence 2026-10-10 15:22:56 +00:00
Devin AI 2e4dfa4d87 feat: require complete terminal Graph message lookups before warmup retention cleanup and reject malformed identities or continuation metadata with bounded native retryable uncertainty 2026-10-10 15:10:05 +00:00
Devin AI 44cd8f8254 feat: preserve complete IMAP folder presence across bounded sync and stop Graph passes on authoritative provider retry deadlines 2026-10-10 14:42:21 +00:00
Matthew Meszaros 89e456dfb5 feat: preserve IMAP folder sync state on incomplete provider status and honor Graph service retry windows 2026-10-10 13:58:24 +00:00
Matthew Meszaros c85b2bcb23 Merge pull request #909 from warmbly/devin/1791603696-post-release-sync-errors
feat: recover partial Outlook ID upgrades and stop reporting arrival waits as exceptions
2026-10-10 07:23:06 +00:00
q 2c6c379feb feat: step IMAP change queries down to the next form when a server refuses one against a folder that stayed selected, retrying in the same pass, while transient NO answers stay retryable, with refusal and transient dialect tests 2026-10-10 08:45:30 +02:00
q 695ac14b24 feat: read IMAP CONDSTORE changes with UID FETCH CHANGEDSINCE and step down through ESEARCH, plain SEARCH and UIDNEXT when a server's reply cannot be parsed, log unreadable replies with the host, test a per-server dialect table, and document mail server dialects in AGENTS.md and the mailboxes guide 2026-10-10 08:43:16 +02:00