Matthew Meszaros bea3f21da2 feat: replace the worker's flat mailbox-sync rate limiter, which fetched every message of every IMAP folder oldest first on first sight and then deactivated the mailbox after 100 new emails in five minutes with no resume state while Gmail and Outlook imported nothing historical at all, with a per-mailbox sync governor and a real backfill on every provider: three Redis fixed-window lanes shared across workers (priority for mail in a conversation the mailbox owns, resolved through the internal own-conversation endpoint and never held behind ordinary inbound mail; live for new mail after connect with burst, hourly, per-mailbox daily and per-organization daily budgets; backfill paced per minute and charged to the org budget) that defer over-budget mail with the provider cursor pinned before the first deferred message rather than dropping it, escalate to the existing EMAIL_RATE_LIMITED path only for a flood (SyncFloodPerHour new live messages seen in one hour, deduped through a lane cache so a held backlog is not re-counted every pass) or chronic overage (per-mailbox daily budget exhausted on three of the last seven days), cancel the mailbox context on termination so the sync goroutine no longer keeps ticking against a removed entry, and fail open on a Redis outage; an initial import newest first inside the policy window and cap with a resumable per-provider cursor relayed as SYNC_STATE, walking IMAP via UID SEARCH SINCE per eligible folder (skipping trash, drafts, junk and Gmail's All Mail) below a saved UID floor, Gmail via messages.list with a page token, and Graph via a filtered ordered folder listing over inbox, sent items and archive; IMAP live sync moved from CHANGEDSINCE window fetches to UID SEARCH MODSEQ plus envelope-only fetches with bodies read only for admitted messages, first sight of a folder now baselining its HIGHESTMODSEQ instead of walking it, and saved folder cursors seeded from the payload; Gmail history and Graph delta callbacks reshaped to offer ids first so dedupe and admission happen before hydration, with history checkpoints and delta links only advancing past fully stored records and pages, bounded pages per pass, Graph now tracking sent items live so a thread shows both sides, Graph priming walking uncapped and unpersisted to its deltaLink, and a Graph 404 on a just-deleted message treated as a skip; Gmail and Graph bodies capped at MaxEmailBodySize like IMAP already was; a provider 429 during sync backing the loop off instead of being relayed as a rate-limit event that deactivated the mailbox; the loop interval now adaptive with jitter and a five minute ceiling while held; a republished ADD_EMAIL applying a changed policy to an already loaded mailbox; and the dead StartImapWorker removed

2026-08-18 08:43:35 -07:00

..

envsample

New Repository: Add Backend Code

2026-01-17 14:11:14 +00:00

main.go

feat: replace the worker's flat mailbox-sync rate limiter, which fetched every message of every IMAP folder oldest first on first sight and then deactivated the mailbox after 100 new emails in five minutes with no resume state while Gmail and Outlook imported nothing historical at all, with a per-mailbox sync governor and a real backfill on every provider: three Redis fixed-window lanes shared across workers (priority for mail in a conversation the mailbox owns, resolved through the internal own-conversation endpoint and never held behind ordinary inbound mail; live for new mail after connect with burst, hourly, per-mailbox daily and per-organization daily budgets; backfill paced per minute and charged to the org budget) that defer over-budget mail with the provider cursor pinned before the first deferred message rather than dropping it, escalate to the existing EMAIL_RATE_LIMITED path only for a flood (SyncFloodPerHour new live messages seen in one hour, deduped through a lane cache so a held backlog is not re-counted every pass) or chronic overage (per-mailbox daily budget exhausted on three of the last seven days), cancel the mailbox context on termination so the sync goroutine no longer keeps ticking against a removed entry, and fail open on a Redis outage; an initial import newest first inside the policy window and cap with a resumable per-provider cursor relayed as SYNC_STATE, walking IMAP via UID SEARCH SINCE per eligible folder (skipping trash, drafts, junk and Gmail's All Mail) below a saved UID floor, Gmail via messages.list with a page token, and Graph via a filtered ordered folder listing over inbox, sent items and archive; IMAP live sync moved from CHANGEDSINCE window fetches to UID SEARCH MODSEQ plus envelope-only fetches with bodies read only for admitted messages, first sight of a folder now baselining its HIGHESTMODSEQ instead of walking it, and saved folder cursors seeded from the payload; Gmail history and Graph delta callbacks reshaped to offer ids first so dedupe and admission happen before hydration, with history checkpoints and delta links only advancing past fully stored records and pages, bounded pages per pass, Graph now tracking sent items live so a thread shows both sides, Graph priming walking uncapped and unpersisted to its deltaLink, and a Graph 404 on a just-deleted message treated as a skip; Gmail and Graph bodies capped at MaxEmailBodySize like IMAP already was; a provider 429 during sync backing the loop off instead of being relayed as a rate-limit event that deactivated the mailbox; the loop interval now adaptive with jitter and a five minute ceiling while held; a republished ADD_EMAIL applying a changed policy to an already loaded mailbox; and the dead StartImapWorker removed

2026-08-18 08:43:35 -07:00

uuid_test.go

worker: multi-IP support (one systemd unit per Primary IP)

2026-05-27 14:43:20 +00:00