mirror of
https://github.com/warmbly/warmbly.git
synced 2026-11-19 00:18:56 +00:00
153 lines
17 KiB
Plaintext
153 lines
17 KiB
Plaintext
---
|
|
title: Realtime mailbox sync
|
|
description: How IMAP IDLE and provider change notifications wake a mailbox's worker within seconds, how to turn on Gmail and Microsoft push, and what keeps polling as the safety net.
|
|
---
|
|
|
|
The worker that holds a mailbox syncs it on a timer. When the mailbox's provider can announce a change, the worker is woken and syncs within seconds instead. Polling never stops: it is the safety net for a notification that arrives late or not at all, because every provider documents that both can happen.
|
|
|
|
## What each mailbox gets
|
|
|
|
| Mailbox | Change signal | Poll while the signal holds | Default |
|
|
|---|---|---|---|
|
|
| IMAP and SMTP | `IDLE` on INBOX, on a second connection | every 3 minutes, for the other folders | on; `IMAP_IDLE=false` turns it off |
|
|
| Microsoft 365 and Outlook | Microsoft Graph change notifications | `SYNC_PUSH_FALLBACK_INTERVAL` (10 minutes) | off until `GRAPH_PUSH=true` and a public HTTPS backend URL |
|
|
| Gmail and Google Workspace | Gmail `users.watch` into Cloud Pub/Sub | `SYNC_PUSH_FALLBACK_INTERVAL` (10 minutes) | off until `GMAIL_PUSH_TOPIC` is set on the backend |
|
|
|
|
A mailbox without a live signal is polled every minute, as before. That covers an IMAP server without `IDLE`, a Gmail or Outlook mailbox on an instance that has not configured push, and any mailbox whose channel the provider refused.
|
|
|
|
A self-host therefore gets IMAP `IDLE` after an upgrade and nothing else changes. Gmail and Microsoft push are opt-in because each needs something only the operator can set up: a Google Cloud topic, or a backend Microsoft can reach. Every variable is listed in the [configuration reference](/development/configuration/#mailbox-sync).
|
|
|
|
Each variable is read by one service: `GRAPH_PUSH`, `GMAIL_PUSH_TOPIC` and `GMAIL_PUSH_SERVICE_ACCOUNT` by the backend, `GMAIL_PUSH_SUBSCRIPTION` and `GOOGLE_APPLICATION_CREDENTIALS_JSON` by the consumer, `IMAP_IDLE` and `SYNC_PUSH_FALLBACK_INTERVAL` by the workers. With the installer's Compose file every service reads `.env`. A joined worker node receives `IMAP_IDLE` and `SYNC_PUSH_FALLBACK_INTERVAL` from the control plane's own environment. A joined consumer is not handed the Gmail settings, because they carry a Google credential, so set those in its `/etc/warmbly/node.local.env` ([settings the control plane cannot know](/development/configuration/#settings-the-control-plane-cannot-know)).
|
|
|
|
## How a wake reaches the worker
|
|
|
|
1. The provider signals a change. Graph posts to the backend, Gmail publishes to Pub/Sub, which pushes to the backend or is pulled by the consumer, and an IMAP `IDLE` session runs inside the worker itself.
|
|
2. The backend or consumer finds the worker holding the mailbox and sends it a [`SYNC_WAKE`](/development/events/#worker-commands) command on that worker's `w.<worker-id>` topic. Repeat wakes for one mailbox within 2 seconds are dropped. That check uses Redis, and when Redis is unavailable every wake is sent.
|
|
3. The worker waits 2 seconds so one pass covers a burst of changes, then runs the ordinary sync pass: Gmail history, Graph delta or IMAP CONDSTORE.
|
|
4. New mail the pass stores reaches the dashboard through the consumer, which drains stored arrivals in batches of 32 and starts the next batch at once while full ones are waiting, rather than pausing between them.
|
|
|
|
A wake carries no mail. The pass reads the provider exactly as a timed pass does, under the same [sync budgets](/guides/mailboxes/#what-gets-synced), so an extra wake costs one pass and a lost one is caught by the poll. A wake never shortens a backoff: after a failed pass, or inside a retry window the provider asked for, the worker waits it out.
|
|
|
|
Only workers that report `sync_wake_protocol` 1 on their heartbeat are sent `SYNC_WAKE`, and only for active mailboxes on a worker that is heartbeating.
|
|
|
|
## IMAP IDLE
|
|
|
|
Every IMAP mailbox with sync turned on opens a second connection that does nothing but hold INBOX open read-only in `IDLE`. The sync connection is unchanged. The watcher wakes the sync when mail arrives in INBOX, is removed from it, or has its flags changed, and re-issues `IDLE` every 20 minutes, inside the 29 minutes RFC 2177 allows a server to wait.
|
|
|
|
- **Servers without `IDLE`** (no `IDLE` or `IMAP4rev2` capability) keep the one-minute poll. The watcher stops for that mailbox until it is loaded again.
|
|
- **Other folders** (sent, spam, archive and your own) are still found by polling, every 3 minutes while `IDLE` holds and every minute when it does not.
|
|
- **A dropped session** reconnects with backoff, from 30 seconds up to 30 minutes. A refused login waits 30 minutes, since the sync connection reports credential problems on its own.
|
|
- **Logins are paced.** Each mailbox's first `IDLE` login lands at a random point in the minute after it loads, and a worker opens at most about 20 per second, so restarting a worker does not sign every mailbox in at once.
|
|
|
|
Each IMAP mailbox now holds two connections to its server. A host that caps concurrent IMAP sessions per account counts both. If a host refuses the second one, the mailbox keeps syncing on the poll; set `IMAP_IDLE=false` on the workers to stop the attempts.
|
|
|
|
## Microsoft 365 and Outlook
|
|
|
|
With push on, the worker holding each Outlook mailbox opens a Microsoft Graph subscription on all of the mailbox's messages (created, updated and deleted) whatever folder they are in. Graph posts notifications and lifecycle events to:
|
|
|
|
```
|
|
<BACKEND_PUBLIC_URL>/api/v1/mailbox-push/graph/<mailbox-id>
|
|
```
|
|
|
|
To turn it on, set on the backend:
|
|
|
|
```bash
|
|
GRAPH_PUSH=true
|
|
BACKEND_PUBLIC_URL=https://api.example.com # or API_PUBLIC_URL when they are the same
|
|
```
|
|
|
|
Push stays off unless the resolved URL starts with `https://`. Microsoft must be able to reach it from the internet, so an instance served only on a private network or a tailnet cannot use Graph push; its Outlook mailboxes keep polling. No new Microsoft permission is needed: the `Mail.ReadWrite` the mailbox already granted covers subscriptions, for single-mailbox sign-in and [whole-organization grants](/development/whole-domain-connect/) alike.
|
|
|
|
How the channel behaves:
|
|
|
|
- Each notification is authenticated by a client state derived from `AUTH_SECRET` and the mailbox id. An item without the matching value is ignored.
|
|
- The endpoint answers `202` well inside Graph's 3-second limit. When the wake cannot be sent it answers `503`, and Graph retries for up to 4 hours.
|
|
- A subscription lasts just under 7 days and is renewed with 2 days left. A `reauthorizationRequired` or `subscriptionRemoved` lifecycle event renews or reopens it at once; a `missed` event triggers a sync pass.
|
|
- Changing `AUTH_SECRET` changes every client state. Workers reopen their subscriptions the next time the mailbox is reloaded, within about 30 minutes, and the 10-minute fallback poll covers the gap.
|
|
- A subscription Microsoft could not create is retried after 30 minutes, or later when Microsoft sends `Retry-After`. One it refuses outright (a `4xx` other than throttling) is retried after 6 hours. The mailbox polls every minute meanwhile, and a changed notification URL or client state is tried at once.
|
|
|
|
## Gmail and Google Workspace
|
|
|
|
With push on, the worker holding each Gmail mailbox calls `users.watch` daily (a watch lasts 7 days), and Gmail publishes a small notification to a Cloud Pub/Sub topic whenever the mailbox changes. The notification carries only the mailbox address and a history id, never mail. Mailboxes connected with an app password are IMAP mailboxes and use `IDLE` instead.
|
|
|
|
Pub/Sub delivers notifications one of two ways:
|
|
|
|
- **Push to the backend** (use this when the backend has a public HTTPS URL). Pub/Sub POSTs each notification to `<BACKEND_PUBLIC_URL>/api/v1/mailbox-push/gmail`. Every request carries a token Google signs for a service account you choose; the backend accepts it only when Google's signature checks out, the audience is that URL and the sender is `GMAIL_PUSH_SERVICE_ACCOUNT`. It answers 204 once the worker is woken and 503 when it cannot be, and Pub/Sub retries with backoff until it gets a 2xx.
|
|
- **Pull from the consumer** (for an instance Pub/Sub cannot reach). The consumer holds a streaming connection to a pull subscription and reads notifications from it.
|
|
|
|
Both end the same way: the mailbox's worker is sent `SYNC_WAKE` and runs its usual Gmail sync. Only one of the two should be set up for a topic. `GMAIL_PUSH_TOPIC` alone turns the watch on and moves Gmail mailboxes to the 10-minute poll, and the backend cannot tell whether anything reads the topic. Set it only once a push subscription with `GMAIL_PUSH_SERVICE_ACCOUNT` or a pull subscription with `GMAIL_PUSH_SUBSCRIPTION` is in place, or Gmail mail can take up to 10 minutes to appear.
|
|
|
|
Gmail names a mailbox only by its address, so a notification reaches a mailbox once its channel is stored. When the consumer stores a Gmail channel it wakes that mailbox once, so a change that landed while the watch was opening is not left for the poll.
|
|
|
|
Set up, once per instance, in the Google Cloud project that holds the Gmail OAuth client (`BOX_GOOGLE_CLIENT_ID`). Gmail only accepts a topic in the project making the watch call. Mailboxes connected with Google sign-in all watch it. Mailboxes on a [whole-domain grant](/development/whole-domain-connect/) watch it only when the delegation service account (`GOOGLE_WORKSPACE_DELEGATION_KEY`) belongs to the topic's project; otherwise they are never asked to and keep polling:
|
|
|
|
1. Under **APIs & Services > Library**, enable the **Cloud Pub/Sub API**.
|
|
2. Under **Pub/Sub > Topics**, create a topic, for example `gmail-push`. Its full name is `projects/<project>/topics/gmail-push`.
|
|
3. On the topic's **Permissions**, grant the **Pub/Sub Publisher** role to `gmail-api-push@system.gserviceaccount.com`, Gmail's push service account. An organization policy that restricts sharing to your own domain (**Domain restricted sharing**) refuses this grant; ask an Organization Policy Administrator for an exception on this project.
|
|
4. Create the subscription:
|
|
- **Push:** create a service account for signing, for example `gmail-push-signer`, and a **push** subscription on the topic with endpoint `<BACKEND_PUBLIC_URL>/api/v1/mailbox-push/gmail`, **Enable authentication** on, that service account, and the audience left empty (it defaults to the endpoint). The Pub/Sub service agent needs **Service Account Token Creator** on that account, which the console offers to grant.
|
|
- **Pull:** create a **pull** subscription, for example `warmbly-consumer`, and give the identity the consumer runs as the **Pub/Sub Subscriber** role on it. That is either a service account whose JSON key you pass as `GOOGLE_APPLICATION_CREDENTIALS_JSON`, or the Application Default Credentials of the machine the consumer runs on.
|
|
|
|
Then set, for push:
|
|
|
|
```bash
|
|
# backend: workers watch into this topic, and push requests must be signed by this account
|
|
GMAIL_PUSH_TOPIC=projects/<project>/topics/gmail-push
|
|
GMAIL_PUSH_SERVICE_ACCOUNT=gmail-push-signer@<project>.iam.gserviceaccount.com
|
|
```
|
|
|
|
or, for pull:
|
|
|
|
```bash
|
|
# backend
|
|
GMAIL_PUSH_TOPIC=projects/<project>/topics/gmail-push
|
|
|
|
# consumer: where it reads notifications, and as whom
|
|
GMAIL_PUSH_SUBSCRIPTION=projects/<project>/subscriptions/warmbly-consumer
|
|
GOOGLE_APPLICATION_CREDENTIALS_JSON='{"type":"service_account",...}' # optional with ADC
|
|
```
|
|
|
|
No new Google scope is needed: `users.watch` accepts the `gmail.modify` grant mailboxes already have. A notification names only the mailbox address, so every watched Gmail mailbox with that address is woken and the pass works out what changed.
|
|
|
|
Pub/Sub is billed by Google by volume (a list price of about $40 per TiB, with a 1 KB minimum per message). Push bills each delivery separately, so it costs somewhat more than pull at the same volume; both stay small next to the Gmail API work they replace. The Gmail API quota per message synced is unchanged, and most of the once-a-minute history polling goes away.
|
|
|
|
With push, a backend that is down or slow is retried by Pub/Sub, which keeps unacknowledged notifications for up to 7 days. With pull, a consumer that loses its Pub/Sub connection restarts the subscriber after 30 seconds.
|
|
|
|
## Turning push off
|
|
|
|
Unset `GRAPH_PUSH` or `GMAIL_PUSH_TOPIC` on the backend and restart it. Each worker closes the channel it holds for a mailbox the next time that mailbox is reloaded, within about 30 minutes, and returns it to the one-minute poll. `IMAP_IDLE=false` on a worker stops its `IDLE` sessions when it restarts.
|
|
|
|
## Upgrades and rollback
|
|
|
|
Migration `000351` adds `fleet_nodes.sync_wake_protocol` (default `0`) and the `mailbox_push_channels` table. The previous release ignores both.
|
|
|
|
- **Workers that have not upgraded** report protocol `0` and are never sent `SYNC_WAKE`. They ignore the push settings inside `ADD_EMAIL` and keep polling every minute.
|
|
- **Upgrade the backend and consumer before the workers**, as with any release. A worker reports each channel it opens, renews or closes as a `PUSH_STATE` event, which the consumer stores in `mailbox_push_channels`. The backend hands it back in every `ADD_EMAIL`, so a reloaded or moved mailbox renews the same channel instead of opening another. A consumer from before this release drops `PUSH_STATE`; a mailbox that moves meanwhile opens a fresh channel, and a subscription nobody tracks expires on its own within 7 days.
|
|
- **Rolling back** leaves open channels to expire on their own, within 7 days. Until then an older backend has no Graph route and Graph's posts fail, and Gmail notifications wait unread in the subscription. Mailboxes poll every minute either way.
|
|
|
|
`mailbox_push_channels` does not travel in a [workspace export](/guides/workspace-export-import/#what-does-not-travel): a channel points at the instance that opened it, and the destination's workers open their own.
|
|
|
|
## Troubleshooting
|
|
|
|
When a worker cannot open a mailbox's Gmail watch or Graph subscription, it reports the failure to error tracking (PostHog or Sentry, when configured) at most once a day per mailbox, tagged `component=mailbox_push`. The report names the provider call and its HTTP status, with the provider's own message, for example `gmail users.watch: HTTP 400 Invalid topicName ...` or `graph subscription create: HTTP 429`. A throttle there limits opening the channel, not sending. The mailbox keeps polling meanwhile, and the worker asks again after 30 minutes, after the provider's `Retry-After`, or after 6 hours when the provider refused the request outright.
|
|
|
|
| Symptom | Cause and fix |
|
|
|---|---|
|
|
| Outlook mail still takes about a minute | Check that `GRAPH_PUSH=true` and that `BACKEND_PUBLIC_URL` (or `API_PUBLIC_URL`) is an `https://` URL. Plain `http` leaves push off. From outside your network, `curl -X POST '<url>/api/v1/mailbox-push/graph/x?validationToken=ok'` must answer `ok`; Microsoft sends the same request when it creates a subscription |
|
|
| Graph subscriptions are never created | Microsoft cannot reach the URL, or the proxy in front of the backend does not forward `/api/v1/mailbox-push/`. Workers retry as described above and log each refusal at debug level |
|
|
| Gmail watch fails with `Invalid topicName` | The topic is in a different project from the OAuth client that issued the mailbox's grant, normally `BOX_GOOGLE_CLIENT_ID`. Create it in that project. A mailbox whose grant came from another Google project keeps polling. Whole-domain grant mailboxes are only asked to watch when the delegation key's `project_id` matches the topic's project |
|
|
| Gmail watch fails with a permission error | Gmail's push service account `gmail-api-push@system.gserviceaccount.com` lacks **Pub/Sub Publisher** on the topic, often because domain restricted sharing refused the grant |
|
|
| Pub/Sub shows push deliveries failing with `401` | The subscription signs with a different service account than `GMAIL_PUSH_SERVICE_ACCOUNT`, authentication is off, or its audience was set to something other than the endpoint URL |
|
|
| Pub/Sub shows push deliveries failing with `404` | `GMAIL_PUSH_SERVICE_ACCOUNT` is unset on the backend, or `BACKEND_PUBLIC_URL` is not `https` |
|
|
| Consumer logs `Gmail push subscriber stopped` | `GMAIL_PUSH_SUBSCRIPTION` is not `projects/<project>/subscriptions/<id>`, the subscription does not exist, or the consumer's identity lacks **Pub/Sub Subscriber** on it |
|
|
| An IMAP mailbox stays on the one-minute poll | Its server does not offer `IDLE`, refuses a second session for the account, or `IMAP_IDLE=false` is set on the worker |
|
|
| Mailboxes on one worker never react to push | The worker predates this release, so it reports `sync_wake_protocol` 0. Update it |
|
|
|
|
## Code references
|
|
|
|
- Settings and wake coalescing: `internal/config/mailbox_push.go`, `internal/app/mailpush/`
|
|
- Graph endpoint: `internal/api/handler/mailbox_push.go`
|
|
- Worker channels and `IDLE` watcher: `internal/app/worker/wmail/push.go`, `internal/client/smtpimap/imap/watch.go`
|
|
- Provider calls: `internal/client/msgraph/subscription.go`, `internal/client/goog/watch.go`
|
|
- Stored channels: `internal/repository/pg_mailbox_push.go`
|