windmill/docs/ai-session-backups.md

# AI session backups

AI sessions live in the browser: the session list (`windmill-sessions`), chat transcripts and
image blobs (`copilot-chat-history`) and artifacts (`copilot-artifacts`), all per-user IndexedDB
stores. This is the design of their backup in the workspace's object storage, and the
constraints future work on either side must keep.

Backend: `backend/windmill-api/src/ai_sessions.rs` (`/w/{w}/ai/sessions/{list,pull,push}`).
Frontend: `frontend/src/lib/components/sessions/sessionMirror*.ts`.

## Why it is lazy

A session changes at the local write rate: a transcript write every 2 s while streaming, a
session-record write per new message on screen. An object in S3 is replaced whole and every PUT
is billed, so the backup deliberately does not follow that rate. Local writes only mark a session
dirty (`sessionMirrorSignal.ts`, import-free so the stores never depend on the backup). A flush
runs 15 s after the marks go quiet, at most 2 min after the first unflushed mark, when the tab is
hidden, and 10 s after load for marks a crash left behind. Marks are persisted in localStorage
(shared by the user's tabs) for that reason, one key per mark: a shared blob would let two tabs
marking different sessions at once rewrite each other's mark away. A dirty mark is a counter
bumped on every write; a push retires it by recording the counter it covered on the session's
sync row rather than deleting the mark, since two localStorage calls cannot compare-and-delete
and a bump landing between them would be lost; retired marks are not reclaimed (one small key per
session ever backed up), and the marks of unsent drafts and of workspaces that are off stay too,
each costing one lookup per flush. Only a session gone from the store has its mark deleted. Losing the last
seconds of a device that never comes back is accepted; a tab that closes normally keeps its marks.
A signal names the user whose store the write landed in (read off the store's scoped name), so
a write that completes after the logged-in user changed marks that user's session, for their
next load, rather than the current user's.

A flush plans and sends one session at a time, filling requests of about 8 MB as it goes, so a
first backfill of a large history never holds more than one request's worth of records and
images in memory.

## What a push carries

The pure planner (`sessionMirrorPlan.ts`) compares each piece against the marker of what was
last pushed, kept per session in the `windmill-sessions-mirror` store:

| Piece | Object | Sent when |
|---|---|---|
| session record | `sessions/{sid}/head.json` | its signature changed |
| chat | `sessions/{sid}/chats/{cid}.json` | its `lastModified` moved |
| artifacts | `sessions/{sid}/artifacts.json` | their fingerprint changed |
| image | `images/{sid}/{cid}/{iid}` | never pushed before (write-once) |
| index marker | `index/{sid}/{epoch}` | last, by the part that completes a push of the session (empty; named by the record's move count) |

All under `windmill_ai_sessions/{w_id}/g{generation}/{sha256(email)}/` in the workspace's
primary storage (the generation is what a key rotation moves, see below).
The listing reads only `index/`: one object per session whatever the session holds, so a
session with many chats cannot crowd newer ones out of a bounded scan, and its
`last_modified` is the session's `updated_at`. Written last, and only by an entry no unsent
part follows (a session split over several entries says `partial` on all but the last, and
names the push on each, or the part is refused), it
lists a session only once a whole push landed; the parts of a session after a failed one are
not written either, on the server within one push and on the client across pushes, so the
marker on the last part never lists a session missing a chat, and a new session whose last part
never lands is not listed at all. A push of the session whole (no sync row, or a stale one,
`whole` on every part) opens with the head on the first: that part replaces the backup (the
marker goes first, then everything under the session), so what an old storage still held of
the session and the push does not carry is gone. An incremental part rides on a listed
session, and the server refuses it with `needs_whole`, writing nothing, when none is listed
(a removal deletes the marker first), rather than write a marker over a session missing what
earlier parts or earlier pushes carried; one that changes more than one object unlists the
session before its writes and lists it again after them, so a pull between two of the writes,
or after one failed, finds it absent rather than a mix of old and new pieces (one object
changing is one write, and stays listed). A push split over several parts, whole or
incremental, names itself on each with a token the browser draws (`push`, `opens` on the
first): the opening part unlists the session, so a pull between two parts finds it absent
rather than a mix of old and new pieces, the last part lists it again, and a later part is
written only while that token is the one there, so two devices pushing the session at once
cannot list a mix of their pieces (the push that opened later wins; the other is refused with
`needs_whole` and goes again), and one abandoned leaves the session unlisted, so the next
push of it goes whole. A push and a removal of one session
are serialized on the server by a Postgres advisory lock keyed on the session's prefix, so
the two never interleave object by object.

The head signature leaves out `name` (a per-browser counter the sessions page routes by),
the unsent-draft fields, `workspace_root_id` (recomputed on import), and the two fields reading
a session bumps (`lastSeenCount`, `lastActivityAt`). Reading a session must never cost a push;
keep that property when adding fields to `Session`.

Unsent drafts (no `workspace_id`) and attached files (Blobs, directory handles) are not backed up.

## Encryption and access

Every object is encrypted with a key derived from the workspace key and the user
(`build_crypt_with_key_suffix` with the email hash), because workspace storage credentials are
shared far more widely than a user's transcripts: `public_resource` storages and legacy-mode
READ/WRITE hand any member the bucket. The key is per user rather than per workspace so that a
member who copies another user's ciphertext under their own prefix gets nothing from `pull`; an
object that does not decrypt for its reader is treated as absent. Rotating the workspace key
(`set_encryption_key`) does not re-key the backups the way it re-encrypts the workspace's
secrets. The objects live under a prefix named by a generation
(`workspace_settings.ai_sessions_backup_generation`) that the rotation bumps in the
transaction committing the new key; once committed, the routes read and write under the new
generation's prefix, the answers name it (`backup_generation`, below; `storage_id` names the
storage and does not change), so every browser marks its sync rows stale and pushes its
sessions whole again there, and every older generation, which nothing writes to any more, is deleted off the
request at leisure (`windmill-api-workspaces/src/ai_session_backups.rs`). Sessions no browser
holds any more are lost. A generation is never reused, so no deletion, however late, can
touch live objects; a rotation that fails before its commit bumps nothing and deletes
nothing; two rotations racing serialize on the key row; the same key set again bumps
nothing. A rotation is rare, and the alternative,
rewriting every object in place while pushes, restarts, storage switches and further
rotations race the rewrite, is where the complexity would be; with this, nothing but the
current key ever reads an object. The
server builds every key from ids it validated
(`[A-Za-z0-9_-]{1,64}`) and the caller's own email; the client never names a key, and the
workspace storage permission rules are not consulted (the same stance as volumes). Only an
unscoped user token may reach the routes: a job token can carry an `on_behalf_of` identity and
every scoped token (guest, embed, app policy, MCP) was minted for something narrower.

The backup is keyed by the email like the browser's own stores are (`userScopedDb` scopes
IndexedDB by it): a user whose email changes starts from an empty history on both sides, and
the objects under the old hash stay in the bucket unread. Carrying them over would need a
server-side re-key (decrypt with the old suffix, encrypt with the new, move every object) in the
email-change flow, which this design leaves out.

An image is accepted only as a base64 data URL of at most 4 MB and stored verbatim, so it
serializes back into a pull answer at its stored size; anything JSON would escape could grow
several times and defeat the pull budget.

`push` carries `owner`, the email the browser prepared the batch for, and the server refuses a
mismatch with 409: an in-place account switch must not file one user's sessions under another's
prefix. The client captures its user at flush start and checks every store handle's name
against it for the same reason.

The feature is on wherever the workspace has primary storage, and off with
`ai_config.sessions_storage_disabled` (the `copilot_disabled` pattern: no migration, carried by
settings export and the CLI). A build without `parquet` has no routes (404), a workspace without
storage answers `enabled: false`; either turns the backup off for ten minutes, after which the
page asks again on its own (a flush for whatever is pending, and a restore), and the AI
settings page tells the mirror at once when the switch is saved there (the off state is
forgotten, the rows that went stale are marked again, a restore runs).

## Conflicts and deletion

Last write wins across devices. The head carries no manifest; `pull` lists the session's prefix
instead, so a stale device that renames or archives a session rewrites only the head and cannot
hide chats a newer device wrote. Two devices continuing the same chat still collide.

Restore brings back only sessions the browser does not have (`importSessions` is write-if-absent,
and skips ids the user deleted in this page) and never overwrites or deletes a local one from
remote state. It covers the workspace and its forks together, and only once every one of them
that keeps backups has listed (a listing that failed leaves the family for the next page load
or workspace switch, or the copy that did list could be the stale one): a session listed by
two of them (moved between them, the old copy not yet removed, since that mark is the moving
browser's, which may never come back) is brought back from the copy that moved last (`epoch`,
the record's move count, which names the marker), the storage's own modification time
deciding between two of the same count, and not from the other, which would otherwise take
the id first and keep the later copy out for good. A workspace's records land together once
its pulls are done, and just before they do the whole family is listed again (members whose
backups were off included, since a move from another device can land in a workspace between
the first listings and the pulls; a family of one, with nowhere else for a copy to show up, is
not): a session a later copy of which showed up elsewhere is left, with the family, for the
next time. Only a user-initiated `deleteSession` removes the backup; the next push from
another device that still has the session is refused with `needs_whole` (nothing of it is
written), its row goes stale without a backoff, and that device's next flush sends the session
whole; the workspace-lifecycle
removals (`reconcileSessionsLifecycle`, `deleteSessionsForWorkspace`) leave it, so a session
dropped by a wrong reconcile comes back on the next restore. Objects of deleted workspaces stay
in the bucket. A session moved to another workspace is pushed whole into the new one, and once
that push has landed the copy in the old one gets a removal mark of its own, naming the storages holding
that copy (the row that knew is the new workspace's by then), retried independently until
each of them has answered, even when the old workspace's backups are off at the time (they
may hold the copy still). Filing the removal only after the new copy is acknowledged keeps the
session backed up somewhere at every point.

A restore writes a session's artifacts and chats before its record, and records nothing for a
session whose pieces could not be written: recording it would let the next flush push the
half-empty local state over the backup.

Every answer names the storage it came from (`storage_id`, a hash of what locates the objects,
endpoint, region and bucket, not the credentials, which rotate) and the backup generation a
key rotation bumps (`backup_generation`). A sync row records both, and a row naming another
storage or generation goes stale and its session is marked again: a workspace pointed at a
new bucket, or whose key was rotated, holds nothing, and the server looks nowhere else, so
the next flush carries the session whole. A switch leaves the old copy where it was, so the
row rewritten under the new storage records the old one (`alsoIn`, one entry per storage
the workspace was on), and a removal is done only once every storage holding a copy answered
it, whatever the generation (a rotation deleted the older generation's copy anyway): each
answer narrows the row to the storages still holding one, and the mark waits for them to
answer, so a switch back never brings a deleted session back. That includes the rows a flush has just written, when a later answer of the
same flush names another storage or the session was pushed in part on top of a row from the
old one; a session whose own parts were answered from different storages is not settled at
all. The listing a restore starts with runs the same check, so a storage switch is noticed at
the first push after it or on the next page load, whichever comes first.

## Limits

Push bodies are packed to about 8 MB (UTF-8 bytes as sent), at most 100 entries, 200 removals and
4000 pieces each (the server's caps, with 32 MB on the body, and 100 chats, 500 images or 1000
deletes per entry, since every piece is an object-store call); an entry that
outgrows the target is split into chat-only parts (the artifacts and deletes on the last, the
head on the last too for an incremental push and on the first part, whatever it carries, for a
push of the session whole), and deletes
past the per-entry cap are carried over to the next push, which the session stays marked for. A chat above
16 MB or a session's artifacts above 8 MB are left out with a console warning; a chat that grew
past the cap after it was backed up has its copy deleted, so a restore never presents the old
transcript as the current one. A 413 fails only the sessions of that request. A
request the server refuses (any other 4xx but 404/403/409) stops the backup for the page but keeps
the marks and the sync state, so the next load tries again; a session the server reports it could
not store stays marked and is retried with backoff. A move files the old workspace's removal
mark before recording the new copy's row, so a mark that could not be written leaves the move
to be planned again. A workspace that answers `enabled: false`
marks its sync rows stale (the next push after storage returns carries every session whole,
since a new storage may be a new bucket) and leaves its dirty marks where they are (a move into
it must still remember the old copy); it keeps the removal marks of sessions that had been backed
up, so one deleted while backups are off does not come back once they are on, and drops the
removals of sessions never backed up from this browser, so a storage-less instance does not
collect one mark per deleted session forever. A user delete whose removal mark cannot be written
(localStorage full) is carried by the session's sync row instead (`removed`), which the flush
and the restore read like a mark; a session without a row yet (its first push may be in
flight) gets a row saying only that, and every row write keeps a removal filed meanwhile, so
the push's own row cannot erase it. Pull bodies are
capped at 64 KB. Pull answers up to 20 ids within a 32 MB
budget: a session's size is known from the listings before anything of it is read, one that
would not fit is deferred unless it is the first of the answer, in which case it comes in
pages: the answer carries what fits in key order (at least one object, so every page makes
progress) and names where the next picks up (`next`, a cursor the browser sends back as
`resume` with that session alone). Every page carries a fingerprint of the session's listing
(marker, keys, sizes, modification times, entity tags and versions, since a store reports
modification times coarsely and an object rewritten at the same size within that grain would
otherwise fingerprint the same) taken before anything of it is read, and the server
takes it again once the page is read: a page the backup moved under (a push landing object by
object) is read again, a few times, then answered as `moved`, and the browser starts the
session over on that or on two pages whose fingerprints differ. The browser writes each page's pieces as it arrives, over whatever
an earlier restore cut short had staged (the session is absent locally, so its pieces have no
local edits to keep, and the backup may have moved on), and the record, which is what makes
the session visible, only with the last page. A restore in progress keeps a staging row for
the session (the ids of every chat, image, artifact and version it wrote), which outlives it
if it is cut short; the next restore deletes the staged pieces the backup no longer has, by id
and never by clock, before the record lands (once the record is there no restore looks at the
session again, and a flush would push them back), and a prune that could not run leaves the
session, its pieces and its staging row for the restore after. A restore holds the user's tab
lock while it runs, so two tabs cannot each write the same absent session's pieces over the
other's, and runs only where Web Locks exist (a secure context: https, or localhost); on a plain
http origin the browser still backs up, and its sessions come back on a secure one. A
restore never writes an older record over a newer
one; between pages it holds nothing but the sync
row being assembled, whose chats also admit the images of a later page. Every page carries a
fingerprint of the session's listing (`listing`), taken before anything of the page is
listed or read, so an object landing after it is in the next page's; a session whose
fingerprint moved between two of its pages (a chat added by another device could sort before
the cursor and be missed) starts over, up to three times, then waits for the next restore. An object that grew
since the listing (a push replaced it) ends its page just before it and the answer names that
spot, so the next page sizes it anew rather than the session being imported without it. A pull sees every key of a session's listing but keeps the 5000 smallest
past its cursor (a page is defined by key order, and the store promises none), so a session
grown without bound by valid pushes cannot grow the answer's memory through its metadata
either; removing a prefix and a rotation's deletion stream their listings. `list` scans at most 50 000 index markers, keeps the newest 500 as it goes and answers with
them (`truncated` says when there were more); the restore takes 50 of them. Every read checks the object's size before buffering it: one larger than any push writes
(32 MB) is planted, whatever its listing said, and skipped, since whoever holds the bucket's
credentials can put anything at a predictable key; one larger than its listing said grew
since (a push replaced it) and ends its page, for the next pull to size anew. A dirty mark that cannot be written
(localStorage full) records its bump on the session's sync row instead (`extraV`, counted
with the mark's counter and kept by every row write, so a push in flight cannot retire it);
a session without a row yet (its first push in flight) keeps the bump in the page, and the
next row write takes it onto the row in the same transaction, so the row the push writes
cannot retire the mark with the bump unseen; every load's backfill marks again a session
without a row, with a stale one, or with one carrying bumps no push has covered.
A mark or removal for another user (a write that landed after a switch) reaches that user's
rows through a connection of its own, since the shared handle follows the current user. Nothing is read past the budget, whatever a session holds. A
restore takes the newest 50 sessions per workspace: every visible session gets a runtime, and
each runtime's history load reads the whole chat store. On CE the push checks the storage quota
and bumps usage by bytes written (an over-count on overwrites; the periodic recount settles it).