windmill/docs/ai-session-backups.md

# AI session backups

AI sessions live in the browser: the session list (`windmill-sessions`), chat transcripts and
image blobs (`copilot-chat-history`) and artifacts (`copilot-artifacts`), all per-user IndexedDB
stores. This is the design of their backup in the workspace's object storage, and the
constraints future work on either side must keep.

Backend: `backend/windmill-api/src/ai_sessions.rs` (`/w/{w}/ai/sessions/{list,pull,push}`).
Frontend: `frontend/src/lib/components/sessions/sessionMirror*.ts`.

## Why it is lazy

A session changes at the local write rate: a transcript write every 2 s while streaming, a
session-record write per new message on screen. An object in S3 is replaced whole and every PUT
is billed, so the backup deliberately does not follow that rate. Local writes only mark a session
dirty (`sessionMirrorSignal.ts`, import-free so the stores never depend on the backup). A flush
runs 15 s after the marks go quiet, at most 2 min after the first unflushed mark, when the tab is
hidden, and 10 s after load for marks a crash left behind. Marks are persisted in localStorage
(shared by the user's tabs) for that reason, one key per mark: a shared blob would let two tabs
marking different sessions at once rewrite each other's mark away. A dirty mark is a counter
bumped on every write; a push retires it by recording the counter it covered on the session's
sync row rather than deleting the mark, since two localStorage calls cannot compare-and-delete
and a bump landing between them would be lost; retired marks are not reclaimed (one small key per
session ever backed up), and the marks of unsent drafts and of workspaces that are off stay too,
each costing one lookup per flush. Only a session gone from the store has its mark deleted. Losing the last
seconds of a device that never comes back is accepted; a tab that closes normally keeps its marks.
A signal names the user whose store the write landed in (read off the store's scoped name), so
a write that completes after the logged-in user changed marks that user's session, for their
next load, rather than the current user's.

A flush plans and sends one session at a time, filling requests of about 8 MB as it goes, so a
first backfill of a large history never holds more than one request's worth of records and
images in memory.

## What a push carries

The pure planner (`sessionMirrorPlan.ts`) compares each piece against the marker of what was
last pushed, kept per session in the `windmill-sessions-mirror` store:

| Piece | Object | Sent when |
|---|---|---|
| session record | `sessions/{sid}/head.json` | its signature changed |
| chat | `sessions/{sid}/chats/{cid}.json` | its `lastModified` moved |
| artifacts | `sessions/{sid}/artifacts.json` | their fingerprint changed |
| image | `images/{sid}/{cid}/{iid}` | never pushed before (write-once) |
| index marker | `index/{sid}/{epoch}` | last, by the part that completes a push of the session (empty; named by the record's move count) |

All under `windmill_ai_sessions/{w_id}/g{generation}/{sha256(email)}/` in the workspace's
primary storage (the generation is what a key rotation moves, see below).
The listing reads only `index/`: one object per session whatever the session holds, so a
session with many chats cannot crowd newer ones out of a bounded scan, and its
`last_modified` is the session's `updated_at`. Written last, and only by an entry no unsent
part follows (a session split over several entries says `partial` on all but the last, and
names the push on each, or the part is refused), it
lists a session only once a whole push landed; the parts of a session after a failed one are
not written either, on the server within one push and on the client across pushes, so the
marker on the last part never lists a session missing a chat, and a new session whose last part
never lands is not listed at all. A push of the session whole (no sync row, or a stale one,
`whole` on every part) opens with the head on the first: that part replaces the backup (the
marker goes first, then everything under the session), so what an old storage still held of
the session and the push does not carry is gone. An incremental part rides on a listed
session, and the server refuses it with `needs_whole`, writing nothing, when none is listed
(a removal deletes the marker first), rather than write a marker over a session missing what
earlier parts or earlier pushes carried; one that changes more than one object unlists the
session before its writes and lists it again after them, so a pull between two of the writes,
or after one failed, finds it absent rather than a mix of old and new pieces (one object
changing is one write, and stays listed). A push split over several parts, whole or
incremental, names itself on each with a token the browser draws (`push`, `opens` on the
first): the opening part unlists the session, so a pull between two parts finds it absent
rather than a mix of old and new pieces, the last part lists it again, and a later part is
written only while that token is the one there, so two devices pushing the session at once
cannot list a mix of their pieces (the push that opened later wins; the other is refused with
`needs_whole` and goes again), and one abandoned leaves the session unlisted, so the next
push of it goes whole. A push and a removal of one session
are serialized on the server by a Postgres advisory lock keyed on the session's prefix, so
the two never interleave object by object.

The head signature leaves out `name` (a per-browser counter the sessions page routes by),
the unsent-draft fields, `workspace_root_id` (recomputed on import), and the two fields reading
a session bumps (`lastSeenCount`, `lastActivityAt`). Reading a session must never cost a push;
keep that property when adding fields to `Session`.

Unsent drafts (no `workspace_id`) and attached files (Blobs, directory handles) are not backed up.

## Encryption and access

Every object is encrypted with a key derived from the workspace key and the user
(`build_crypt_with_key_suffix` with the email hash), because workspace storage credentials are
shared far more widely than a user's transcripts: `public_resource` storages and legacy-mode
READ/WRITE hand any member the bucket. The key is per user rather than per workspace so that a
member who copies another user's ciphertext under their own prefix gets nothing from `pull`; an
object that does not decrypt for its reader is treated as absent. Rotating the workspace key
(`set_encryption_key`) does not re-key the backups the way it re-encrypts the workspace's
secrets. The objects live under a prefix named by a generation
(`workspace_settings.ai_sessions_backup_generation`) that the rotation bumps in the
transaction committing the new key; once committed, the routes read and write under the new
generation's prefix, the answers name it (`backup_generation`, below; `storage_id` names the
storage and does not change), so every browser marks its sync rows stale and pushes its
sessions whole again there, and every older generation, which nothing writes to any more, is deleted off the
request at leisure (`windmill-api-workspaces/src/ai_session_backups.rs`). Sessions no browser
holds any more are lost. A generation is never reused, so no deletion, however late, can
touch live objects; a rotation that fails before its commit bumps nothing and deletes
nothing; two rotations racing serialize on the key row; the same key set again bumps
nothing. A rotation is rare, and the alternative,
rewriting every object in place while pushes, restarts, storage switches and further
rotations race the rewrite, is where the complexity would be; with this, nothing but the
current key ever reads an object. The
server builds every key from ids it validated
(`[A-Za-z0-9_-]{1,64}`) and the caller's own email; the client never names a key, and the
workspace storage permission rules are not consulted (the same stance as volumes). Only an
unscoped user token may reach the routes: a job token can carry an `on_behalf_of` identity and
every scoped token (guest, embed, app policy, MCP) was minted for something narrower.

The backup is keyed by the email like the browser's own stores are (`userScopedDb` scopes
IndexedDB by it): a user whose email changes starts from an empty history on both sides, and
the objects under the old hash stay in the bucket unread. Carrying them over would need a
server-side re-key (decrypt with the old suffix, encrypt with the new, move every object) in the
email-change flow, which this design leaves out.

An image is accepted only as a base64 data URL of at most 4 MB and stored verbatim, so it
serializes back into a pull answer at its stored size; anything JSON would escape could grow
several times and defeat the pull budget.

`push` carries `owner`, the email the browser prepared the batch for, and the server refuses a
mismatch with 409: an in-place account switch must not file one user's sessions under another's
prefix. The client captures its user at flush start and checks every store handle's name
against it for the same reason.

The feature is on wherever the workspace has primary storage, and off with
`ai_config.sessions_storage_disabled` (the `copilot_disabled` pattern: no migration, carried by
settings export and the CLI). A build without `parquet` has no routes (404), a workspace without
storage and nothing to stand in for it answers `enabled: false`; either turns the backup off
for ten minutes, after which the
page asks again on its own (a flush for whatever is pending, and a restore), and the AI
settings page tells the mirror at once when the switch is saved there (the off state is
forgotten, the rows that went stale are marked again, a restore runs).

## The instance store standing in

A workspace without storage of its own keeps its backups in the instance object store
(`object_store_cache_config`, loaded the way every other use of it is, so never with
`DISABLE_S3_STORE`; the plan is checked on every request and Pro never falls back, since a
store loaded before a switch to Pro stays loaded), under the same layout and the same
per-user key,
while the instance setting `ai_sessions_instance_storage_fallback` allows it (on unless set
to false; the instance settings page shows it under Object Storage). A build without
`private` has neither workspace storage nor the quota below, and never falls back. Every
answer says which kind of store it came from (`fallback`), and the instance store is named
(`storage_id`) by what locates its objects, the endpoint, region and bucket its settings
resolve to, in a namespace of its own: moving the instance store to another endpoint under
the same bucket name is a storage switch for the browsers, and a workspace bucket is never
taken for it. The location is kept with the loaded store, so a server whose reload is still
pending names the store it writes to. A route decides between the workspace's storage and
the instance store from the row it reads the generation from, so a push lands in the
instance store only under a generation read while the workspace had no storage. The store a
workspace's backups live in is resolved in one place
(`ai_session_backups::workspace_store`), for the routes and for the rotation's deletion of
older generations alike.

Configuring a storage for a workspace that had none (`edit_large_file_storage_config`) bumps
the backup generation in the transaction that sets it, so everything the workspace left in
any instance store sits under a generation the routes never read again: a later return to
the instance store, whichever it is by then, starts from a newer one. Every storage settings
change then deletes from the instance store, off the request, the workspace's generations
older than the one it committed, whether the setting is on or off (copies from when it was
on may be there). Nothing live is older, whatever happens next: a deletion that is slow, cut
short, or overtaken by the storage being dropped or pointed at the instance store's own
bucket touches only generations nothing reads. Dropping the storage bumps nothing and is a
switch like any other: the rows go stale, the sessions are pushed whole into the instance
store, and the old bucket keeps its copy. On the browser side a removal owed to an instance
store (the row names it apart, `storageName` in `sessionMirrorPlan.ts`) is retired by any
answer from the workspace's own storage, since that storage being there means the
generation moved past the copy; one owed to a workspace storage still waits for that
storage, whatever the instance store answered. Copies a deletion missed stay in the
operator's bucket unread, as a deleted workspace's copies do.

On CE the bytes in the instance store count toward the workspace's storage quota under a
storage name of their own (`_ai_sessions_fallback_`, listed by the periodic recount while
the workspace has no storage of its own, and left out when there are none), so a member
cannot fill the operator's bucket past what the workspace may use; on EE, where workspace
storage has no quota either, nothing bounds them but the per-push caps and the instance
setting.

## Conflicts and deletion

Last write wins across devices. The head carries no manifest; `pull` lists the session's prefix
instead, so a stale device that renames or archives a session rewrites only the head and cannot
hide chats a newer device wrote. Two devices continuing the same chat still collide.

Restore brings back only sessions the browser does not have (`importSessions` is write-if-absent,
and skips ids the user deleted in this page) and never overwrites or deletes a local one from
remote state. It covers the workspace and its forks together, and only once every one of them
that keeps backups has listed (a listing that failed leaves the family for the next page load
or workspace switch, or the copy that did list could be the stale one): a session listed by
two of them (moved between them, the old copy not yet removed, since that mark is the moving
browser's, which may never come back) is brought back from the copy that moved last (`epoch`,
the record's move count, which names the marker), the storage's own modification time
deciding between two of the same count, and not from the other, which would otherwise take
the id first and keep the later copy out for good. A workspace's records land together once
its pulls are done, and just before they do the whole family is listed again (members whose
backups were off included, since a move from another device can land in a workspace between
the first listings and the pulls; a family of one, with nowhere else for a copy to show up, is
not): a session a later copy of which showed up elsewhere is left, with the family, for the
next time. Only a user-initiated `deleteSession`, and the retention sweep below, remove the
backup; the next push from
another device that still has the session is refused with `needs_whole` (nothing of it is
written), its row goes stale without a backoff, and that device's next flush sends the session
whole; the workspace-lifecycle
removals (`reconcileSessionsLifecycle`, `deleteSessionsForWorkspace`) leave it, so a session
dropped by a wrong reconcile comes back on the next restore. Objects of deleted workspaces stay
in the bucket. A session moved to another workspace is pushed whole into the new one, and once
that push has landed the copy in the old one gets a removal mark of its own, naming the storages holding
that copy (the row that knew is the new workspace's by then), retried independently until
each of them has answered, even when the old workspace's backups are off at the time (they
may hold the copy still). Filing the removal only after the new copy is acknowledged keeps the
session backed up somewhere at every point.

A restore writes a session's artifacts and chats before its record, and records nothing for a
session whose pieces could not be written: recording it would let the next flush push the
half-empty local state over the backup.

Every answer names the storage it came from (`storage_id`, a hash of what locates the objects,
endpoint, region and bucket, not the credentials, which rotate; the instance store standing
in for a workspace without one is named apart, see above) and the backup generation a
key rotation bumps (`backup_generation`). A sync row records both, and a row naming another
storage or generation goes stale and its session is marked again: a workspace pointed at a
new bucket, or whose key was rotated, holds nothing, and the server looks nowhere else, so
the next flush carries the session whole. A switch leaves the old copy where it was, so the
row rewritten under the new storage records the old one (`alsoIn`, one entry per storage
the workspace was on), and a removal is done only once every storage holding a copy answered
it, whatever the generation (a rotation deleted the older generation's copy anyway): each
answer narrows the row to the storages still holding one, and the mark waits for them to
answer, so a switch back never brings a deleted session back. That includes the rows a flush has just written, when a later answer of the
same flush names another storage or the session was pushed in part on top of a row from the
old one; a session whose own parts were answered from different storages is not settled at
all. The listing a restore starts with runs the same check, so a storage switch is noticed at
the first push after it or on the next page load, whichever comes first.

## Retention

`ai_config.sessions_retention_days` (per workspace, in the AI settings; unset by default;
the `sessions_storage_disabled` pattern: no migration, carried by settings export and the
CLI; 1 to 3650) puts an age on sessions, counted from their last activity. Each side applies
it with its own clock against its own timestamps, so no clock is compared with another
machine's, and the two do not time the same event: the server counts the last push that
completed, a browser its last local activity, which includes reading new messages and is not
pushed. A backup swept while a browser still reads its copy comes back once that browser
writes to the session again (its incremental push is refused and goes whole):

- The server sweeps the object store (`sweep_expired_ai_session_backups`, from the monitor
  about every 40 minutes on each server, one pass at a time under a session-level advisory
  lock). For every workspace with a retention it takes the store its backups live in, its
  own storage or the instance store standing in, decided from the row it reads the
  generation from as the routes do, names the users under the generation prefix
  (`list_with_delimiter`) and lists each user's `index/` once: one object
  per session, nothing of what the sessions hold. A session whose marker is older than the
  retention is removed under its lock (`lock_session`), once its markers, listed again
  there, are still all older: a push that renewed the session between the walk and the lock
  keeps it, and one split over parts either holds the lock or has the session unlisted with
  its token next to the markers (`index/{sid}/push`), which the sweep leaves alone while the
  token is younger than the retention: an older one is a push a browser abandoned, whose
  landed parts nothing lists, and it goes the same way. Before deleting anything the sweep
  writes a record next to
  the markers (`index/{sid}/sweep`, not an epoch, so neither `list` nor `pull` counts it),
  and `remove_session` deletes it last: a removal cut short, its markers already gone, is
  found by the next pass and finished, unless a push listed the session again first. At
  most 1000 sessions per workspace and pass; the rest wait for the next. `list` leaves an
  expired marker out of its answer meanwhile, so a browser never restores a session the
  sweep has not reached. The marker's modification time is the storage's clock and the
  cutoff the server's. The sweep reaches only the backups the routes would: a deleted
  workspace's stay in its storage, and so do those a workspace keeps in the instance store
  once `ai_sessions_instance_storage_fallback` is set to false.
- The browser sweeps its own stores when a tab resolves the logged-in user
  (`sweepExpiredSessions`, from the one `onUserChange` in `sessionState.svelte.ts`), before
  that tab reads a single session. A session whose last activity is older than the retention
  by the browser's clock is deleted locally, record, chats, images, attached files and
  artifacts. A restored session carries the backup's time as its last activity, the storage's
  clock, so it counts from the later of that and the moment it was restored here
  (`restoredAt`): a browser clock ahead of the storage's never deletes a session it just
  brought back. Archived sessions count like any other, and persisted unsent drafts by their
  pending workspace.

  The stores are shared by the user's tabs, and each keeps copies of the sessions in memory,
  so every tab holds a shared Web Lock from before it reads them until it stops using them,
  and the sweep deletes only while holding that lock exclusively, requested if available:
  granted exactly when no tab of the user has the sessions loaded, which is why the sweep
  runs where it does and nowhere else. Nothing holds a copy of what it deletes and nothing
  writes the stores meanwhile, so it deletes one record at a time and without re-reading. It
  also takes the tab lock the flush and the restore take, again only if available, so neither
  plans nor stages a session half deleted; like the restore, it does not run where Web Locks
  do not exist. With several tabs open nothing is swept, until one of them reloads alone.

  The hold is only as good as the tabs that take it, so a tab still running a build from before
  it has the sessions loaded and holds nothing. A tab loaded after that one, across a deploy,
  can sweep a session the older tab has in memory, and a write there afterwards brings the
  record back without its chats, which the next flush pushes. It needs a tab left open across a
  deploy, a session untouched for the whole retention, and the user going back to that session
  in the older tab; the next sweep deletes it again. The same window is open to the
  workspace-lifecycle delete in `reconcileSessionsLifecycle`, which no lock guards at all.

  What deletes is the retention the server gives as the sweep runs, asked for under both locks
  (`POST /workspaces/session_workspace_retention`, its own route rather than a field on the
  lifecycle status, whose answer a tab loaded before this version still reads). Never a
  remembered one: a retention raised or cleared since would otherwise delete a session that is
  now within it, and a persisted unsent draft has no backup to come back from. What the sweep
  keeps in localStorage decides only whether to ask again — it asks when it has asked nothing
  yet, when the answer it has is a day old, or when that answer marks a session expired — so
  an ordinary load costs no request at all. An answer that does not arrive within five seconds
  leaves the sessions for the next load rather than delete on what this browser guessed. That
  route answers for a workspace the caller can be authed into, unlike the status: a status is
  what to do with the caller's own sessions, a setting is the workspace's to tell, so a
  disabled membership is told nothing though its sessions still reconcile.

  Each session's record goes before its pieces, so nothing plans a push for it afterwards,
  and a localStorage key written before the record and removed once every piece is gone makes
  a later sweep finish a deletion that failed, unless a restore brought the session back
  since. The record is deleted without the tombstone a user delete leaves, which is what lets
  a restore bring it back. The session's dirty mark and sync row go with it (`sessionSwept`),
  unless the row still carries a removal or a restore's staging. Nothing is sent to the storage: the local
  copy's age says nothing about another device's, which may have pushed the session since,
  and the server applies the rule to the backup on its own. A session swept here that the
  storage still lists comes back on the next restore.

## Limits

Push bodies are packed to about 8 MB (UTF-8 bytes as sent), at most 100 entries, 200 removals and
4000 pieces each (the server's caps, with 32 MB on the body, and 100 chats, 500 images or 1000
deletes per entry, since every piece is an object-store call); an entry that
outgrows the target is split into chat-only parts (the artifacts and deletes on the last, the
head on the last too for an incremental push and on the first part, whatever it carries, for a
push of the session whole), and deletes
past the per-entry cap are carried over to the next push, which the session stays marked for. A chat above
16 MB or a session's artifacts above 8 MB are left out with a console warning; a chat that grew
past the cap after it was backed up has its copy deleted, so a restore never presents the old
transcript as the current one. A 413 fails only the sessions of that request. A
request the server refuses (any other 4xx but 404/403/409) stops the backup for the page but keeps
the marks and the sync state, so the next load tries again; a session the server reports it could
not store stays marked and is retried with backoff. A move files the old workspace's removal
mark before recording the new copy's row, so a mark that could not be written leaves the move
to be planned again. A workspace that answers `enabled: false`
marks its sync rows stale (the next push after storage returns carries every session whole,
since a new storage may be a new bucket) and leaves its dirty marks where they are (a move into
it must still remember the old copy); it keeps the removal marks of sessions that had been backed
up, so one deleted while backups are off does not come back once they are on, and drops the
removals of sessions never backed up from this browser, so a storage-less instance does not
collect one mark per deleted session forever. A user delete whose removal mark cannot be written
(localStorage full) is carried by the session's sync row instead (`removed`), which the flush
and the restore read like a mark; a session without a row yet (its first push may be in
flight) gets a row saying only that, and every row write keeps a removal filed meanwhile, so
the push's own row cannot erase it. Pull bodies are
capped at 64 KB. Pull answers up to 20 ids within a 32 MB
budget: a session's size is known from the listings before anything of it is read, one that
would not fit is deferred unless it is the first of the answer, in which case it comes in
pages: the answer carries what fits in key order (at least one object, so every page makes
progress) and names where the next picks up (`next`, a cursor the browser sends back as
`resume` with that session alone). Every page carries a fingerprint of the session's listing
(marker, keys, sizes, modification times, entity tags and versions, since a store reports
modification times coarsely and an object rewritten at the same size within that grain would
otherwise fingerprint the same) taken before anything of it is read, and the server
takes it again once the page is read: a page the backup moved under (a push landing object by
object) is read again, a few times, then answered as `moved`, and the browser starts the
session over on that or on two pages whose fingerprints differ. The browser writes each page's pieces as it arrives, over whatever
an earlier restore cut short had staged (the session is absent locally, so its pieces have no
local edits to keep, and the backup may have moved on), and the record, which is what makes
the session visible, only with the last page. A restore in progress keeps a staging row for
the session (the ids of every chat, image, artifact and version it wrote), which outlives it
if it is cut short; the next restore deletes the staged pieces the backup no longer has, by id
and never by clock, before the record lands (once the record is there no restore looks at the
session again, and a flush would push them back), and a prune that could not run leaves the
session, its pieces and its staging row for the restore after. A restore holds the user's tab
lock while it runs, so two tabs cannot each write the same absent session's pieces over the
other's, and runs only where Web Locks exist (a secure context: https, or localhost); on a plain
http origin the browser still backs up, and its sessions come back on a secure one. A
restore never writes an older record over a newer
one; between pages it holds nothing but the sync
row being assembled, whose chats also admit the images of a later page. Every page carries a
fingerprint of the session's listing (`listing`), taken before anything of the page is
listed or read, so an object landing after it is in the next page's; a session whose
fingerprint moved between two of its pages (a chat added by another device could sort before
the cursor and be missed) starts over, up to three times, then waits for the next restore. An object that grew
since the listing (a push replaced it) ends its page just before it and the answer names that
spot, so the next page sizes it anew rather than the session being imported without it. A pull sees every key of a session's listing but keeps the 5000 smallest
past its cursor (a page is defined by key order, and the store promises none), so a session
grown without bound by valid pushes cannot grow the answer's memory through its metadata
either; removing a prefix and a rotation's deletion stream their listings. `list` scans at most 50 000 index markers, keeps the newest 500 as it goes and answers with
them (`truncated` says when there were more); the restore takes 50 of them. Every read checks the object's size before buffering it: one larger than any push writes
(32 MB) is planted, whatever its listing said, and skipped, since whoever holds the bucket's
credentials can put anything at a predictable key; one larger than its listing said grew
since (a push replaced it) and ends its page, for the next pull to size anew. A dirty mark that cannot be written
(localStorage full) records its bump on the session's sync row instead (`extraV`, counted
with the mark's counter and kept by every row write, so a push in flight cannot retire it);
a session without a row yet (its first push in flight) keeps the bump in the page, and the
next row write takes it onto the row in the same transaction, so the row the push writes
cannot retire the mark with the bump unseen; every load's backfill marks again a session
without a row, with a stale one, or with one carrying bumps no push has covered.
A mark or removal for another user (a write that landed after a switch) reaches that user's
rows through a connection of its own, since the shared handle follows the current user. Nothing is read past the budget, whatever a session holds. A
restore takes the newest 50 sessions per workspace: every visible session gets a runtime, and
each runtime's history load reads the whole chat store. On CE the push checks the storage quota
and bumps usage by bytes written (an over-count on overwrites; the periodic recount settles it).