windmill/docs/fork-object-storage.md

# Fork-isolated workspace object storage

A fork clones its parent's `large_file_storage` settings, so without isolation it reads and
writes the parent's objects: a script iterating in a fork overwrites production's files. Each
storage the fork isolates (a `fork_storage_namespace` row, chosen at fork creation, default
isolated) instead maps logical keys:

- **Writes** (put, multipart, delete, copy destination) go to `__wm_forks/<fork segment>/<key>`,
  the same one-component segment DuckLake's fork namespaces use (`fork_data_dir_segment`).
- **Reads** try the fork's own key, then each ancestor's mapping, nearest first, then the root's
  raw key. They stop at the first ancestor whose storage is a different one
  (`lfs_entry_storage_ref`): its keys are not in this bucket. A shared fork maps like its parent.
- **Listings** show the workspace's own objects only, as logical keys.
- **Deletes** remove the fork's copy; an ancestor's object stays readable through it.

The mapping (`windmill_common::fork_storage::ForkStorageKeys`) is applied wherever Windmill
turns a workspace-storage key into a storage operation: the `ForkStore` object-store wrapper
(`windmill_object_store::fork_store::workspace_object_store`) for every server-side helper, and
key rewriting in the S3 proxy's raw S3 forwarding (URI key, list `prefix`/`start-after`/`marker`
and response keys, `x-amz-copy-source`). A named resource (`s3_resource_path`) is not workspace
storage and is never mapped. A workspace whose keys are mapped is refused its workspace
storage's own credentials (`s3_resource_info`, the duckdb/polars settings endpoints): a client
holding them would write past the mapping.

Not mapped, on purpose: AI sessions and volumes (their keys already carry the workspace id),
SMTP attachments (job-id keys), the storage usage count (it counts the whole bucket). Known
gaps: the worker's result-cache hash HEADs the logical key unmapped, and a read of an object
the fork does not own costs one HEAD per chain level before the GET.

## Invariants

- **`__wm_forks/` is reachable only through the workspace's own chain.** A key under it maps to
  itself when it is in the workspace's own area (writes) or an area of the chain (reads), and is
  refused otherwise; listings hide it. This is also what keeps DuckLake working: its fork data
  paths arrive already prefixed, and its defer attaches read ancestors' raw keys through the
  read chain. Every workspace gets the rule, isolating or not, so pre-existing forks' lakes keep
  working.
- **No `.` or `..` key segments**, in any workspace: the proxy forwards mapped keys in a URL,
  where they collapse and would climb out of the fork's prefix.
- **The prefix derives from the workspace id**, so an isolating fork's id cannot be changed
  (`change_workspace_id` refuses): under a new id it would see none of its objects. Nor can an
  isolated storage be repointed, since the purge targets the storage recorded at creation.
- **A shared lake needs its storage shared**: its files must land at the parent's keys, where the
  parent's catalog looks for them. Fork creation refuses the other combination.
- **Children before parents**: deleting a fork that has forks of its own is refused when anything
  in its chain isolates a storage, shared forks included. A child reads through, or writes into,
  the isolating prefix via that link, and without it the child's chain would map like the root's.
- **Deletion and same-id creation are serialized** on the `fork:<id>` advisory lock, and a purge
  skips an id that exists again: the rows are then the new fork's. The proxy caches only the raw
  client, never a mapping, so fork creation and deletion invalidate the one mapping cache.
- **Deletion always purges** the fork's area of each isolated storage, post-commit, from the
  storage identity recorded at creation: nothing can reach it once the fork is gone. Rows have
  no FK and outlive the workspace when the purge fails; creating a fork with the same id retries
  them and refuses while they cannot be purged, since a same-id fork would read them.
- Isolation is a per-fork-creation choice (`shared_storages` on create-fork), never inherited.
  Forks created before the registry existed have no rows and keep sharing.