windmill/docs/npm-proxy-storage.md

# npm proxy: where cached registry content belongs

Design rationale for `backend/windmill-api-npm-proxy`. Read this before changing what the
proxy caches or where it keeps it.

## Context

The npm proxy serves an instance's configured private registry to two browser-side
consumers: the raw app editor's package installer, and the script editor's type
acquisition (ATA). Both are chatty by nature. Opening one script asks for a package's file
tree and then for every `.d.ts` in it, which for `@types/lodash` is several hundred
requests against one archive.

Everything the proxy caches today lives in the API process's memory:

| Cache | Budget | TTL | Holds |
|---|---|---|---|
| `PACKAGE_JSON_CACHE` | 64 MiB | 60 s | registry documents (packuments) |
| `TARBALL_CACHE` | 256 MiB | 300 s | per package version: the compressed archive, its file list, and the retained `.d.ts` and manifest |

That is roughly 320 MiB of steady state, plus a transient of `reads in flight x
RETAINED_BYTES` while a read builds its map before the cache weighs it. On a 2 GB API
container the steady state alone is 16% of the box, and the transient scales with request
concurrency rather than with any budget.

## The two things being cached are not the same shape

This is the whole design.

**Packuments** are small (10 to 100 KB in npm's abbreviated form), hot, and mutable: a new
version publishes and the document changes. They exist to save a registry round trip, so
moving them to a store that itself costs a round trip buys nothing. They belong in memory,
with a short TTL, and their budget can be small.

**Package archives** are large (10 KB to 50 MB), cold between sessions, and immutable: a
published version's tarball never changes. They are re-fetchable from the registry at any
time. This is blob-shaped data, and memory is the wrong tier for it.

## Decision

Keep packuments in memory. Move archive-derived content to disk, backed by the instance
object store when one is configured.

### Store extracted files, not the archive

Storing the tarball as the cached unit is the obvious move and it is wrong: every `/file`
request would still download and inflate the whole archive to reach one `.d.ts`. For
`aws-sdk@2.1692.0` that is 50 MB of inflation per declaration, and ATA asks for 442 of
them.

Store instead what the endpoints actually serve, which is already what the current code
chooses to retain:

```
<cache root>/npm_proxy/<registry hash>/<package>/<version>/
  manifest.json      the file list and the resolved entry point
  files/<path>       each retained .d.ts and the package.json
```

`/filetree` then reads one small file and `/file` reads one small file. No inflation on
the request path at all, and nothing large resident.

The extraction rule stays as it is: the manifest plus `.d.ts`, because that is exactly what
`ata/index.ts` reads (`treeToDTSFiles` filters to `.d.ts`, and the caller loop fetches each
dependency's `package.json`). Anything else in the archive is walked past. A request for a
path outside that set still works, by walking the archive once on demand.

### Tiering

Follow the order `windmill-worker/src/global_cache.rs` already uses for dependency caches:

1. **Local disk** first. Fast, per replica, survives nothing but a container restart.
2. **Instance object store** on a local miss, which then populates disk. Shared across
   replicas, survives restarts.
3. **Registry** on a miss in both, which populates disk and the object store.

The object store layer stores a tar of the extracted subset rather than individual objects.
That subset is small (kilobytes to a few MB, not the 50 MB archive), one PUT and one GET
per package version instead of hundreds, and it matches `build_tar_and_push` /
`pull_from_tar` which already exist.

### Keys are scoped to the registry

The cache key must include the registry identity, not just package and version. The `npmrc`
instance setting can be changed to point at a different registry that serves a different
artifact under the same name and version. Hash the resolved registry URL into the path.

Hashing it in is only half of it: a miss has to resolve the setting once and use that one
snapshot for the key, the packument and the tarball. Resolving a second time lets the
setting change in between, and since the tarball's origin is only checked by host, two
repositories on one host will write the second registry's files under the first one's key,
where immutability then keeps them.

Immutability means the archive tiers need no TTL. The packument cache still does, since
that is what discovers new versions.

## What already exists

Little of this is new machinery.

- `windmill_object_store::get_object_store() -> Option<Arc<dyn ObjectStore>>` is the
  instance store accessor, and returning `Option` is exactly the "no object store
  configured" branch.
- `windmill_common::worker::extract_tar` and `atomic_publish_dir` handle unpacking and the
  concurrent half-extracted directory problem. `atomic_publish_dir` has a thundering-herd
  test already.
- `windmill_common::worker::ROOT_CACHE_DIR` is the existing on-disk cache root convention.
- `global_cache::load_cache` / `save_cache` / `build_tar_and_push` / `pull_from_tar` are the
  disk-then-object-store pattern in worker form. They live in `windmill-worker`, so either
  the shared parts move to `windmill-common` or the proxy grows a small equivalent.

## CE does not get the object store tier

`windmill-object-store` is compiled entirely under `#[cfg(feature = "parquet")]`, and every
call site gates on `#[cfg(all(feature = "enterprise", feature = "parquet"))]`. Instance
object storage is therefore an EE capability.

This matters more than usual here: the users most likely to configure a private npm
registry are self-hosted, and a good number of them are on CE. The disk tier is not a
nicety for them, it is the whole feature. Disk has to work well on its own, and the object
store has to be a pure addition on top.

## Failure modes disk has and memory does not

- **Eviction.** A sweep trims to 2 GiB, least recently used first, rate limited to once per
  ten minutes or 256 MiB written, whichever comes first. It measures allocated blocks
  rather than file lengths: a package of many tiny declarations costs a block each and
  would otherwise look nearly free.
- **Recency is `mtime`, marked explicitly.** Cache volumes mount `relatime`, so access time
  does not move on a read and an LRU built on it is silently a FIFO. A cache hit touches
  the manifest instead.
- **ENOSPC and read-only cache directories degrade, they do not fail.** Every write is best
  effort; when one fails the archive is walked a second time into an in-memory map for that
  request. Only on that path, since building the map alongside a successful disk fill would
  put back the per-request heap this exists to remove.
- **Debris, distinguished from live writers.** A killed writer leaves a `.tmp.` directory
  that nothing will read. The sweep removes those, but only once they are an hour old, and
  never treats one as an eviction candidate: concurrent cold fills are exactly when a sweep
  is most likely to run, and deleting a scratch directory underneath its writer would
  publish a package missing whatever it had already written.
- **The live path holds a whole package or nothing.** Publishing renames a fully written
  temp directory into place, so a partial one never looks complete. Losing that rename is
  success only when the destination has a manifest, which means another writer published the
  same immutable content; a destination without one is debris, and it is renamed aside and
  replaced. Eviction is the same rename in reverse rather than a recursive delete in place.
- **Transfers stream, and everything that scales with a package stays off the runtime.**
  Pulls go to a temp file, and unpacking, measuring and publishing all run on a blocking
  thread; pushes build a tar on disk and upload it with a bound on parts in flight, not
  bytes queued. Buffering either, or walking a tree of thousands of declarations on a
  runtime worker, would put back what this exists to remove.
- **A malicious archive is not a broken disk.** An entry whose path escapes its directory
  is refused before either sink, rather than degrading the way an unwritable cache does.

- **Eviction races the reads.** A package can be swept between the lookup and the manifest
  read, so a missing manifest is treated as a miss and repopulated rather than raised. A
  missing file already falls through to the on-demand walk.

- **A recursive delete has no atomic point, so nothing deletes on the live path.**
  `remove_dir_all` unlinks in readdir order, and either half of a package it can leave
  behind is permanent, which is why publishing and eviction both move whole directories with
  `rename`:
  - *Files gone, manifest left* reads as a complete package forever. The manifest
    short-circuits the lookup, so the object store is never consulted, and every declaration
    it names falls through to the on-demand walk, re-fetching and inflating the archive per
    request.
  - *Manifest gone, files left* reads as a miss, but `rename` refuses a non-empty
    destination, so every tree published afterwards is discarded in favour of the debris and
    the path answers 500 with a valid copy sitting in the bucket.

  A sweep that cannot rename a candidate aside leaves it whole rather than falling back to
  deleting it where it stands: the cap is a bound the next sweep rechecks, and a package
  half-deleted under it is not.

Still open: **single flight**. Concurrent misses for the same package each fetch and
extract. Publishing by rename and per-pull scratch directories make that safe, just
wasteful.

## Alternative considered: just lower the budgets

The constants were sized for throughput without weighing a 2 GB container. Cutting
`TARBALL_CACHE_BYTES` to 64 MiB and `PACKAGE_JSON_CACHE_BYTES` to 16 MiB is a four line
change with no new failure modes, and the only cost is re-fetching large packages more
often.

It does not fix the transient, which scales with concurrency rather than with the budget,
and it does not survive a restart or share across replicas. Take it if the storage work is
not worth doing now; it is strictly a smaller version of the same trade.

## Status

Implemented in `store.rs`. The archive cache is gone from memory: extraction streams each
entry straight to the package directory, so a read holds one entry rather than the whole
retained set, and `PACKAGE_JSON_CACHE` at 16 MiB is all that is left in the heap.

Two things settled during implementation and are worth keeping in mind before changing it:

- **A warm package must not cost a round trip.** The registry is resolved from settings, and
  the packument is fetched only on a cache miss. Fetching it first, to reach the tarball URL,
  made a fully cached package still hit the registry once per minute.
- **The manifest is exempt from the retention budget.** `next` orders its `package.json`
  3748th, so a budget spent on the declarations ahead of it would leave the entry point
  defaulted, which is a wrong answer rather than a slow one.