docs(experiment): oliphaunt/pglite-oxide lightweight dev-mode harness (WIN-2130)

Standalone dev/embedded-db harness (boots embedded pglite Postgres, execs windmill against it; plus a schema-compat probe) and a write-up of the experiment. The enabling migration-bootstrap fix landed separately in #9970 (now on main).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

dev/embedded-db/.gitignore

@@ -0,0 +1,3 @@
+/target
+.wm-embedded-pgdata/
+.schema-compat-pgdata/

dev/embedded-db/Cargo.toml

@@ -0,0 +1,25 @@
+# Standalone experiment crate (WIN-2130). Intentionally NOT a member of the
+# backend workspace: it pulls the heavy wasmer/wasix tree and pins alpha
+# versions, which must not leak into the main backend Cargo.lock.
+[package]
+name = "wm-embedded-db"
+version = "0.1.0"
+edition = "2021"
+publish = false
+
+[[bin]]
+name = "wm-embedded-db"
+path = "src/main.rs"
+
+[[bin]]
+name = "schema-compat"
+path = "src/schema_compat.rs"
+
+[dependencies]
+pglite-oxide = "0.5"
+anyhow = "1"
+tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
+sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "postgres", "migrate"] }
+
+# Detach from any parent workspace so this keeps its own isolated lockfile.
+[workspace]

dev/embedded-db/README.md

@@ -0,0 +1,54 @@
+# wm-embedded-db — oliphaunt/pglite-oxide dev-mode harness (WIN-2130)
+
+Experiment harness for running Windmill against an **embedded Postgres**
+([`pglite-oxide`](https://crates.io/crates/pglite-oxide), the crates.io release of
+[`oliphaunt`](https://github.com/f0rr0/oliphaunt)) — Postgres 17.5 compiled to
+WASIX, no Docker, no Postgres install.
+
+See the write-up and conclusions in
+[`docs/experiments/oliphaunt-lightweight-dev-mode.md`](../../docs/experiments/oliphaunt-lightweight-dev-mode.md).
+**Short version: it works** — with the migration-bootstrap fix (merged in #9970),
+a full Windmill server+worker boots on the embedded single-connection database and
+runs scripts and flows.
+
+This crate is intentionally **not** a member of the backend workspace: it pulls
+the heavy `wasmer`/`wasix` tree and pins alpha versions, which must not leak into
+the main backend `Cargo.lock`. The committed `Cargo.lock` here carries the
+required `virtual-net = 0.702.0-alpha.3` pin (without it, `wasmer-wasix` fails to
+compile — see the write-up).
+
+## Binaries
+
+### `wm-embedded-db` (launcher)
+
+Boots an embedded Postgres, exports `DATABASE_URL`, and execs the command after
+`--` with that env set. The DB lives in this parent process and shuts down when
+the child exits.
+
+```bash
+cd dev/embedded-db
+
+# Just boot the DB and print its URL, then idle (Ctrl-C to stop):
+cargo run
+
+# Boot the DB, then run the windmill binary against it (requires the migration
+# fix from #9970, now on main):
+cargo run -- ../../backend/target/debug/windmill
+# -> windmill migrates, boots server+worker, and runs scripts/flows on pglite.
+#    Run the windmill binary with DATABASE_CONNECTIONS=1 to match pglite's model.
+```
+
+Env:
+- `WM_PGDATA` — data dir (default `./.wm-embedded-pgdata`; set empty for a temp DB)
+
+### `schema-compat` (migration-compatibility probe)
+
+Boots embedded Postgres and applies **Windmill's full migration set** on a single
+connection, then samples core tables. This is the test that proved the schema is
+Postgres-17 compatible (all migrations apply in ~1 s). It neutralizes the single
+`uuid-ossp` `CREATE EXTENSION` line the same way Windmill's own migrator does.
+
+```bash
+cargo run --bin schema-compat
+# -> [schema-compat] ALL 1182 migrations applied OK in ~950ms
+```

dev/embedded-db/src/main.rs

@@ -0,0 +1,70 @@
+//! Lightweight dev-mode launcher (WIN-2130 experiment).
+//!
+//! Boots an embedded PostgreSQL (via `pglite-oxide` / oliphaunt — Postgres
+//! compiled to WASIX, no Docker, no local Postgres install), exports
+//! `DATABASE_URL`, and execs the command passed after `--` with that env set.
+//! The embedded server lives in this parent process and is shut down when the
+//! child exits.
+//!
+//! Usage:
+//!   wm-embedded-db                       # just boot the DB and print the URL, then idle
+//!   wm-embedded-db -- ./target/debug/windmill   # boot DB, then run windmill against it
+//!
+//! Env:
+//!   WM_PGDATA   data dir (default: ./.wm-embedded-pgdata; empty => temporary)
+
+use std::process::Command;
+
+fn main() -> anyhow::Result<()> {
+    let mut args = std::env::args().skip(1);
+    let mut child_cmd: Vec<String> = Vec::new();
+    let mut seen_sep = false;
+    for a in args.by_ref() {
+        if !seen_sep && a == "--" {
+            seen_sep = true;
+            continue;
+        }
+        if seen_sep {
+            child_cmd.push(a);
+        }
+    }
+
+    let pgdata = std::env::var("WM_PGDATA").unwrap_or_else(|_| "./.wm-embedded-pgdata".to_string());
+
+    eprintln!("[wm-embedded-db] booting embedded Postgres (pglite-oxide)...");
+    let t0 = std::time::Instant::now();
+    let mut builder = pglite_oxide::PgliteServer::builder();
+    if pgdata.is_empty() {
+        builder = builder.temporary();
+    } else {
+        builder = builder.path(&pgdata);
+    }
+    let server = builder.start()?;
+    let url = server.database_url();
+    eprintln!(
+        "[wm-embedded-db] embedded Postgres ready in {:?}",
+        t0.elapsed()
+    );
+    eprintln!("[wm-embedded-db] DATABASE_URL={url}");
+    eprintln!(
+        "[wm-embedded-db] NOTE: pglite-oxide serves ONE backend connection at a time; \
+         point pools at a single connection."
+    );
+
+    if child_cmd.is_empty() {
+        eprintln!("[wm-embedded-db] no command given; idling. Ctrl-C to stop.");
+        loop {
+            std::thread::sleep(std::time::Duration::from_secs(3600));
+        }
+    }
+
+    eprintln!("[wm-embedded-db] exec: {}", child_cmd.join(" "));
+    let status = Command::new(&child_cmd[0])
+        .args(&child_cmd[1..])
+        .env("DATABASE_URL", &url)
+        .status()?;
+    eprintln!("[wm-embedded-db] child exited: {status}");
+
+    server.shutdown().ok();
+    std::process::exit(status.code().unwrap_or(1));
+}

dev/embedded-db/src/schema_compat.rs

@@ -0,0 +1,101 @@
+//! Schema-compatibility probe (WIN-2130).
+//!
+//! Boots embedded Postgres (pglite-oxide) and applies Windmill's full migration
+//! set on a single connection, then samples core tables. Proves the schema is
+//! Postgres-17 compatible independent of the runtime connection-concurrency
+//! blocker documented in docs/experiments/oliphaunt-lightweight-dev-mode.md.
+//!
+//! Uses sqlx's runtime `Migrator::new(path)` (not the compile-time macro) so it
+//! can read a patched copy of the migrations with the single `uuid-ossp`
+//! CREATE EXTENSION line neutralized — the same thing Windmill's own migrator
+//! does via OVERRIDDEN_MIGRATIONS (windmill-api/src/db.rs).
+
+use pglite_oxide::PgliteServer;
+use sqlx::migrate::Migrator;
+use sqlx::postgres::PgPoolOptions;
+use std::path::PathBuf;
+use std::time::Duration;
+
+const UUID_OSSP_LINE: &str =
+    "create extension if not exists \"uuid-ossp\"      with schema extensions;";
+
+#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
+async fn main() -> anyhow::Result<()> {
+    let manifest = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
+    let src = manifest.join("../../backend/migrations").canonicalize()?;
+    eprintln!("[schema-compat] source migrations: {}", src.display());
+
+    // Copy the (flat) migrations dir to a temp location and neutralize the one
+    // uuid-ossp line pglite can't satisfy.
+    let tmp = std::env::temp_dir().join("wm-embedded-db-migrations");
+    let _ = std::fs::remove_dir_all(&tmp);
+    std::fs::create_dir_all(&tmp)?;
+    let mut patched = 0usize;
+    for entry in std::fs::read_dir(&src)? {
+        let entry = entry?;
+        if !entry.file_type()?.is_file() {
+            continue;
+        }
+        let name = entry.file_name();
+        let mut body = std::fs::read_to_string(entry.path())?;
+        if body.contains(UUID_OSSP_LINE) {
+            body = body.replace(
+                UUID_OSSP_LINE,
+                "-- uuid-ossp unavailable in pglite; neutralized (see OVERRIDDEN_MIGRATIONS)",
+            );
+            patched += 1;
+        }
+        std::fs::write(tmp.join(name), body)?;
+    }
+    eprintln!("[schema-compat] copied migrations to temp, patched {patched} file(s)");
+
+    let server = PgliteServer::builder()
+        .path(
+            manifest
+                .join(".schema-compat-pgdata")
+                .to_string_lossy()
+                .to_string(),
+        )
+        .start()?;
+    let url = server.database_url();
+    eprintln!("[schema-compat] embedded Postgres up: {url}");
+
+    let pool = PgPoolOptions::new()
+        .min_connections(1)
+        .max_connections(1)
+        .acquire_timeout(Duration::from_secs(120))
+        .connect(&url)
+        .await?;
+
+    let migrator = Migrator::new(tmp.as_path()).await?;
+    eprintln!(
+        "[schema-compat] applying {} migrations (single connection)...",
+        migrator.migrations.len()
+    );
+    let t0 = std::time::Instant::now();
+    match migrator.run(&pool).await {
+        Ok(()) => eprintln!(
+            "[schema-compat] ALL {} migrations applied OK in {:?}",
+            migrator.migrations.len(),
+            t0.elapsed()
+        ),
+        Err(e) => {
+            eprintln!("[schema-compat] FAILED after {:?}: {e}", t0.elapsed());
+            pool.close().await;
+            server.shutdown().ok();
+            std::process::exit(1);
+        }
+    }
+
+    for t in ["workspace", "script", "flow", "v2_job", "usr"] {
+        let q = format!("select count(*) from {t}");
+        match sqlx::query_scalar::<_, i64>(&q).fetch_one(&pool).await {
+            Ok(c) => eprintln!("[schema-compat]   table {t}: OK ({c} rows)"),
+            Err(e) => eprintln!("[schema-compat]   table {t}: MISSING/ERR: {e}"),
+        }
+    }
+
+    pool.close().await;
+    server.shutdown().ok();
+    Ok(())
+}

docs/experiments/oliphaunt-lightweight-dev-mode.md

@@ -0,0 +1,159 @@
+# Experiment: oliphaunt / pglite-oxide as a lightweight dev-mode database (WIN-2130)
+
+**Goal:** find out whether Windmill can run against
+[`oliphaunt`](https://github.com/f0rr0/oliphaunt) — published on crates.io as
+[`pglite-oxide`](https://crates.io/crates/pglite-oxide) — as a "lightweight dev
+mode": an embedded Postgres with no Docker and no separate Postgres install, so
+that `cargo run` alone gives you a working instance. The bar was "ideally get
+scripts and flows running".
+
+**Verdict (TL;DR): it works.** With a small migration-bootstrap fix (merged to
+`main` in #9970), a full Windmill server+worker boots on the embedded
+`pglite-oxide` database (`PostgreSQL 17.5 on wasm32-unknown-wasix`) and **runs
+scripts and flows** — over a single database connection.
+
+```
+SCRIPT: {"started":true,"success":true,"completed":true,"result":"6x7=42"}
+FLOW:   {"started":true,"success":true,"completed":true,"result":"step-b-ok-flow-complete"}
+PostgreSQL version: PostgreSQL 17.5 on wasm32-unknown-wasix … 32-bit
+```
+
+The reproducible harness lives in [`dev/embedded-db/`](../../dev/embedded-db).
+
+---
+
+## What oliphaunt / pglite-oxide is
+
+- **PostgreSQL 17.5 compiled to WASIX** (PGlite lineage), run in-process through a
+  bundled `wasmer` runtime. No Docker, no local Postgres, no runtime build.
+- `PgliteServer::builder().path(dir).start()` boots a local server and
+  `server.database_url()` returns a normal `postgresql://…` URL for SQLx /
+  `tokio-postgres` / `psql`.
+- Bundles `pgvector`, `pg_trgm`, `hstore`, `citext`, `ltree`, `pg_dump`.
+- **Serves one client connection at a time** — the crate is explicit:
+  *"the server owns one embedded backend, so downstream pools should use a single
+  connection."* This single-connection property is the whole story below.
+
+---
+
+## The key insight: single-connection ≡ `max_connections=1`
+
+pglite's "one backend at a time" behaves **exactly** like pointing Windmill at any
+normal Postgres with a `max_connections=1` SQLx pool: the first physical
+connection is served; a *concurrently-needed* second one blocks. That equivalence
+let the whole investigation run against the fast local dev Postgres (with
+`DATABASE_CONNECTIONS=1`) and only use pglite for final confirmation.
+
+### The runtime does NOT need two connections
+
+Running Windmill standalone against a normal Postgres with the runtime pool capped
+at one connection (`DATABASE_CONNECTIONS=1`), the server and worker share that
+single connection and **scripts and flows execute fine** — the instance holds
+exactly one live Postgres connection the whole time (verified via
+`pg_stat_activity`). Everything else people assume "needs" concurrency — HTTP
+handlers, the worker job-poll loop, the health checker — just *serializes* on the
+pool. That's slower under load, but perfectly correct for dev.
+
+So most of what looks like a concurrency requirement is only pool *contention*
+that queues. The real blockers are the few places that **hold one connection open
+and then await a second acquisition** — a genuine self-deadlock at
+`max_connections=1`. There are exactly three, and all three are in the **one-time
+migration bootstrap**:
+
+| # | Site | Problem |
+|---|---|---|
+| 1 | `migrate()` housekeeping (`windmill-api/src/db.rs`) | Holds `db.acquire()` (conn #1) while running `DELETE … _sqlx_migrations` via `.execute(db)` (needs conn #2) |
+| 2 | `migrate()` stale-migration cleanup (same file) | Same pattern inside the per-migration loop |
+| 3 | `fix_flow_versioning_migration` (`windmill-api/src/live_migrations.rs`) | Holds the migrator connection (with the migration advisory lock) while doing `fetch_one(db)` / `db.begin()` on the pool |
+
+`initial_connection()` also hardcodes `max_connections(2)`, but that's only a
+*ceiling* — once the three sites above stop asking for a second connection, the
+migration phase only ever uses one, so the ceiling is never hit.
+
+---
+
+## The fix (merged in #9970)
+
+Route those three migration-phase queries onto the connection the migrator has
+**already checked out** (and, for #3, already holds the advisory lock on) instead
+of re-acquiring from the pool. A `CustomMigrator::connection()` accessor
+(`windmill-api/src/db.rs`) exposes that held connection; `migrate()` and
+`fix_flow_versioning_migration` (`windmill-api/src/live_migrations.rs`) use it.
+
+This is not a pglite-specific hack — it is a strict improvement for **any**
+connection-constrained Postgres (managed providers with tight `max_connections`,
+PgBouncer transaction pooling, etc.): fewer connections during migration, and for
+#3 the existence-check and the write now run on the same advisory-locked
+connection (tighter, not looser). Default behavior is unchanged for normal
+multi-connection setups.
+
+With it in place, Windmill boots on pglite and runs scripts and flows (see the
+TL;DR output). No `SKIP_MIGRATION`, no schema edits.
+
+---
+
+## What else was needed (already handled by Windmill)
+
+- **`uuid-ossp`** is the only extension pglite lacks, and Windmill doesn't need it:
+  `uuid_generate_v*` is never used (the schema uses built-in `gen_random_uuid()`),
+  and Windmill already strips that `CREATE EXTENSION` line in
+  `OVERRIDDEN_MIGRATIONS` (a compat path for managed providers that forbid
+  `CREATE EXTENSION`). All 1182 migrations otherwise apply cleanly — verified
+  independently by the `schema-compat` probe (~950 ms on a single connection).
+
+---
+
+## Caveats / known limitations
+
+- **Serialized on one connection** — correct but not fast under concurrent load.
+  Fine for a single-developer dev instance; not for anything multi-user.
+- **32-bit wasm backend** (`wasm32-unknown-wasix`) — 4 GiB address-space ceiling
+  and lower memory limits than native Postgres.
+- **Alpha dependency pin.** `pglite-oxide 0.5.1` pulls `wasmer`/`wasmer-wasix`
+  `0.702.0-alpha.3`; cargo resolves the transitive `virtual-net` to stable
+  `0.702.0` (which added `NetworkError::MessageSize`), and the alpha
+  `wasmer-wasix` doesn't handle it → won't compile. Pin with
+  `cargo update -p virtual-net --precise 0.702.0-alpha.3` (the harness commits a
+  `Cargo.lock` with this pin).
+- **Language runtimes are orthogonal.** Scripts/flows run to the extent their
+  language is compiled into the binary (e.g. a `--features quickjs`-only build
+  runs bash but not Python) — unrelated to the database.
+
+---
+
+## Recommendation
+
+- The migration-bootstrap fix is worth taking on its own merits — it removes an
+  unnecessary second-connection requirement during migration that also affects
+  tightly connection-limited managed Postgres, not just pglite.
+- With it, `pglite-oxide` is a viable **zero-dependency dev database**: one binary,
+  no Docker, no Postgres install, boots in ~200 ms, runs scripts and flows. A
+  natural next step is a first-class `--embedded-db` dev flag that boots pglite and
+  wires `DATABASE_URL` automatically (the harness in `dev/embedded-db` prototypes
+  exactly this).
+- Keep the single-connection caveat in mind: it's a dev-mode convenience, not a
+  path to running production or load tests embedded.
+
+---
+
+## Reproducing
+
+See [`dev/embedded-db/README.md`](../../dev/embedded-db/README.md). In short:
+
+```bash
+# Prove the schema migrates on a single connection:
+cd dev/embedded-db && cargo run --bin schema-compat
+
+# Boot a real Windmill (built with the migration fix) on embedded pglite:
+cargo run -- ../../backend/target/debug/windmill
+# then hit http://localhost:$PORT and run a script / flow
+```
+
+Isolate the blockers yourself without pglite, using the equivalence above:
+
+```bash
+# fresh empty DB + runtime pool capped at 1 connection
+DATABASE_URL=postgres://postgres:changeme@127.0.0.1:5432/<empty_db> \
+DATABASE_CONNECTIONS=1 PORT=8273 ./backend/target/debug/windmill
+# scripts and flows run; pg_stat_activity shows a single connection
+```