windmill/docs/pipeline-local-dev.md

Local development for Data Pipelines

Status: draft PR, validated end-to-end. Both the headless CLI paths and the live browser preview (pipeline dev → /pipeline_dev) have been exercised against a running EE + MinIO stack — headless pipeline run --local, browser Run / Run + downstream cascades writing real assets, live-reload on save, failure/cascade UX, parameterized run-forms, and the --frontend flag. This document captures the design and the remaining follow-ups.

What & why

Windmill "data pipelines" are folders of scripts marked with a // pipeline comment, wired together by asset annotations (// on <asset-uri>, // partitioned, // schedule). Until now they could only be built/run in the browser at /pipeline/<folder> (the PipelineGraphEditor), and the CLI only inspected/ran the deployed workspace (wmill pipeline list|show|run).

This adds the local edit → preview → run loop, the pipeline analog of wmill dev (flows/scripts) and wmill app dev (raw apps), usable both from a code editor and from an agentic loop — building a pipeline from working-tree files, seeing the same graph the UI shows, and running it, all without deploying.

The key design decision (no backend changes)

Full body inference is obtained in the CLI from the same wasm the frontend uses: windmill-parser-wasm-asset (parse_assets_ts | parse_assets_py | parse_assets_sql). That wasm returns the entire serialized Rust ParseAssetsOutput — assets (with r/w/rw access) and the parsed pipeline annotations (in_pipeline, triggers, partition, …) in one call. The CLI already loads sibling wasm parsers via loadParser(); we just added the -asset dep.

Running local content reuses the existing preview API: runScriptPreview({ content, language, path, args: { _wmill_skip_asset_dispatch: true }, temp_script_refs }) per node in topological order. Data flows through real asset storage; _wmill_skip_asset_dispatch makes the client own the whole cascade so the backend dispatcher never double-fires.

⇒ No new backend endpoint, no Rust changes, no TS re-port of the annotation parser.

Surfaces

Headless CLI (agentic loop) — cli/src/commands/pipeline/

• pipeline show <folder> --local — render the DAG from working-tree files (fully offline).
• pipeline run <folder> --local [--from/--to/--dry-run/--json] — run the cascade via preview of local content, reusing the boundedCascade.ts topo/lineage engine. Scripts whose only trigger needs caller input or per-event fanout (data_upload/webhook/kafka/…) are skipped by default.
• pipeline run … --upload <script>[:<param>]=<local-file|s3://storage/key> — bind an object to a data_upload/webhook entry point so it (and its downstream) runs: a local path is uploaded to the workspace store; an s3://<storage>/<key> source binds an existing object (authority = named storage, s3:///key for the default store). The target S3Object arg is inferred when the script declares exactly one (else name it with :<param>). Repeatable.
• pipeline docs <folder> [--local] — write PIPELINE.md + AGENTS.md/CLAUDE.md pointers (graph + datatable schemas) so an editor/agent has the same context the UI surfaces.

Browser live-preview — pipeline dev + /pipeline_dev

• pipeline dev [folder] — folder arg or cwd auto-detect; watches the folder, rebuilds the graph on each save, pushes {type:'pipeline', folder, graph, scripts, temp_script_refs} over a WebSocket (direct mode, default port 3201), opens the dev page.
• /pipeline_dev route → PipelineDevView.svelte renders the same PipelineGraphEditor (mode view) fed by the pushed graph; runs the cascade via preview. Editing stays in the user's editor; each save live-reloads.

Files

New

• cli/src/commands/pipeline/localGraph.ts — the enabler. Walks f/<folder> scripts, wasm-infers each (parse_assets_<lang>), assembles the /assets/graph-shaped payload (runnables, assets, edges, triggers). Exports buildLocalPipelineGraph, collectScripts, inferScriptAssets, workspaceRoot, and the canonical graph types.
• cli/src/commands/pipeline/docs.ts — pipeline docs.
• cli/src/commands/pipeline/dev.ts — pipeline dev watcher + WS server.
• cli/test/pipeline_local_graph_unit.test.ts — unit tests for the builder.
• frontend/src/lib/components/assets/AssetGraph/cascadeRun.ts — reusable run primitives (makeLaunch, makeWaitJobTerminal, runDownstreamCascade, runBoundedCascade) wrapping cascadeOrchestrator + graphTraversal.
• frontend/src/lib/components/assets/AssetGraph/PipelineDevView.svelte + frontend/src/routes/pipeline_dev/+page.svelte.

Modified

• cli/src/commands/pipeline/pipeline.ts — --local on show/run; show split into graph acquisition + renderGraph() + deployed-only enrichRootMarkers(); registers docs/dev; graph types moved to localGraph.ts.
• cli/package.json (+ windmill-parser-wasm-asset), cli/package-lock.json.
• system_prompts/auto-generated/*, cli/src/guidance/skills.gen.ts — regenerated via python system_prompts/generate.py (required by AGENTS.md for CLI command changes).

Language coverage

inferScriptAssets mirrors frontend/src/lib/infer.ts:inferAssets: ts (bun/deno/nativets), python3, and SQL dialects all get wasm inference (SQL dialects route to parse_assets_sql; its comment-header annotation scan is dialect-independent). go/bash have no wasm asset parser, so they fall back to a minimal // pipeline + // on scan (annotation-only). Inferred asset paths must match to connect nodes, and all S3 URI forms canonicalize to one key: parse_asset_syntax (shared by the native and wasm parsers) strips leading slashes from S3 paths, so the SDK object forms — TS writeS3File({s3:"x"}) and python write_s3_file(S3Object(s3="x")), which resolve to s3:///x (empty default storage) — the triple-slash annotation // on s3:///x, DuckDB read_csv('s3://x') / COPY ... TO 's3://x', and // on s3://x all yield path x. A TS/Python writer and a DuckDB reader of the same object therefore connect regardless of which URI form each side uses. (Only leading slashes are stripped — so a canonical key never starts with /, which keeps the identity stable through // on trigger-ref reconstruction — while Hive-partition keys like s3://bucket/y=2024/f.parquet and the explicit-storage form s3://storage/key keep their bucket/… / storage/key paths.) Tradeoff of collapsing to one canonical key: the explicit-storage form s3://storage/key and the default-storage nested-key form s3:///storage/key now alias to the same node storage/key, even though they name different objects (a bucket storage vs. an object under the storage/ prefix in default storage). This only collides when a storage config is named to match a default-storage prefix — unlikely, and acceptable for a best-effort lineage graph that already doesn't split the first segment as a storage name.

How to test

Run the CLI from source (alias wmilld='bun run /home/rfiszel/windmill/cli/src/main.ts').

Headless (works directly against any remote, e.g. internal.windmill.dev / data-pipelines):

wmilld workspace add internal data-pipelines https://internal.windmill.dev/   # paste a token
# build an example (connected all-DuckDB DAG):
mkdir -p ~/pl-demo/f/demo_pipeline && cd ~/pl-demo && printf 'defaultTs: bun\n' > wmill.yaml
#   ingest.duckdb.sql:    -- pipeline\nCOPY (SELECT 1 id,'a' name) TO 's3://demo/raw.csv';
#   transform.duckdb.sql: -- pipeline\n-- on s3://demo/raw.csv\nCOPY (SELECT * FROM read_csv('s3://demo/raw.csv')) TO 's3://demo/clean.csv';
#   report.duckdb.sql:    -- pipeline\n-- on s3://demo/clean.csv\nSELECT count(*) FROM read_csv('s3://demo/clean.csv');
wmilld pipeline show demo_pipeline --local
wmilld pipeline run  demo_pipeline --local --dry-run
wmilld pipeline run  demo_pipeline --local          # writes real s3 assets; needs object storage
wmilld pipeline docs demo_pipeline --local   # --local: document the working tree (default path queries the deployed graph)

Or pull real pipelines: wmilld sync pull --yes && wmilld pipeline list && wmilld pipeline show <folder> --local.

Browser preview: against a remote whose deployed frontend predates the /pipeline_dev route, the auto-opened <remote>/pipeline_dev?… 404s. Run THIS branch's frontend locally and point the dev page at it with --frontend:

cd frontend && REMOTE=https://internal.windmill.dev npm run dev      # or a local backend
cd ~/pl-demo && wmilld pipeline dev demo_pipeline --frontend http://localhost:3000

pipeline dev then opens (and prints) the full URL — copy that printed URL, don't hand-write it. It carries both wm_token (workspace auth) and ws_token (the per-session WS token); the dev server rejects the WebSocket without ws_token, so the page would sit disconnected:

http://localhost:3000/pipeline_dev?workspace=<WS>&wm_token=<TOK>&folder=demo_pipeline&port=3201&ws_token=<WS_TOK>

Validation status

• ✅ pipeline show --local verified offline against a fixture (renders a connected DAG).
• ✅ localGraph unit tests (7) + the CLI suite pass; CLI tsc clean; frontend npm run check 0 errors on changed files; svelte-autofixer clean.
• ✅ CLI agent docs regenerated.
• ✅ Live pipeline dev → /pipeline_dev browser preview exercised against a running EE + MinIO stack: graph render + live-reload on save, single-node Run and Run + downstream cascades writing real assets, parameterized run-forms, failure/cascade UX, responsive layout, and the --frontend flag. Screenshots in the PR body.

TODO for the takeover agent

(Items 1–2 below are done; left here for context.)

1. Verify the browser preview end-to-end — done (see Validation status; screenshots in PR).
2. Dev-page route reachability on remotes — done: added --frontend <origin> so pipeline dev can open the page on a locally-run frontend while the API/token target the remote.
3. Route-page dedup: optionally refactor frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte to consume cascadeRun.ts (its launchCascadeScript/runDraftAwareCascade/waitJobTerminal are the source of the extraction) — eliminates duplication. Keep behavior identical.
4. Proxy mode for pipeline dev: currently direct-mode only. Port dev/dev.ts's startProxyServer for embedders that need a localhost origin (e.g. Claude Code preview).
5. pipeline dev editing: the dev page is view+run only (editing stays in the user's editor). If in-browser editing with file round-trip is wanted, mirror flow-dev's handleFlowRoundTrip.
6. Asset-path normalization: done — the python parser resolves the S3Object(s3=…, storage=…?) constructor / dict-literal forms to the same canonical path as the TS {s3, storage} object form, and parse_asset_syntax now strips leading slashes from S3 keys so the SDK-form s3:///x (/x) and the bare-URI no-slash form (x, DuckDB and // on s3://x) canonicalize to one key (see Language coverage). SDK writes and DuckDB reads of the same object connect regardless of URI convention.

Plan reference

Original plan: /home/rfiszel/.claude/plans/tingly-wandering-whistle.md (local to the author).