Files
Ruben Fiszel 2d3a773441 feat(pipelines): mid-DAG selective execution (dbt model+) for pipeline runs (#9945)
* feat(pipelines): mid-DAG selective execution (dbt `model+`) for pipeline runs

Relax the root-only constraint on bounded-cascade starts so `--from` can name
any node in a pipeline — not just a schedule/manual root. A mid-DAG start runs
that node plus its transitive downstream and never re-runs upstream, giving
dbt's most common gesture (`dbt run --select model+`) a direct form:

    wmill pipeline run f/orders --from fct_orders_daily

Previously this errored with "Starts must be schedule-triggered or manual
roots". The bounded-run engine already computed downstream/path-between sets
generically; only the eligibility gate was root-only.

- Shared engine (`boundedCascade.ts`, CLI + frontend mirror): add
  `validFromStarts` — every autorun-able script (roots AND mid-DAG asset
  subscribers / pure readers), excluding only event/input-only handlers
  (kafka/mqtt/…/webhook/data_upload) that can't run with empty args.
- CLI: `--from` accepts any `validFromStarts` node; asset `--from` and
  non-autorun handlers still rejected (the latter runnable via `--upload`). An
  explicit mid-DAG start is protected from the barrier cut. Help text + regenerated
  system_prompts describe the new surface.
- Frontend graph UI parity: any node with downstream now offers "Run + downstream…"
  (was roots-only). With no end picked the bounded-run bar runs the full downstream
  closure (`model+`); picking end(s) still bounds the path-between set.
- Unit tests for the new selection semantics in both engines.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipelines): address CI review — scheduled-root --from regression + pick-mode barrier parity

Codex review findings on #9945:

- P1: explicit `--from` rejected a scheduled root that also carries a secondary
  non-autorun trigger (e.g. `// on schedule` + `// on data_upload`), even though
  it stays a valid IMPLICIT start. `validFromStarts` excluded anything in
  `nonAutorunTriggerScripts`; now it unions in `validStarts` (which lets the
  schedule identity win over the secondary trigger), so a scheduled root is
  `--from`-eligible in both CLI and the graph UI. Regression tests added in both
  engines.

- P2: bounded-pick mode built `eligible` (pickable end bounds) from raw
  `descendants`, so an event handler — or a node only reachable through one —
  could be clicked as an end yet be silently dropped from the barrier-cut run.
  `eligible` is now the barrier-cut closure, so those nodes are dimmed and
  non-pickable. The highlighted `bounded` ring now also reflects the actual
  (barrier-cut) run set, including the no-ends "Run + downstream" case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipelines): frontend barrier set must exclude all valid roots, not just the picked start

Codex review follow-up: the frontend `boundReachable` barrier set only protected
the picked start (`id !== boundPickStart`), while the CLI protects every valid
root (`!starts.has(id)`). So a scheduled root that also carries an event trigger,
reached downstream from another start, was wrongly treated as a barrier — the UI
dimmed/skipped it and its downstream, diverging from the CLI run set.

Exclude `validStarts` from the barrier set too (a scheduled/manual root runs on
its own identity even with a secondary event trigger). Regression test asserts a
scheduled-event root and its downstream stay reachable from an upstream start,
and that the naive (start-only) barrier set would have dropped them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipelines): frontend must exclude webhook/data_upload as mid-DAG autorun starts

Codex review follow-up: the frontend `validFromStarts` only excluded
`EVENT_TRIGGER_KINDS`, so a mid-DAG `webhook`/`data_upload` subscriber was added
by the new eligibility loop — the UI would offer "Run + downstream" and launch it
with empty args (no uploaded S3Object / webhook body). The CLI mirror already
excludes these input-only kinds.

Add a frontend `NON_AUTORUN_TRIGGER_KINDS` (event kinds + webhook + data_upload),
mirroring the CLI, and use it in both `validFromStarts` (exclude such mid-DAG
handlers from starts) and `nonAutorunTriggerScripts` (cut them as barriers).
When the marker is visible (editor overlay / draft) these are now handled
exactly as the CLI does; the deployed-graph blind spot (no webhook/data_upload
rows) remains the documented pre-existing `validStarts` limitation.

Regression test: a `data_upload`/`webhook` mid-DAG subscriber is not an eligible
start and is barrier-cut (with its exclusive downstream) when running from an
upstream root.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 00:11:25 +02:00
..

System Prompts

This directory contains the single source of truth for AI system prompts used by both the frontend copilot and CLI guidance.

Structure

system_prompts/
├── base/              # Core instruction templates (manually written)
│   ├── flow-base.md   # Shared OpenFlow structure guidance
│   └── flow-cli.md    # CLI/local-agent workflow guidance for write-flow skill
├── languages/         # Language-specific instructions (manually written)
└── auto-generated/    # Auto-generated files (DO NOT EDIT)
    ├── sdks/          # SDK documentation
    ├── cli/           # CLI command documentation
    ├── prompts.ts     # TypeScript exports
    └── index.ts       # Helper functions

Usage

Regenerating Prompts

When SDK methods or the OpenFlow schema change, run:

python system_prompts/generate.py

To also refresh the standalone skills in a Claude plugin checkout:

python system_prompts/generate.py --plugin-dir ~/windmill-claude-plugin

--plugin-dir accepts:

  • the windmill-claude-plugin repo root
  • a plugin root such as plugins/windmill
  • a direct skills/ directory

To regenerate the public docs repo (consumed by context7):

python system_prompts/generate.py --context7-dir ~/windmill-cli-docs

--context7-dir writes a fully-rendered snapshot (AGENTS.md, cli-commands.md, skills/<name>/SKILL.md, README.md, manifest.json with the Windmill version) with all template placeholders resolved — suitable for ingestion by docs aggregators. In CI this runs from .github/workflows/publish-cli-docs.yml on every release tag. The generator refuses to wipe the target directory unless it's empty or has a context7 marker (context7.json, manifest.json, or a windmill-cli-docs git remote), so a typo can't delete unrelated files.

This will:

  1. Parse TypeScript and Python SDK files to extract function signatures
  2. Parse the OpenFlow YAML schema
  3. Parse the CLI commands
  4. Assemble complete prompts from markdown files
  5. Generate TypeScript exports in auto-generated/
  6. Optionally refresh plugin-ready standalone SKILL.md files in the target directory

Scope

These system prompts contain ONLY:

  • How to write Windmill scripts (language syntax, conventions, SDK usage)
  • How to structure Windmill flows (OpenFlow schema, module types, data flow)
  • Resource type handling, S3 operations

They DO NOT contain:

  • Tool usage instructions (edit_code, set_flow_json, etc.)
  • IDE/editor specific commands
  • Testing tool invocations

Tool instructions are added separately by the frontend and CLI.

CLI-only workflow instructions live in base/flow-cli.md and are included in the generated write-flow skill for wmill init. They are intentionally excluded from the frontend flow chat prompt.

Integration

Frontend

Uses Vite path alias $system_prompts pointing to auto-generated/:

import { FLOW_GUIDANCE } from "$system_prompts/flow";
import { getLangContext } from "$system_prompts/languages";

CLI

Generates /cli/src/guidance/skills.gen.ts with embedded skill content for wmill init.

Editing Guidelines

  • Edit markdown files in base/, languages/
  • Never edit files in auto-generated/ directly
  • After editing, run generate.py to update exports