Ruben Fiszel 2d3a773441 feat(pipelines): mid-DAG selective execution (dbt model+) for pipeline runs (#9945) ...

* feat(pipelines): mid-DAG selective execution (dbt `model+`) for pipeline runs

Relax the root-only constraint on bounded-cascade starts so `--from` can name
any node in a pipeline — not just a schedule/manual root. A mid-DAG start runs
that node plus its transitive downstream and never re-runs upstream, giving
dbt's most common gesture (`dbt run --select model+`) a direct form:

    wmill pipeline run f/orders --from fct_orders_daily

Previously this errored with "Starts must be schedule-triggered or manual
roots". The bounded-run engine already computed downstream/path-between sets
generically; only the eligibility gate was root-only.

- Shared engine (`boundedCascade.ts`, CLI + frontend mirror): add
  `validFromStarts` — every autorun-able script (roots AND mid-DAG asset
  subscribers / pure readers), excluding only event/input-only handlers
  (kafka/mqtt/…/webhook/data_upload) that can't run with empty args.
- CLI: `--from` accepts any `validFromStarts` node; asset `--from` and
  non-autorun handlers still rejected (the latter runnable via `--upload`). An
  explicit mid-DAG start is protected from the barrier cut. Help text + regenerated
  system_prompts describe the new surface.
- Frontend graph UI parity: any node with downstream now offers "Run + downstream…"
  (was roots-only). With no end picked the bounded-run bar runs the full downstream
  closure (`model+`); picking end(s) still bounds the path-between set.
- Unit tests for the new selection semantics in both engines.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipelines): address CI review — scheduled-root --from regression + pick-mode barrier parity

Codex review findings on #9945:

- P1: explicit `--from` rejected a scheduled root that also carries a secondary
  non-autorun trigger (e.g. `// on schedule` + `// on data_upload`), even though
  it stays a valid IMPLICIT start. `validFromStarts` excluded anything in
  `nonAutorunTriggerScripts`; now it unions in `validStarts` (which lets the
  schedule identity win over the secondary trigger), so a scheduled root is
  `--from`-eligible in both CLI and the graph UI. Regression tests added in both
  engines.

- P2: bounded-pick mode built `eligible` (pickable end bounds) from raw
  `descendants`, so an event handler — or a node only reachable through one —
  could be clicked as an end yet be silently dropped from the barrier-cut run.
  `eligible` is now the barrier-cut closure, so those nodes are dimmed and
  non-pickable. The highlighted `bounded` ring now also reflects the actual
  (barrier-cut) run set, including the no-ends "Run + downstream" case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipelines): frontend barrier set must exclude all valid roots, not just the picked start

Codex review follow-up: the frontend `boundReachable` barrier set only protected
the picked start (`id !== boundPickStart`), while the CLI protects every valid
root (`!starts.has(id)`). So a scheduled root that also carries an event trigger,
reached downstream from another start, was wrongly treated as a barrier — the UI
dimmed/skipped it and its downstream, diverging from the CLI run set.

Exclude `validStarts` from the barrier set too (a scheduled/manual root runs on
its own identity even with a secondary event trigger). Regression test asserts a
scheduled-event root and its downstream stay reachable from an upstream start,
and that the naive (start-only) barrier set would have dropped them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipelines): frontend must exclude webhook/data_upload as mid-DAG autorun starts

Codex review follow-up: the frontend `validFromStarts` only excluded
`EVENT_TRIGGER_KINDS`, so a mid-DAG `webhook`/`data_upload` subscriber was added
by the new eligibility loop — the UI would offer "Run + downstream" and launch it
with empty args (no uploaded S3Object / webhook body). The CLI mirror already
excludes these input-only kinds.

Add a frontend `NON_AUTORUN_TRIGGER_KINDS` (event kinds + webhook + data_upload),
mirroring the CLI, and use it in both `validFromStarts` (exclude such mid-DAG
handlers from starts) and `nonAutorunTriggerScripts` (cut them as barriers).
When the marker is visible (editor overlay / draft) these are now handled
exactly as the CLI does; the deployed-graph blind spot (no webhook/data_upload
rows) remains the documented pre-existing `validStarts` limitation.

Regression test: a `data_upload`/`webhook` mid-DAG subscriber is not an eligible
start and is barrier-cut (with its exclusive downstream) when running from an
upstream root.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-06 00:11:25 +02:00

..

cli-commands.md

feat(pipelines): mid-DAG selective execution (dbt model+) for pipeline runs (#9945)

2026-07-06 00:11:25 +02:00