Files
windmill/backend/windmill-common/src/assets.rs
T
Ruben FiszelandClaude Opus 4.8 3dcd3949a1 feat(pipelines): auto-derive cascade edges from ducklake/s3 reads (+ muted-read badge) (#9963)
* feat(pipelines): auto-derive cascade trigger edges from ducklake/s3 reads

Within a `// pipeline`, a read of a ducklake table or s3 object now
auto-wires its cascade trigger edge straight from the FROM clause, so
`// on <asset>` is only needed for edges inference can't see (dynamic SQL)
or to carry per-edge opts. Two opt-outs: `// mute <asset>` suppresses a
single derived edge (a lookup / SCD input read every run but not cascaded
on), and `// mute all` opts the script out of derivation entirely (back to
explicit-`// on`-only). Explicit `// on` still wins the dedup.

Scoped to ducklake + s3 reads; resource/datatable/volume stay explicit.
Read-write (RW) and write inputs are excluded so a self-referential
merge can't loop-trigger itself; ambiguous (None) access is skipped.

- parser: `mute` / `mute_all` in PipelineAnnotations (Rust + TS mirror)
- deploy: derive_pipeline_asset_trigger_refs → script_trigger rows
- frontend: resolveGraph mirrors derivation for the live edit-mode canvas
- tests: shared parity corpus + derive-helper units + resolveGraph overlays

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(pipelines): mark auto-derived cascade edges with a persisted derived flag + "auto" badge

Persist script_trigger.derived (deploy: true for ducklake/s3-read derivation,
false for explicit // on) and return it from the asset-graph endpoint so the
canvas renders a Sparkles "auto" badge on auto-wired edges — the inference is
now visible on both the deployed graph and the live edit canvas, not just
implied. Dispatch (fetch_subscribers) ignores the flag, so a derived edge fires
identically to an explicit // on. Also copy derived in the workspace-clone
trigger copy, and backfill muteAssets/muteAll into two empty PipelineAnnotations
literals the base commit left stale (check:fast).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipelines): derive cascade edge from effective (alt-fallback) asset access

derive_pipeline_asset_trigger_refs gated on the raw parser access_type, but the
persisted asset.usage_access_type and the frontend canvas both use
access_type.or(alt_access_type). An ambiguous parse with a manual read override
was persisted/drawn as a read yet derived no edge, so the auto edge silently
vanished on deploy. Gate on the effective access type for parity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(pipelines): badge muted reads instead of auto-derived edges

Auto-derivation is the default now, so badging every derived cascade edge is
noise. Drop the "auto" badge and the persisted `script_trigger.derived` flag
(migration + insert param + graph field + clone copy) that only powered it, and
instead badge the exception: a ducklake/s3 asset a script reads but does NOT
cascade — `// mute <asset>` / `// mute all`. `computeMutedReadKeys` marks a
read-only ('r') supported read with no cascade trigger and no self-write; the
canvas renders a bell-off "muted" badge on that read edge.

Also fixes two review parity nits:
- TS `// on` parser now strips trailing `key=value` opts (e.g. `debounce=60s`)
  like the Rust `split_trailing_kv_opts`, so the ref dedups against inference.
- A `// materialize` producer reading its own target is upgraded to `rw`
  (deploy) / excluded via the materialize write refs (canvas), so it neither
  self-cascades nor shows as a muted read.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipelines): drop redundant // on for auto-derived reads; gate muted badge to pipeline scripts

- Templates no longer scaffold `// on <asset>` for a ducklake/s3 input the body
  reads — the read auto-wires the cascade now that derivation is the default.
  Kept for datatable/resource (not auto-derived) and native triggers. The
  discoverability hint now mentions `// mute` (the newly relevant annotation).
- computeMutedReadKeys only badges reads by `// pipeline` scripts. A plain
  script or flow reading a ducklake/s3 asset never had an auto trigger to
  suppress, so it must render as ordinary lineage, not "muted" (Codex review).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipelines): only drop template // on when the body actually reads the input

The redundant-`// on` removal assumed the generated body reads the ducklake/s3
input, but postgres/bash/generic bodies (and `data_upload`, which reads the
picker file) ignore `input` — dropping `// on` there left the asset-created
script with no cascade at all. Gate the drop on READS_INPUT_LANGS
(bun/deno/python/duckdb) so non-reading templates keep the explicit trigger.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 18:12:12 +00:00

664 lines
26 KiB
Rust

use std::collections::HashSet;
use sqlx::{PgExecutor, Postgres, Transaction};
use crate::{error, scripts::ScriptHash};
pub use windmill_parser::asset_parser::{
merge_column_lineage, parse_pipeline_annotations, ColumnLineage, ColumnRef, DataTest,
OnSchemaChange, PartitionKind, PipelineAnnotations, RetrySpec, TriggerSpec, PARTITION_TOKEN,
};
pub use windmill_types::assets::*;
// --- Workspace DuckDB macro registry cache (worker hot path) ---
// Every DuckDB job reads the `macro_definition` registry; cascades can run
// many jobs per second. Primary invalidation is the
// `notify_macro_registry_change` event, emitted transactionally with every
// registry mutation and dispatched to this cache in main.rs — the TTL is a
// backstop for mutation paths that don't emit (manual SQL edits).
#[derive(Clone, Debug, sqlx::FromRow)]
pub struct MacroRegistryEntry {
pub name: String,
pub params: String,
pub body: String,
pub is_table_macro: bool,
pub provider_path: String,
}
#[derive(Clone)]
pub struct ExpiringMacroRegistry {
pub rows: std::sync::Arc<Vec<MacroRegistryEntry>>,
pub expires_at: std::time::Instant,
}
pub const MACRO_REGISTRY_TTL: std::time::Duration = std::time::Duration::from_secs(60);
// Tests sharing one process across several databases must disable the cache:
// it is keyed by workspace id alone, and a notify from one DB can't evict
// entries populated from another (same hazard as DEPLOYED_SCRIPT_CACHE_DISABLED).
pub static MACRO_REGISTRY_CACHE_DISABLED: std::sync::atomic::AtomicBool =
std::sync::atomic::AtomicBool::new(false);
lazy_static::lazy_static! {
pub static ref MACRO_REGISTRY_CACHE: quick_cache::sync::Cache<String, ExpiringMacroRegistry> =
quick_cache::sync::Cache::new(1000);
}
#[derive(sqlx::Type, Debug, Clone, Copy, PartialEq)]
#[sqlx(type_name = "SCRIPT_TRIGGER_KIND", rename_all = "lowercase")]
pub enum ScriptTriggerKind {
Asset,
Schedule,
Webhook,
Email,
Kafka,
Mqtt,
Nats,
Postgres,
Sqs,
Gcp,
}
pub async fn insert_static_asset_usage<'e>(
executor: impl PgExecutor<'e>,
workspace_id: &str,
asset: &AssetWithAltAccessType,
usage_path: &str,
usage_kind: AssetUsageKind,
) -> error::Result<()> {
// Convert columns BTreeMap to JSONB format
let columns_json = asset
.columns
.as_ref()
.map(|cols| serde_json::to_value(cols).unwrap_or(serde_json::Value::Null));
// Invalidate the per-workspace producer-writes cache only when this insert
// actually adds a write producer: the cache (asset_dispatch::
// ASSET_PRODUCER_WRITES_CACHE) tracks script rows with 'w'/'rw' access, so
// a row that was a no-op (ON CONFLICT skipped), a flow usage, or read-only
// can't change it. Emitting in the same statement keeps the notify atomic
// with the insert and visible to pollers only on commit. See the matching
// delete-side guard in clear_static_asset_usage.
sqlx::query!(
r#"WITH ins AS (
INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)
VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT DO NOTHING
RETURNING usage_kind, usage_access_type
)
INSERT INTO notify_event (channel, payload)
SELECT 'notify_asset_producer_change', $1
FROM ins WHERE usage_kind = 'script' AND usage_access_type IN ('w', 'rw')"#,
workspace_id,
asset.path,
asset.kind as AssetKind,
(asset.access_type.or(asset.alt_access_type)) as Option<AssetUsageAccessType>,
usage_path,
usage_kind as AssetUsageKind,
columns_json as Option<serde_json::Value>
)
.execute(executor)
.await?;
Ok(())
}
pub async fn clear_static_asset_usage<'e>(
executor: impl PgExecutor<'e>,
workspace_id: &str,
usage_path: &str,
usage_kind: AssetUsageKind,
) -> error::Result<()> {
// Invalidate the per-workspace producer-writes cache that gates the
// asset-trigger dispatch hook (windmill-queue
// asset_dispatch::ASSET_PRODUCER_WRITES_CACHE). That cache only tracks
// script rows with 'w'/'rw' access, so emit the notify only when the
// delete actually removed such a write producer: flow usage, read-only
// usage, and deletes that matched no producer row leave the cache
// unchanged (the common case — most deploys touch no write asset). The
// matching add side lives in insert_static_asset_usage. Emitting in the
// same statement keeps the notify atomic with the delete and visible to
// pollers only on commit.
sqlx::query!(
r#"WITH del AS (
DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3
RETURNING usage_access_type
)
INSERT INTO notify_event (channel, payload)
SELECT 'notify_asset_producer_change', $1
WHERE $3 = 'script'
AND EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))"#,
workspace_id,
usage_path,
usage_kind as AssetUsageKind
)
.execute(executor)
.await?;
Ok(())
}
pub async fn clear_static_asset_usage_by_script_hash<'e>(
executor: impl PgExecutor<'e>,
workspace_id: &str,
script_hash: ScriptHash,
) -> error::Result<()> {
// Always script usage → invalidate the producer-writes cache for this
// workspace, but only when the delete actually removed a write producer
// ('w'/'rw'); see clear_static_asset_usage. Atomic with the delete.
sqlx::query!(
r#"WITH del AS (
DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script'
AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)
RETURNING usage_access_type
)
INSERT INTO notify_event (channel, payload)
SELECT 'notify_asset_producer_change', $1
WHERE EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))"#,
workspace_id,
script_hash.0
)
.execute(executor)
.await?;
Ok(())
}
fn is_write_access(access: Option<AssetUsageAccessType>) -> bool {
matches!(
access,
Some(AssetUsageAccessType::W) | Some(AssetUsageAccessType::RW)
)
}
/// Kinds whose *read* usage auto-derives a cascade trigger edge inside a
/// `// pipeline`. Scoped to the two intra-pipeline data kinds — a ducklake
/// table read (the core case) and an s3 object read (file-ingestion
/// producers). Resource / datatable / volume reads stay explicit-`// on`:
/// a config/lookup read cascading is more often surprising than wanted.
fn is_auto_trigger_kind(kind: AssetKind) -> bool {
matches!(kind, AssetKind::Ducklake | AssetKind::S3Object)
}
/// Trigger refs auto-derived from a pipeline script's inferred reads, so the
/// FROM clause alone wires the cascade edge (no redundant `// on <asset>`).
///
/// Included: an input read read-*only* (`R`) of a supported kind
/// ([`is_auto_trigger_kind`]). The effective access type is
/// `access_type.or(alt_access_type)` — same precedence as the persisted
/// `asset.usage_access_type` and the frontend mirror's `access_type ??
/// alt_access_type`, so a manual read override on an ambiguous parse still
/// derives an edge (and the live canvas and the deployed graph agree).
/// Excluded, each for a reason:
/// - `RW` / `W` — the script also writes the asset; an edge would be a
/// self-triggering loop.
/// - `None` access — usage is ambiguous (poisoned merge) with no override;
/// can't confirm a read, so fail safe and don't cascade.
/// - already in `explicit_refs` — the author wrote `// on <asset>`, which
/// wins (it carries the per-edge debounce/opts).
/// - in `muted_refs` — a `// mute <asset>` opt-out (lookup / SCD input).
///
/// `mute_all` (from `// mute all`) short-circuits to no derivation, leaving
/// only the explicit `// on` edges. Returns canonical refs (e.g.
/// `ducklake://main.orders`), deduped, in input order.
pub fn derive_pipeline_asset_trigger_refs(
assets: &[AssetWithAltAccessType],
explicit_refs: &HashSet<String>,
muted_refs: &HashSet<String>,
mute_all: bool,
) -> Vec<String> {
if mute_all {
return vec![];
}
let mut out = vec![];
let mut seen = HashSet::new();
for a in assets {
// Effective access mirrors the persisted `usage_access_type` and the
// frontend derivation: an explicit parse wins, else the manual override.
let access = a.access_type.or(a.alt_access_type);
if access != Some(AssetUsageAccessType::R) || !is_auto_trigger_kind(a.kind) {
continue;
}
let Some(prefix) = a.kind.canonical_prefix() else {
continue;
};
let r = format!("{}{}", prefix, a.path);
if explicit_refs.contains(&r) || muted_refs.contains(&r) {
continue;
}
if seen.insert(r.clone()) {
out.push(r);
}
}
out
}
/// Clear and reinsert the full static-asset usage set of a script in one tx,
/// invalidating the producer-writes cache at most once and only on a real
/// change. The cache (asset_dispatch::ASSET_PRODUCER_WRITES_CACHE) keys a
/// workspace by the set of (kind, path) script rows with 'w'/'rw' access, so a
/// redeploy that keeps the same write producers must leave it untouched. We
/// diff the old write set (captured from the clearing DELETE's RETURNING)
/// against the new one and emit a single notify only when they differ —
/// emitting per statement, as clear/insert_static_asset_usage do, would fire
/// twice on every write-asset redeploy (the clear removes the row, the reinsert
/// adds it back). The notify rides the deploy tx, so it stays atomic with the
/// DML and visible to pollers only on commit.
///
/// DELETE and INSERT of the same primary key cannot share a single statement
/// (both would read the pre-statement snapshot, so the reinsert's ON CONFLICT
/// would silently drop the row), which is why this clears and reinserts as
/// separate statements rather than one CTE.
///
/// Performs no authorization itself: `tx` must already be scoped to a caller
/// authorized for `workspace_id`/`usage_path` (e.g. `user_db.begin(&authed)`,
/// which applies RLS), exactly like the sibling clear/insert helpers.
pub async fn replace_static_asset_usage(
tx: &mut Transaction<'_, Postgres>,
workspace_id: &str,
usage_path: &str,
assets: &[AssetWithAltAccessType],
) -> error::Result<()> {
let cleared = sqlx::query!(
r#"DELETE FROM asset
WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = 'script'
RETURNING kind AS "kind!: AssetKind", path,
usage_access_type AS "usage_access_type: AssetUsageAccessType""#,
workspace_id,
usage_path,
)
.fetch_all(&mut **tx)
.await?;
let old_writes: HashSet<(AssetKind, String)> = cleared
.into_iter()
.filter(|r| is_write_access(r.usage_access_type))
.map(|r| (r.kind, r.path))
.collect();
// Build new_writes from rows actually inserted (RETURNING), not the
// requested slice: ON CONFLICT DO NOTHING is first-writer-wins, so a payload
// with duplicate (kind, path) entries at conflicting access types persists
// only the first. Since the DELETE above emptied this usage_path, every
// non-conflicting insert lands, so the inserted rows are exactly the new
// persisted set.
let mut new_writes: HashSet<(AssetKind, String)> = HashSet::new();
for asset in assets {
let access = asset.access_type.or(asset.alt_access_type);
let columns_json = asset
.columns
.as_ref()
.map(|cols| serde_json::to_value(cols).unwrap_or(serde_json::Value::Null));
let inserted = sqlx::query!(
r#"INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)
VALUES ($1, $2, $3, $4, $5, 'script', $6) ON CONFLICT DO NOTHING
RETURNING usage_access_type AS "usage_access_type: AssetUsageAccessType""#,
workspace_id,
asset.path,
asset.kind as AssetKind,
access as Option<AssetUsageAccessType>,
usage_path,
columns_json as Option<serde_json::Value>,
)
.fetch_optional(&mut **tx)
.await?;
if let Some(row) = inserted {
if is_write_access(row.usage_access_type) {
new_writes.insert((asset.kind, asset.path.clone()));
}
}
}
if old_writes != new_writes {
sqlx::query!(
r#"INSERT INTO notify_event (channel, payload)
VALUES ('notify_asset_producer_change', $1)"#,
workspace_id,
)
.execute(&mut **tx)
.await?;
}
Ok(())
}
// Wipe all pipeline trigger declarations held by the given runnable. Used at
// deploy time: redeploying a script wipes its prior `// on` annotations so
// removing them implicitly un-declares those edges.
pub async fn clear_script_triggers<'e>(
executor: impl PgExecutor<'e>,
workspace_id: &str,
runnable_path: &str,
runnable_kind: AssetUsageKind,
) -> error::Result<()> {
sqlx::query!(
r#"DELETE FROM script_trigger
WHERE workspace_id = $1 AND runnable_kind = $2 AND runnable_path = $3"#,
workspace_id,
runnable_kind as AssetUsageKind,
runnable_path,
)
.execute(executor)
.await?;
Ok(())
}
// Insert a single trigger declaration. Caller is expected to wipe first.
// `join_all` is the script-level `// trigger all` flag (AND join barrier);
// `retry_count` / `retry_delay_s` are the `// retry <n> [<delay>]` policy.
// All three are script-level — the same value for every row of a given
// runnable — but stored per-row to keep the wipe-and-reinsert pattern and a
// single-query subscriber lookup.
pub async fn insert_script_trigger<'e>(
executor: impl PgExecutor<'e>,
workspace_id: &str,
runnable_kind: AssetUsageKind,
runnable_path: &str,
trigger_kind: ScriptTriggerKind,
trigger_ref: &str,
join_all: bool,
debounce_s: Option<i32>,
retry_count: Option<i16>,
retry_delay_s: Option<i32>,
) -> error::Result<()> {
sqlx::query!(
r#"INSERT INTO script_trigger
(workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all,
debounce_s, retry_count, retry_delay_s)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)"#,
workspace_id,
runnable_kind as AssetUsageKind,
runnable_path,
trigger_kind as ScriptTriggerKind,
trigger_ref,
join_all,
debounce_s,
retry_count,
retry_delay_s,
)
.execute(executor)
.await?;
Ok(())
}
/// Parse a debounce duration into whole seconds. Accepts a bare integer
/// (seconds) or an `<n>` with an `s`/`m`/`h`/`d` suffix (e.g. `30s`,
/// `5m`, `2h`, `1d`). Returns `None` for empty / malformed / non-positive
/// input — the caller treats `None` as "no debounce" (fan-out), so a typo
/// fails safe rather than silently debouncing.
pub fn parse_duration_secs(s: &str) -> Option<i32> {
let s = s.trim();
if s.is_empty() {
return None;
}
let (num, mult): (&str, i64) = match s.as_bytes().last() {
Some(b's') => (&s[..s.len() - 1], 1),
Some(b'm') => (&s[..s.len() - 1], 60),
Some(b'h') => (&s[..s.len() - 1], 3600),
Some(b'd') => (&s[..s.len() - 1], 86400),
Some(c) if c.is_ascii_digit() => (s, 1),
_ => return None,
};
let n: i64 = num.trim().parse().ok()?;
let secs = n.checked_mul(mult)?;
if secs <= 0 || secs > i32::MAX as i64 {
return None;
}
Some(secs as i32)
}
#[cfg(test)]
mod debounce_duration_tests {
use super::parse_duration_secs;
#[test]
fn parses_units_and_bare_seconds() {
assert_eq!(parse_duration_secs("60"), Some(60));
assert_eq!(parse_duration_secs("30s"), Some(30));
assert_eq!(parse_duration_secs("5m"), Some(300));
assert_eq!(parse_duration_secs("2h"), Some(7200));
assert_eq!(parse_duration_secs(" 1d "), Some(86400));
// Explicit plus sign comes free with i64 parsing; the TS mirror
// (parseDurationSecs) matches it — keep the two in lockstep.
assert_eq!(parse_duration_secs("+5m"), Some(300));
assert_eq!(parse_duration_secs("+45"), Some(45));
}
#[test]
fn rejects_garbage_and_nonpositive() {
assert_eq!(parse_duration_secs(""), None);
assert_eq!(parse_duration_secs("abc"), None);
assert_eq!(parse_duration_secs("0"), None);
assert_eq!(parse_duration_secs("-5"), None);
assert_eq!(parse_duration_secs("10x"), None);
assert_eq!(parse_duration_secs("s"), None);
}
}
#[cfg(test)]
mod derive_trigger_tests {
use super::{derive_pipeline_asset_trigger_refs, AssetKind, AssetUsageAccessType};
use std::collections::HashSet;
use windmill_types::assets::AssetWithAltAccessType;
fn asset(
kind: AssetKind,
path: &str,
at: Option<AssetUsageAccessType>,
) -> AssetWithAltAccessType {
AssetWithAltAccessType {
path: path.to_string(),
kind,
access_type: at,
alt_access_type: None,
columns: None,
}
}
fn derive(assets: &[AssetWithAltAccessType]) -> Vec<String> {
derive_pipeline_asset_trigger_refs(assets, &HashSet::new(), &HashSet::new(), false)
}
#[test]
fn read_only_ducklake_and_s3_derive_an_edge() {
use AssetUsageAccessType::R;
let a = [
asset(AssetKind::Ducklake, "main.orders", Some(R)),
asset(AssetKind::S3Object, "raw/events", Some(R)),
];
assert_eq!(
derive(&a),
vec![
"ducklake://main.orders".to_string(),
"s3://raw/events".to_string()
]
);
}
#[test]
fn writes_and_rw_are_skipped_to_avoid_self_edges() {
use AssetUsageAccessType::*;
// W (pure producer) and RW (reads *and* writes the same table — a
// self-cascade if edged) both derive nothing.
let a = [
asset(AssetKind::Ducklake, "main.out", Some(W)),
asset(AssetKind::Ducklake, "main.self", Some(RW)),
];
assert!(derive(&a).is_empty());
}
#[test]
fn ambiguous_access_and_unsupported_kinds_are_skipped() {
use AssetUsageAccessType::R;
let a = [
asset(AssetKind::Ducklake, "main.ambiguous", None), // poisoned merge
asset(AssetKind::Resource, "f/db", Some(R)), // out of scope
asset(AssetKind::DataTable, "main.dt", Some(R)), // out of scope
];
assert!(derive(&a).is_empty());
}
#[test]
fn manual_read_override_on_ambiguous_parse_derives_an_edge() {
use AssetUsageAccessType::{R, W};
// Parser can't confirm access (`access_type: None`) but the user manually
// overrode it. Effective access = `access_type.or(alt_access_type)`, the
// same value persisted to `asset.usage_access_type` and used by the
// frontend canvas — so a read override derives an edge (parity, no
// silently-vanishing edge on deploy) and a write override does not.
let read_override = AssetWithAltAccessType {
path: "main.override_r".to_string(),
kind: AssetKind::Ducklake,
access_type: None,
alt_access_type: Some(R),
columns: None,
};
let write_override = AssetWithAltAccessType {
path: "main.override_w".to_string(),
kind: AssetKind::Ducklake,
access_type: None,
alt_access_type: Some(W),
columns: None,
};
assert_eq!(
derive(&[read_override, write_override]),
vec!["ducklake://main.override_r".to_string()]
);
}
#[test]
fn explicit_and_muted_refs_are_excluded() {
use AssetUsageAccessType::R;
let a = [
asset(AssetKind::Ducklake, "main.explicit", Some(R)),
asset(AssetKind::Ducklake, "main.muted", Some(R)),
asset(AssetKind::Ducklake, "main.keep", Some(R)),
];
let explicit: HashSet<String> = ["ducklake://main.explicit".to_string()].into();
let muted: HashSet<String> = ["ducklake://main.muted".to_string()].into();
assert_eq!(
derive_pipeline_asset_trigger_refs(&a, &explicit, &muted, false),
vec!["ducklake://main.keep".to_string()]
);
}
#[test]
fn mute_all_derives_nothing() {
use AssetUsageAccessType::R;
let a = [asset(AssetKind::Ducklake, "main.orders", Some(R))];
assert!(
derive_pipeline_asset_trigger_refs(&a, &HashSet::new(), &HashSet::new(), true)
.is_empty()
);
}
#[test]
fn duplicate_reads_dedup() {
use AssetUsageAccessType::R;
let a = [
asset(AssetKind::Ducklake, "main.orders", Some(R)),
asset(AssetKind::Ducklake, "main.orders", Some(R)),
];
assert_eq!(derive(&a), vec!["ducklake://main.orders".to_string()]);
}
}
#[cfg(test)]
mod trigger_ref_roundtrip_tests {
use super::{parse_asset_trigger_ref, trigger_spec_to_row, AssetKind, ScriptTriggerKind};
use windmill_parser::asset_parser::{parse_asset_syntax, AssetKind as PAssetKind, TriggerSpec};
// `trigger_spec_to_row` rebuilds a stored ref as `s3://<path>`, and
// `parse_asset_trigger_ref` parses it back. The two must be inverse for
// every S3 URI form, or a consumer's `// on` trigger lands on a different
// graph node than the producer's inferred write. Because `parse_asset_syntax`
// strips ALL leading slashes, a canonical path never starts with `/`, so the
// naive `prefix + path` rebuild round-trips — including the `S3Object(s3="/x")`
// quad-slash case that previously desynced (path `/x` rebuilt to `s3:///x`,
// which re-parsed to `x`).
fn roundtrip(uri: &str) -> String {
let (pkind, path) = parse_asset_syntax(uri, false).expect("parse uri");
assert_eq!(pkind, PAssetKind::S3Object);
let spec = TriggerSpec::Asset { asset_kind: pkind, path: path.to_string(), debounce: None };
let (kind, stored) = trigger_spec_to_row(&spec).expect("to row");
assert_eq!(kind, ScriptTriggerKind::Asset);
let (rkind, rpath) = parse_asset_trigger_ref(&stored).expect("parse ref");
assert_eq!(rkind, AssetKind::S3Object);
// The producer path and the round-tripped consumer path must match.
assert_eq!(
rpath, path,
"round-trip diverged for {uri} (stored {stored})"
);
rpath
}
#[test]
fn s3_trigger_ref_roundtrips_for_every_uri_form() {
assert_eq!(roundtrip("s3:///exports/x"), "exports/x"); // SDK default storage
assert_eq!(roundtrip("s3://exports/x"), "exports/x"); // DuckDB / bare
assert_eq!(roundtrip("s3://mybucket/exports/x"), "mybucket/exports/x"); // explicit
assert_eq!(roundtrip("s3:////x"), "x"); // S3Object(s3="/x") quad-slash
assert_eq!(roundtrip("s3:///y=2024/f.parquet"), "y=2024/f.parquet"); // Hive
}
}
// Inverse of trigger_spec_to_row for the Asset variant: parses a stored
// trigger_ref (e.g. `s3://foo`, `$res:bar`) back into the (kind, path) pair
// used as a graph node id. Returns None for refs that don't match any known
// asset prefix — callers should skip those edges.
pub fn parse_asset_trigger_ref(s: &str) -> Option<(AssetKind, String)> {
let (kind, path) = windmill_parser::asset_parser::parse_asset_syntax(s, false)?;
Some((asset_kind_from_parser(kind), path.to_string()))
}
// Convert a parser TriggerSpec into the `(kind, ref)` pair stored in
// script_trigger. Asset refs get their canonical prefix back so the
// trigger_ref matches what downstream lookups expect.
//
// Returns `None` for native trigger kinds (Kafka, Mqtt, Postgres, …) —
// those annotations are marker-only and don't produce a `script_trigger`
// row. The actual binding lives on the trigger row's own `script_path`
// column; the graph endpoint looks it up directly per kind.
pub fn trigger_spec_to_row(spec: &TriggerSpec) -> Option<(ScriptTriggerKind, String)> {
match spec {
TriggerSpec::Asset { asset_kind, path, .. } => {
// Single source of truth for the canonical prefix lives on the
// common AssetKind; map the parser kind across first. The parser
// enum has no Variable variant, so canonical_prefix is always Some.
let prefix = asset_kind_from_parser(*asset_kind).canonical_prefix()?;
Some((ScriptTriggerKind::Asset, format!("{}{}", prefix, path)))
}
// Schedule joins the native-trigger family — no script_trigger row
// is inserted for the annotation. The binding lives on the schedule
// row's own `script_path` field, same as kafka/mqtt/etc.
TriggerSpec::Schedule
| TriggerSpec::Webhook
| TriggerSpec::Email
| TriggerSpec::Kafka
| TriggerSpec::Mqtt
| TriggerSpec::Nats
| TriggerSpec::Postgres
| TriggerSpec::Sqs
| TriggerSpec::Gcp
// data_upload is a UI-first marker — no event source, no trigger row.
// The script's S3Object input + auto-generated S3 picker drive it.
| TriggerSpec::DataUpload => None,
}
}
pub fn asset_kind_from_parser(parser_kind: windmill_parser::asset_parser::AssetKind) -> AssetKind {
match parser_kind {
windmill_parser::asset_parser::AssetKind::S3Object => AssetKind::S3Object,
windmill_parser::asset_parser::AssetKind::Resource => AssetKind::Resource,
windmill_parser::asset_parser::AssetKind::Ducklake => AssetKind::Ducklake,
windmill_parser::asset_parser::AssetKind::DataTable => AssetKind::DataTable,
windmill_parser::asset_parser::AssetKind::Volume => AssetKind::Volume,
}
}
pub fn asset_access_type_from_parser(
parser_kind: windmill_parser::asset_parser::AssetUsageAccessType,
) -> AssetUsageAccessType {
match parser_kind {
windmill_parser::asset_parser::AssetUsageAccessType::R => AssetUsageAccessType::R,
windmill_parser::asset_parser::AssetUsageAccessType::W => AssetUsageAccessType::W,
windmill_parser::asset_parser::AssetUsageAccessType::RW => AssetUsageAccessType::RW,
}
}