Ruben Fiszel 5a661279a3 feat(pipelines): add managed SCD2 history materialize strategy (#9850) ...

* feat(pipelines): add managed SCD2 history materialize strategy

`// materialize ducklake://... key=<col> history [track=...]` (alias: `scd2`)
upgrades the keyed merge to SCD type 2: diff the current snapshot against live
rows, close changed versions (valid_to/is_current) and open new ones in one
transaction, keeping full history. Adds a consumer-convenience <dim>_current
view; effective-dated joins via native ASOF JOIN >= valid_from. Managed, so
// data_test and schema capture work (unlike manual mode). Non-partitioned v1,
soft-delete on absence.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(pipelines): document scd2 track= spacing, reserved _current suffix, schema-freeze

Addresses non-blocking CI-review nits on the new SCD2 public surface.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipelines): null-safe scd2 key matching + create _current view inside txn

Addresses CI review: (1) Codex P1 — NULL natural keys were flagged as changed
but silently dropped because `key IN (...)` never matches NULL; close/open now
match with `IS NOT DISTINCT FROM` via correlated EXISTS. (2) cubic P2 — the
`<dim>_current` view was created after COMMIT and CREATE VIEW advances the
DuckLake snapshot, so the summary recorded the view's snapshot instead of the
data write; the view is now created inside the write transaction. Validated both
against a real DuckLake (NULL key materialized; one snapshot per run).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipelines): create scd2 _current view with IF NOT EXISTS to keep no-change runs no-op

Addresses CI review (Codex P2): CREATE OR REPLACE VIEW advances the DuckLake
snapshot every run, so an unchanged rerun still minted/recorded a snapshot. The
view definition is static, so IF NOT EXISTS creates it once (folded into the
first data-write snapshot) and is a true no-op thereafter — verified an unchanged
rerun keeps max(snapshot_id) constant. Also softens the reserved-name collision:
IF NOT EXISTS skips silently instead of erroring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(pipelines): add scd2 deletes=close (hard-delete-close)

Opt-in `deletes=close` closes the current version of a key that disappears from
the snapshot (dbt's hard_deletes=close); default stays soft-delete. Codegen adds
a vanished-key temp set (current keys EXCEPT snapshot keys) + a second null-safe
close UPDATE with no reopen; a reappearing key opens a fresh version (validity
gap = correct SCD2). Wired through both parsers with parity fixtures/tests, worker
derivation, unit + codegen tests, and docs. Verified end-to-end against a real
DuckLake incl. delete-close + reactivation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipelines): align materialize deploy precedence warning with runtime (scd2>append>merge)

The deploy-time conflict warning only knew append>key, so
warned 'append wins' while the runtime (duckdb_executor) runs SCD2 (history wins).
Warn for history+append (history wins, append ignored) before the append+key case,
mirroring the runtime strategy precedence. (Pi review P2.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipelines): register scd2 _current view as a produced asset for cascade dispatch

The docs present the companion <dim>_current view as a subscribable produced asset
(// on ducklake://.../<dim>_current), but deploy registered only the base table as
a write asset, so a subscriber on the view would never be dispatched (the cascade
fans out from deploy-time asset rows). Register <dim>_current as a produced write
asset when scd2 so those subscribers fire. (Codex review P1.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipelines): don't register _current asset for manual+history (no view created)

Manual mode short-circuits before the scd2 codegen, so no <dim>_current view is
created; gate the produced-asset registration on !manual so a contradictory
// materialize manual ... history doesn't leave a false write edge dispatching
subscribers on a nonexistent view. (Codex review P2.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-07-01 10:15:00 +02:00

..

src

feat(pipelines): add managed SCD2 history materialize strategy (#9850)

2026-07-01 10:15:00 +02:00

Cargo.toml

feat: ducklake materialization for data pipelines (#9689)

2026-06-20 15:42:03 +02:00