feat: recompute computed column rows whose inputs changed

refresh_column fills nulls, so once a row has a value nothing revisits
it: an update to one of its inputs, or a definition change, leaves the
computed value stale for good.

This stamps the column's field metadata with the definition it was
computed under and a per-fragment signature of the input storage it was
read from (input data files and overlays; not the deletion file, since
a delete changes no surviving value). A refresh recomputes every live
row of a fragment whose stamp disagrees with the manifest, then records
what it computed from in a second commit after the fill. A compacted
fragment built from signed fragments inherits their freshness through
the Rewrite lineage; one built from an unsigned fragment recomputes. A
column declared before the stamps existed keeps the null-fill contract
on its first refresh, which enrolls it as it stood.

The stamp is a metadata-only commit on the computed columns, so a
materialized view's drift check treats it like the fill. The core lives
in `table::freshness` so a remote refresh can share the contract.
This commit is contained in:
Wyatt Alt
2026-09-10 22:35:01 +00:00
parent d3077b7641
commit 2caf8aaf47
12 changed files with 1334 additions and 115 deletions
+8 -8
View File
@@ -74,10 +74,10 @@ now: the column is committed with no values, and rows get them from
[Table#refreshColumn](Table.md#refreshcolumn). Declaring one therefore costs the same on a
large table as on an empty one.
A refresh does not revisit rows it has already filled, so mutating an
input leaves the value computed at fill time; recomputing means dropping
the column and declaring it again. While a declaration reads a column,
that column cannot be renamed, retyped or dropped.
A refresh also recomputes the rows whose inputs changed since they were
computed, so a mutated input is reflected by the next refresh. While a
declaration reads a column, that column cannot be renamed, retyped or
dropped.
On LanceDB Cloud and Enterprise the expression is planned by the
server, and the refresh runs as a server job -- see
@@ -854,10 +854,10 @@ abstract refreshColumn(column): Promise<RefreshColumnResult>
Fill the rows of a computed column that hold no value yet.
Rows appended since the last refresh are filled by the next one; rows
already filled are left as they are, so the call is idempotent and does
not observe a mutated input. Local tables only: a remote refresh runs
as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
Rows appended since the last refresh are filled by the next one, and
rows whose inputs changed since they were computed are recomputed;
everything else is left as it is. Local tables only: a remote refresh
runs as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
#### Parameters