mirror of
https://github.com/lancedb/lancedb.git
synced 2026-09-11 07:42:26 +00:00
feat: recompute computed column rows whose inputs changed
refresh_column fills nulls, so once a row has a value nothing revisits it: an update to one of its inputs, or a definition change, leaves the computed value stale for good. This stamps the column's field metadata with the definition it was computed under and a per-fragment signature of the input storage it was read from (input data files and overlays; not the deletion file, since a delete changes no surviving value). A refresh recomputes every live row of a fragment whose stamp disagrees with the manifest, then records what it computed from in a second commit after the fill. A compacted fragment built from signed fragments inherits their freshness through the Rewrite lineage; one built from an unsigned fragment recomputes. A column declared before the stamps existed keeps the null-fill contract on its first refresh, which enrolls it as it stood. The stamp is a metadata-only commit on the computed columns, so a materialized view's drift check treats it like the fill. The core lives in `table::freshness` so a remote refresh can share the contract.
This commit is contained in:
@@ -74,10 +74,10 @@ now: the column is committed with no values, and rows get them from
|
||||
[Table#refreshColumn](Table.md#refreshcolumn). Declaring one therefore costs the same on a
|
||||
large table as on an empty one.
|
||||
|
||||
A refresh does not revisit rows it has already filled, so mutating an
|
||||
input leaves the value computed at fill time; recomputing means dropping
|
||||
the column and declaring it again. While a declaration reads a column,
|
||||
that column cannot be renamed, retyped or dropped.
|
||||
A refresh also recomputes the rows whose inputs changed since they were
|
||||
computed, so a mutated input is reflected by the next refresh. While a
|
||||
declaration reads a column, that column cannot be renamed, retyped or
|
||||
dropped.
|
||||
|
||||
On LanceDB Cloud and Enterprise the expression is planned by the
|
||||
server, and the refresh runs as a server job -- see
|
||||
@@ -854,10 +854,10 @@ abstract refreshColumn(column): Promise<RefreshColumnResult>
|
||||
|
||||
Fill the rows of a computed column that hold no value yet.
|
||||
|
||||
Rows appended since the last refresh are filled by the next one; rows
|
||||
already filled are left as they are, so the call is idempotent and does
|
||||
not observe a mutated input. Local tables only: a remote refresh runs
|
||||
as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
|
||||
Rows appended since the last refresh are filled by the next one, and
|
||||
rows whose inputs changed since they were computed are recomputed;
|
||||
everything else is left as it is. Local tables only: a remote refresh
|
||||
runs as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
|
||||
|
||||
#### Parameters
|
||||
|
||||
|
||||
Reference in New Issue
Block a user