rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2026-09-11 07:42:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: recompute computed column rows whose inputs changed

Browse Source 
refresh_column fills nulls, so once a row has a value nothing revisits
it: an update to one of its inputs, or a definition change, leaves the
computed value stale for good.

This stamps the column's field metadata with the definition it was
computed under and a per-fragment signature of the input storage it was
read from (input data files and overlays; not the deletion file, since
a delete changes no surviving value). A refresh recomputes every live
row of a fragment whose stamp disagrees with the manifest, then records
what it computed from in a second commit after the fill. A compacted
fragment built from signed fragments inherits their freshness through
the Rewrite lineage; one built from an unsigned fragment recomputes. A
column declared before the stamps existed keeps the null-fill contract
on its first refresh, which enrolls it as it stood.

The stamp is a metadata-only commit on the computed columns, so a
materialized view's drift check treats it like the fill. The core lives
in `table::freshness` so a remote refresh can share the contract.
This commit is contained in:
Wyatt Alt
2026-09-10 22:35:01 +00:00
parent d3077b7641
commit 2caf8aaf47
12 changed files with 1334 additions and 115 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nodejs/lancedb/table.ts
+8 -8
View File
@@ -542,10 +542,10 @@ export abstract class Table {
* {@link Table#refreshColumn}. Declaring one therefore costs the same on a
* large table as on an empty one.
*
* A refresh does not revisit rows it has already filled, so mutating an
* input leaves the value computed at fill time; recomputing means dropping
* the column and declaring it again. While a declaration reads a column,
* that column cannot be renamed, retyped or dropped.
* A refresh also recomputes the rows whose inputs changed since they were
* computed, so a mutated input is reflected by the next refresh. While a
* declaration reads a column, that column cannot be renamed, retyped or
* dropped.
*
* On LanceDB Cloud and Enterprise the expression is planned by the
* server, and the refresh runs as a server job -- see
@@ -576,10 +576,10 @@ export abstract class Table {
/**
* Fill the rows of a computed column that hold no value yet.
*
* Rows appended since the last refresh are filled by the next one; rows
* already filled are left as they are, so the call is idempotent and does
* not observe a mutated input. Local tables only: a remote refresh runs
* as a server job, through {@link Table#refreshColumnAsync}.
* Rows appended since the last refresh are filled by the next one, and
* rows whose inputs changed since they were computed are recomputed;
* everything else is left as it is. Local tables only: a remote refresh
* runs as a server job, through {@link Table#refreshColumnAsync}.
* @param {string} column The name of the computed column to fill.
* @returns {Promise<RefreshColumnResult>} A promise that resolves to the
* number of rows filled and the new version number of the table.