feat: recompute computed column rows whose inputs changed

refresh_column fills nulls, so once a row has a value nothing revisits
it: an update to one of its inputs, or a definition change, leaves the
computed value stale for good.

This stamps the column's field metadata with the definition it was
computed under and a per-fragment signature of the input storage it was
read from (input data files and overlays; not the deletion file, since
a delete changes no surviving value). A refresh recomputes every live
row of a fragment whose stamp disagrees with the manifest, then records
what it computed from in a second commit after the fill. A compacted
fragment inherits freshness through the Rewrite lineage when every
fragment it was built from was signed, or was appended since the stamp,
never had an input moved, and left its rows of the product unfilled (a
raw append may supply a value; the product's data is the evidence, and
the null fill covers those rows); otherwise it recomputes. A
column declared before the stamps existed keeps the null-fill contract
on its first refresh, which enrolls it as it stood.

The stamp is a metadata-only commit on the computed columns, so a
materialized view's drift check treats it like the fill. The core lives
in `table::freshness` so a remote refresh can share the contract.

Cargo.lock

@@ -5553,6 +5553,7 @@ dependencies = [
  "serde_json",
  "serde_with",
  "serial_test",
+ "sha2 0.10.9",
  "snafu 0.8.9",
  "tempfile",
  "test-log",

docs/src/js/classes/Table.md

@@ -74,10 +74,10 @@ now: the column is committed with no values, and rows get them from
 [Table#refreshColumn](Table.md#refreshcolumn). Declaring one therefore costs the same on a
 large table as on an empty one.
 
-A refresh does not revisit rows it has already filled, so mutating an
-input leaves the value computed at fill time; recomputing means dropping
-the column and declaring it again. While a declaration reads a column,
-that column cannot be renamed, retyped or dropped.
+A refresh also recomputes the rows whose inputs changed since they were
+computed, so a mutated input is reflected by the next refresh. While a
+declaration reads a column, that column cannot be renamed, retyped or
+dropped.
 
 On LanceDB Cloud and Enterprise the expression is planned by the
 server, and the refresh runs as a server job -- see
@@ -854,10 +854,10 @@ abstract refreshColumn(column): Promise<RefreshColumnResult>
 
 Fill the rows of a computed column that hold no value yet.
 
-Rows appended since the last refresh are filled by the next one; rows
-already filled are left as they are, so the call is idempotent and does
-not observe a mutated input. Local tables only: a remote refresh runs
-as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
+Rows appended since the last refresh are filled by the next one, and
+rows whose inputs changed since they were computed are recomputed;
+everything else is left as it is. Local tables only: a remote refresh
+runs as a server job, through [Table#refreshColumnAsync](Table.md#refreshcolumnasync).
 
 #### Parameters
 

nodejs/lancedb/table.ts

@@ -542,10 +542,10 @@ export abstract class Table {
    * {@link Table#refreshColumn}. Declaring one therefore costs the same on a
    * large table as on an empty one.
    *
-   * A refresh does not revisit rows it has already filled, so mutating an
-   * input leaves the value computed at fill time; recomputing means dropping
-   * the column and declaring it again. While a declaration reads a column,
-   * that column cannot be renamed, retyped or dropped.
+   * A refresh also recomputes the rows whose inputs changed since they were
+   * computed, so a mutated input is reflected by the next refresh. While a
+   * declaration reads a column, that column cannot be renamed, retyped or
+   * dropped.
    *
    * On LanceDB Cloud and Enterprise the expression is planned by the
    * server, and the refresh runs as a server job -- see
@@ -576,10 +576,10 @@ export abstract class Table {
   /**
    * Fill the rows of a computed column that hold no value yet.
    *
-   * Rows appended since the last refresh are filled by the next one; rows
-   * already filled are left as they are, so the call is idempotent and does
-   * not observe a mutated input. Local tables only: a remote refresh runs
-   * as a server job, through {@link Table#refreshColumnAsync}.
+   * Rows appended since the last refresh are filled by the next one, and
+   * rows whose inputs changed since they were computed are recomputed;
+   * everything else is left as it is. Local tables only: a remote refresh
+   * runs as a server job, through {@link Table#refreshColumnAsync}.
    * @param {string} column The name of the computed column to fill.
    * @returns {Promise<RefreshColumnResult>} A promise that resolves to the
    * number of rows filled and the new version number of the table.

python/python/lancedb/table.py

@@ -2188,10 +2188,10 @@ class Table(ABC):
             Declaring one therefore costs the same on a large table as on an
             empty one.
 
-            A refresh does not revisit rows it has already filled, so mutating
-            an input leaves the value computed at fill time; recomputing means
-            dropping the column and declaring it again. While a declaration
-            reads a column, that column cannot be renamed, retyped or dropped.
+            A refresh also recomputes the rows whose inputs changed since they
+            were computed, so a mutated input is reflected by the next refresh.
+            While a declaration reads a column, that column cannot be renamed,
+            retyped or dropped.
 
             On LanceDB Cloud and Enterprise the expression is planned by the
             server, and the refresh runs as a server job -- see
@@ -2211,7 +2211,7 @@ class Table(ABC):
         >>> table.add_columns(computed={"doubled": "x * 2"})
         AddColumnsResult(version=2)
         >>> table.refresh_column("doubled")
-        RefreshColumnResult(rows_filled=2, version=3)
+        RefreshColumnResult(rows_filled=2, version=4)
         >>> table.to_arrow().sort_by("x").to_pandas()
            x  doubled
         0  1        2
@@ -2225,8 +2225,8 @@ class Table(ABC):
 
         Declared with ``add_columns(computed=...)``, a column starts empty and
         gets its values here. Rows appended since the last refresh are filled
-        by the next one; rows already filled are left as they are, so the call
-        is idempotent and does not observe a mutated input.
+        by the next one, and rows whose inputs changed since they were computed
+        are recomputed; everything else is left as it is.
 
         Local tables only: a remote refresh runs as a server job, through
         [`refresh_column_async`][lancedb.table.Table.refresh_column_async].
@@ -4318,13 +4318,14 @@ class LanceTable(Table):
         return LOOP.run(self._table.add_columns(transforms, computed=computed))
 
     def refresh_column(self, column: str) -> "RefreshColumnResult":
-        """Fill a computed column's unfilled rows. See
+        """Fill a computed column's unfilled rows and recompute those whose
+        inputs changed. See
         [`AsyncTable.refresh_column`][lancedb.AsyncTable.refresh_column]."""
         return LOOP.run(self._table.refresh_column(column))
 
     def refresh_column_async(self, column: str) -> Job[RefreshColumnJobResult]:
-        """Fill a computed column's unfilled rows, returning a handle to the
-        refresh job. See
+        """Fill a computed column's unfilled rows and recompute those whose
+        inputs changed, returning a handle to the refresh job. See
         [`Table.refresh_column_async`][lancedb.table.Table.refresh_column_async].
         """
         return Job(LOOP.run(self._table.refresh_column_async(column)))
@@ -6312,10 +6313,10 @@ class AsyncTable:
             them from
             [`refresh_column`][lancedb.table.AsyncTable.refresh_column].
 
-            A refresh does not revisit rows it has already filled, so mutating
-            an input leaves the value computed at fill time. While a
-            declaration reads a column, that column cannot be renamed, retyped
-            or dropped.
+            A refresh also recomputes the rows whose inputs changed since they
+            were computed, so a mutated input is reflected by the next refresh.
+            While a declaration reads a column, that column cannot be renamed,
+            retyped or dropped.
 
             On LanceDB Cloud and Enterprise the expression is planned by
             the server. Cannot be combined with ``transforms``.
@@ -6377,8 +6378,8 @@ class AsyncTable:
 
         Declared with ``add_columns(computed=...)``, a column starts empty and
         gets its values here. Rows appended since the last refresh are filled
-        by the next one; rows already filled are left as they are, so the call
-        is idempotent and does not observe a mutated input.
+        by the next one, and rows whose inputs changed since they were computed
+        are recomputed; everything else is left as it is.
 
         Local tables only: a remote refresh runs as a server job, through
         [`refresh_column_async`][lancedb.table.Table.refresh_column_async].

python/python/tests/test_table.py

@@ -4183,13 +4183,14 @@ def test_refresh_column_async_returns_job(tmp_path):
     assert result.rows_failed == 0
     assert result.rows_remaining == 0
     assert result.source_version == 2
-    assert result.published_version == 3
+    # The fill lands at 3; the stamp recording its inputs is published at 4.
+    assert result.published_version == 4
     assert job.status() == "finished"
     assert sorted(table.to_arrow()["doubled"].to_pylist()) == [2, 4]
 
     no_op = table.refresh_column_async("doubled").wait()
     assert no_op.rows_assigned == 0
-    assert no_op.source_version == 3
+    assert no_op.source_version == 4
     assert no_op.published_version is None
 
     # Bad input raises at the call, not through the job.
@@ -4208,6 +4209,6 @@ async def test_refresh_column_async_job_async_table(tmp_path):
     assert isinstance(result, lancedb.RefreshColumnResult)
     assert result.rows_assigned == 1
     assert result.source_version == 2
-    assert result.published_version == 3
+    assert result.published_version == 4
     assert await job.status() == "finished"
     assert (await table.to_arrow())["tripled"].to_pylist() == [9]

rust/lancedb/Cargo.toml

@@ -95,13 +95,14 @@ candle-transformers = { version = "0.9.1", optional = true }
 candle-nn = { version = "0.9.1", optional = true }
 tokenizers = { version = "0.19.1", optional = true }
 semver = { workspace = true }
+roaring = "0.11.4"
+sha2 = "0.10"
 
 [dev-dependencies]
 anyhow = "1"
 lance-testing = { workspace = true }
 tempfile = { workspace = true }
 random_word = { version = "0.4.3", features = ["en"] }
-roaring = "0.11.4"
 tokio = { workspace = true, features = ["io-util", "macros", "net", "test-util"] }
 uuid = { workspace = true }
 walkdir = "2"

rust/lancedb/src/materialized_view/refresh.rs

@@ -1124,8 +1124,9 @@ struct RowScope {
 
 /// Whether every commit on the view after `recorded` is a fill of its
 /// computed columns: a column rewrite or data replacement touching only
-/// those fields and neither adding nor removing rows. A version whose
-/// transaction cannot be read is not proven, so it counts as drift.
+/// those fields and neither adding nor removing rows, or the freshness
+/// stamp a fill leaves on them. A version whose transaction cannot be read
+/// is not proven, so it counts as drift.
 async fn only_computed_rewrites_since(view_ds: &Dataset, recorded: u64) -> Result<bool> {
     // A fill may write any field under a computed column, so the whole
     // subtree counts, not only the root.
@@ -1176,6 +1177,19 @@ async fn only_computed_rewrites_since(view_ds: &Dataset, recorded: u64) -> Resul
                                 .all(|field| computed_fields.contains(&(*field as u32)))
                     })
             }
+            // The stamp `refresh_column` writes after its fill (see
+            // `table::freshness`): field metadata on computed columns, no data.
+            Operation::UpdateConfig {
+                config_updates: None,
+                table_metadata_updates: None,
+                schema_metadata_updates: None,
+                field_metadata_updates,
+            } => {
+                !field_metadata_updates.is_empty()
+                    && field_metadata_updates
+                        .keys()
+                        .all(|field| computed_fields.contains(&(*field as u32)))
+            }
             _ => false,
         };
         if !fill {
@@ -3359,6 +3373,46 @@ mod tests {
         );
     }
 
+    /// Field metadata on `field` only, the commit shape of the freshness
+    /// stamp `refresh_column` leaves after its fill.
+    async fn commit_field_metadata(view: &MaterializedView, field: &str, key: &str) {
+        let native = view.table().as_native().unwrap();
+        native.dataset.reload().await.unwrap();
+        let mut dataset = native.dataset.get().await.unwrap().as_ref().clone();
+        dataset
+            .update_field_metadata()
+            .update(field, [(key.to_string(), "{}".to_string())])
+            .unwrap()
+            .await
+            .unwrap();
+    }
+
+    /// The stamp is metadata on the computed column and rewrites nothing
+    /// refresh certifies, so it is not drift; the same commit shape on a
+    /// projected column is, like any other write to it.
+    #[tokio::test]
+    async fn test_a_freshness_stamp_is_not_drift() {
+        let conn = connect("memory://").execute().await.unwrap();
+        let view = refreshed_computed_view(&conn).await;
+
+        commit_field_metadata(
+            &view,
+            "emb",
+            crate::table::computed_columns::SOURCE_SIGNATURE_META_KEY,
+        )
+        .await;
+        assert_eq!(
+            view.refresh().execute().await.unwrap().mode,
+            RefreshMode::NoOp
+        );
+
+        commit_field_metadata(&view, "id", "probe").await;
+        assert_eq!(
+            view.refresh().execute().await.unwrap().mode,
+            RefreshMode::Rebuild
+        );
+    }
+
     /// The fill job's commit rewrites only computed columns. It is the one
     /// commit on a view that is not drift: the next refresh carries on from
     /// its watermark instead of rebuilding, which would null what the fill
@@ -3524,9 +3578,9 @@ mod tests {
     }
 
     /// A SQL declaration is filled by `refresh_column` on the view, which
-    /// commits a data replacement; the next refresh continues from its
-    /// watermark and keeps what the fill wrote, and only rows the view added
-    /// since come back unfilled.
+    /// commits a data replacement and then its freshness stamp; the next
+    /// refresh continues from its watermark and keeps what the fill wrote,
+    /// and only rows the view added since come back unfilled.
     #[tokio::test]
     async fn test_a_sql_fill_is_not_drift() {
         use crate::materialized_view::tests::{people, sql_field};

rust/lancedb/src/table.rs

@@ -75,6 +75,7 @@ mod create_index;
 pub mod datafusion;
 pub(crate) mod dataset;
 pub mod delete;
+pub mod freshness;
 pub mod lsm_stats;
 pub mod merge;
 pub mod optimize;
@@ -778,7 +779,8 @@ pub trait BaseTable: std::fmt::Display + std::fmt::Debug + Send + Sync {
             message: "Function columns are supported only on LanceDB Cloud and Enterprise".into(),
         })
     }
-    /// Fill a computed column's unfilled rows.
+    /// Fill a computed column's unfilled rows and recompute those whose
+    /// inputs changed.
     ///
     /// The default returns `NotSupported`; Lance-backed tables override it.
     async fn refresh_column(&self, _column: &str) -> Result<RefreshColumnResult> {
@@ -786,8 +788,8 @@ pub trait BaseTable: std::fmt::Display + std::fmt::Debug + Send + Sync {
             message: "computed columns are supported only on local tables".into(),
         })
     }
-    /// Fill a computed column's unfilled rows, returning a [`Job`] tracking
-    /// the operation.
+    /// Fill a computed column's unfilled rows and recompute those whose
+    /// inputs changed, returning a [`Job`] tracking the operation.
     async fn refresh_column_async(
         &self,
         _column: &str,
@@ -1749,9 +1751,10 @@ impl Table {
     /// Declared with
     /// [`AddColumnsBuilder::computed`](add_columns::AddColumnsBuilder::computed),
     /// a column starts empty and gets its values here. Fragments appended
-    /// since the last refresh are filled by the next one; fragments already
-    /// filled are left as they are, so the call is idempotent and does not
-    /// observe a mutated input.
+    /// since the last refresh are filled by the next one, and fragments whose
+    /// inputs changed since they were computed are recomputed (see
+    /// [`freshness`](crate::table::freshness)); everything else is left as
+    /// it is.
     ///
     /// Local tables only: a remote refresh runs as a server job, through
     /// [`Table::refresh_column_async`].

rust/lancedb/src/table/add_columns.rs

@@ -60,10 +60,11 @@ impl AddColumnsBuilder {
     /// every fragment that has none -- including fragments appended since the
     /// last refresh.
     ///
-    /// Refresh does not revisit a fragment it has filled, so mutating an input
-    /// leaves the value computed at fill time; recomputing means dropping the
-    /// column and declaring it again. An input cannot be renamed, retyped or
-    /// dropped while a declaration reads it, since the expression names it.
+    /// A refresh also recomputes the rows of a fragment whose inputs changed
+    /// since it was computed (see [`freshness`](super::freshness)), so a
+    /// mutated input is reflected by the next refresh. An input cannot be
+    /// renamed, retyped or dropped while a declaration reads it, since the
+    /// expression names it.
     ///
     /// On LanceDB Cloud and Enterprise the expression is planned by the
     /// server, and the refresh runs as a server job -- see

rust/lancedb/src/table/computed_columns.rs

@@ -71,6 +71,22 @@ pub const FUNCTION_BINDINGS_META_KEY: &str = "lancedb::function_bindings";
 /// Version of the schema-level Function binding envelope.
 pub const FUNCTION_BINDINGS_VERSION: u32 = 1;
 
+/// Field metadata key holding `{fragment id -> input signature}` as JSON,
+/// recorded by the refresh that last computed each fragment. Outside the
+/// declaration namespace on purpose: a declaration is immutable through
+/// metadata edits, this is rewritten by every refresh. Seeded empty at
+/// declaration, so a column is tracked from birth; a column without it was
+/// declared before signatures existed.
+pub const SOURCE_SIGNATURE_META_KEY: &str = "computed_refresh.source_signature";
+
+/// Field metadata key holding the definition digest a column was last
+/// computed under. A change to it makes every row stale.
+pub const DEFINITION_VERSION_META_KEY: &str = "computed_refresh.definition_version";
+
+/// Field metadata key holding the table version the signature map describes:
+/// where a refresh starts following compactions to carry freshness forward.
+pub const RECORDED_AT_VERSION_META_KEY: &str = "computed_refresh.recorded_at_version";
+
 /// Value of [`KIND_META_KEY`] for a column defined by a SQL expression.
 pub const SQL_KIND: &str = "sql";
 
@@ -139,6 +155,7 @@ fn computed_column_metadata(expression: &str, inputs: &[String]) -> HashMap<Stri
             INPUTS_META_KEY.to_string(),
             serde_json::to_string(inputs).unwrap_or_else(|_| "[]".to_string()),
         ),
+        (SOURCE_SIGNATURE_META_KEY.to_string(), "{}".to_string()),
     ])
 }
 
@@ -163,6 +180,7 @@ pub fn function_computed_column_metadata(
             INPUTS_META_KEY.to_string(),
             serde_json::to_string(inputs).unwrap_or_else(|_| "[]".to_string()),
         ),
+        (SOURCE_SIGNATURE_META_KEY.to_string(), "{}".to_string()),
     ])
 }
 
@@ -1295,7 +1313,8 @@ pub(crate) fn ensure_not_an_input(schema: &SchemaRef, paths: &[&str]) -> Result<
 }
 
 /// Reject a write that supplies values for a computed column directly:
-/// only refresh materializes one, and refresh never revisits a filled row.
+/// only refresh materializes one, and only refresh decides what it
+/// recomputes.
 pub(crate) fn ensure_not_written<'a>(
     schema: &ArrowSchema,
     written: impl IntoIterator<Item = &'a str>,
@@ -1470,7 +1489,9 @@ fn ensure_no_foreign_declaration(field: &ArrowField) -> Result<()> {
 /// kind, the expression, the inputs -- would bypass that validation or move
 /// a binding out from under a refresh. Drop the column and declare it again.
 pub(crate) fn is_declaration_key(key: &str) -> bool {
-    key == COMPUTED_COLUMN_META_KEY || key.starts_with("computed_column.")
+    key == COMPUTED_COLUMN_META_KEY
+        || key.starts_with("computed_column.")
+        || key.starts_with("computed_refresh.")
 }
 
 /// Reject retyping a computed column itself.

rust/lancedb/src/table/refresh.rs

@@ -3,9 +3,9 @@
 
 //! Filling computed columns.
 //!
-//! A row without a value gets one; a row that has one keeps it. Refresh is
-//! therefore idempotent and does not observe input mutation -- once a row is
-//! filled, changing what the expression reads leaves the stored result alone.
+//! A row without a value gets one; a row that has one keeps it unless its
+//! fragment's inputs moved since it was computed, which `freshness` decides
+//! from the manifest and stamps after every fill.
 //!
 //! A column's computed inputs are filled first -- the dependency graph is
 //! walked once, each reachable column filled once in dependency order, each
@@ -31,6 +31,7 @@
 
 use std::collections::HashSet;
 use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
 
 use arrow_array::{
     Array, ArrayRef, BooleanArray, LargeBinaryArray, RecordBatch, RecordBatchOptions, StructArray,
@@ -48,6 +49,7 @@ use lance_core::datatypes::{BlobHandling, Schema as LanceSchema};
 use serde::{Deserialize, Serialize};
 
 use super::computed_columns::{BoundExpression, ComputedColumnKind, computed_column_from_field};
+use super::freshness::{self, SignatureMap, StalenessPlan};
 use super::{BaseTable, NativeTable};
 use crate::job::Job;
 use crate::{Error, Result};
@@ -110,28 +112,81 @@ async fn execute_refresh_column_with_source(
     };
     let output_is_blob = field.is_blob_v2();
 
+    // Which fragments the null filter cannot speak for: their inputs moved
+    // since they were computed, or the definition did. Decided once, from
+    // the manifest the values are read from.
+    let inputs = freshness::fields_for_paths(dataset.schema(), &bound.inputs)?;
+    let definition = freshness::definition_version(&expression);
+    let staleness = freshness::staleness_against(&dataset, column, &definition, &inputs).await?;
+
     let mut rows_filled = 0u64;
     let mut replacements = Vec::new();
+    // Fragments this refresh computed in full, signed at the version read.
+    let mut computed = SignatureMap::new();
     for fragment in dataset.get_fragments() {
-        let gained = count_fragment_gains(&dataset, &fragment, &bound, column).await?;
-        if gained == 0 {
-            continue;
+        let fragment_id = u32::try_from(fragment.id()).map_err(|_| Error::Runtime {
+            message: format!("fragment id {} does not fit a signature map", fragment.id()),
+        })?;
+        // A recompute rewrites every live row, so it is staged without the
+        // probe and counted as it fills; a null fill probes first, since a
+        // fragment with nothing to gain is not worth a write.
+        let recompute = staleness.is_dirty(fragment_id);
+        let whole = recompute || {
+            let (gained, unfilled) =
+                count_fragment_gains(&dataset, &fragment, &bound, column).await?;
+            if gained == 0 {
+                continue;
+            }
+            rows_filled += gained;
+            unfilled == u64::try_from(fragment.count_rows(None).await?).unwrap_or(u64::MAX)
+        };
+        if whole {
+            computed.insert(
+                fragment_id,
+                freshness::fragment_input_signature(fragment.metadata(), &inputs)?,
+            );
         }
-        rows_filled += gained;
-        let values =
-            fill_stream(&dataset, &fragment, bound.clone(), column, output_is_blob).await?;
+        let gained = Arc::new(AtomicU64::new(0));
+        let values = fill_stream(
+            &dataset,
+            &fragment,
+            bound.clone(),
+            column,
+            output_is_blob,
+            recompute,
+            gained.clone(),
+        )
+        .await?;
         replacements.push(fragment.write_columns(values, &column_schema).await?);
+        if recompute {
+            rows_filled += gained.load(Ordering::Relaxed);
+        }
     }
 
     let source_version = dataset.version().version;
     if replacements.is_empty() {
+        // Nothing to fill; the stamp may still have something to record -- a
+        // column not yet enrolled, or fragments a compaction carried.
+        let mut latest = (*dataset).clone();
+        let stamped = record(
+            &mut latest,
+            (&dataset, &staleness),
+            column,
+            &definition,
+            &inputs,
+            computed,
+        )
+        .await;
+        if stamped.is_some() {
+            table.dataset.update(latest);
+        }
         return Ok(RefreshExecution {
             result: RefreshColumnResult {
                 rows_filled: 0,
-                version: source_version,
+                version: stamped.unwrap_or(source_version),
             },
             source_version,
-            published_version: None,
+            published_version: stamped,
         });
     }
 
@@ -149,7 +204,17 @@ async fn execute_refresh_column_with_source(
     )
     .await?;
 
-    let version = new_dataset.version().version;
+    let mut new_dataset = new_dataset;
+    let version = record(
+        &mut new_dataset,
+        (&dataset, &staleness),
+        column,
+        &definition,
+        &inputs,
+        computed,
+    )
+    .await
+    .unwrap_or(new_dataset.version().version);
     table.dataset.update(new_dataset);
     Ok(RefreshExecution {
         result: RefreshColumnResult {
@@ -161,6 +226,31 @@ async fn execute_refresh_column_with_source(
     })
 }
 
+/// Stamp the input state the refresh computed from (see
+/// [`freshness::record_freshness`]); the version the stamp landed at, which
+/// is the last one the refresh wrote. Never fails the refresh: the values
+/// are committed, and a missing stamp only costs a recompute next time.
+async fn record(
+    latest: &mut Dataset,
+    pinned: (&Dataset, &StalenessPlan),
+    column: &str,
+    definition: &str,
+    inputs: &freshness::InputFields,
+    computed: SignatureMap,
+) -> Option<u64> {
+    match freshness::record_freshness(latest, Some(pinned), column, definition, inputs, computed)
+        .await
+    {
+        Ok(record) => record.version,
+        Err(error) => {
+            log::warn!(
+                "could not record the input state computed column '{column}' was refreshed from ({error}); its fragments will recompute on the next refresh"
+            );
+            None
+        }
+    }
+}
+
 /// Refuse while a computed input still has rows a refresh of it would fill:
 /// read now, its placeholder null would be evaluated as a value and kept.
 async fn ensure_inputs_filled(
@@ -188,7 +278,9 @@ async fn ensure_inputs_filled(
         let input_bound = super::computed_columns::bind(schema.clone(), input, expression)?;
         let mut unfilled = 0u64;
         for fragment in dataset.get_fragments() {
-            unfilled += count_fragment_gains(dataset, &fragment, &input_bound, input).await?;
+            unfilled += count_fragment_gains(dataset, &fragment, &input_bound, input)
+                .await?
+                .0;
         }
         if unfilled > 0 {
             return Err(Error::InvalidInput {
@@ -436,32 +528,35 @@ fn blob_array_from_binary(
 /// Scans only the unfilled live rows -- deleted rows never reach the
 /// expression here, the filter having already excluded them -- and counts the
 /// non-null results. Exact, so it is both the staging decision and the
-/// fragment's contribution to `rows_filled`.
+/// fragment's contribution to `rows_filled`. Returns the gains and the rows
+/// scanned.
 async fn count_fragment_gains(
     dataset: &Dataset,
     fragment: &FileFragment,
     bound: &BoundExpression,
     column: &str,
-) -> Result<u64> {
+) -> Result<(u64, u64)> {
     let mut scanner = dataset.scan();
     scanner
         .with_fragments(vec![fragment.metadata().clone()])
         .with_row_id()
-        .filter(&format!("{} IS NULL", quote_identifier(column)))?
-        .project(&bound.roots)?;
+        .project(&bound.roots)?
+        .filter(&format!("{} IS NULL", quote_identifier(column)))?;
     configure_blob_inputs(&mut scanner, dataset.schema(), bound, None)?;
 
     let mut gained = 0u64;
+    let mut considered = 0u64;
     let mut batches = scanner.try_into_stream().await?;
     while let Some(batch) = batches.try_next().await? {
         let evaluated = evaluate(bound, &evaluation_batch(&batch, bound, None)?)?;
         gained += (batch.num_rows() - evaluated.null_count()) as u64;
+        considered += batch.num_rows() as u64;
     }
-    Ok(gained)
+    Ok((gained, considered))
 }
 
 /// Stream one fragment's column in physical order, filling the unfilled live
-/// rows and keeping every other value.
+/// rows -- every live row, for a recompute -- and keeping every other value.
 ///
 /// Deleted rows are carried through so the values line up positionally with
 /// the fragment's data files; they are never read back, but the column file
@@ -472,6 +567,8 @@ async fn fill_stream(
     bound: Arc<BoundExpression>,
     column: &str,
     output_is_blob: bool,
+    recompute: bool,
+    gained: Arc<AtomicU64>,
 ) -> Result<impl Stream<Item = lance_core::Result<RecordBatch>> + Send + use<>> {
     let mut projection: Vec<String> = bound.roots.clone();
     projection.push(column.to_string());
@@ -521,14 +618,20 @@ async fn fill_stream(
             .column_by_name(ROW_ID)
             .ok_or_else(|| missing(ROW_ID))?;
 
-        // Only an unfilled live row gains a value; a deleted row has a null
-        // row id and keeps its (null) slot.
-        let unfilled = arrow::compute::is_null(existing.as_ref())?;
+        // Only an unfilled live row gains a value, or every live row under a
+        // recompute; a deleted row has a null row id and keeps its (null) slot.
         let live = arrow::compute::is_not_null(row_ids.as_ref())?;
-        let fill = arrow::compute::and(&unfilled, &live)?;
+        let fill = if recompute {
+            live
+        } else {
+            let unfilled = arrow::compute::is_null(existing.as_ref())?;
+            arrow::compute::and(&unfilled, &live)?
+        };
         let keep = arrow::compute::not(&fill)?;
 
         let computed = evaluate(&bound, &evaluation_batch(&batch, &bound, Some(&keep))?)?;
+        let values = arrow::compute::and(&fill, &arrow::compute::is_not_null(&computed)?)?;
+        gained.fetch_add(values.true_count() as u64, Ordering::Relaxed);
         let merged = arrow_select::zip::zip(&fill, &computed, existing)?;
         let merged = if output_is_blob {
             blob_array_from_binary(&merged, projected.field(0))?
@@ -737,7 +840,8 @@ mod tests {
             .await
             .unwrap();
         assert_eq!(no_op.rows_assigned, 0);
-        assert_eq!(no_op.source_version, 3);
+        // The fill, then the stamp recording what it computed from.
+        assert_eq!(no_op.source_version, 4);
         assert_eq!(no_op.published_version, None);
     }
 
@@ -777,8 +881,8 @@ mod tests {
     }
 
     /// A row is filled only by gaining a value, so an expression yielding null
-    /// settles at once instead of re-selecting the same rows forever. Nothing
-    /// is staged, so the version does not move either.
+    /// settles at once instead of re-selecting the same rows forever: the
+    /// second refresh finds the fragment signed and moves nothing.
     #[tokio::test]
     async fn test_refresh_converges_on_a_null_result() {
         let table = table_with("refresh_null_result", vec![1, 2, 3]).await;
@@ -792,28 +896,145 @@ mod tests {
 
         let first = table.refresh_column("maybe").await.unwrap();
         assert_eq!(first.rows_filled, 0);
-        assert_eq!(first.version, declared);
+        assert!(first.version > declared);
         assert_eq!(read(&table, "maybe").await, vec![None, None, None]);
 
         let again = table.refresh_column("maybe").await.unwrap();
         assert_eq!(again.rows_filled, 0);
-        assert_eq!(again.version, declared);
+        assert_eq!(again.version, first.version);
     }
 
-    /// The contract's boundary: a filled fragment is not revisited, so
-    /// mutating an input leaves the value computed at fill time.
+    /// A filled row whose input moved is recomputed: the update rewrites
+    /// the row into a fragment the stamp never signed, and only that one.
     #[tokio::test]
-    async fn test_refresh_does_not_observe_input_mutation() {
-        let table = table_with("refresh_mutation", vec![1]).await;
+    async fn test_refresh_recomputes_a_row_whose_input_moved() {
+        let table = table_with("refresh_mutation", vec![1, 2]).await;
         declare_doubled(&table).await.unwrap();
+        append(&table, vec![5]).await;
         table.refresh_column("doubled").await.unwrap();
-        assert_eq!(read(&table, "doubled").await, vec![Some(2)]);
+        assert_eq!(
+            read(&table, "doubled").await,
+            vec![Some(2), Some(4), Some(10)]
+        );
 
-        table.update().column("x", "3").execute().await.unwrap();
+        table
+            .update()
+            .column("x", "7")
+            .only_if("x = 5")
+            .execute()
+            .await
+            .unwrap();
+
+        let again = table.refresh_column("doubled").await.unwrap();
+        assert_eq!(again.rows_filled, 1);
+        assert_eq!(
+            read(&table, "doubled").await,
+            vec![Some(2), Some(4), Some(14)]
+        );
+
+        let settled = table.refresh_column("doubled").await.unwrap();
+        assert_eq!(settled.rows_filled, 0);
+        assert_eq!(settled.version, again.version);
+    }
+
+    /// A deleted row is never computed and the rows that stay keep their
+    /// values: a delete recomputes nothing and stamps nothing.
+    #[tokio::test]
+    async fn test_a_delete_recomputes_nothing() {
+        let table = table_with("refresh_delete", vec![1, 2, 3]).await;
+        declare_doubled(&table).await.unwrap();
+        let filled = table.refresh_column("doubled").await.unwrap();
+
+        table.delete("x = 2").await.unwrap();
+        let deleted = table.version().await.unwrap();
 
         let again = table.refresh_column("doubled").await.unwrap();
         assert_eq!(again.rows_filled, 0);
-        assert_eq!(read(&table, "doubled").await, vec![Some(2)]);
+        assert_eq!(again.version, deleted);
+        assert!(deleted > filled.version);
+        assert_eq!(read(&table, "doubled").await, vec![Some(2), Some(6)]);
+    }
+
+    /// Compaction copies inputs unchanged, so a fragment it builds from
+    /// signed ones is fresh: the refresh recomputes nothing and only records
+    /// the new fragment.
+    #[tokio::test]
+    async fn test_a_compaction_of_signed_fragments_recomputes_nothing() {
+        let table = table_with("refresh_compact_signed", vec![1, 2]).await;
+        declare_doubled(&table).await.unwrap();
+        append(&table, vec![5]).await;
+        table.refresh_column("doubled").await.unwrap();
+
+        table
+            .optimize(crate::table::OptimizeAction::Compact {
+                options: crate::table::CompactionOptions::default(),
+                remap_options: None,
+            })
+            .await
+            .unwrap();
+        let compacted = table.version().await.unwrap();
+
+        let carried = table.refresh_column("doubled").await.unwrap();
+        assert_eq!(carried.rows_filled, 0);
+        assert_eq!(carried.version, compacted + 1);
+        let settled = table.refresh_column("doubled").await.unwrap();
+        assert_eq!(settled.version, carried.version);
+        assert_eq!(
+            read(&table, "doubled").await,
+            vec![Some(2), Some(4), Some(10)]
+        );
+    }
+
+    /// A column declared before signatures existed has no map. Its first
+    /// refresh keeps the null-fill contract and enrolls what it read from;
+    /// from then on a moved input is recomputed like any other.
+    #[tokio::test]
+    async fn test_an_unsigned_column_is_enrolled_by_its_first_refresh() {
+        let table = table_with("refresh_legacy", vec![1, 2]).await;
+        declare_doubled(&table).await.unwrap();
+        table.refresh_column("doubled").await.unwrap();
+        table
+            .update()
+            .column("x", "3")
+            .only_if("x = 1")
+            .execute()
+            .await
+            .unwrap();
+
+        let native = table.as_native().unwrap();
+        let mut dataset = (*native.dataset.get().await.unwrap()).clone();
+        let declaration = dataset
+            .schema()
+            .field("doubled")
+            .unwrap()
+            .metadata
+            .iter()
+            .filter(|(key, _)| !key.starts_with("computed_refresh."))
+            .map(|(key, value)| (key.clone(), value.clone()))
+            .collect::<Vec<_>>();
+        dataset
+            .update_field_metadata()
+            .replace("doubled", declaration)
+            .unwrap()
+            .await
+            .unwrap();
+        table.checkout_latest().await.unwrap();
+
+        // Null-fill only: the moved row keeps the value it was filled with.
+        let enrolled = table.refresh_column("doubled").await.unwrap();
+        assert_eq!(enrolled.rows_filled, 0);
+        assert_eq!(read(&table, "doubled").await, vec![Some(2), Some(4)]);
+
+        table
+            .update()
+            .column("x", "5")
+            .only_if("x = 3")
+            .execute()
+            .await
+            .unwrap();
+        let again = table.refresh_column("doubled").await.unwrap();
+        assert_eq!(again.rows_filled, 1);
+        assert_eq!(read(&table, "doubled").await, vec![Some(4), Some(10)]);
     }
 
     /// A row rewrite before the first refresh materializes the declared
@@ -831,11 +1052,11 @@ mod tests {
         assert_eq!(read(&table, "doubled").await, vec![Some(6)]);
     }
 
-    /// The contract holds row by row, not fragment by fragment: revisiting a
-    /// fragment to fill one row must not recompute a filled row sitting beside
-    /// it, even where the input behind it has since changed.
+    /// A fragment compacted out of one the stamp never signed cannot vouch
+    /// for any of its rows: every live row is recomputed, the moved one
+    /// included.
     #[tokio::test]
-    async fn test_refresh_does_not_recompute_a_filled_row_beside_an_unfilled_one() {
+    async fn test_a_compaction_of_an_unsigned_fragment_recomputes_it() {
         let table = table_with("refresh_mixed", vec![1, 2]).await;
         declare_doubled(&table).await.unwrap();
         table.refresh_column("doubled").await.unwrap();
@@ -857,18 +1078,70 @@ mod tests {
             .unwrap();
 
         let result = table.refresh_column("doubled").await.unwrap();
-        assert_eq!(result.rows_filled, 1);
-        // 2 is the mutated row keeping the value it was filled with, not 200.
+        assert_eq!(result.rows_filled, 3);
+        assert_eq!(
+            read(&table, "doubled").await,
+            vec![Some(4), Some(10), Some(200)]
+        );
+    }
+
+    /// The gate's reproducer: a raw lance append may carry a value for the
+    /// computed column. Compaction cannot certify it, so the product is
+    /// recomputed and the supplied value replaced.
+    #[tokio::test]
+    async fn test_raw_append_values_are_not_trusted_after_compaction() {
+        use arrow_array::RecordBatchIterator;
+        use lance::Dataset;
+        use lance::dataset::{WriteMode, WriteParams};
+
+        let dir = tempfile::tempdir().unwrap();
+        let conn = connect(dir.path().to_str().unwrap())
+            .execute()
+            .await
+            .unwrap();
+        let batch = record_batch!(("x", Int32, [1, 2])).unwrap();
+        let table = conn
+            .create_table("raw_append", batch)
+            .execute()
+            .await
+            .unwrap();
+        declare_doubled(&table).await.unwrap();
+        table.refresh_column("doubled").await.unwrap();
+
+        let batch = record_batch!(("x", Int32, [5]), ("doubled", Int32, [Some(999_i32)])).unwrap();
+        let schema = batch.schema();
+        let uri = table.uri().await.unwrap();
+        Dataset::write(
+            RecordBatchIterator::new(vec![Ok(batch)], schema),
+            &uri,
+            Some(WriteParams {
+                mode: WriteMode::Append,
+                ..Default::default()
+            }),
+        )
+        .await
+        .unwrap();
+        table.checkout_latest().await.unwrap();
+        table
+            .optimize(crate::table::OptimizeAction::Compact {
+                options: crate::table::CompactionOptions::default(),
+                remap_options: None,
+            })
+            .await
+            .unwrap();
+
+        let result = table.refresh_column("doubled").await.unwrap();
+        assert_eq!(result.rows_filled, 3);
         assert_eq!(
             read(&table, "doubled").await,
             vec![Some(2), Some(4), Some(10)]
         );
     }
 
-    /// Filling a fragment must not disturb the values it already holds, which
-    /// is what makes a compaction-mixed fragment safe to revisit.
+    /// An appended fragment holds no values, so compacting it into a signed
+    /// one leaves the product fresh: only the appended rows are filled.
     #[tokio::test]
-    async fn test_refresh_preserves_already_filled_rows() {
+    async fn test_a_compaction_with_an_appended_fragment_fills_only_its_rows() {
         let table = table_with("refresh_preserves", vec![1, 2]).await;
         declare_doubled(&table).await.unwrap();
         table.refresh_column("doubled").await.unwrap();
@@ -995,7 +1268,8 @@ mod tests {
         assert_eq!(result.rows_failed, 0);
         assert_eq!(result.rows_remaining, 0);
         assert_eq!(result.source_version, 2);
-        assert_eq!(result.published_version, Some(3));
+        // The fill lands at 3; the stamp recording its inputs is published at 4.
+        assert_eq!(result.published_version, Some(4));
         assert_eq!(job.status().await.unwrap(), "finished");
         assert_eq!(
             read(&table, "doubled").await,
@@ -1059,31 +1333,32 @@ mod tests {
         assert_eq!(read(&table, "quotient").await, vec![Some(10)]);
     }
 
-    /// The gate's reproducer: an already-filled row's value must not be
-    /// re-evaluated either -- its input may have mutated into one the
-    /// expression chokes on.
+    /// A filled row whose input moved is re-evaluated, and a row whose
+    /// input did not move is not: the untouched fragment is never read, so
+    /// its poison input is never reached.
     #[tokio::test]
-    async fn test_a_filled_rows_value_is_never_evaluated() {
-        let table = table_with("refresh_filled_poison", vec![1, 2]).await;
+    async fn test_only_a_moved_rows_value_is_re_evaluated() {
+        let table = table_with("refresh_filled_poison", vec![1, 0]).await;
         table
             .add_columns()
-            .computed("quotient", "10 / x")
+            .computed("quotient", "10 / coalesce(nullif(x, 0), 1)")
             .execute()
             .await
             .unwrap();
         table.refresh_column("quotient").await.unwrap();
+        assert_eq!(read(&table, "quotient").await, vec![Some(10), Some(10)]);
 
+        append(&table, vec![5]).await;
         table
             .update()
-            .column("x", "0")
+            .column("x", "2")
             .only_if("x = 1")
             .execute()
             .await
             .unwrap();
-        append(&table, vec![5]).await;
 
         let result = table.refresh_column("quotient").await.unwrap();
-        assert_eq!(result.rows_filled, 1);
+        assert_eq!(result.rows_filled, 2);
         assert_eq!(
             read(&table, "quotient").await,
             vec![Some(2), Some(5), Some(10)]