From 7f892aaed6033193bde234079b884159429c63dc Mon Sep 17 00:00:00 2001 From: Wyatt Alt Date: Sat, 15 Aug 2026 07:12:00 -0700 Subject: [PATCH] docs: state what cross-process refresh actually leaves behind The caveat said two refreshes "would each append the same rows", which reads as a warning about wasted work. Both appends are durable: lance rebases the loser's append instead of rejecting it, so the check that follows the write reports an abort after the rows have landed, and its "unrecorded" wording describes the watermark rather than the data. Say that, and say what closing it needs -- one transaction carrying both the fragments and the watermark, which no current lance operation expresses. Co-Authored-By: Claude Fable 5 --- rust/lancedb/src/materialized_view/refresh.rs | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/rust/lancedb/src/materialized_view/refresh.rs b/rust/lancedb/src/materialized_view/refresh.rs index a6ddb2ba4..0c8402bc3 100644 --- a/rust/lancedb/src/materialized_view/refresh.rs +++ b/rust/lancedb/src/materialized_view/refresh.rs @@ -22,9 +22,22 @@ //! and the next refresh rebuilds rather than trusting any of it. //! //! Refreshes of one view are serialized within a process by a per-view lock. -//! Across processes nothing serializes them, and two incremental refreshes -//! planned at the same watermark would each append the same rows; run one -//! process's refreshes against a view at a time. +//! Across processes nothing serializes them: two incremental refreshes +//! planned at the same watermark each append the same rows, and both +//! appends become durable. The version check that follows a write cannot +//! prevent this -- lance rebases a conflicting append rather than rejecting +//! it, so the loser's rows land and only then does it report an abort. Its +//! error says the refresh is "unrecorded", which is true of the watermark +//! and false of the rows: the view serves duplicates until something +//! rebuilds it. +//! +//! Closing this needs the data and the watermark to commit as one +//! transaction, which no current lance operation expresses (`Append` and +//! `Update` carry fragments, `UpdateConfig` carries schema metadata, and +//! `Overwrite` -- which carries config -- rebases over concurrent metadata +//! commits). Until then, run one process's refreshes against a view at a +//! time. `differential::concurrent_refreshes_hold_each_row_once` is the +//! reproducer. use std::collections::{HashMap, HashSet}; use std::sync::atomic::{AtomicU64, Ordering};