A scan reads a limit of zero as no limit at all, so a view declared to
hold nothing was refreshed with every selectable row instead. Answer that
cap without a scan, and convert the cap with a checked conversion rather
than a cast that would wrap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eviction committed on its own, ahead of the rows replacing it, so a reader
between the two saw the view missing every row an update touched -- and
with the predicate chunked, that gap spanned several commits. Only the
rebuild path was ever atomic.
The removals are staged instead and ride in the same Update as the new
fragments, which also lets lance reject a racing write on its own conflict
rules rather than a version check after the fact. The eviction predicate
becomes an expression rather than SQL text that grows with the delta, and
the watermark stamp is now the only follow-up commit: a crash before it
leaves the data correct and merely uncertified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two ways a refresh could drop rows it never materialized.
A cap stops rows being written once the view is full, and the watermark
then advances past them, so room freed by a later delete could never be
refilled from any delta -- the view sat incomplete under its own cap. A
capped view now rebuilds when the source removes or changes rows.
Separately, a transaction file carries placeholder ids for the fragments
it creates. Merging into an empty source made those collide with the ids
the commit went on to assign, so the first rows read as
already-materialized and were skipped while the watermark still moved.
Only ids of fragments that already existed are treated as real; rewritten
rows are excluded by creation version, as they already were.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two defects in incremental refresh. New and recomputed rows were capped
separately, so their sum could push a limited view past its limit; they
now draw on one budget, with recomputed rows -- which replace rows the
view already held -- ahead of new ones when it cannot cover both.
Separately, eviction commits before the data does, and an unrelated write
landing in between was invisible: the refresh would publish and stamp a
watermark certifying a view generation it had not accounted for. Every
commit a refresh makes is now checked against the generation it planned
from, and it aborts rather than certify drift.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The classifier only marked an Update as changing rows, but merge_insert
commits an Update too and its by-source arm removes rows rather than
changing them. A merge that deleted source rows left the view holding
them, since nothing enabled deletion-delta processing for that span.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The module doc described appends as the only incremental case and
cross-process refresh as unserialized, with a note that no lance operation
could express the fix. Both statements are stale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The incremental path staged its rows as a plain append, and two appends
never conflict, so two refreshes computing the same source rows both
committed and the view held each row twice. Commit them instead as an
update carrying an inserted-rows filter over the provenance column, which
lance rejects on key overlap while leaving refreshes over disjoint source
rows free to proceed. That subsumes the hand-rolled version check the
append path used to certify itself with.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A refresh only ever appended, so a source delete or update forced a full
rebuild of the view: the watermark proves what has been materialized, not
what has since left or changed underneath it. That made the cost of any
mutation proportional to the view rather than to the change.
The classifier now admits Delete and Update alongside Append, and the plan
carries whether either was seen. Rows the source dropped are found through
the version delta and evicted from the view by provenance id; rows it
changed are evicted the same way and recomputed into the same pass, so the
view converges without rereading what did not move. New rows are selected
by creation version rather than by fragment identity, which is what keeps
the eviction and the append from double-counting a rewritten fragment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scaffolding. Incremental refresh needs DatasetDelta::get_deleted_row_ids,
which is not in a released lance yet. The deny allow-list entry pairs with
the patch section; drop both and bump the pin once it lands upstream.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The caveat said two refreshes "would each append the same rows", which
reads as a warning about wasted work. Both appends are durable: lance
rebases the loser's append instead of rejecting it, so the check that
follows the write reports an abort after the rows have landed, and its
"unrecorded" wording describes the watermark rather than the data. Say
that, and say what closing it needs -- one transaction carrying both the
fragments and the watermark, which no current lance operation expresses.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ordinary reads on a MemWAL table merge rows from un-compacted tiers, but
refresh plans from committed fragments, so it could certify a source
version while missing rows those reads expose -- and a view with such
state holds rows outside the swap/append model refresh maintains. Reject
an active write spec or retained un-compacted rows (the catch-up flag
outlives unset) on the source at create and on both sides at every
refresh, and refuse installing a spec over a view, mirroring the
computed-column boundary in both directions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A DataOverlay replaces cell values without changing any file path, so the
fragment-signature fallback classified an overlaid source as a pure append
and stamped it current. Overlay identity now joins the signature, scoped to
the columns the view reads like data files are.
Rebuilds also no longer overwrite: lance rebases an overwrite over a
concurrent schema-metadata commit, so a racing definition update could be
durably erased by the stale schema the overwrite carries. Every rebuild is
now the fragment swap -- which carries no schema metadata and already
aborts unless it lands on the planned generation -- with the watermark
stamped separately; an unstamped view rebuilds on the next refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reloaded definition is now re-planned against the current source schema
under the lock, and its planned output must be exactly the view's physical
schema -- otherwise the refresh is refused rather than certifying rows the
stored table cannot represent. Planning also re-derives the inputs and
re-applies the immutability check, so a definition written by another
producer gets the same scrutiny as one written here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Lance rebases an append over an intervening delete or update, so an
incremental data commit could absorb a foreign mutation and the stamp would
certify it. The append and the indexed fragment swap now require landing at
exactly the planned view version, aborting unrecorded otherwise -- the next
refresh rebuilds from the drifted state.
Refresh also re-reads the definition from the reloaded view under the lock:
what it executes and what it stamps must be one generation, not whatever a
stale handle cached at open.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two ways a refresh could still certify state it did not produce: a second
handle's lazy cache let it plan from a pre-commit snapshot even under the
per-view lock, double-appending or no-opping over a mutated view -- the
lock now force-reloads the view before planning; and the watermark stamp,
on discovering another commit raced in between, retried on top of it and
certified that commit as the refresh's own generation -- it now aborts
unrecorded, so the next refresh rebuilds from the drifted state.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The watermark now carries two witnesses recorded atomically with it: the
view version its refresh produced (predicted, then verified, so a racing
commit cannot leave it pointing at someone else's work) and the commit
timestamp of the watermark's source manifest. Any later view commit or a
source whose manifest timestamp disagrees -- a dropped and recreated table
reuses version numbers, never timestamps -- invalidates the watermark and
the refresh rebuilds instead of reporting NoOp over stale rows.
Refreshes of one view are also serialized within the process by a per-view
lock, closing the same-watermark double-append; cross-process remains the
caller's boundary, stated in the module docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A declared view holds no rows; refresh computes them. It pins one source
version and brings the view to exactly the definition's result at that
version, recording the version as a watermark in the view's schema
metadata. The refresh is incremental when the source changed by nothing but
appends and compactions since the watermark: a transaction-log walk
separates the two, so only genuinely appended fragments are computed --
compaction outputs are already-materialized rows rearranged and cost
nothing, which is what keeps routine background compaction from rebuilding
the view. One subtlety shapes the walk: transaction files record an
Append's fragments with placeholder ids (real ids are assigned at commit),
while a Rewrite's ids are reserved beforehand and real, so appends are
derived as new-at-head minus rewrite outputs rather than read from the log.
A fragment-signature check scoped to the columns the view reads is the
fallback for deltas the walk cannot classify, and passes changes to
unrelated columns.
Anything else rebuilds: deletes, updates, a vacuumed watermark version, or
an append a later compaction swallowed (its rows cannot be told apart from
already-materialized ones without a provenance scan). A rebuild of an
indexed view stages the new fragments uncommitted and commits one Update
swapping out every old fragment, so index definitions are never absent and
readers never observe an empty view; unindexed rebuilds overwrite, with the
watermark riding the same commit. Refresh also accepts a pinned source
version, and a row limit counts already-held rows.
Two bounds, stated in the module docs: on the append path the watermark
moves in a follow-up commit, so a crash between the two re-appends those
rows on the next refresh; and concurrent refreshes of one view can
double-append, since lance appends do not conflict -- run one at a time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A view field was rebuilt from name and type alone, so a column projected
as itself lost its field metadata: default-projecting a blob column turned
it into a plain struct that blob discovery no longer recognized. A
projection that reads one source field directly -- a bare column or a path
of struct accesses -- now keeps that field's metadata.
Schema-structure declarations are stripped at every depth instead. A
view's fields are always nullable, which an unenforced primary key
forbids, and a declaration buried in a struct child binds as hard as one
on top. Embedding configuration is carried only for entries whose columns
the view projects directly, renamed to the names the view gives them; a
configuration naming a column the view does not carry describes a table
that does not exist.
A limit is also refused at declaration when it exceeds what a scan can
express, rather than being stored and failing every refresh afterwards.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The connection-level entry points hold a local-only boundary, but
from_table does not, and that is the entry both bindings use to refresh.
A remote table therefore reached a describe request and came back as a
plain table rather than as unsupported, so the caller paid a round trip
for a misleading error.
Check the boundary before the schema read.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reopen-failure arm of canonicalization had no direct regression: a
fallback to the caller handle would revive the bug unseen. The module doc
also named a refresh method this layer does not define.
An external surface declaring views (a SQL layer, a service) needs to
produce declarations byte-identical to this crate's, but exporting the
raw planner and serializer would leave the assembly invariants -- the
provenance column, the metadata embedding, the source checks -- to every
caller. prepare_declaration canonicalizes the source through the
coordinate a refresh will resolve (its own database, root namespace,
bare name), plans against the table that resolution reaches, and rejects
what the definition cannot record: a namespaced source, or a handle that
does not resolve back to itself. Consuming the result through
PreparedDeclaration::create is the only way to materialize it, and it
creates in the retained source database -- there is no target parameter
to misuse -- through the stable-row-id create-and-verify path. The
builder itself now flows through both calls.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mixed case: a creation key stripped from an accessor holding both
static store options and a provider must leave an accessor carrying both.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Focused coverage for each strip outcome: creation keys removed while store
keys survive, an all-creation-key request leaves no accessor to fork the
store, an accessor carrying no creation keys is preserved untouched, an
emptied provider-backed accessor remains first-fetch rather than caching an
empty map, and end to end a creation-key-only create stays reachable on the
connection's store with the option applied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The strip rebuilt the accessor unconditionally, so a provider-backed
accessor with no static options became one caching an empty map, and its
credentials were never fetched. The accessor is now rebuilt only when a
creation key was actually removed, and a provider-only accessor is rebuilt
with the provider alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The request-level new_table_enable_stable_row_ids option outranks a
connection-level override, so a view can no longer be created unstable and
then destructively dropped by name in cleanup. What made this possible: the
new-table keys used to ride in the store options and fork a fresh store
connection for the request; prepare_write_params now strips them, since
they are creation configuration, not store credentials. The manifest is
still verified as defense against Database impls that ignore the option,
but a failure only reports -- nothing is rolled back.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arrow_typeof, arrow_field and arrow_metadata are labeled immutable but
compute from schema state -- name, type, nullability, field metadata --
which can change without any data change a refresh could observe, mixing
evaluations across an incremental refresh. The guard's criterion is now
stated as it is: a definition may only call functions whose result is
determined by row values alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DataFusion marks version() immutable, but it returns the build string:
identical-across-refreshes has to hold across upgrades too, and an
incremental view would mix values from different builds. The immutability
guard now also excludes known build-dependent functions by name.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Incremental maintenance preserves rows from earlier refreshes, so a
definition containing random() or now() would make the view a mix of
evaluations no single source version explains. Every function in a
projection or filter must be immutable, checked on the parsed expression
before optimization -- the simplifier folds a stable call like now() into
a literal that would hide it while the stored definition keeps the call.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Filters must now type as Boolean; nested references resolve by root field
so struct columns can be declared; listing returns name-and-kind entries so
a newer-kind view no longer disappears; remote connections are refused
before any request reaches the network; and view creation verifies the
created manifest has stable row ids, failing loudly when a connection-level
override would otherwise produce an unchainable view.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A materialized view is a table whose contents are defined by a query over
one source table and maintained by refresh rather than by writes. This adds
the declaration half: create_materialized_view(name, source) resolves a
projected/filtered/limited definition against the source schema -- output
types and input columns are derived with the DataFusion planner, never
user-supplied -- and commits an empty table carrying the definition as
kind-tagged JSON in its schema metadata. The tag is what lets a later
definition kind read back as a view this version cannot refresh instead of
as a plain table, mirroring the computed-column convention. Views open and
list as ordinary tables, so queries, indexes and search apply unchanged.
Sources must have stable row ids, checked here because the property cannot
be enabled after a table exists: every view row records the source row that
produced it in __source_row_id, and that provenance only survives source
compactions, updates and deletes when row ids are stable. The refresh
engine and its provenance writes land separately; a declared view is inert
but fully introspectable (open_materialized_view, list_materialized_views,
the definition read back from the stored schema).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LanceDB Cloud and Enterprise support computed columns through the REST
API,
so declaration dispatches per backend: local tables plan the expression
themselves, remote ones send {name, computed} entries for the server to
plan. A remote refresh is the server's backfill job --
refresh_column_async
submits it and returns a handle whose successful wait establishes a
read-freshness baseline on the submitting handle, unless a checkout has
pinned the handle by the time the job completes; the blocking form
refuses
rather than invent a fill count the server does not report.
Declaration entries are built from the namespace client's
AddColumnsEntry
model (lance-namespace 0.11.0, via the lance beta.13 pin), so the
payload
shape is compile-checked against the published contract.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
Updates the Lance Rust workspace dependencies and Java lance-core
dependency to
[v11.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.13).
Adds the required `ListTablesResponse.context` compatibility field and
validates the workspace with Clippy warnings denied.
Mirrors create_index's dual surface: the blocking refresh_column keeps
returning {rows_filled, version}, and refresh_column_async returns the
same
Job handle create_index uses, running the refresh as an in-process task.
Invalid input is reported by the submitting call rather than by the job.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
add_columns().computed("doubled", "x * 2") stores the expression in
field
metadata and commits the column empty; a later refresh fills it. Type
and
inputs are derived from the expression.
The declaration stays authoritative for its lifetime: writes that would
give
the column a value (append, update, merge, SQL insert), schema changes
that
would break the stored expression or reshape its output, metadata edits,
volatile expressions, declaration metadata arriving through any path but
the
validated declare call, and LSM write specs in either order against
latest
committed state are all refused. The LSM check also refuses on the
mem-wal
catch-up feature flag, which outlives unset and marks retained SSTable
rows.
Simultaneous declare/install interleavings conflict at commit via
lance's
mem-wal rule (lance#8539). Local tables only.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
> Stacked on #3780. Blocked only on #3922 (`lance` → `v11.0.0-beta.6`),
so CI
> stays red until that lands.
## Missing coverage must mean "not known to be covered"
#3780 caps the SSTable exclusion watermark at an index's recorded
catch-up when
there is one, and silently ignores the case where there is none. On a
table that
requires catch-up, an absent entry means the index is *not* known to
hold the
compacted rows — and the LSM base arm reads base through the index
(`fast_search`, no brute-force tail), so dropping that SSTable loses
those rows
for that query.
```rust
Some(caught_up) => watermark = watermark.min(caught_up),
None if catchup_required => watermark = 0, // retain everything
None => {}
```
`catchup_required` reads the manifest feature bit directly, and requires
both
words: a half-set manifest is treated as legacy, which is the
conservative side.
Without the bit the field is not maintained at all, so absence carries
no
information and behaviour is unchanged.
## Activation, as a table-level entry point
`Table::require_mem_wal_index_catchup()` performs the one-way switch,
separate
from `set_lsm_write_spec`: a table carrying the bit retains every
generation
until something records catch-up, so it has to follow the deployment of
whatever
repairs coverage, not the creation of the table.
This is a convenience, not the only path — a writer holding the dataset
calls
the equivalent on `DatasetMemWalExt`, which is what the WAL pod does.
Lance
enforces the preconditions either way: the MemWAL index must exist, and
the
table must not already carry `compacted_sstables` from before this
protocol,
since those numbers cannot be validated.
## Still correct after the Lance rework
lance-format/lance#8481 replaced the transmitted `IndexCatchupAdvance`
with a
position derived at commit time from the version a transaction read.
That
changed how a writer earns coverage; it did not change what a reader may
conclude from its absence. The rule here, and the field it reads, are
unchanged.
## Tests
Existing `exclusion_watermarks` unit tests carry the new argument.
Coverage
against a real dataset follows once #3922 lands and this can build.
Updates the Rust workspace Lance dependencies and Java lance-core
dependency to v11.0.0-beta.10. No compatibility fixes were required;
workspace clippy with all features and Rust formatting pass.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.10
## Summary
- add `drop_table_async` and return a job handle while preserving
`drop_table`
- consume remote 202 responses with cleanup job IDs and retain
older-server compatibility
- expose the API through Python and TypeScript connection wrappers
Say what resolving "latest" actually does: pick the newest release,
preferring stable over pre-release, and skip the run if it is not newer
than the version pinned in Cargo.toml.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`exclusion_watermarks` resolved a single index and capped SSTable
exclusion at that index's catch-up watermark. It now takes every index
the query relies on and retains to the **lowest** of them, and the
resolver collects arms together rather than returning at the first
match.
This is groundwork, not a fix for a reachable bug: `reject_unsupported`
refuses hybrid search, so the vector and full-text arms are mutually
exclusive and the list never holds more than one entry today. The
generalisation is what the remaining work below plugs into.
Unchanged: a plain scan uses the compaction watermark alone, an index
with no catch-up entry contributes no cap, and a caught-up index falls
back to the compaction watermark. Taking a minimum over more indexes can
only lower a watermark, so the failure direction is "read an SSTable
unnecessarily", never "miss rows".
## Tests
Three in `lsm`: the existing lagging-index test updated for the new
signature;
`exclusion_watermark_takes_the_minimum_across_every_index_used` (two
indexes at 7 and 4 against compaction at 9 — each alone stops at its own
watermark, together the lower governs, order-independent); and
`an_untracked_index_does_not_widen_a_lagging_sibling`.
`cargo test -p lancedb --lib` — 45 lsm tests, 484 in the crate. `cargo
fmt --check` clean.
## Follow-ups
This crate pins lance to a released tag, so anything needing unreleased
Lance symbols waits for a bump.
1. **Select legacy versus strict semantics from the feature bit.** On a
table with `FLAG_MEM_WAL_INDEX_CATCHUP` set, a *missing* entry must mean
"not caught up" and retain the SSTables, instead of leaving the
compaction watermark unchanged. Needs the bit from
lance-format/lance#8263. **This must land before any table is
activated** — otherwise the bit is set while queries still read
permissively.
2. **Collect scalar and bitmap-family prefilter indexes.** The genuinely
multi-index query is a vector search with a scalar prefilter, and it is
gated on the vector index alone today. Identifying the others needs the
planner's chosen indexes, not the columns the filter names, so it needs
a Lance-side helper.
3. **Verify a retained SSTable can actually answer.** Both base and
SSTable arms use `fast_search`; a source without a compatible index
contributes nothing, so retention alone does not guarantee its rows are
returned. Needs a flat-search fallback or an explicit error in Lance's
`LsmScanner`.
4. **Planner-level integration tests.** Current tests exercise the
watermark arithmetic directly. End-to-end coverage over real queries —
prefilter forms, legacy versus activated, missing index and missing
shard entries — depends on 1–3.
## What is the bug?
#3731 tries to distinguish a missing table from a corrupt table after
Lance returns `DatasetNotFound`. It does that by listing the database
parent and treating a physical `<name>.lance` entry as evidence that the
table exists.
That premise is not sound for a listing database. Table creation writes
data before atomically committing the first manifest, so the same
physical prefix can represent a live concurrent create, abandoned
uncommitted data, or an old empty directory. It is not evidence of a
committed table. The parent listing also makes every missing-table open,
including the create-on-miss path, perform work proportional to the
number of sibling tables. Cloud `list_with_delimiter` exhausts all pages
before returning.
## How does this PR fix the problem?
This PR makes the committed Lance manifest the sole table-existence
authority for listing-database opens:
- `DatasetNotFound` maps directly to `TableNotFound`; no parent or
target storage probe runs.
- Other Lance load errors continue to propagate unchanged.
- A physical directory, object prefix, or uncommitted data file alone
does not block `Create`.
- Concurrent `Create` requests are arbitrated by the conditional
version-1 manifest commit: one succeeds and the loser receives
`TableAlreadyExists`.
- `table_names` is documented as physical discovery, not an atomic
table-existence check. Its snapshot can contain an entry that is still
being created, has only uncommitted storage, or is concurrently dropped.
This removes the need for a new Lance object-store capability. LanceDB
remains on the official Lance `v11.0.0-beta.6` dependency from `main`;
the merge commit for lance-format/lance#7722 is an ancestor of that tag,
so the ambiguous-GCS-500 corruption-prevention fix is retained.
## Performance evidence
Lower is better. The benchmark uses real `.lance` directories with
marker objects on the local filesystem; fixture creation and teardown
are outside the timed region. Baseline is `origin/main` at `6fb976cf`,
candidate is `e1240751`. Both were built from the same lockfile on the
same macOS arm64 machine with the repository's `release` profile (fat
LTO), then executed in alternating baseline/candidate order for three
pairs. Each run used 10 warmups and 100 distinct missing-table opens per
scale. The table reports the median of the three run-level percentiles.
| Scenario / metric | Baseline | This PR | Benefit |
| --- | ---: | ---: | ---: |
| 1,000 real sibling directories, p50 | 11.905 ms | 21.042 us | 566x
speedup |
| 10,000 real sibling directories, p50 | 143.630 ms | 18.375 us | 7,817x
speedup |
| 100,000 real sibling directories, p50 | 1.991 s | 19.917 us | 99,984x
speedup |
| 100,000 real sibling directories, p95 | 2.346 s | 25.792 us | 90,965x
speedup |
These results validate removal of the sibling-cardinality dependency in
this local-filesystem workload; they are not an extrapolation to
production GCS latency. A structural object-store regression test
separately asserts that opening one missing table performs zero
parent-scoped `list`, `list_with_offset`, or `list_with_delimiter`
calls.
Run with:
```bash
BENCH_SIBLINGS=1000,10000,100000 BENCH_WARMUPS=10 BENCH_TRIALS=100 \
cargo run --locked --release --quiet -p lancedb --example bench_open_missing_table
```
## Correctness and compatibility boundaries
- An empty `.lance` directory or orphan data without a committed
manifest now opens as `TableNotFound` and may be replaced by a
successful `Create`.
- Two synchronized creators sharing one object store deterministically
produce one success and one conditional-manifest conflict mapped to
`TableAlreadyExists`.
- A readable manifest remains authoritative; non-`DatasetNotFound`
corruption, external-manifest, authorization, and object-store errors
are not folded into `TableNotFound`.
- `TableCorrupted` remains in the public error enum for compatibility,
but this listing-database fallback no longer synthesizes it from an
ambiguous physical footprint.
- Reliably distinguishing `Missing`, `Creating`, and `Corrupt` would
require explicit authoritative lifecycle/catalog metadata (for example a
leased creation record). It cannot be inferred from a directory or
prefix, and is outside this incident fix.
## Validation
- `cargo fmt --all -- --check`
- `cargo check --quiet --locked -p lancedb --features remote --tests
--examples`
- `cargo clippy --quiet --locked -p lancedb --features remote --tests
--examples -- -D warnings`
- `cargo test --quiet --locked -p lancedb --features remote --tests`
- library: 843 passed, 1 ignored
- integration groups: 39 passed, 6 passed, 5 passed
- focused coverage for empty directories, orphan data, physical listing
snapshots, zero parent listings, and concurrent manifest arbitration
Closes#3704
## Problem
Transforms can fail on bad data (e.g. nulls/NaNs from incomplete user
surveys). Today any transform exception aborts iteration, and there is
no way to skip invalid rows during loading.
## Solution
New `on_transform_error` parameter on `StreamingDataset`:
- `"raise"` (default, matches current behavior and the convention in
tf.data / WebDataset / Ray Data)
- `"skip"` — drop the failing rows and continue
- `"warn"` — like skip, plus a logged warning per failing batch
- a WebDataset-style callable `handler(exc) -> bool`, so users can skip
only expected error types
Key design points:
- **Row-granular skipping**: when a batch fails, the transform is re-run
on single-row slices so only the rows that actually fail are dropped
(avoids Ray-style whole-block loss). Skips are counted in a new
`rows_skipped` property.
- **No crash on uneven skips**: the round-robin loop now ends the epoch
at the last cycle where every split still has a row, instead of hitting
`IndexError` when a split runs dry early.
- **Exact resumability under skips**: checkpoints are now
position-based. `state_dict` gains `positions_consumed_per_split` (exact
for owned splits), and a new `merge_state_dicts` static method combines
per-rank states via elementwise max for elastic resume across topology
changes. Old checkpoints without the new key still load. Positions equal
sample counts when nothing is skipped, so existing behavior is
unchanged.
- **Guardrail**: transforms returning the wrong number of rows now raise
a clear `ValueError` instead of silently corrupting split accounting.
### Answers to the issue's open questions
- *Can we do this?* Yes — all transforms funnel through one guarded call
in the Stage 2 pipeline.
- *What do other libraries do?* tf.data `ignore_errors()`, WebDataset
`handler=`, Ray `max_errored_blocks`; MosaicML StreamingDataset offers
nothing (skipping conflicts with its determinism model). This design
follows the common conventions: raise by default, opt-in skipping,
count/log drops.
- *Error handling or pre-filtering?* Both: the existing `filter=`
remains the recommended tool for predictable bad data (splits are built
post-filter, so all guarantees hold — now documented);
`on_transform_error` covers failures not expressible as a predicate.
- *Impact on splits / elastic determinism?* Per-split sample sequences
stay deterministic (skips are data-dependent, not topology-dependent).
With unequal bad-row counts across splits the last few global steps of
an epoch can differ across topologies (bounded by the skew), which is
documented on the parameter. With equal counts per split, full
determinism is preserved — covered by a test.
## Testing
15 new tests in `test_elastic_dataloader.py` covering: default raise,
invalid values, uniform and uneven skips (including epoch-end
truncation), warn logging, selective callable handlers, wrong-row-count
guardrail, determinism across runs and across world sizes (1/2/3/4) with
skips, exact mid-epoch resume with skips on the same topology, elastic
resume via `merge_state_dicts` (ws=2 → ws=1), merge validation, and
backward-compat loading of old checkpoints.
Note: relying on CI for the test run — my local machine OOMs during the
final link of the native extension. The change itself is pure Python.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
PyO3 defaults native extension classes to `builtins`, so
mkdocstrings/Griffe could not resolve the newly documented
`lancedb.Session` alias and `Deploy docs to Pages` failed on `main`.
Declare the extension module for the public native types referenced by
the Python API docs so Griffe resolves them through `lancedb._lancedb`
and Pages can build again.
Validated with the docs toolchain used by CI (`griffe==0.49.0`,
`mkdocstrings==0.25.2`, and `mkdocs==1.6.1`); `PYTHONPATH=. mkdocs
build` succeeds.
## Summary
`opendal 0.58.1` (the version pulled in transitively via Lance) already
ships
`goosefs-sdk 0.1.9`, which includes the upstream fix for the 0.1.6
compile
break. The explicit version pin that lancedb has been carrying since the
GooseFS feature was introduced is therefore no longer necessary and is
now
redundant work to maintain.
## Changes
- Remove the direct `goosefs-sdk` dependency from
`rust/lancedb/Cargo.toml`
(it was pinned to `=0.1.9` with a comment referencing the 0.1.6 compile
break).
- Remove the `dep:goosefs-sdk` entry from the `goosefs` cargo feature,
since
no source file in lancedb imports the crate directly.
- Refresh `Cargo.lock`; `goosefs-sdk 0.1.9` now resolves transitively
through
`lance` → `opendal 0.58.1`.
## Verification
- `cargo fmt --all` — clean
- `cargo check --features remote,goosefs --tests --examples` — passes
- `Cargo.lock` confirms `goosefs-sdk 0.1.9` is still resolved (now
transitively), so the `goosefs` feature continues to enable the same set
of
Lance/IOPaths as before.
## Backwards compatibility
No public API changes. The `goosefs` cargo feature still activates
`lance/goosefs`, `lance-io/goosefs`, and
`lance-namespace-impls/dir-goosefs`,
and the same `goosefs-sdk 0.1.9` version is selected by the resolver.
Wyatt AltandClaude Fable 5
LanceDB Robot
XY Zhan
Lance Release
Jack Ye
Yang Cen
Sravan AvvaruandClaude Fable 5
Xuanwo
ForwardXu