Closes#3704
## Problem
Transforms can fail on bad data (e.g. nulls/NaNs from incomplete user
surveys). Today any transform exception aborts iteration, and there is
no way to skip invalid rows during loading.
## Solution
New `on_transform_error` parameter on `StreamingDataset`:
- `"raise"` (default, matches current behavior and the convention in
tf.data / WebDataset / Ray Data)
- `"skip"` — drop the failing rows and continue
- `"warn"` — like skip, plus a logged warning per failing batch
- a WebDataset-style callable `handler(exc) -> bool`, so users can skip
only expected error types
Key design points:
- **Row-granular skipping**: when a batch fails, the transform is re-run
on single-row slices so only the rows that actually fail are dropped
(avoids Ray-style whole-block loss). Skips are counted in a new
`rows_skipped` property.
- **No crash on uneven skips**: the round-robin loop now ends the epoch
at the last cycle where every split still has a row, instead of hitting
`IndexError` when a split runs dry early.
- **Exact resumability under skips**: checkpoints are now
position-based. `state_dict` gains `positions_consumed_per_split` (exact
for owned splits), and a new `merge_state_dicts` static method combines
per-rank states via elementwise max for elastic resume across topology
changes. Old checkpoints without the new key still load. Positions equal
sample counts when nothing is skipped, so existing behavior is
unchanged.
- **Guardrail**: transforms returning the wrong number of rows now raise
a clear `ValueError` instead of silently corrupting split accounting.
### Answers to the issue's open questions
- *Can we do this?* Yes — all transforms funnel through one guarded call
in the Stage 2 pipeline.
- *What do other libraries do?* tf.data `ignore_errors()`, WebDataset
`handler=`, Ray `max_errored_blocks`; MosaicML StreamingDataset offers
nothing (skipping conflicts with its determinism model). This design
follows the common conventions: raise by default, opt-in skipping,
count/log drops.
- *Error handling or pre-filtering?* Both: the existing `filter=`
remains the recommended tool for predictable bad data (splits are built
post-filter, so all guarantees hold — now documented);
`on_transform_error` covers failures not expressible as a predicate.
- *Impact on splits / elastic determinism?* Per-split sample sequences
stay deterministic (skips are data-dependent, not topology-dependent).
With unequal bad-row counts across splits the last few global steps of
an epoch can differ across topologies (bounded by the skew), which is
documented on the parameter. With equal counts per split, full
determinism is preserved — covered by a test.
## Testing
15 new tests in `test_elastic_dataloader.py` covering: default raise,
invalid values, uniform and uneven skips (including epoch-end
truncation), warn logging, selective callable handlers, wrong-row-count
guardrail, determinism across runs and across world sizes (1/2/3/4) with
skips, exact mid-epoch resume with skips on the same topology, elastic
resume via `merge_state_dicts` (ws=2 → ws=1), merge validation, and
backward-compat loading of old checkpoints.
Note: relying on CI for the test run — my local machine OOMs during the
final link of the native extension. The change itself is pure Python.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
PyO3 defaults native extension classes to `builtins`, so
mkdocstrings/Griffe could not resolve the newly documented
`lancedb.Session` alias and `Deploy docs to Pages` failed on `main`.
Declare the extension module for the public native types referenced by
the Python API docs so Griffe resolves them through `lancedb._lancedb`
and Pages can build again.
Validated with the docs toolchain used by CI (`griffe==0.49.0`,
`mkdocstrings==0.25.2`, and `mkdocs==1.6.1`); `PYTHONPATH=. mkdocs
build` succeeds.
## Summary
`opendal 0.58.1` (the version pulled in transitively via Lance) already
ships
`goosefs-sdk 0.1.9`, which includes the upstream fix for the 0.1.6
compile
break. The explicit version pin that lancedb has been carrying since the
GooseFS feature was introduced is therefore no longer necessary and is
now
redundant work to maintain.
## Changes
- Remove the direct `goosefs-sdk` dependency from
`rust/lancedb/Cargo.toml`
(it was pinned to `=0.1.9` with a comment referencing the 0.1.6 compile
break).
- Remove the `dep:goosefs-sdk` entry from the `goosefs` cargo feature,
since
no source file in lancedb imports the crate directly.
- Refresh `Cargo.lock`; `goosefs-sdk 0.1.9` now resolves transitively
through
`lance` → `opendal 0.58.1`.
## Verification
- `cargo fmt --all` — clean
- `cargo check --features remote,goosefs --tests --examples` — passes
- `Cargo.lock` confirms `goosefs-sdk 0.1.9` is still resolved (now
transitively), so the `goosefs` feature continues to enable the same set
of
Lance/IOPaths as before.
## Backwards compatibility
No public API changes. The `goosefs` cargo feature still activates
`lance/goosefs`, `lance-io/goosefs`, and
`lance-namespace-impls/dir-goosefs`,
and the same `goosefs-sdk 0.1.9` version is selected by the resolver.
<!-- lance-gatekeeper-fix:v1 agent=613a074d606e626c5169d601373a32d8
generation=1 -->
## Root cause
When LanceDB accepted an Arrow table created by a different installed
Arrow package, its compatibility sanitizer rebuilt each Data node
without converting the foreign type or preserving nested children. It
also dropped the separate dictionary vector payload and did not preserve
identity shared by dictionary schema types, vector wrappers, or growing
dictionary chunks.
## Fix
Recursively sanitize nested Arrow data types and child data. Use one
table-scoped sanitization context to rebuild and memoize source type
objects, dictionary vectors, and Data nodes in the local Arrow realm,
preserving all identities required by Arrow IPC.
Add Arrow 15 through 18 regressions for list serialization, ordinary
dictionaries, dictionaries shared across fields and batches, growing
dictionaries, and IPC round trips.
## Validation
- pnpm test __test__/arrow.test.ts --runInBand (188 passed)
- pnpm lint
- pnpm build
- pnpm test --runInBand (706 passed, 5 skipped)
- pnpm run docs
Fixes#2256
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Some issues:
- file_size_bytes is optional in the manifest, so if it's not there (old
writer I guess) it'll under-report the table size.
- it changes results a little bit from the old way by including per-file
footers and metadata (probably not a big difference at real scale)
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
## What
`JinaEmbeddings._generate_image_input_dict()` crashes with
`AttributeError: 'function' object has no attribute 'urlparse'` on any
image given as a URL string, local path string, or `pathlib.Path` — i.e.
every documented `jina-clip-v1` image-embedding use case except raw
`bytes`.
## Why
```python
from urllib.parse import urlparse
...
parsed = urlparse.urlparse(image)
```
`urlparse` is imported as a function, then called as if it were the
`urllib.parse` module (`urlparse.urlparse(...)`). The module-level
`is_valid_url()` a few lines above does it correctly (`urlparse(text)`),
which is why this reads as a typo rather than intentional. Fixed to
`urlparse(str(image))` — `str()` is needed because `urlparse()` only
accepts `str`/`bytes` and raises a different `AttributeError` on a raw
`Path`.
## Testing
Added `test_jina_generate_image_input_dict_local_path`, which fails with
the original `AttributeError` before the fix and passes after, covering
both a `str` path and a `pathlib.Path`. Verified locally (built the Rust
extension, ran red→green, then the full `test_embeddings.py` file: 15
passed / 8 skipped, no regressions) and with `ruff check`/`ruff format`.
---
Disclosure: this PR was drafted with AI assistance (Claude); I reviewed,
tested, and take responsibility for the change.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
`LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`:
| value | meaning |
|---|---|
| `None` (new default) | every index the MemWAL supports, resolved when
the spec is installed |
| `Some([])` | maintain nothing — a scan/filter-only WAL table |
| `Some([..])` | exactly these, taken verbatim |
`with_maintained_indexes` keeps its signature;
`with_no_maintained_indexes()` is new. Surfaced through the remote path
(null on the wire), Python, and Node.
## Why
Callers had to state the maintained set by hand every time, which is
both tedious and easy to get wrong — the common case is "maintain what I
already built."
Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating
to lance's `is_maintainable_index_type`. This is load-bearing rather
than cosmetic: lance does **not** skip an index type its memtable cannot
build, it errors when the shard writer opens, so sweeping up a bitmap
index would fail every memtable claim and leave the table unwritable.
The inferred set excludes those, and an explicit list naming one is now
rejected at spec time instead of at claim time.
## Behavior change
A freshly constructed spec used to maintain **nothing**; it now
maintains **everything supported**. This flipped because napi collapses
`undefined` and `null` to `None`, so TypeScript cannot express "absent
means nothing, null means all" — any other choice makes the bindings
disagree with the wire. The error direction also favors it: an unwanted
maintained index costs memory, while a silently unmaintained one
degrades FTS to an unscored scan.
Three existing tests encoded the old default and are updated rather than
worked around.
## Caveat
The resolved set is a snapshot, not a subscription. An index created
after the spec is installed is not maintained until the spec is unset
and set again. `get_lsm_write_spec` therefore always reports a concrete
list — `None` never round-trips.
## Dependency
Needs a lance release carrying `is_maintainable_index_type`
(lance-format/lance#8095) before this builds against the pinned tag.
Draft until then.
## Testing
38 Rust LSM tests and 10 Python tests pass against a local lance build,
including new coverage that a bitmap index is excluded from inference
and rejected when named, and that `[]` stays distinguishable from null
on the wire.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Xuanwo
Sravan Avvaru
ForwardXu
lancedb-gatefixer[bot]
Dan Tasse
Lance Release
Andrew Chen
Dan Rammer