**Blocked on the corrected Lance release:**
https://github.com/lance-format/lance/pull/10370 restores the `Send`
contract required by downstream Function workers. Refresh this upgrade
with that release before merging.
Upgrade the Rust, Java, and Python Lance dependencies to
https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.2. This
brings in bounded vector-index consolidation without retraining
(https://github.com/lance-format/lance/pull/10335) and deletion
filtering for index entries whose fragment coverage was pruned
(https://github.com/lance-format/lance/pull/10325).
Synchronize the vendored `lance-arrow` with the release and retain its
offset-safe validity patch. The existing `nullable_struct_merge`
regression still panics with the unpatched release, so removing that
patch would regress nullable nested queries. Refresh both lockfiles,
including the `zstd-sys` version required by the new Lance encoding
dependency.
Update the custom memtable bitmap test adapter to train on row
addresses, matching the new bitmap contract; remove its obsolete row-ID
conversion.
Preserve the Blob APIs' existing missing-row error messages when newer
Lance rejects a missing fragment earlier in the read path. Retain
upstream attribution in vendored sources while satisfying the repository
license-header check.
Validation of the compatibility fixes: all 58 Rust Blob integration
tests and 219 Python Blob tests pass (one Python test skipped). Strict
workspace Clippy, formatting, and diff checks pass.
## Root cause
The pinned Lance Arrow crate rebuilt a sliced struct child with its
original Arrow offset but supplied a validity bitmap containing only the
visible rows. Arrow requires the bitmap to cover the offset as well, so
filtered vector searches involving nullable nested fields could panic
with `null_bit_buffer size too small`.
## Fix
- Carry an offset-aligned validity bitmap when propagating parent nulls
to a sliced child.
- Vendor the pinned `lance-arrow` crate until a Lance release includes
the fix; include it in Maturin source distributions.
- Add unit regressions for sliced children and parent bitmaps, plus a
record-batch merge regression.
## Validation
- The issue's 8,600-row Python reproducer completes its filtered flat
dot search and returns 8,448 rows without a panic.
- The targeted Rust merge regression and two focused Python query tests
pass.
- Rust formatting, Ruff formatting and lint, the vendored crate tests,
and the workspace Cargo checks passed. The Maturin source archive
includes the patched crate.
Fixes#4260
<!-- lance-gatekeeper-fix:v1 agent=e6045f701d9b708ee26528ba36353aba
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Xuanwo
lancedb-gatefixer[bot]