## Summary
Lance can now plan multiple byte ranges for the same blob in one
`read_blob_ranges` operation, but LanceDB users currently cannot expose
a complete set of logical ranges to that planner.
This complements `BlobFile`: file-like consumers such as PyAV can
continue to discover ranges dynamically, while callers that already know
the ranges for a batch can submit them together.
## Motivating example
A training table may store a large video blob together with a small
application-level clip index:
```text
video: blob
clips: [{offset, length}, ...]
```
The caller can select the videos and clips for a batch, obtain their row
IDs from the query, and read all of the selected windows together:
```python
rows = (
table.search()
.select(["clips"])
.with_row_id(True)
.limit(64)
.to_arrow()
.to_pylist()
)
requests = []
for row in rows:
clip = sample_clip(row["clips"])
requests.append(
(row["_rowid"], clip["offset"], clip["length"])
)
chunks = table.fetch_blob_ranges("video", requests)
```
Here, `_rowid` comes from the LanceDB query, while `offset` and `length`
come from the application's clip index and are relative to that row's
video blob. The caller describes only the logical reads; Lance still
handles validation, source grouping, coalescing, scheduling, and byte
backpressure.
Lance v10.0.0-beta.5 returns one logical result per blob selector or
range request and explicitly distinguishes null blobs from valid empty
values. LanceDB consumes that aligned result contract directly and only
adds a cardinality check for unresolved row IDs.
This PR exposes batched blob-range reads on local Rust and Python
tables. Results preserve request identity, duplicates, null slots, and
valid empty ranges while allowing Lance to execute the physical reads
out of order. Scheduler buffer sizing remains an internal Lance concern,
so the LanceDB API does not expose `io_buffer_size`.
Cloud tables continue to report this operation as unsupported until
there is a corresponding remote API.
Updates the Rust workspace Lance dependencies and Java lance-core from
v9.1.0-beta.5 to v9.1.0-beta.7, including the generated Cargo lockfile.
No LanceDB compatibility changes were required for this release. See the
[Lance v9.1.0-beta.7
release](https://github.com/lance-format/lance/releases/tag/v9.1.0-beta.7).
Bumps Lance to v9.0.0-beta.19, which includes lance-format/lance#7687
for side-effect-free DirectoryNamespace read paths.
This fixes root-level read-only table opens that previously could
trigger `__manifest` creation through directory namespace construction,
including Hugging Face bucket reads with read-only tokens. A LanceDB
regression test now covers root listing operations without creating
`__manifest`.
Fixes#3633.
## Summary
This PR extends nested-field regression coverage across Rust
local/remote, Python sync/async, and Node so canonical escaped paths
stay consistent across scalar, vector, and FTS index lifecycle behavior.
It also aligns LanceDB's LabelList type gate with Lance by accepting
`LargeList<primitive>` columns while keeping `List<Struct<...>>`
unsupported until Lance defines stable membership semantics for struct
labels.
Part of #3406.
Moves the Lance dependency bump process into an in-repository skill so
local agents and GitHub Actions share the same workflow definition.
The update workflow is now an explicit, optional-tag entrypoint;
latest-release resolution, duplicate PR handling, Java/Rust dependency
updates, and Sophon follow-up are documented in the skill and backed by
a small deterministic helper.
This PR makes remote LanceDB tables usable from PyTorch multiprocessing
workers. Remote tables now carry enough safe JSON connection state to
reopen themselves after pickle/spawn or fork, and permutations lazily
rebuild their reader from restored tables instead of trying to reuse
process-local handles.
This addresses the remote-table gap in the PyTorch dataset path while
preserving the explicit connection factory escape hatch for custom
worker-side credential loading or non-serializable header providers.
Validated with targeted remote table, permutation, and PyTorch
DataLoader tests.
This updates the workspace Lance dependencies from `v7.1.0-beta.4` to
`v7.2.0-beta.3` and refreshes `Cargo.lock`.
The lockfile now points at Lance commit
`7c070f760fa8e24c8015cb2afbd22c5e6b7898e8` and includes the transitive
dependency updates required by the new beta.
Update the Rust workspace Lance git dependencies and Java lance-core
dependency to v7.2.0-beta.1.
This keeps LanceDB aligned with the latest Lance beta release and
refreshes the Cargo lockfile for the new Lance dependency graph.
The latest main Python workflow fails across multiple matrix jobs
because `test_remote_create_index_new_api` opens a remote table whose
mocked schema only exposes `id`, while the new `create_index(...,
config=...)` path validates the requested indexed columns.
This updates the remote-table fixture to include the indexed columns
used by the smoke test and checks the emitted column payloads, keeping
the test aligned with the schema-aware API path.
Fixes#3407.
Remote tables now resolve create-index field paths against the table
schema before sending requests, so nested, escaped, and case-insensitive
inputs use the same canonical path contract as local tables. Remote
`list_indices()` also canonicalizes returned columns against the current
schema, and the remote query tests lock explicit nested vector and FTS
request payloads.
LanceDB default vector column discovery only considered top-level
fields, so tables with a single nested vector leaf still required users
to pass an explicit field path. This updates Rust and Python discovery
to recurse into struct fields, return canonical field paths, and
preserve actionable errors when no default or multiple defaults exist.
The explicit nested path flow for index creation and search remains
supported across Rust, Python, and Node, with regression coverage for
single nested vector leaves, multiple candidate leaves, and schemas
without vector leaves.
Closes#3405.
Adds regression coverage for Python FTS APIs targeting nested text
leaves, including sync and async match, phrase, and hybrid query paths.
This also locks in the intended error boundary: nested text leaf paths
are valid, while struct containers, non-text leaves, and missing paths
remain rejected.
Fixes#3404.
Index metadata APIs now resolve stored field ids back to Lance canonical
field paths instead of leaf names, so nested indexes such as
`metadata.user_id` and escaped literal-dot fields round-trip through
`list_indices()`. Native index creation also canonicalizes the input
path before handing it to Lance, keeping local metadata consistent with
the field-path contract while remote responses continue to expose
server-provided canonical columns.
Fixes#3403.
Native index creation was resolving requested columns through top-level
Arrow schema lookup before handing the request to Lance, which rejected
nested paths and could collapse a nested field to its leaf name. This PR
resolves index targets with Lance field-path semantics, passes the
canonical path through to Lance, and reports indexed columns from field
ids as canonical full paths.
This also removes the Python native FTS guard that rejected dotted paths
so scalar, vector, and FTS index creation share the same nested-field
contract. Related to #3402.
This updates the agent instructions to codify the CI gates that failed
before code review started: PR titles must satisfy Conventional Commits,
and Python changes need root-level ruff format/lint checks.
It also makes the touched-language preflight explicit so mixed Rust,
Python, and TypeScript changes run the checks CI expects before opening
a PR.
This wires Lance's existing `jieba/*` and `lindera/*` native FTS
tokenizers through the Python SDK instead of leaving them behind
disabled features and narrow public typing. It also documents the
`LANCE_LANGUAGE_MODEL_HOME` model layout and adds Python coverage for
successful CJK indexing plus missing-model error guidance.
Closes#2168.
This follows the Rust-side Tantivy removal by deleting the remaining
Python Tantivy runtime, tests, and packaging references.
It also turns the legacy Python-only Tantivy parameters into explicit
errors and stops reading legacy `_indices/fts` directories so Python FTS
is fully native-only.
This PR migrates all Rust crates in the workspace to Rust 2024 edition
and addresses the resulting compatibility updates. It also fixes all
clippy warnings surfaced by the workspace checks so the codebase remains
warning-free under the current lint configuration.
Context:
- Scope: workspace edition bump (`2021` -> `2024`) plus follow-up
refactors required by new edition and clippy rules.
- Validation: `cargo fmt --all` and `cargo clippy --quiet --features
remote --tests --examples -- -D warnings` both pass.
This PR fixes the npm publish dry-run failure for prerelease versions
without changing the existing workflow trigger behavior. The publish
step now detects prerelease versions from `nodejs/package.json` and
always appends `--tag preview` when needed.
Context:
- On `main` pushes, the workflow still runs `npm publish --dry-run` by
design.
- Recent failures were caused by prerelease versions (for example
`0.27.0-beta.3`) running without `--tag`, which npm rejects.
- The previous `refs/tags/v...-beta...` check did not apply on branch
pushes, so dry-run could fail even though release tags worked.
clippy::string_to_string has been replaced by implicit_clone, so lancedb
will raise a build error in Rust 1.91. This PR suppresses it.
---
**This PR was primarily authored with Codex using GPT-5-Codex and then
hand-reviewed by me. I AM responsible for every change made in this PR.
I aimed to keep it aligned with our goals, though I may have missed
minor issues. Please flag anything that feels off, I'll fix it
quickly.**
Signed-off-by: Xuanwo <github@xuanwo.io>
This PR will migrate macos CI runners.
---
**This PR was primarily authored with Codex using GPT-5-Codex and then
hand-reviewed by me. I AM responsible for every change made in this PR.
I aimed to keep it aligned with our goals, though I may have missed
minor issues. Please flag anything that feels off, I'll fix it
quickly.**
Signed-off-by: Xuanwo <github@xuanwo.io>
This PR will add a timely lance release check for lancedb which will
auto bump lance while new tags released.
---
**This PR was primarily authored with Codex using GPT-5-Codex and then
hand-reviewed by me. I AM responsible for every change made in this PR.
I aimed to keep it aligned with our goals, though I may have missed
minor issues. Please flag anything that feels off, I'll fix it
quickly.**
---------
Signed-off-by: Xuanwo <github@xuanwo.io>
This PR will add an action that allow codex to auto upgrade lance.
---
**This PR was primarily authored with Codex using GPT-5-Codex and then
hand-reviewed by me. I AM responsible for every change made in this PR.
I aimed to keep it aligned with our goals, though I may have missed
minor issues. Please flag anything that feels off, I'll fix it
quickly.**
Signed-off-by: Xuanwo <github@xuanwo.io>
Xuanwo