A remote backfill submission validates its target column against a table
snapshot, but it did not carry the existing read-after-write freshness
headers. Immediately after `add_columns`, a stale query node could
therefore reject the newly committed column.
Route backfill submission through the remote table read fence so it
carries the version returned by the preceding write. The shared remote
submission path gives synchronous and asynchronous client surfaces the
same freshness guarantee.
The Windows wheel job is the slowest job in the PyPI release workflow.
Fat LTO of the cdylib is single-threaded and the peak-memory step of the
build, so it does not get faster with more cores — and it has already
caused rustc-LLVM OOM on the Windows runners for the nodejs builds.
Switch the job to thin LTO with 16 codegen units on a
`windows-2025-8x-x64` runner, trading some runtime performance on our
least performance-sensitive platform for build time. This matches what
the nodejs Windows builds in `npm-publish.yml` already do.
`pypi-publish.yml` is in this workflow's `pull_request` paths filter, so
this PR triggers a dry-run build that shows the new timing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arrow-rs and datafusion crates are released in lockstep, but
Dependabot has been opening one PR per sub-crate for them — the 58.3.0
to 58.4.0 wave produced four separate PRs for `arrow`, `arrow-array`,
`arrow-schema`, and `arrow-buffer`. The existing `rust-minor-patch`
group did not catch them because it only filters on `update-types` and
declares no patterns.
This PR adds an explicit `arrow-datafusion` group matching `arrow*`,
`parquet*`, `datafusion*`, and `object_store`, so those bumps arrive as
a single PR. It is listed before `rust-minor-patch` because a dependency
joins the first group it matches, and it deliberately omits
`update-types` so major bumps are grouped too.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bugs in Node's reading of the embedding_functions schema metadata.
First, parseFunctions keyed its result map by function name, so a table
whose metadata configures the same function for two vector columns came
back with only the last one. It now keys by the vector column, the
convention Python's parser already uses.
Second, Node could not read metadata written by the Python bindings at
all, which spell the keys snake_case: configs parsed with both columns
undefined, breaking embedding application on add() and leaving only
query-side embedding working. The parse now accepts both spellings.
Both fixes land in one shared parser used by every reader --
parseFunctions and the makeArrowTable schema validator, which had its
own private camelCase-only parse -- so the wire contract cannot fork
between entry points. A config naming no source or vector column is an
error at the boundary rather than a default downstream, as are two
configs claiming one column. The "vector" fallback remains only on the
optional field of user-supplied configs.
Breaking: parseFunctions is exported and its map keys change from
function name to vector column.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Remote Function bindings can expose a nullable parameter schema even
when the source table column is non-nullable. Binding validation rebuilt
the exact input schema from table nullability and rejected this safe
widening.
Accept non-null table columns for nullable Function parameters while
continuing to reject nullable table columns for non-null parameters. All
other input schema fields remain exact, including named multi-input
ordering, names, types, and metadata.
Remote Function catalog requests used singular endpoints that are not
exposed by Phalanx. Route registration to `POST /v1/functions/create`
and exact-version lookup to `POST /v1/functions/get`, while preserving
the existing typed Job submission and wait behavior.
As in https://github.com/lancedb/lancedb/pull/3977, we're trying to
reduce anything in the lancedb skill that duplicates other docs. So this
shrinks the branch-ops logic down to a few lines that mostly just point
the agent to fetch the branching docs from lancedb.github.io.
Run stats (2 runs each):
<img width="1001" height="232" alt="Screenshot 2026-08-20 at 5 02 34 PM"
src="https://github.com/user-attachments/assets/a3f4d305-278e-4093-b153-07f0af57b251"
/>
This is out of order, rearranged:
|condition|time (sec)|cost|
|---|---|---|
|No branch_ops.md|250|1.33|
|No branch_ops.md|227|1.26|
|Old branch_ops.md|116|0.83|
|Old branch_ops.md|127|0.87|
|New branch_ops.md|135|0.86|
|New branch_ops.md|147|0.93|
Averaged between each of the two runs:
<img width="775" height="337" alt="Screenshot 2026-08-20 at 5 34 15 PM"
src="https://github.com/user-attachments/assets/4f2fb2a8-3112-4614-87d9-8dbf807f3b75"
/>
It seems helpful to have *some* doc about branching; otherwise the model
gets a little confused about our branch model and what methods to call.
But it looks like the new one (in this PR; all just references to
current docs) is basically as good as the old one (lots of duplicative
text).
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Problem
Generic job result propagation exposed the PyO3 representation of Rust's
unit value as `()` in Python. Unit jobs therefore returned an empty
tuple instead of `None`, breaking the documented `Job.wait()` contract
and the Python doctest workflow.
## Behavior
Unit job completion now converts explicitly to Python `None`. Typed job
results continue to pass through unchanged, with synchronous and
asynchronous regression coverage.
Follows lance-format/lance#8680, which removes
`FLAG_MEM_WAL_INDEX_CATCHUP`.
With one set of semantics there is no mode to switch into.
## Removed
`require_mem_wal_index_catchup` — the activation entry point — from the
trait,
from `Table`, and from the LSM merge module.
## The read path
`exclusion_watermarks` loses its `catchup_required` argument and keeps
the
conservative branch: an index with no entry is not known to hold these
rows, so
every generation stays readable from its SSTable. Nothing is excluded
until an
index records that it covers those generations, so a table that has
never
recorded catch-up reads every row from its SSTables rather than assuming
the
base covers them.
## One guard needed a replacement, not deletion
`refresh_column` and computed-column declaration refuse a table whose
rows sit
in un-compacted tiers, since refresh enumerates base fragments and would
silently omit them. They keyed on the feature bit because
`unset_lsm_write_spec` **drops the MemWAL index** — after an unset the
write
spec no longer describes such a table, and the bit was the only marker
that
outlived it. Two tests covered this, so deleting the term would have
dropped a
tested property.
Both guards now check for MemWAL shard directories on storage, which
outlive
the index. That is strictly wider than the bit ever was: the bit only
marked
tables where activation had run.
## Two tests conflated two different things
An index that is *caught up* and one that is *untracked* both fell back
to the
compaction watermark, because absence carried no information without the
bit.
Absence now means "not caught up", so untracked retains everything.
`an_untracked_index_does_not_widen_a_lagging_sibling` becomes
`an_untracked_index_retains_everything`, with the genuinely-caught-up
case
asserted separately.
## Testing
933 `lancedb` lib tests. `cargo fmt` clean. (The pre-existing
`Error::Http`
build failure in `job.rs` without the `remote` feature is unrelated and
untouched.)
The Linux Rust job can exhaust its disk after restoring a large fallback
target cache and compiling multiple feature graphs into one target
directory.
Run remote tests in an independent job with registry-only caching, and
run the simple example with all features so it reuses the preceding
build artifacts. This preserves remote coverage and fork behavior while
preventing all-features and remote-only artifacts from accumulating
together.
Failure evidence:
https://github.com/lancedb/lancedb/actions/runs/32467317540/job/96726650990
## Problem
The canonical Function wire values and typed remote Job contract do not
yet provide a Python authoring surface or catalog client, so users
cannot package a scalar callable, register it, or reopen the exact
immutable Function version.
## Behavior
This adds scalar-only `@udf` authoring with deterministic annotation or
explicit Arrow schema validation, content-addressed Python artifacts,
and an internal scalar-to-Arrow-batch adapter descriptor. Registration
payloads model non-secret environment values and secret names only.
Remote connections can submit `create_function_async` and receive a
typed `Job<FunctionVersion>`, then reopen that exact version by name and
version ID. Synchronous connections can call `create_function` to submit
and wait for the immutable version in one operation. Local Function
catalog operations return a stable `NotSupported` error. Shared
Rust/Python golden payloads and mocked catalog responses freeze the
request, typed terminal result, and exact lookup contract.
## Validation
- Rust formatting, remote check, clippy, and focused LDB-1/LDB-2 tests
- Python formatting, lint, and focused LDB-1/LDB-2 tests
- Python API documentation build
Function applications from the canonical remote contract cannot
currently declare scalar or grouped computed-column outputs atomically.
This adds the remote-only declaration contract for scalar,
struct-as-one-column, and expanded named-struct outputs. It validates
result mappings, fixes exact input/output Arrow schemas in the request,
persists grouped sibling metadata, and keeps local Function execution
unsupported. Unknown newer application or binding metadata remains
readable, while schema-changing mutations fail closed instead of
rewriting it.
Stable Lance field IDs are deliberately not a declaration prerequisite
in this slice. Inputs bind by parameter name and field path; Sophon
remains responsible for exact-version validation, atomic all-NULL
sibling creation, binding identity and revision allocation, and
persisted output identities.
LanceDB's Python SDK now requires Pydantic `>=2.7.4,<3` and uses the v2
APIs throughout. This removes dual-version behavior from schema
conversion, query serialization, embedding models, and Function wire
models while preserving their existing public and canonical-wire
behavior.
The minimum-dependencies CI job pins Pydantic 2.7.4 so the declared
compatibility floor remains covered.
Local benchmarks currently inherit the release profile's fat LTO and
single codegen unit, making local iteration pay release-artifact build
costs.
Provide repository-defined profiles for no-LTO local work and cheaper
benchmark builds, and document when each profile is appropriate. Release
artifacts continue to use fat LTO.
PyO3 exposed `LsmWriteSpec` with its default `builtins` module, causing
mkdocstrings to resolve the public `lancedb.LsmWriteSpec` re-export as
`builtins.LsmWriteSpec` and fail the documentation build. Declare the
native extension module and pin the public re-export with a regression
test.
This also applies the repository's current Ruff formatter to seven
previously unformatted Python scripts.
## Problem
Enterprise Function-backed computed columns need a stable SDK contract
before Sophon catalog and execution endpoints can be added. The existing
`Job` API can only represent unit terminal results, and there is no
shared Rust/Python wire definition for immutable Function versions,
applications, bindings, or refresh results.
## Behavior
This introduces remote-only canonical Function values in Rust and
Python, evolves `Job<T = ()>` to decode typed remote terminal results
while keeping local spawned operations unit-typed, and fixes the
cross-language contract with shared JSON golden fixtures. Unknown fields
and discriminator values remain forward-decodable, while canonical
output contains only fields known to the client. Function models contain
secret names only.
Sophon remains the sole owner of catalog persistence, environment bake,
secret resolution, execution, and publication. This PR does not add
authoring/catalog endpoints, local execution, refresh runners, or live
Sophon E2E coverage.
Background: if we keep adding stuff to the lancedb skill that repeats
other knowledge, we're basically creating a whole new docs site, which
means one more thing that can get out of date. Worse, if it gets out of
date, it will tell agents to do the wrong thing.
These files were added without a ton of analysis of whether they'd be
improving agent performance at all. It looks like they don't really:
<img width="644" height="90" alt="Screenshot 2026-08-20 at 5 21 03 PM"
src="https://github.com/user-attachments/assets/44e60436-b7ad-498b-8e73-0181385c7c60"
/>
(top run is without these docs, bottom run is with them - arguably these
docs might even make the agent a little slower! that's probably noise
though; I'd just say at least they're unnecessary.)
So this PR just removes them. We'll more judiciously add bits we need
and/or point to preexisting docs, to avoid duplication.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`cargo deny` did not check crate-level dependency declarations against
`[workspace.dependencies]`, so a crate used by both the core crate and
the bindings could be declared independently in each one and drift. For
example `tokio` was pinned at `1.23` in `rust/lancedb` and `1.40` in
`python`, and `pin-project` at `1.0.7` in the workspace table but
`1.1.5` in `python`.
This PR turns on cargo-deny's `bans.workspace-dependencies` lint, which
fails when a dependency is used by more than one member without going
through `workspace = true`, and when a `[workspace.dependencies]` entry
is used by nobody.
Enabling it surfaced 12 violations. Fixing them means adding `bytes`,
`lancedb`, `serde`, `serde_json`, `tempfile`, `tokio`, and `uuid` to
`[workspace.dependencies]`, and pointing the `arrow`, `arrow-buffer`,
`async-trait`, `chrono`, and `pin-project` declarations at the entries
that already existed. `Cargo.lock` is unchanged, so resolution is the
same as before.
The shared `chrono` entry now carries `default-features = false,
features = ["clock"]`, matching what `nodejs` and `python` already asked
for — cargo ignores a member's `default-features = false` unless the
workspace entry sets it too. On the targets we build, `clock` covers
everything `rust/lancedb` was getting from chrono's defaults.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
<!-- lance-gatekeeper-fix:v1 agent=40e5cf476a59265c71653574eda834d2
generation=1 -->
## Summary
- preserve incoming PyArrow `arrow.json` fields while schema
sanitization aligns input to a stored `lance.json` schema
- let Lance perform the required JSONB encoding instead of relabeling
raw JSON bytes as encoded storage
- cover both merge insert and the conditional add sanitization path with
end-to-end regression tests
## Root cause
Python schema sanitization aligns incoming data to the table schema
before passing it to Lance. Merge insert always takes this path, while
add takes it conditionally for preprocessing such as non-default
bad-vector handling or embedding functions. For JSON columns, the cast
changed logical `arrow.json` strings into the table's JSONB-backed
`lance.json` storage type without encoding the bytes, so Lance treated
raw JSON text as JSONB.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_table.py
-k 'merge_insert or add_sanitization_encodes_json' -q`
- targeted schema-cast and JSON encoding tests
- `ruff check .`
- `ruff format --check python/python/lancedb/table.py
python/python/tests/test_table.py`
Fixes#3923
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Why
Four of the eight LSM methods are **remote-only in the core**. `impl
BaseTable for NativeTable` implements only
`set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`,
`compact_lsm` and `get_lsm_stats` fall through to trait defaults
returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and
`checkpoint_lsm` is built on all three.
That explains the state of the bindings: Node had bound the four that
work against a local table and stopped, so a Cloud user could install an
LSM write spec but had no way to observe fresh-tier state or drive a
checkpoint. Java had none of it at all.
| SDK | set/unset/get spec | closeWriters | flush | compact | getStats |
checkpoint |
|---|---|---|---|---|---|---|
| Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Node *(before)* | ✅ | ✅ | — | — | — | — |
| **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** |
| Java *(before)* | — | — | — | — | — | — |
| **Java (after)** | **new** | n/a | **new** | **new** | **new** |
**new** |
Go and C are separate repos and are out of scope here. `closeLsmWriters`
drains cached in-process shard writers, so it has no meaning for Java,
which is a pure REST client.
## Node
Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and
`getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats`
/ `MemtableStats` objects — typed rather than a JSON blob, matching the
existing `LsmWriteSpec` object in the same file, with `u64` cast to
`i64` per that file's convention.
Because these four are remote-only, the new tests assert each binding
reaches the core and surfaces `NotSupported` against a local table. That
covers the wiring; behavior against a real endpoint stays covered by the
mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`.
## Python
No new methods. All eight are on `LanceTable`, `AsyncTable` and
`RemoteTable` — the last four landed on the sync `RemoteTable` in #3961,
which is merged into this branch.
What was missing here was reachability. `LsmWriteSpec` was importable
only from the private `lancedb._lancedb`, appearing in `table.py` solely
under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no
mention of it, which per the repo's docs guidance means it rendered
nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in
`__all__`, and documented.
## Java
Java reaches LanceDB purely over REST through the generated Lance
Namespace client, and these routes are not in that spec, so they are
issued through a small dedicated client rather than added to the spec.
That call is revisitable — LSM is one of four unspecified route families
alongside `multipart_write`, `page_cache/prewarm` and
`branches/diff|merge`. If those are ever regularized into the spec as a
group, `LanceDbTableLsm` is one file that gets deleted.
`LsmWriteSpec` here is deliberately **not**
`org.lance.memwal.InitializeMemWalParams`. That type defaults to
maintaining *no* indexes where a spec here defaults to maintaining
*every* index, and it cannot express the `null` that asks the server to
resolve the set:
| Value | On the wire | Meaning |
|---|---|---|
| unset (null) | `null` | Server resolves **every** maintainable index |
| `Collections.emptyList()` | `[]` | Maintain **none** |
| `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those |
A dedicated test pins null and `[]` as distinct on the wire, since
collapsing them is the failure mode that motivated a LanceDB-owned type.
`checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs`
with its constants and status semantics intact: 429/503 retried in place
against an 8-budget, 421 restarting from flush against a 3-budget, 5s
poll, and a target watermark fixed after the seal so it terminates under
write load.
`getLsmStats` returns typed `LsmStats` / `BucketStats` /
`GenerationStats` / `MemtableStats`, mirroring the Rust structs in
`rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes.
Decoding is strict — see below.
## Review feedback
Both gatekeeper findings were real. Each was reproduced against the
scripted test server first, and each fix ships with the reproducer as a
regression test.
**The transport was doubling every checkpoint retry budget.**
`HttpClients.createDefault()` installs Apache's default response retry
strategy, whose retryable-status list is exactly 429 and 503 — the two
statuses `isRetryable` owns. A 429 held against `flush_lsm` issued
**18** wire requests where the loop intends 9, and `compact_lsm` was
retried in place despite the loop being built to fall through to a fresh
stats poll instead. Timing confirmed the mechanism: that run took 25.4s
≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry
interval.
Automatic retries are now disabled, so the checkpoint loop is the sole
owner of the 421/429/503 transitions. A side effect worth noting:
`testCheckpointRetriesRetryableStatusInPlace` was passing on a
transport-absorbed 429 and never reaching `issue()`'s retry branch at
all. It now exercises the real path.
**Stats decoding failed open.** `getLsmStats` read the response with
Jackson's `path()`, which yields a missing node that iterates as an
empty array — making "malformed" indistinguishable from "no buckets",
which is indistinguishable from "drained". Four separate payloads made
`checkpointLsm()` report convergence for a checkpoint that never ran:
| Response | Before | Now |
|---|---|---|
| `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ |
| `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` |
| empty response body | **reported success** | `IllegalStateException` |
| bucket missing required fields | **reported success** |
`IllegalStateException` |
The empty-body row is the one to weight: a proxy 200 with no body is a
realistic production event, and it silently reported a checkpoint that
never happened.
Decoding is now strict and fails closed, matching the serde contract on
the Rust side exactly. One deliberate deviation from the review comment,
which asked that *only* explicit JSON `null` count as disabled: Rust has
`#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to
`None` there too. Java now matches that. It is an absent-or-malformed
**`buckets`** that fails closed, which is the case the comment was
actually protecting.
## Testing
- Java: **33 passing** (8 existing + 25 LSM) against a scripted
`com.sun.net.httpserver.HttpServer` — no new test dependency. Wire
assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`;
checkpoint tests cover convergence, not piling onto a latched bucket,
421 restart-from-flush, 429 retry-in-place, terminal-status propagation,
reissue exhaustion, the exact wire-request count against the retry
budget, and five malformed stats payloads.
- Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm
run tsc`, `npm run lint`, `npm run docs` all clean.
- Python: `ruff format --check` and `ruff check` clean.
- Java formatting: `./mvnw -pl lancedb-core spotless:apply` and
`spotless:check` both clean under a JDK 11 toolchain.
## Note: spotless needs a pre-16 JDK
`./mvnw spotless:apply` fails on JDK 16+ with
`JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7,
pinned at `java/pom.xml:34`, predates JDK 16's compiler API change.
**This is pre-existing** and reproduces on a pristine `main` checkout.
It is not a blocker, just a toolchain requirement. Spotless was run
against these sources under JDK 11 and both `spotless:apply` and
`spotless:check` pass on the whole module:
```shell
JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply
```
Bumping the plugin so it works on modern JDKs is still worth doing, but
separately from this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
The sync `RemoteTable` carried `set_lsm_write_spec`,
`unset_lsm_write_spec`, `get_lsm_write_spec`, and `close_lsm_writers`,
but not `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, or
`get_lsm_stats`.
That left the four LSM control methods reachable from `AsyncTable` only.
They are also the four that *only* work against a remote table —
`NativeTable` does not override the `BaseTable` defaults, so on a local
table they return `NotSupported` (`rust/lancedb/src/table.rs:679-701`).
The net effect for sync users:
| | `checkpoint_lsm` / `get_lsm_stats` |
|---|---|
| `LanceTable` (sync, local) | present, but always `NotSupported` |
| `RemoteTable` (sync, remote) | `AttributeError` — method absent |
| `AsyncTable` (remote) | works |
So there was no working sync path at all, despite the Rust `RemoteTable`
implementing every one of these against real endpoints.
## Changes
* Add `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, and `get_lsm_stats`
to `lancedb.remote.table.RemoteTable`, mirroring the delegation style of
their neighbours.
* Correct the docstrings on `set_lsm_write_spec` /
`unset_lsm_write_spec`, which read `"""Not supported on LanceDB
Cloud."""` although `rust/lancedb/src/remote/table.rs:2549-2601`
implements both against `/v1/table/{}/set_lsm_write_spec/` and
`/unset_lsm_write_spec/`. They appear to have been copy-pasted from
`set_unenforced_primary_key` directly above.
No Rust or PyO3 changes — the bindings and the `AsyncTable` methods
already existed. The `Table` ABC is left alone, matching how the
existing `*_lsm_write_spec` methods are declared on the concrete classes
only.
## Tests
Four new tests in `python/python/tests/test_remote_db.py`, against the
existing mock HTTP server:
* `test_get_lsm_stats_sync` — the server payload round-trips into the
dict, and `include_generation_rows` defaults to `False` and is forwarded
when set.
* `test_get_lsm_stats_sync_returns_none_when_lsm_disabled` — a
`{"lsm_stats": null}` envelope yields `None` rather than an error.
* `test_flush_and_compact_lsm_sync` — both are one-shot POSTs answered
`202` with no body.
* `test_checkpoint_lsm_sync` — pins the binding to the endpoints it
drives (`flush_lsm` then `get_lsm_stats`); the convergence loop itself
is already covered in Rust.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Context
Java users opening catalog-backed tables with vended credentials
currently lack a documented workflow. Opening the catalog-returned URI
directly drops the namespace-provided storage options and automatic
credential refresh.
Document the namespace-backed `Dataset.open()` path so temporary object
store credentials are applied and refreshed transparently.
## Summary
`LanceHybridQueryBuilder` (sync hybrid search,
`table.search(query_type="hybrid")`) silently ignored `.offset()`.
`self._offset` was never forwarded to the vector/FTS sub-queries and
never applied when slicing the final combined/reranked result, so
`.offset(N)` behaved identically to `.offset(0)` — no error, just wrong
pagination.
Fixes#3765
## Changes
- `_create_query_builders()`: each sub-query now fetches `limit +
offset` rows so there's enough data to slice the correct window out of
after combining/reranking.
- `_combine_hybrid_results()` / `to_arrow()`: the final table is sliced
with `offset=self._offset` instead of always starting at 0.
## Test plan
- [x] New regression test `test_hybrid_query_offset` in
`python/python/tests/test_hybrid_query.py`
- [x] `uv run --extra tests pytest python/tests/test_hybrid_query.py
-vv` — 13 passed
- [x] `uv run --extra dev ruff format` / `ruff check` — clean
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
## Summary
Fixes naive `lit(datetime)` equality filters against table timestamp
columns on non-UTC hosts, and adds the integration matrix from #3262.
## Failure (before)
On a machine in US Eastern (UTC−4 / EDT), with PyPI `lancedb==0.36.0`:
```python
from datetime import datetime
import lancedb
from lancedb.expr import col, lit
db = lancedb.connect("memory://")
ts = datetime(2024, 7, 1, 10, 0, 0) # naive
table = db.create_table("t", [{"id": 1, "ts": ts}])
rows = table.search().where(col("ts") == lit(ts)).to_list()
# actual: [] (0 rows)
# expected: 1 row
```
### Root cause
In `python/src/expr.rs`, `expr_lit` converted every `datetime` via
Python's `.timestamp()`:
- **naive** `.timestamp()` = local wall → UTC epoch (shifted by host
offset)
- **PyArrow naive** storage = UTC wall-clock microseconds (no local
shift)
So `lit(naive)` became `CAST('2024-07-01 14:00:00' AS TIMESTAMP)` on EDT
while the table held `10:00:00`.
## After
Naive datetimes are interpreted as UTC wall clock
(`replace(tzinfo=timezone.utc).timestamp()`), matching Arrow storage.
Aware datetimes still use `.timestamp()` (correct epoch).
Same repro on this branch: **1 matching row**.
## Tests
Added `TestExprDatetimeTimezoneIntegration` covering:
| Case | Result |
|------|--------|
| both naive | match |
| both same TZ (UTC) | match |
| different TZs, same instant | match |
| table TZ + naive lit | match (wall clock) |
| table naive + aware lit | match |
| naive lit SQL is wall clock, not local-shifted | asserts `10:00:00` in
SQL |
### Verification
```bash
cd python
maturin develop
pytest python/tests/test_expr.py -v
```
**102 passed** (full `test_expr.py`, including the 6 new cases).
Closes#3262
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
LanceDB Cloud and Enterprise support computed columns through the REST
API,
so declaration dispatches per backend: local tables plan the expression
themselves, remote ones send {name, computed} entries for the server to
plan. A remote refresh is the server's backfill job --
refresh_column_async
submits it and returns a handle whose successful wait establishes a
read-freshness baseline on the submitting handle, unless a checkout has
pinned the handle by the time the job completes; the blocking form
refuses
rather than invent a fill count the server does not report.
Declaration entries are built from the namespace client's
AddColumnsEntry
model (lance-namespace 0.11.0, via the lance beta.13 pin), so the
payload
shape is compile-checked against the published contract.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
Updates the Lance Rust workspace dependencies and Java lance-core
dependency to
[v11.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.13).
Adds the required `ListTablesResponse.context` compatibility field and
validates the workspace with Clippy warnings denied.
Mirrors create_index's dual surface: the blocking refresh_column keeps
returning {rows_filled, version}, and refresh_column_async returns the
same
Job handle create_index uses, running the refresh as an in-process task.
Invalid input is reported by the submitting call rather than by the job.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
add_columns().computed("doubled", "x * 2") stores the expression in
field
metadata and commits the column empty; a later refresh fills it. Type
and
inputs are derived from the expression.
The declaration stays authoritative for its lifetime: writes that would
give
the column a value (append, update, merge, SQL insert), schema changes
that
would break the stored expression or reshape its output, metadata edits,
volatile expressions, declaration metadata arriving through any path but
the
validated declare call, and LSM write specs in either order against
latest
committed state are all refused. The LSM check also refuses on the
mem-wal
catch-up feature flag, which outlives unset and marks retained SSTable
rows.
Simultaneous declare/install interleavings conflict at commit via
lance's
mem-wal rule (lance#8539). Local tables only.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
> Stacked on #3780. Blocked only on #3922 (`lance` → `v11.0.0-beta.6`),
so CI
> stays red until that lands.
## Missing coverage must mean "not known to be covered"
#3780 caps the SSTable exclusion watermark at an index's recorded
catch-up when
there is one, and silently ignores the case where there is none. On a
table that
requires catch-up, an absent entry means the index is *not* known to
hold the
compacted rows — and the LSM base arm reads base through the index
(`fast_search`, no brute-force tail), so dropping that SSTable loses
those rows
for that query.
```rust
Some(caught_up) => watermark = watermark.min(caught_up),
None if catchup_required => watermark = 0, // retain everything
None => {}
```
`catchup_required` reads the manifest feature bit directly, and requires
both
words: a half-set manifest is treated as legacy, which is the
conservative side.
Without the bit the field is not maintained at all, so absence carries
no
information and behaviour is unchanged.
## Activation, as a table-level entry point
`Table::require_mem_wal_index_catchup()` performs the one-way switch,
separate
from `set_lsm_write_spec`: a table carrying the bit retains every
generation
until something records catch-up, so it has to follow the deployment of
whatever
repairs coverage, not the creation of the table.
This is a convenience, not the only path — a writer holding the dataset
calls
the equivalent on `DatasetMemWalExt`, which is what the WAL pod does.
Lance
enforces the preconditions either way: the MemWAL index must exist, and
the
table must not already carry `compacted_sstables` from before this
protocol,
since those numbers cannot be validated.
## Still correct after the Lance rework
lance-format/lance#8481 replaced the transmitted `IndexCatchupAdvance`
with a
position derived at commit time from the version a transaction read.
That
changed how a writer earns coverage; it did not change what a reader may
conclude from its absence. The rule here, and the field it reads, are
unchanged.
## Tests
Existing `exclusion_watermarks` unit tests carry the new argument.
Coverage
against a real dataset follows once #3922 lands and this can build.
Updates the Rust workspace Lance dependencies and Java lance-core
dependency to v11.0.0-beta.10. No compatibility fixes were required;
workspace clippy with all features and Rust formatting pass.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.10
## Summary
- add `drop_table_async` and return a job handle while preserving
`drop_table`
- consume remote 202 responses with cleanup job IDs and retain
older-server compatibility
- expose the API through Python and TypeScript connection wrappers
Say what resolving "latest" actually does: pick the newest release,
preferring stable over pre-release, and skip the run if it is not newer
than the version pinned in Cargo.toml.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`exclusion_watermarks` resolved a single index and capped SSTable
exclusion at that index's catch-up watermark. It now takes every index
the query relies on and retains to the **lowest** of them, and the
resolver collects arms together rather than returning at the first
match.
This is groundwork, not a fix for a reachable bug: `reject_unsupported`
refuses hybrid search, so the vector and full-text arms are mutually
exclusive and the list never holds more than one entry today. The
generalisation is what the remaining work below plugs into.
Unchanged: a plain scan uses the compaction watermark alone, an index
with no catch-up entry contributes no cap, and a caught-up index falls
back to the compaction watermark. Taking a minimum over more indexes can
only lower a watermark, so the failure direction is "read an SSTable
unnecessarily", never "miss rows".
## Tests
Three in `lsm`: the existing lagging-index test updated for the new
signature;
`exclusion_watermark_takes_the_minimum_across_every_index_used` (two
indexes at 7 and 4 against compaction at 9 — each alone stops at its own
watermark, together the lower governs, order-independent); and
`an_untracked_index_does_not_widen_a_lagging_sibling`.
`cargo test -p lancedb --lib` — 45 lsm tests, 484 in the crate. `cargo
fmt --check` clean.
## Follow-ups
This crate pins lance to a released tag, so anything needing unreleased
Lance symbols waits for a bump.
1. **Select legacy versus strict semantics from the feature bit.** On a
table with `FLAG_MEM_WAL_INDEX_CATCHUP` set, a *missing* entry must mean
"not caught up" and retain the SSTables, instead of leaving the
compaction watermark unchanged. Needs the bit from
lance-format/lance#8263. **This must land before any table is
activated** — otherwise the bit is set while queries still read
permissively.
2. **Collect scalar and bitmap-family prefilter indexes.** The genuinely
multi-index query is a vector search with a scalar prefilter, and it is
gated on the vector index alone today. Identifying the others needs the
planner's chosen indexes, not the columns the filter names, so it needs
a Lance-side helper.
3. **Verify a retained SSTable can actually answer.** Both base and
SSTable arms use `fast_search`; a source without a compatible index
contributes nothing, so retention alone does not guarantee its rows are
returned. Needs a flat-search fallback or an explicit error in Lance's
`LsmScanner`.
4. **Planner-level integration tests.** Current tests exercise the
watermark arithmetic directly. End-to-end coverage over real queries —
prefilter forms, legacy versus activated, missing index and missing
shard entries — depends on 1–3.
## What is the bug?
#3731 tries to distinguish a missing table from a corrupt table after
Lance returns `DatasetNotFound`. It does that by listing the database
parent and treating a physical `<name>.lance` entry as evidence that the
table exists.
That premise is not sound for a listing database. Table creation writes
data before atomically committing the first manifest, so the same
physical prefix can represent a live concurrent create, abandoned
uncommitted data, or an old empty directory. It is not evidence of a
committed table. The parent listing also makes every missing-table open,
including the create-on-miss path, perform work proportional to the
number of sibling tables. Cloud `list_with_delimiter` exhausts all pages
before returning.
## How does this PR fix the problem?
This PR makes the committed Lance manifest the sole table-existence
authority for listing-database opens:
- `DatasetNotFound` maps directly to `TableNotFound`; no parent or
target storage probe runs.
- Other Lance load errors continue to propagate unchanged.
- A physical directory, object prefix, or uncommitted data file alone
does not block `Create`.
- Concurrent `Create` requests are arbitrated by the conditional
version-1 manifest commit: one succeeds and the loser receives
`TableAlreadyExists`.
- `table_names` is documented as physical discovery, not an atomic
table-existence check. Its snapshot can contain an entry that is still
being created, has only uncommitted storage, or is concurrently dropped.
This removes the need for a new Lance object-store capability. LanceDB
remains on the official Lance `v11.0.0-beta.6` dependency from `main`;
the merge commit for lance-format/lance#7722 is an ancestor of that tag,
so the ambiguous-GCS-500 corruption-prevention fix is retained.
## Performance evidence
Lower is better. The benchmark uses real `.lance` directories with
marker objects on the local filesystem; fixture creation and teardown
are outside the timed region. Baseline is `origin/main` at `6fb976cf`,
candidate is `e1240751`. Both were built from the same lockfile on the
same macOS arm64 machine with the repository's `release` profile (fat
LTO), then executed in alternating baseline/candidate order for three
pairs. Each run used 10 warmups and 100 distinct missing-table opens per
scale. The table reports the median of the three run-level percentiles.
| Scenario / metric | Baseline | This PR | Benefit |
| --- | ---: | ---: | ---: |
| 1,000 real sibling directories, p50 | 11.905 ms | 21.042 us | 566x
speedup |
| 10,000 real sibling directories, p50 | 143.630 ms | 18.375 us | 7,817x
speedup |
| 100,000 real sibling directories, p50 | 1.991 s | 19.917 us | 99,984x
speedup |
| 100,000 real sibling directories, p95 | 2.346 s | 25.792 us | 90,965x
speedup |
These results validate removal of the sibling-cardinality dependency in
this local-filesystem workload; they are not an extrapolation to
production GCS latency. A structural object-store regression test
separately asserts that opening one missing table performs zero
parent-scoped `list`, `list_with_offset`, or `list_with_delimiter`
calls.
Run with:
```bash
BENCH_SIBLINGS=1000,10000,100000 BENCH_WARMUPS=10 BENCH_TRIALS=100 \
cargo run --locked --release --quiet -p lancedb --example bench_open_missing_table
```
## Correctness and compatibility boundaries
- An empty `.lance` directory or orphan data without a committed
manifest now opens as `TableNotFound` and may be replaced by a
successful `Create`.
- Two synchronized creators sharing one object store deterministically
produce one success and one conditional-manifest conflict mapped to
`TableAlreadyExists`.
- A readable manifest remains authoritative; non-`DatasetNotFound`
corruption, external-manifest, authorization, and object-store errors
are not folded into `TableNotFound`.
- `TableCorrupted` remains in the public error enum for compatibility,
but this listing-database fallback no longer synthesizes it from an
ambiguous physical footprint.
- Reliably distinguishing `Missing`, `Creating`, and `Corrupt` would
require explicit authoritative lifecycle/catalog metadata (for example a
leased creation record). It cannot be inferred from a directory or
prefix, and is outside this incident fix.
## Validation
- `cargo fmt --all -- --check`
- `cargo check --quiet --locked -p lancedb --features remote --tests
--examples`
- `cargo clippy --quiet --locked -p lancedb --features remote --tests
--examples -- -D warnings`
- `cargo test --quiet --locked -p lancedb --features remote --tests`
- library: 843 passed, 1 ignored
- integration groups: 39 passed, 6 passed, 5 passed
- focused coverage for empty directories, orphan data, physical listing
snapshots, zero parent listings, and concurrent manifest arbitration
Closes#3704
## Problem
Transforms can fail on bad data (e.g. nulls/NaNs from incomplete user
surveys). Today any transform exception aborts iteration, and there is
no way to skip invalid rows during loading.
## Solution
New `on_transform_error` parameter on `StreamingDataset`:
- `"raise"` (default, matches current behavior and the convention in
tf.data / WebDataset / Ray Data)
- `"skip"` — drop the failing rows and continue
- `"warn"` — like skip, plus a logged warning per failing batch
- a WebDataset-style callable `handler(exc) -> bool`, so users can skip
only expected error types
Key design points:
- **Row-granular skipping**: when a batch fails, the transform is re-run
on single-row slices so only the rows that actually fail are dropped
(avoids Ray-style whole-block loss). Skips are counted in a new
`rows_skipped` property.
- **No crash on uneven skips**: the round-robin loop now ends the epoch
at the last cycle where every split still has a row, instead of hitting
`IndexError` when a split runs dry early.
- **Exact resumability under skips**: checkpoints are now
position-based. `state_dict` gains `positions_consumed_per_split` (exact
for owned splits), and a new `merge_state_dicts` static method combines
per-rank states via elementwise max for elastic resume across topology
changes. Old checkpoints without the new key still load. Positions equal
sample counts when nothing is skipped, so existing behavior is
unchanged.
- **Guardrail**: transforms returning the wrong number of rows now raise
a clear `ValueError` instead of silently corrupting split accounting.
### Answers to the issue's open questions
- *Can we do this?* Yes — all transforms funnel through one guarded call
in the Stage 2 pipeline.
- *What do other libraries do?* tf.data `ignore_errors()`, WebDataset
`handler=`, Ray `max_errored_blocks`; MosaicML StreamingDataset offers
nothing (skipping conflicts with its determinism model). This design
follows the common conventions: raise by default, opt-in skipping,
count/log drops.
- *Error handling or pre-filtering?* Both: the existing `filter=`
remains the recommended tool for predictable bad data (splits are built
post-filter, so all guarantees hold — now documented);
`on_transform_error` covers failures not expressible as a predicate.
- *Impact on splits / elastic determinism?* Per-split sample sequences
stay deterministic (skips are data-dependent, not topology-dependent).
With unequal bad-row counts across splits the last few global steps of
an epoch can differ across topologies (bounded by the skew), which is
documented on the parameter. With equal counts per split, full
determinism is preserved — covered by a test.
## Testing
15 new tests in `test_elastic_dataloader.py` covering: default raise,
invalid values, uniform and uneven skips (including epoch-end
truncation), warn logging, selective callable handlers, wrong-row-count
guardrail, determinism across runs and across world sizes (1/2/3/4) with
skips, exact mid-epoch resume with skips on the same topology, elastic
resume via `merge_state_dicts` (ws=2 → ws=1), merge validation, and
backward-compat loading of old checkpoints.
Note: relying on CI for the test run — my local machine OOMs during the
final link of the native extension. The change itself is pure Python.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
PyO3 defaults native extension classes to `builtins`, so
mkdocstrings/Griffe could not resolve the newly documented
`lancedb.Session` alias and `Deploy docs to Pages` failed on `main`.
Declare the extension module for the public native types referenced by
the Python API docs so Griffe resolves them through `lancedb._lancedb`
and Pages can build again.
Validated with the docs toolchain used by CI (`griffe==0.49.0`,
`mkdocstrings==0.25.2`, and `mkdocs==1.6.1`); `PYTHONPATH=. mkdocs
build` succeeds.
## Summary
`opendal 0.58.1` (the version pulled in transitively via Lance) already
ships
`goosefs-sdk 0.1.9`, which includes the upstream fix for the 0.1.6
compile
break. The explicit version pin that lancedb has been carrying since the
GooseFS feature was introduced is therefore no longer necessary and is
now
redundant work to maintain.
## Changes
- Remove the direct `goosefs-sdk` dependency from
`rust/lancedb/Cargo.toml`
(it was pinned to `=0.1.9` with a comment referencing the 0.1.6 compile
break).
- Remove the `dep:goosefs-sdk` entry from the `goosefs` cargo feature,
since
no source file in lancedb imports the crate directly.
- Refresh `Cargo.lock`; `goosefs-sdk 0.1.9` now resolves transitively
through
`lance` → `opendal 0.58.1`.
## Verification
- `cargo fmt --all` — clean
- `cargo check --features remote,goosefs --tests --examples` — passes
- `Cargo.lock` confirms `goosefs-sdk 0.1.9` is still resolved (now
transitively), so the `goosefs` feature continues to enable the same set
of
Lance/IOPaths as before.
## Backwards compatibility
No public API changes. The `goosefs` cargo feature still activates
`lance/goosefs`, `lance-io/goosefs`, and
`lance-namespace-impls/dir-goosefs`,
and the same `goosefs-sdk 0.1.9` version is selected by the resolver.
<!-- lance-gatekeeper-fix:v1 agent=613a074d606e626c5169d601373a32d8
generation=1 -->
## Root cause
When LanceDB accepted an Arrow table created by a different installed
Arrow package, its compatibility sanitizer rebuilt each Data node
without converting the foreign type or preserving nested children. It
also dropped the separate dictionary vector payload and did not preserve
identity shared by dictionary schema types, vector wrappers, or growing
dictionary chunks.
## Fix
Recursively sanitize nested Arrow data types and child data. Use one
table-scoped sanitization context to rebuild and memoize source type
objects, dictionary vectors, and Data nodes in the local Arrow realm,
preserving all identities required by Arrow IPC.
Add Arrow 15 through 18 regressions for list serialization, ordinary
dictionaries, dictionaries shared across fields and batches, growing
dictionaries, and IPC round trips.
## Validation
- pnpm test __test__/arrow.test.ts --runInBand (188 passed)
- pnpm lint
- pnpm build
- pnpm test --runInBand (706 passed, 5 skipped)
- pnpm run docs
Fixes#2256
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Some issues:
- file_size_bytes is optional in the manifest, so if it's not there (old
writer I guess) it'll under-report the table size.
- it changes results a little bit from the old way by including per-file
footers and metadata (probably not a big difference at real scale)
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
## What
`JinaEmbeddings._generate_image_input_dict()` crashes with
`AttributeError: 'function' object has no attribute 'urlparse'` on any
image given as a URL string, local path string, or `pathlib.Path` — i.e.
every documented `jina-clip-v1` image-embedding use case except raw
`bytes`.
## Why
```python
from urllib.parse import urlparse
...
parsed = urlparse.urlparse(image)
```
`urlparse` is imported as a function, then called as if it were the
`urllib.parse` module (`urlparse.urlparse(...)`). The module-level
`is_valid_url()` a few lines above does it correctly (`urlparse(text)`),
which is why this reads as a typo rather than intentional. Fixed to
`urlparse(str(image))` — `str()` is needed because `urlparse()` only
accepts `str`/`bytes` and raises a different `AttributeError` on a raw
`Path`.
## Testing
Added `test_jina_generate_image_input_dict_local_path`, which fails with
the original `AttributeError` before the fix and passes after, covering
both a `str` path and a `pathlib.Path`. Verified locally (built the Rust
extension, ran red→green, then the full `test_embeddings.py` file: 15
passed / 8 skipped, no regressions) and with `ruff check`/`ruff format`.
---
Disclosure: this PR was drafted with AI assistance (Claude); I reviewed,
tested, and take responsibility for the change.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
`LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`:
| value | meaning |
|---|---|
| `None` (new default) | every index the MemWAL supports, resolved when
the spec is installed |
| `Some([])` | maintain nothing — a scan/filter-only WAL table |
| `Some([..])` | exactly these, taken verbatim |
`with_maintained_indexes` keeps its signature;
`with_no_maintained_indexes()` is new. Surfaced through the remote path
(null on the wire), Python, and Node.
## Why
Callers had to state the maintained set by hand every time, which is
both tedious and easy to get wrong — the common case is "maintain what I
already built."
Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating
to lance's `is_maintainable_index_type`. This is load-bearing rather
than cosmetic: lance does **not** skip an index type its memtable cannot
build, it errors when the shard writer opens, so sweeping up a bitmap
index would fail every memtable claim and leave the table unwritable.
The inferred set excludes those, and an explicit list naming one is now
rejected at spec time instead of at claim time.
## Behavior change
A freshly constructed spec used to maintain **nothing**; it now
maintains **everything supported**. This flipped because napi collapses
`undefined` and `null` to `None`, so TypeScript cannot express "absent
means nothing, null means all" — any other choice makes the bindings
disagree with the wire. The error direction also favors it: an unwanted
maintained index costs memory, while a silently unmaintained one
degrades FTS to an unscored scan.
Three existing tests encoded the old default and are updated rather than
worked around.
## Caveat
The resolved set is a snapshot, not a subscription. An index created
after the spec is installed is not maintained until the spec is unset
and set again. `get_lsm_write_spec` therefore always reports a concrete
list — `None` never round-trips.
## Dependency
Needs a lance release carrying `is_maintainable_index_type`
(lance-format/lance#8095) before this builds against the pinned tag.
Draft until then.
## Testing
38 Rust LSM tests and 10 Python tests pass against a local lance build,
including new coverage that a bitmap index is excluded from inference
and rejected when named, and that `[]` stays distinguishable from null
on the wire.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Updates the Rust workspace Lance dependencies and Java lance-core
dependency to v11.0.0-beta.3. No compatibility fixes were required;
all-features clippy and Rust formatting pass. Triggering tag:
https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.3
Converge a table's LSM write path into its base table, and inspect it.
`checkpoint_lsm` is `flush` then `compact`, repeated until the fresh
tier is empty — and the loop runs **client-side**. Putting it on the
server would mean a background task, which means a single-flight intent,
an intent that leaks on panic, a bounded-iteration policy, an "is it
done" observable, and a story for every way a client can vanish
mid-operation. None of that exists in this shape: each request does a
bounded unit of work and reports what is left, so completion is *carried
in the responses* rather than inferred from a shared counter that cannot
distinguish "converged" from "hasn't started yet".
Best-effort by construction. Nothing is frozen, so `converged` means L0
was empty as of the last pass. It is idempotent, abandonable at any
point with zero consequence, and safe to run on a cadence — an
already-converged table costs one round trip and zero compaction passes,
because `flush` reports `generations_remaining` and the loop is never
entered.
## The failure taxonomy is the load-bearing part
Five distinct conditions used to arrive at a client as one 503.
`Error::LsmRoute` carries a classification read from the response body's
namespace error code **at the point of receipt** — before any generic
helper folds the body into a string and keeps only the status.
| condition | wire | client action |
|---|---|---|
| contention (latch held / pool saturated) | 429, code 21 | retry with
backoff |
| owning node draining | 503, code 19 `InvalidTableState` | **stop** |
| fenced / no slot / transport | 503, code 17 | retry with backoff |
| registry entry vanished | 404 | re-issue from `flush` (capped) |
| table being dropped / not WAL-backed | 409 / 400 | stop |
Draining is terminal because the drain gate is a one-way latch —
retrying spins until the deadline to report a failure that was knowable
on the first response. Transport retry is disabled on these routes for
the same reason: it treats every 503 alike and would burn its budget
before the classifier ever saw the body.
`get_lsm_stats` returns `Option<LsmStats>`, matching
`get_lsm_write_spec` — `None` only when the table has no LSM write path,
since a struct of zeros would read as measurements.
Python bindings mirror all four, preserving per-bucket detail rather
than flattening to a table-level summary.
## Testing
Six new unit tests against the mocked endpoint, plus the taxonomy
round-trip:
- flush into an empty L0 issues **zero** compact calls (asserts the call
count — `generations_consumed: 0` is also true of a loop that ran a
pointless pass)
- the loop drives compact until the server reports zero remaining
- **contention is not draining**: a 429 retries and converges; asserts
the retry count
- a draining node stops after **exactly one** request, no retries
- stats round-trips fully populated; `include_generation_rows` off by
default
- every `(status, code)` pair classifies correctly, including
unparseable 503 bodies falling back to *retryable* rather than terminal
`cargo test -p lancedb --features remote --lib`: 723 passed.
## Notes for review
- Depends on the sibling lance change returning `SealedGeneration` from
`force_seal_active` only at the *server* level — no lance API is used
here.
- The branch is based on `codex/update-lance-10-0-0-beta-5`, so it
carries one extra commit (`chore: update lance dependency to
v10.0.0-beta.5`) that is not part of this change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: lancedb automation <robot@lancedb.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- cover Hugging Face cache layouts where both manifests and Lance data
files are relative symlinks into a blob directory
- reconnect with a fresh session before opening so the test exercises
filesystem discovery instead of cached manifest metadata
- scan the reopened table to verify both manifest recovery and data-file
reads
## Root cause
Lance 3.0.1 recorded Unix symlink metadata as the known manifest size,
so the short link length caused a file size is too small error. The
current Lance v11.0.0-beta.2 dependency repairs this by detecting an
invalid footer from a stale known size and retrying with the target file
metadata. This regression test locks that behavior into the LanceDB
open-table path used by Node.
## Validation
- cargo fmt --all
- cargo test --quiet --features remote -p lancedb --lib
test_open_table_follows_hugging_face_symlinks -- --nocapture
- cargo test --quiet --features remote -p lancedb --lib
database::listing::tests
- cargo clippy --quiet --features remote -p lancedb --lib --tests -- -D
warnings
- cargo check --quiet --features remote --tests --examples
Fixes#3197
<!-- lance-gatekeeper-fix:v1 agent=4aadcf04e9ac93b97d499d7448b67e19
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add Node regression coverage for vector-search offset pagination
- add equivalent coverage for full-text search
- compare later pages with the corresponding complete-result slice and
assert page sizes
## Root cause
The historical query path requested only the user limit from
nearest-neighbor or full-text search before applying the offset, so a
page became empty when its offset reached that limit. The production
query path on current main already incorporates the later fix from
#2592; this change adds the missing Node binding coverage for the
still-open report and protects both affected APIs from regression.
## Validation
- corepack pnpm build
- corepack pnpm test -- query.test.ts --runInBand
--testNamePattern="Search pagination"
- corepack pnpm lint-ci
- corepack pnpm tsc
- corepack pnpm run docs
Fixes#2229
<!-- lance-gatekeeper-fix:v1 agent=8ba8b18a18260a68a3e605d1bbfa518e
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- classify unsupported local-filesystem operations from Lance as a
NotSupported error
- explain that object-storage mounts cannot provide the safe commit
operations Lance requires and direct users to native object-store URIs
- preserve existing error behavior for other local I/O failures and
non-local backends
## Root cause
Mountpoint for Amazon S3 exposes an S3 bucket as a local path but does
not implement atomic rename. Lance uses atomic rename for safe local
commits, and the resulting unsupported I/O error was previously passed
through as a generic Lance error, leaving Python users with an opaque
low-level failure. Transparent support for such mounts is not safe;
direct s3:// access remains the supported path.
## Validation
- cargo test --quiet --features remote -p lancedb error::tests
- cargo test --quiet --features remote -p lancedb --lib (807 passed, 1
ignored)
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples
- cargo fmt --all -- --check
Fixes#2016
<!-- lance-gatekeeper-fix:v1 agent=d53283c18fdb00a3a1b69448b1f40529
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- Add an issue-specific regression for appending generated embeddings to
an empty table with a non-nullable vector field.
- Verify the custom embedding function produces the declared Float64
vectors and both appended rows are readable.
## Root cause
In v0.4.19, records without a vector value were materialized against the
explicit schema before embeddings were inserted. Apache Arrow inferred
the generated batch vector field as nullable while the table retained
the user-provided non-nullable field, then rejected the mismatched
schemas.
The current conversion path excludes the generated field from the
initial record conversion and realigns the completed batch to the stored
schema after embedding, but the reported empty-table append sequence
lacked permanent regression coverage.
## Validation
- `pnpm exec biome format --write __test__/embedding.test.ts`
- `pnpm lint-ci`
- `pnpm test -- --runInBand __test__/embedding.test.ts` (12 passed, 1
skipped integration test)
- `pnpm build`
- `pnpm run docs`
Fixes#1281
<!-- lance-gatekeeper-fix:v1 agent=6b7270aeb92e6b6c6f5b45022fa83f6a
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add an end-to-end regression for indexed vector search after merging a
pandas column
- verify unmatched rows retain a null merged value instead of failing
Arrow batch assembly
## Root cause
Historical Lance readers could assemble schema-evolved columns in
physical data-file order. Indexed row-ID reads after a merge could
therefore omit or misorder the newly merged column for unmatched rows.
The currently pinned Lance release contains the reader correction, but
LanceDB did not cover the reported merge-then-search path.
## Validation
- uv run --extra tests pytest python/tests/test_table.py::test_merge
python/tests/test_table.py::test_search_after_merge -q
- uv run --project python --extra dev ruff check .
- uv run --project python --extra dev ruff format --check
python/python/tests/test_table.py
Fixes#599
<!-- lance-gatekeeper-fix:v1 agent=4e17331e0542c132eae31e86da508629
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Root cause
`Table.to_polars()` disabled PyArrow predicate pushdown by selecting the
non-PyArrow Polars scan callback. Polars 1.32.3 invokes that callback
with `batch_size` both positionally and through its partial, so
collecting the returned lazy frame raises `TypeError:
_scan_pyarrow_dataset_impl() got multiple values for argument
batch_size`.
## Fix
- Keep the compatible PyArrow callback path.
- Add an identity `map_batches` barrier so predicates stay in Polars
instead of reaching the LanceDB adapter as unsupported PyArrow
expressions.
- Extend the tested Polars range through 1.32.3 and retain lazy-frame
regression coverage.
## Validation
- `python/tests/test_table.py::test_polars` with Polars 1.32.3
- `python/tests/test_table.py::test_polars` with the locked Polars 1.3.0
baseline
- `ruff format --check` on the changed Python files
- `ruff check .`
- `uv lock --check`
Fixes#2619
<!-- lance-gatekeeper-fix:v1 agent=0d42bcda944ac42765b25f2c19ff729f
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- exercise float16 sanitization through the reported direct Arrow-data
table creation path
- assert that the inferred fixed-size vector schema remains float16
- retain end-to-end index creation and vector search coverage
## Root cause and fix
PyArrow 16 does not provide an is_nan kernel for half-float arrays, so
passing float16 vector values directly to that kernel raises
ArrowNotImplementedError. LanceDB's sanitizer already carries the
compatibility fix from #837: it casts float16 values to float32 only for
NaN detection while preserving the stored vector type.
The existing end-to-end regression created an empty schema-defined table
and added data afterward. This change aligns that regression with the
issue reproduction by creating a table directly from a
FixedSizeList<float16> Arrow table and verifying the persisted schema.
## Validation
- uv run --extra tests pytest
python/tests/test_table.py::test_create_f16_table_from_arrow_data -q
- direct 1,000-row by 128-dimension float16 Arrow-table reproduction
- PyArrow 16.1 half-float is_nan kernel reproduction
- uvx ruff@0.15.20 format --check python/python/tests/test_table.py
- uvx ruff@0.15.20 check .
Fixes#835
<!-- lance-gatekeeper-fix:v1 agent=dd0a32a959f691f49de958d4333fb29d
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- require Node.js 18-compatible type declarations when TypeScript
consumers install them
- keep the type peer optional for JavaScript-only consumers
- add a regression test tying the Node type peer range to the supported
runtime
## Root cause
LanceDB requires Node.js 18 or newer, and its public types expose Apache
Arrow declarations that import built-ins through the node: scheme. The
package did not declare a matching @types/node peer requirement, so npm
accepted projects pinned to Node 12 declarations and TypeScript then
reported that node:stream and node:fs/promises did not exist.
## Validation
- pnpm lint
- pnpm build
- pnpm run docs
- pnpm test --runInBand (678 passed, 5 skipped)
- packed-package consumer probe rejects @types/node 12.20.55 and
installs with @types/node 18.19.130
Fixes#1713
<!-- lance-gatekeeper-fix:v1 agent=7a2b68f3daad20bed9e46cb8892d6e6c
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- publish the PEP 561 `py.typed` marker so downstream type checkers
consume the inline public annotations
- add a Pyright contract test that distinguishes synchronous `connect`
from awaited `connect_async`
- verify the marker is present in the installed package
## Root cause
The public Python module already annotated `lancedb.connect` as
synchronous and `lancedb.connect_async` as asynchronous. The private
native `_lancedb.connect` stub is intentionally awaitable because it
backs `connect_async`. However, the distribution did not include a PEP
561 marker, so downstream tools such as mypy could ignore the public
inline annotations and expose misleading or incomplete type information.
## Validation
- `python/.venv/bin/ruff format --check python/python/tests/test_db.py
python/python/type_tests/connect.py`
- `python/.venv/bin/ruff check .`
- `cd python && .venv/bin/pytest
python/tests/test_db.py::test_package_includes_pep_561_marker -q`
- `cd python && .venv/bin/pyright --pythonpath .venv/bin/python`
- downstream mypy contract check for both public connection functions
Fixes#2159
<!-- lance-gatekeeper-fix:v1 agent=b07901451487187fc03f61890d3aa6bb
generation=1 -->
Co-authored-by: lancedb-gatefixer[bot] <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- raise a clear `TypeError` when `Vector` is used without a dimension
- preserve normal `Vector(dim)` behavior across Pydantic v1 and v2
- add a regression test that defines a model without importing PyArrow
## Root cause
Pydantic interpreted the bare `Vector` factory as a callable field type
and inspected its postponed annotations in the user model's namespace.
Because that namespace did not define LanceDB's internal `pa` alias,
model construction failed with the misleading `NameError: name 'pa' is
not defined` instead of explaining that `Vector` must be parameterized.
The factory now exposes Pydantic's v1 and v2 schema hooks and rejects
bare use before signature introspection with guidance to use
`Vector(dim)`.
## Validation
- `uvx --from 'ruff==0.15.20' ruff check .`
- `uvx --from 'ruff==0.15.20' ruff format --check
python/python/lancedb/pydantic.py python/python/tests/test_pydantic.py`
- `cd python && uv run --extra tests pytest
python/tests/test_pydantic.py::test_bare_vector_raises_clear_error -q`
- `cd python && uv run --extra tests pytest
python/tests/test_pydantic.py -q`
- compatibility checks with Pydantic 1.10.22, 2.11.4, and 2.13.4
Fixes#2384
<!-- lance-gatekeeper-fix:v1 agent=71e7473e18c91db5137a3c0d3bb73640
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add regression coverage for adding dictionary rows with a nullable
fixed-size-list column
- verify ordinary list columns remain aligned alongside the null
fixed-size-list value
## Root cause
PyArrow infers an all-`None` dictionary column as the generic `null`
type. The original schema-alignment path treated the target
fixed-size-list type as proof that the inferred source was also
list-like and unconditionally accessed `value_field`, which raised
`AttributeError`. Current alignment logic correctly falls back to the
target type when the source is not list-like; this test locks in that
repair for the reported ingestion path.
## Validation
- `uv run --extra tests pytest
python/tests/test_table.py::test_add_with_empty_fixed_size_list_drops_bad_rows
python/tests/test_table.py::test_add_nullable_fixed_size_list_with_none
python/tests/test_table.py::test_add_nullable_struct_with_none -q`
- `uv run --with pyarrow==19.0.1 --extra tests pytest
python/tests/test_table.py::test_add_nullable_fixed_size_list_with_none
-q`
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_table.py`
- `uv run --project python --extra dev ruff check .`
Fixes#2340
<!-- lance-gatekeeper-fix:v1 agent=cb0475e85e764f79bd03b35eb8955ec4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Root cause
The former listing-database table URI builder used OS-native
`Path::join` for object-store URIs. On Windows this inserted backslashes
into `az://` table paths, so `table_names` found slash-delimited objects
while `open_table` addressed a different key. The production path now
builds URI paths with forward slashes after the equivalent S3 report was
fixed in #2575, but #1072 remained open without Azure-specific
regression coverage.
## Fix
- Add Azure URI regression assertions at the Rust table URI construction
boundary.
- Cover connection bases both with and without a trailing slash,
matching the behavior reported in #1072.
- Verify the resulting table URI always uses forward slashes on every
platform.
## Validation
- `cargo fmt --all -- --check`
- `cargo test --quiet -p lancedb --lib
database::listing::tests::test_table_uri`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests` (866 passed, 1 ignored)
Fixes#1072
<!-- lance-gatekeeper-fix:v1 agent=7d385255a072ed89ddc3ff4d08f82218
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add regression coverage for multiple query vectors in the local
synchronous Python API
- verify that each query vector receives its own limited
nearest-neighbor result and `query_index`
## Root cause
In LanceDB v0.16, the local synchronous scanner passed a nested vector
array as one query, unlike the async and remote implementations. The
subsequent sync-to-async table migration supplied the correct shared
runtime path, but this local sync behavior was never regression-tested
and issue #1857 remained open.
## Validation
- `uv run --extra tests pytest
python/tests/test_query.py::test_query_multiple_vectors -q`
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/tests/test_query.py`
- `uv run --project python --extra tests --extra dev ruff check .`
Fixes#1857
<!-- lance-gatekeeper-fix:v1 agent=6b25bc529d76813c3db7627c8be947ef
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Lancedb does not work with any other version of `futures`.
With futures 0.1 it fails like this:
```console
error[E0432]: unresolved imports `futures::StreamExt`, `futures::TryStreamExt`
--> rust/lancedb/src/arrow.rs:21:23
|
21 | use futures::{Stream, StreamExt, TryStreamExt};
| ^^^^^^^^^ ^^^^^^^^^^^^ no `TryStreamExt` in the root
| |
| no `StreamExt` in the root
|
error[E0432]: unresolved import `futures::StreamExt`
--> rust/lancedb/src/data/scannable.rs:24:5
|
24 | use futures::StreamExt;
| ^^^^^^^^^^^^^^^^^^ no `StreamExt` in the root
|
error[E0432]: unresolved import `futures::TryStreamExt`
--> rust/lancedb/src/dataloader/permutation/builder.rs:9:5
|
9 | use futures::TryStreamExt;
| ^^^^^^^^^^^^^^^^^^^^^ no `TryStreamExt` in the root
error[E0432]: unresolved imports `futures::StreamExt`, `futures::TryStreamExt`
--> rust/lancedb/src/dataloader/permutation/reader.rs:25:15
|
25 | use futures::{StreamExt, TryStreamExt};
| ^^^^^^^^^ ^^^^^^^^^^^^ no `TryStreamExt` in the root
| |
| no `StreamExt` in the root
|
error[E0432]: unresolved imports `futures::StreamExt`, `futures::TryStreamExt`
--> rust/lancedb/src/dataloader/permutation/shuffle.rs:8:15
|
8 | use futures::{StreamExt, TryStreamExt};
| ^^^^^^^^^ ^^^^^^^^^^^^ no `TryStreamExt` in the root
| |
| no `StreamExt` in the root
|
error[E0432]: unresolved imports `futures::StreamExt`, `futures::TryStreamExt`
--> rust/lancedb/src/dataloader/permutation/split.rs:12:15
|
12 | use futures::{StreamExt, TryStreamExt};
| ^^^^^^^^^ ^^^^^^^^^^^^ no `TryStreamExt` in the root
| |
| no `StreamExt` in the root
|
error[E0432]: unresolved import `futures::TryStreamExt`
--> rust/lancedb/src/dataloader/permutation/util.rs:9:5
|
9 | use futures::TryStreamExt;
| ^^^^^^^^^^^^^^^^^^^^^ no `TryStreamExt` in the root
error[E0432]: unresolved imports `futures::StreamExt`, `futures::TryFutureExt`
--> rust/lancedb/src/io/object_store.rs:8:15
|
8 | use futures::{StreamExt, TryFutureExt, stream::BoxStream};
| ^^^^^^^^^ ^^^^^^^^^^^^ no `TryFutureExt` in the root
| |
| no `StreamExt` in the root
|
error[E0432]: unresolved imports `futures::FutureExt`, `futures::TryFutureExt`, `futures::TryStreamExt`, `futures::try_join`
--> rust/lancedb/src/query.rs:12:15
|
12 | use futures::{FutureExt, TryFutureExt, TryStreamExt, stream, try_join};
| ^^^^^^^^^ ^^^^^^^^^^^^ ^^^^^^^^^^^^ ^^^^^^^^ no `try_join` in the root
| | | |
| | | no `TryStreamExt` in the root
| | no `TryFutureExt` in the root
| no `FutureExt` in the root
|
error[E0432]: unresolved imports `futures::StreamExt`, `futures::TryStreamExt`
--> rust/lancedb/src/remote/table/blobs.rs:13:15
|
13 | use futures::{StreamExt, TryStreamExt};
| ^^^^^^^^^ ^^^^^^^^^^^^ no `TryStreamExt` in the root
| |
| no `StreamExt` in the root
|
error[E0432]: unresolved imports `futures::SinkExt`, `futures::StreamExt`
--> rust/lancedb/src/remote/table/insert.rs:20:15
|
20 | use futures::{SinkExt, StreamExt};
| ^^^^^^^ ^^^^^^^^^ no `StreamExt` in the root
| |
| no `SinkExt` in the root
|
error[E0432]: unresolved imports `futures::StreamExt`, `futures::TryStreamExt`
--> rust/lancedb/src/remote/table.rs:58:15
|
58 | use futures::{StreamExt, TryStreamExt};
| ^^^^^^^^^ ^^^^^^^^^^^^ no `TryStreamExt` in the root
| |
| no `StreamExt` in the root
|
error[E0432]: unresolved import `futures::StreamExt`
--> rust/lancedb/src/remote/util.rs:5:23
|
5 | use futures::{Stream, StreamExt};
| ^^^^^^^^^ no `StreamExt` in the root
|
error[E0432]: unresolved import `futures::StreamExt`
--> rust/lancedb/src/table.rs:14:5
|
14 | use futures::StreamExt;
| ^^^^^^^^^^^^^^^^^^ no `StreamExt` in the root
|
error[E0432]: unresolved import `futures::TryStreamExt`
--> rust/lancedb/src/table/datafusion/insert.rs:20:5
|
20 | use futures::TryStreamExt;
| ^^^^^^^^^^^^^^^^^^^^^ no `TryStreamExt` in the root
error[E0432]: unresolved import `futures::TryStreamExt`
--> rust/lancedb/src/table/datafusion/scannable_exec.rs:14:5
|
14 | use futures::TryStreamExt;
| ^^^^^^^^^^^^^^^^^^^^^ no `TryStreamExt` in the root
error[E0432]: unresolved imports `futures::TryFutureExt`, `futures::TryStreamExt`
--> rust/lancedb/src/table/datafusion.rs:25:15
|
25 | use futures::{TryFutureExt, TryStreamExt};
| ^^^^^^^^^^^^ ^^^^^^^^^^^^ no `TryStreamExt` in the root
| |
| no `TryFutureExt` in the root
error[E0432]: unresolved import `futures::FutureExt`
--> rust/lancedb/src/table/delete.rs:3:5
|
3 | use futures::FutureExt;
| ^^^^^^^^^^^^^^^^^^ no `FutureExt` in the root
|
error[E0432]: unresolved imports `futures::FutureExt`, `futures::TryFutureExt`
--> rust/lancedb/src/table/merge.rs:9:15
|
9 | use futures::{FutureExt, TryFutureExt};
| ^^^^^^^^^ ^^^^^^^^^^^^ no `TryFutureExt` in the root
| |
| no `FutureExt` in the root
|
error[E0432]: unresolved import `futures::future::try_join_all`
--> rust/lancedb/src/table/query.rs:24:5
|
24 | use futures::future::try_join_all;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no `try_join_all` in `future`
|
error[E0432]: unresolved import `futures::FutureExt`
--> rust/lancedb/src/utils/background_cache.rs:12:5
|
12 | use futures::FutureExt;
| ^^^^^^^^^^^^^^^^^^ no `FutureExt` in the root
|
error[E0432]: unresolved import `futures::FutureExt`
--> rust/lancedb/src/utils/mod.rs:12:15
|
12 | use futures::{FutureExt, Stream};
| ^^^^^^^^^ no `FutureExt` in the root
|
error[E0433]: cannot find `join` in `futures`
--> rust/lancedb/src/remote/table/insert.rs:504:55
|
504 | let (producer_result, send_result) = futures::join!(producer, send);
| ^^^^ could not find `join` in `futures`
error[E0407]: method `poll_next` is not a member of trait `Stream`
--> rust/lancedb/src/arrow.rs:108:5
|
108 | / fn poll_next(
109 | | self: Pin<&mut Self>,
110 | | cx: &mut std::task::Context<'_>,
111 | | ) -> std::task::Poll<Option<Self::Item>> {
112 | | let this = self.project();
113 | | this.stream.poll_next(cx)
114 | | }
| |_____^ not a member of trait `Stream`
error[E0407]: method `poll_next` is not a member of trait `Stream`
--> rust/lancedb/src/utils/mod.rs:362:5
|
362 | / fn poll_next(
363 | | mut self: std::pin::Pin<&mut Self>,
364 | | cx: &mut std::task::Context<'_>,
365 | | ) -> std::task::Poll<Option<Self::Item>> {
... |
391 | | }
| |_____^ not a member of trait `Stream`
error[E0407]: method `poll_next` is not a member of trait `Stream`
--> rust/lancedb/src/utils/mod.rs:433:5
|
433 | / fn poll_next(
434 | | mut self: Pin<&mut Self>,
435 | | cx: &mut std::task::Context<'_>,
436 | | ) -> std::task::Poll<Option<Self::Item>> {
... |
470 | | }
| |_____^ not a member of trait `Stream`
error[E0425]: cannot find function `try_unfold` in module `futures::stream`
--> rust/lancedb/src/remote/table/insert.rs:230:39
|
230 | let stream = futures::stream::try_unfold(
| ^^^^^^^^^^ not found in `futures::stream`
error[E0433]: cannot find `channel` in `futures`
--> rust/lancedb/src/remote/table/insert.rs:418:22
|
418 | futures::channel::mpsc::channel::<Result<Vec<u8>, std::io::Error>>(2);
| ^^^^^^^ could not find `channel` in `futures`
|
error[E0425]: cannot find function `try_join_all` in module `futures::future`
--> rust/lancedb/src/remote/table.rs:1062:40
|
1062 | let streams = futures::future::try_join_all(futures);
| ^^^^^^^^^^^^
|
::: $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/future/join_all.rs:76:1
|
76 | / pub fn join_all<I>(i: I) -> JoinAll<I>
77 | | where I: IntoIterator,
78 | | I::Item: IntoFuture,
| |______________________________- similarly named function `join_all` defined here
|
error[E0425]: cannot find function `try_join_all` in module `futures::future`
--> rust/lancedb/src/remote/table.rs:1660:40
|
1660 | let results = futures::future::try_join_all(futures).await?;
| ^^^^^^^^^^^^
|
::: $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/future/join_all.rs:76:1
|
76 | / pub fn join_all<I>(i: I) -> JoinAll<I>
77 | | where I: IntoIterator,
78 | | I::Item: IntoFuture,
| |______________________________- similarly named function `join_all` defined here
|
error[E0425]: cannot find function `try_join_all` in module `futures::future`
--> rust/lancedb/src/remote/table.rs:2243:43
|
2243 | let plan_texts = futures::future::try_join_all(futures).await?;
| ^^^^^^^^^^^^
|
::: $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/future/join_all.rs:76:1
|
76 | / pub fn join_all<I>(i: I) -> JoinAll<I>
77 | | where I: IntoIterator,
78 | | I::Item: IntoFuture,
| |______________________________- similarly named function `join_all` defined here
|
error[E0425]: cannot find function `try_join_all` in module `futures::future`
--> rust/lancedb/src/remote/table.rs:2290:53
|
2290 | let analyze_result_texts = futures::future::try_join_all(futures).await?;
| ^^^^^^^^^^^^
|
::: $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/future/join_all.rs:76:1
|
76 | / pub fn join_all<I>(i: I) -> JoinAll<I>
77 | | where I: IntoIterator,
78 | | I::Item: IntoFuture,
| |______________________________- similarly named function `join_all` defined here
|
error[E0425]: cannot find function `try_unfold` in module `futures::stream`
--> rust/lancedb/src/remote/util.rs:21:35
|
21 | let stream = futures::stream::try_unfold(
| ^^^^^^^^^^ not found in `futures::stream`
error[E0191]: the value of the associated type `Error` in `futures::Stream` must be specified
--> rust/lancedb/src/arrow.rs:70:50
|
70 | pub type SendableRecordBatchStream = Pin<Box<dyn RecordBatchStream + Send>>;
| ^^^^^^^^^^^^^^^^^
|
help: specify the associated type
|
70 | pub type SendableRecordBatchStream = Pin<Box<dyn RecordBatchStream<Error = /* Type */> + Send>>;
| ++++++++++++++++++++
error[E0107]: type alias takes 0 lifetime arguments but 1 lifetime argument was supplied
--> rust/lancedb/src/utils/background_cache.rs:15:31
|
15 | type SharedFut<V, E> = Shared<BoxFuture<'static, Result<V, Arc<E>>>>;
| ^^^^^^^^^ ------- help: remove the lifetime argument
| |
| expected 0 lifetime arguments
|
note: type alias defined here, with 0 lifetime parameters
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/future/mod.rs:106:14
|
106 | pub type BoxFuture<T, E> = ::std::boxed::Box<Future<Item = T, Error = E> + Send>;
| ^^^^^^^^^
error[E0107]: type alias takes 2 generic arguments but 1 generic argument was supplied
--> rust/lancedb/src/utils/background_cache.rs:15:31
|
15 | type SharedFut<V, E> = Shared<BoxFuture<'static, Result<V, Arc<E>>>>;
| ^^^^^^^^^ ----------------- supplied 1 generic argument
| |
| expected 2 generic arguments
|
note: type alias defined here, with 2 generic parameters: `T`, `E`
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/future/mod.rs:106:14
|
106 | pub type BoxFuture<T, E> = ::std::boxed::Box<Future<Item = T, Error = E> + Send>;
| ^^^^^^^^^ - -
help: add missing generic argument
|
15 | type SharedFut<V, E> = Shared<BoxFuture<'static, Result<V, Arc<E>>, E>>;
| +++
error[E0046]: not all trait items implemented, missing: `Error`, `poll`
--> rust/lancedb/src/arrow.rs:105:1
|
105 | impl<S: Stream<Item = Result<arrow_array::RecordBatch>>> Stream for SimpleRecordBatchStream<S> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing `Error`, `poll` in implementation
|
= help: implement the missing item: `type Error = /* Type */;`
= help: implement the missing item: `fn poll(&mut self) -> std::result::Result<Async<std::option::Option<<Self as futures::Stream>::Item>>, <Self as futures::Stream>::Error> { todo!() }`
error[E0107]: type alias takes 0 lifetime arguments but 1 lifetime argument was supplied
--> rust/lancedb/src/io/object_store.rs:97:46
|
97 | fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>> {
| ^^^^^^^^^ ------- help: remove the lifetime argument
| |
| expected 0 lifetime arguments
|
note: type alias defined here, with 0 lifetime parameters
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/mod.rs:132:14
|
132 | pub type BoxStream<T, E> = ::std::boxed::Box<Stream<Item = T, Error = E> + Send>;
| ^^^^^^^^^
error[E0107]: type alias takes 2 generic arguments but 1 generic argument was supplied
--> rust/lancedb/src/io/object_store.rs:97:46
|
97 | fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>> {
| ^^^^^^^^^ ------------------ supplied 1 generic argument
| |
| expected 2 generic arguments
|
note: type alias defined here, with 2 generic parameters: `T`, `E`
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/mod.rs:132:14
|
132 | pub type BoxStream<T, E> = ::std::boxed::Box<Stream<Item = T, Error = E> + Send>;
| ^^^^^^^^^ - -
help: add missing generic argument
|
97 | fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>, E> {
| +++
error[E0107]: type alias takes 0 lifetime arguments but 1 lifetime argument was supplied
--> rust/lancedb/src/io/object_store.rs:107:20
|
107 | locations: BoxStream<'static, Result<Path>>,
| ^^^^^^^^^ ------- help: remove the lifetime argument
| |
| expected 0 lifetime arguments
|
note: type alias defined here, with 0 lifetime parameters
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/mod.rs:132:14
|
132 | pub type BoxStream<T, E> = ::std::boxed::Box<Stream<Item = T, Error = E> + Send>;
| ^^^^^^^^^
error[E0107]: type alias takes 2 generic arguments but 1 generic argument was supplied
--> rust/lancedb/src/io/object_store.rs:107:20
|
107 | locations: BoxStream<'static, Result<Path>>,
| ^^^^^^^^^ ------------ supplied 1 generic argument
| |
| expected 2 generic arguments
|
note: type alias defined here, with 2 generic parameters: `T`, `E`
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/mod.rs:132:14
|
132 | pub type BoxStream<T, E> = ::std::boxed::Box<Stream<Item = T, Error = E> + Send>;
| ^^^^^^^^^ - -
help: add missing generic argument
|
107 | locations: BoxStream<'static, Result<Path>, E>,
| +++
error[E0107]: type alias takes 0 lifetime arguments but 1 lifetime argument was supplied
--> rust/lancedb/src/io/object_store.rs:108:10
|
108 | ) -> BoxStream<'static, Result<Path>> {
| ^^^^^^^^^ ------- help: remove the lifetime argument
| |
| expected 0 lifetime arguments
|
note: type alias defined here, with 0 lifetime parameters
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/mod.rs:132:14
|
132 | pub type BoxStream<T, E> = ::std::boxed::Box<Stream<Item = T, Error = E> + Send>;
| ^^^^^^^^^
error[E0107]: type alias takes 2 generic arguments but 1 generic argument was supplied
--> rust/lancedb/src/io/object_store.rs:108:10
|
108 | ) -> BoxStream<'static, Result<Path>> {
| ^^^^^^^^^ ------------ supplied 1 generic argument
| |
| expected 2 generic arguments
|
note: type alias defined here, with 2 generic parameters: `T`, `E`
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/mod.rs:132:14
|
132 | pub type BoxStream<T, E> = ::std::boxed::Box<Stream<Item = T, Error = E> + Send>;
| ^^^^^^^^^ - -
help: add missing generic argument
|
108 | ) -> BoxStream<'static, Result<Path>, E> {
| +++
error[E0599]: no method named `map_err` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/dataloader/permutation/builder.rs:208:32
|
208 | let stream = df_stream.map_err(|e| Error::Other {
| ----------^^^^^^^ method not found in `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>`
|
::: $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.32/src/stream/try_stream/mod.rs:248:8
|
248 | fn map_err<E, F>(self, f: F) -> MapErr<Self, F>
| ------- the method is available for `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` here
|
error[E0599]: no method named `try_collect` found for struct `DatasetRecordBatchStream` in the current scope
--> rust/lancedb/src/dataloader/permutation/reader.rs:220:28
|
220 | let batches = data.try_collect::<Vec<_>>().await?;
| ^^^^^^^^^^^
|
error[E0599]: no method named `map_err` found for struct `DatasetRecordBatchStream` in the current scope
--> rust/lancedb/src/dataloader/permutation/reader.rs:287:14
|
286 | let mut stream = row_ids
| __________________________-
287 | | .map_err(Error::from)
| | -^^^^^^^ method not found in `DatasetRecordBatchStream`
| |_____________|
|
error[E0599]: the method `chain` exists for struct `futures::stream::Once<_, _>`, but its trait bounds were not satisfied
--> rust/lancedb/src/dataloader/permutation/reader.rs:307:81
|
307 | let stream = futures::stream::once(std::future::ready(Ok(first_batch))).chain(stream);
| ^^^^^ method cannot be called on `futures::stream::Once<_, _>` due to unsatisfied trait bounds
error[E0308]: mismatched types
--> rust/lancedb/src/dataloader/permutation/shuffle.rs:120:35
|
120 | futures::stream::once(async move { Ok(shuffled) }),
| --------------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `Result<_, _>`, found `async` block
| |
| arguments to this function are incorrect
|
= note: expected enum `std::result::Result<_, _>`
found `async` block `{async block@rust/lancedb/src/dataloader/permutation/shuffle.rs:120:35: 120:45}`
note: function defined here
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/once.rs:20:8
|
20 | pub fn once<T, E>(item: Result<T, E>) -> Once<T, E> {
| ^^^^
help: try wrapping the expression in a variant of `std::result::Result`
|
120 | futures::stream::once(Ok(async move { Ok(shuffled) })),
| +++ +
120 | futures::stream::once(Err(async move { Ok(shuffled) })),
| ++++ +
error[E0271]: type mismatch resolving `<Range<u64> as IntoIterator>::Item == Result<_, _>`
--> rust/lancedb/src/dataloader/permutation/shuffle.rs:228:44
|
228 | let stream = futures::stream::iter(0..num_files)
| --------------------- ^^^^^^^^^^^^ expected `Result<_, _>`, found `u64`
| |
| required by a bound introduced by this call
|
= note: expected enum `std::result::Result<_, _>`
found type `u64`
note: required by a bound in `futures::stream::iter`
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/iter.rs:31:27
|
30 | pub fn iter<J, T, E>(i: J) -> Iter<J::IntoIter>
| ---- required by a bound in this function
31 | where J: IntoIterator<Item=Result<T, E>>,
| ^^^^^^^^^^^^^^^^^ required by this bound in `iter`
error[E0599]: no method named `then` found for struct `IterStream<I>` in the current scope
--> rust/lancedb/src/dataloader/permutation/shuffle.rs:229:14
|
228 | let stream = futures::stream::iter(0..num_files)
| ______________________-
229 | | .then(move |file_index| {
| | -^^^^ method not found in `IterStream<std::ops::Range<u64>>`
| |_____________|
|
error[E0599]: no method named `try_collect` found for struct `Pin<Box<dyn lance::io::RecordBatchStream>>` in the current scope
--> rust/lancedb/src/dataloader/permutation/shuffle.rs:258:26
|
250 | let batches = reader
| ___________________________________-
251 | | .read_stream(
252 | | ReadBatchParams::RangeFull,
253 | | reader.num_rows() as u32,
... |
257 | | .await?
258 | | .try_collect::<Vec<_>>()
| |_________________________-^^^^^^^^^^^
error[E0599]: no method named `and_then` found for associated type `impl Future<Output = Result<Arc<...>, ...>> + Send` in the current scope
--> rust/lancedb/src/query.rs:766:14
|
765 | / self.create_plan(QueryExecutionOptions::default())
766 | | .and_then(|plan| std::future::ready(Ok(plan.schema())))
| |_____________-^^^^^^^^
|
::: $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.32/src/future/try_future/mod.rs:395:8
|
395 | fn and_then<Fut, F>(self, f: F) -> AndThen<Self, Fut, F>
| -------- the method is available for `impl std::future::Future<Output = std::result::Result<Arc<(dyn ExecutionPlan + 'static)>, error::Error>> + std::marker::Send` here
error[E0599]: no method named `boxed` found for `async` block `{async block@rust/lancedb/src/query.rs:1492:33: 1492:43}` in the current scope
--> rust/lancedb/src/query.rs:1493:18
|
1492 | let hybrid_result = async move { self.execute_hybrid(options).await }
| _________________________________-
1493 | | .boxed()
| | -^^^^^ method not found in `{async block@rust/lancedb/src/query.rs:1492:33: 1492:43}`
| |_________________|
error[E0271]: expected `{closure@blobs.rs:181:58}` to return `Result<_, _>`, but it returns `impl Future<Output = Result<Bytes, Error>>`
--> rust/lancedb/src/remote/table/blobs.rs:181:66
|
181 | futures::stream::iter(ranges.iter().cloned().map(|range| self.read_range(range)))
| --------------------- ------- ^^^^^^^^^^^^^^^^^^^^^^ expected `Result<_, _>`, found future
| | |
| | this closure
| required by a bound introduced by this call
error[E0599]: no method named `buffered` found for struct `IterStream<I>` in the current scope
--> rust/lancedb/src/remote/table/blobs.rs:182:14
|
181 | / futures::stream::iter(ranges.iter().cloned().map(|range| self.read_range(range)))
182 | | .buffered(BLOB_REQUEST_CONCURRENCY)
| | -^^^^^^^^ method not found in `Iter<Map<Cloned<Iter<'_, Range<u64>>>, {closure@...}>>`
| |_____________|
error[E0599]: no method named `try_next` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/remote/table/blobs.rs:379:40
|
379 | while let Some(batch) = stream.try_next().await? {
| ^^^^^^^^ method not found in `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>`
error[E0271]: type mismatch resolving `<Vec<...> as IntoIterator>::Item == Result<_, _>`
--> rust/lancedb/src/remote/table/blobs.rs:481:27
|
481 | futures::stream::iter(probe_futures)
| --------------------- ^^^^^^^^^^^^^ expected `Result<_, _>`, found future
| |
| required by a bound introduced by this call
error[E0599]: no method named `buffered` found for struct `IterStream<I>` in the current scope
--> rust/lancedb/src/remote/table/blobs.rs:482:10
|
481 | / futures::stream::iter(probe_futures)
482 | | .buffered(BLOB_REQUEST_CONCURRENCY)
| | -^^^^^^^^ method not found in `Iter<IntoIter<impl Future<Output = Result<..., ...>>>>`
| |_________|
error[E0599]: no method named `next` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/remote/table/insert.rs:324:37
|
324 | let mut first = match input.next().await {
| ^^^^ method not found in `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>`
error[E0599]: no method named `next` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/remote/table/insert.rs:345:33
|
345 | first = match input.next().await {
| ^^^^ method not found in `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>`
error[E0599]: the method `next` exists for mutable reference `&mut Pin<Box<dyn RecordBatchStream + Send>>`, but its trait bounds were not satisfied
--> rust/lancedb/src/remote/table/insert.rs:446:41
|
446 | None => match input.next().await {
| ^^^^ method cannot be called on `&mut Pin<Box<dyn RecordBatchStream + Send>>` due to unsatisfied trait bounds
|
= note: the following trait bounds were not satisfied:
`Pin<Box<(dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send + 'static)>>: Iterator`
which is required by `&mut Pin<Box<(dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send + 'static)>>: Iterator`
error[E0599]: no method named `map_err` found for struct `IterStream<I>` in the current scope
--> rust/lancedb/src/remote/table.rs:688:53
|
688 | let stream = futures::stream::iter(batches).map_err(DataFusionError::from);
| ^^^^^^^ method not found in `Iter<Box<dyn Iterator<Item = Result<..., ...>> + Send>>`
error[E0599]: no method named `try_collect` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/remote/table.rs:1378:49
|
1378 | let result: Result<Vec<_>> = stream.try_collect().await.map_err(Error::from);
| ^^^^^^^^^^^
error[E0599]: no method named `next` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/remote/table.rs:1509:48
|
1509 | while let Some(batch) = stream.next().await {
| ^^^^ method not found in `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>`
error[E0599]: no method named `boxed` found for opaque type `impl Future<Output = Result<DeleteResult, Error>>` in the current scope
--> rust/lancedb/src/table/delete.rs:35:51
|
35 | let delete_result = dataset.delete(s).boxed().await?;
| ^^^^^ method not found in `impl Future<Output = Result<DeleteResult, Error>>`
error[E0599]: no variant, associated function, or constant named `Left` found for enum `Either<A, B>` in the current scope
--> rust/lancedb/src/table/merge.rs:292:17
|
292 | Either::Left(tokio::time::timeout(timeout, future).map(|res| match res {
| ^^^^ variant, associated function, or constant not found in `Either<_, _>`
error[E0599]: `Timeout<impl Future<Output = Result<(Arc<...>, ...), ...>>>` is not an iterator
--> rust/lancedb/src/table/merge.rs:292:60
|
292 | Either::Left(tokio::time::timeout(timeout, future).map(|res| match res {
| --------------------------------------^^^ `Timeout<impl Future<Output = Result<(Arc<...>, ...), ...>>>` is not an iterator
|
::: $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/pin-project-lite-0.2.17/src/lib.rs:745:9
|
745 | / $vis struct $ident $($def_generics)*
746 | | $(where
747 | | $($where_clause)*)?
... |
751 | | ),+
752 | | }
| |_________- doesn't satisfy `_: Iterator`
|
= note: the following trait bounds were not satisfied:
`tokio::time::Timeout<impl std::future::Future<Output = std::result::Result<(Arc<lance::Dataset>, MergeStats), lance::Error>>>: Iterator`
which is required by `&mut tokio::time::Timeout<impl std::future::Future<Output = std::result::Result<(Arc<lance::Dataset>, MergeStats), lance::Error>>>: Iterator`
error[E0599]: no variant, associated function, or constant named `Right` found for enum `Either<A, B>` in the current scope
--> rust/lancedb/src/table/merge.rs:301:17
|
301 | Either::Right(job.execute_reader(new_data).map_err(|e| e.into()))
| ^^^^^ variant, associated function, or constant not found in `Either<_, _>`
error[E0599]: no method named `map_err` found for opaque type `impl Future<Output = Result<(Arc<Dataset>, ...), ...>>` in the current scope
--> rust/lancedb/src/table/merge.rs:301:52
|
301 | Either::Right(job.execute_reader(new_data).map_err(|e| e.into()))
| ^^^^^^^ method not found in `impl Future<Output = Result<(Arc<Dataset>, ...), ...>>`
error[E0277]: the trait bound `Iter<Map<IntoIter<RecordBatch>, ...>>: Stream` is not satisfied
--> rust/lancedb/src/table/query.rs:681:38
|
681 | Ok(DatasetRecordBatchStream::new(record_batch_stream))
| ^^^^^^^^^^^^^^^^^^^ the trait `futures_core::stream::Stream` is not implemented for `Iter<Map<IntoIter<RecordBatch>, ...>>`
error[E0277]: the trait bound `TimeoutStream: futures_core::stream::Stream` is not satisfied
--> rust/lancedb/src/utils/mod.rs:353:28
|
353 | impl RecordBatchStream for TimeoutStream {
| ^^^^^^^^^^^^^ unsatisfied trait bound
error[E0046]: not all trait items implemented, missing: `Error`, `poll`
--> rust/lancedb/src/utils/mod.rs:359:1
|
359 | impl Stream for TimeoutStream {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing `Error`, `poll` in implementation
|
= help: implement the missing item: `type Error = /* Type */;`
= help: implement the missing item: `fn poll(&mut self) -> std::result::Result<Async<std::option::Option<<Self as futures::Stream>::Item>>, <Self as futures::Stream>::Error> { todo!() }`
error[E0277]: the trait bound `MaxBatchLengthStream: futures_core::stream::Stream` is not satisfied
--> rust/lancedb/src/utils/mod.rs:424:28
|
424 | impl RecordBatchStream for MaxBatchLengthStream {
| ^^^^^^^^^^^^^^^^^^^^ unsatisfied trait bound
error[E0046]: not all trait items implemented, missing: `Error`, `poll`
--> rust/lancedb/src/utils/mod.rs:430:1
|
430 | impl Stream for MaxBatchLengthStream {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ missing `Error`, `poll` in implementation
|
= help: implement the missing item: `type Error = /* Type */;`
= help: implement the missing item: `fn poll(&mut self) -> std::result::Result<Async<std::option::Option<<Self as futures::Stream>::Item>>, <Self as futures::Stream>::Error> { todo!() }`
error[E0599]: no method named `map` found for type parameter `I` in the current scope
--> rust/lancedb/src/arrow.rs:75:45
|
72 | impl<I: lance::io::RecordBatchStream + 'static> From<I> for SendableRecordBatchStream {
| - method `map` not found for this type parameter
...
75 | let mapped_stream = Box::pin(stream.map(|r| r.map_err(Into::into)));
| ^^^
error[E0599]: no method named `poll_next` found for struct `Pin<&mut S>` in the current scope
--> rust/lancedb/src/arrow.rs:113:21
|
113 | this.stream.poll_next(cx)
| ^^^^^^^^^
|
= help: items from traits can only be used if the trait is implemented and in scope
= note: the following traits define an item `poll_next`, perhaps you need to implement one of them:
candidate #1: `futures_core::stream::Stream`
candidate #2: `sorts::stream::PartitionedStream`
help: there is a method `collect` with a similar name, but with different arguments
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/mod.rs:563:5
|
563 | / fn collect(self) -> Collect<Self>
564 | | where Self: Sized
| |_________________________^
error[E0599]: the method `map_err` exists for struct `Pin<Box<dyn Stream<Item = Result<..., ...>> + Send>>`, but its trait bounds were not satisfied
--> rust/lancedb/src/arrow.rs:150:29
|
150 | let stream = stream.map_err(|err| Error::Arrow { source: err });
| ^^^^^^^ method cannot be called due to unsatisfied trait bounds
error[E0308]: mismatched types
--> rust/lancedb/src/data/scannable.rs:80:26
|
80 | stream: once(async move { Ok(batch) }),
| ---- ^^^^^^^^^^^^^^^^^^^^^^^^ expected `Result<_, _>`, found `async` block
| |
| arguments to this function are incorrect
|
= note: expected enum `std::result::Result<_, _>`
found `async` block `{async block@rust/lancedb/src/data/scannable.rs:80:26: 80:36}`
note: function defined here
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/once.rs:20:8
|
20 | pub fn once<T, E>(item: Result<T, E>) -> Once<T, E> {
| ^^^^
help: try wrapping the expression in a variant of `std::result::Result`
|
80 | stream: once(Ok(async move { Ok(batch) })),
| +++ +
80 | stream: once(Err(async move { Ok(batch) })),
| ++++ +
error[E0308]: mismatched types
--> rust/lancedb/src/data/scannable.rs:107:30
|
107 | stream: once(async {
| _________________________----_^
| | |
| | arguments to this function are incorrect
108 | | Err(Error::InvalidInput {
109 | | message: "Cannot scan an empty Vec<RecordBatch>".to_string(),
110 | | })
111 | | }),
| |_________________^ expected `Result<_, _>`, found `async` block
|
= note: expected enum `std::result::Result<_, _>`
found `async` block `{async block@rust/lancedb/src/data/scannable.rs:107:30: 107:35}`
note: function defined here
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/once.rs:20:8
|
20 | pub fn once<T, E>(item: Result<T, E>) -> Once<T, E> {
| ^^^^
help: try wrapping the expression in a variant of `std::result::Result`
|
107 ~ stream: once(Ok(async {
108 | Err(Error::InvalidInput {
109 | message: "Cannot scan an empty Vec<RecordBatch>".to_string(),
110 | })
111 ~ })),
|
107 ~ stream: once(Err(async {
108 | Err(Error::InvalidInput {
109 | message: "Cannot scan an empty Vec<RecordBatch>".to_string(),
110 | })
111 ~ })),
|
error[E0271]: expected `Ok` to return `Result<Result<RecordBatch, Error>, _>`, but it returns `Result<RecordBatch, _>`
--> rust/lancedb/src/data/scannable.rs:117:52
|
117 | Box::pin(SimpleRecordBatchStream { schema, stream })
| ^^^^^^ expected `Result<Result<RecordBatch, Error>, _>`, found `Result<RecordBatch, _>`
error[E0308]: mismatched types
--> rust/lancedb/src/data/scannable.rs:158:59
|
158 | let stream = futures::stream::unfold(rx, |mut rx| async move {
| ___________________________________________________________^
159 | | rx.recv().await.map(|batch| (batch, rx))
160 | | })
| |_________^ expected `Option<_>`, found `async` block
|
= note: expected enum `std::option::Option<_>`
found `async` block `{async block@rust/lancedb/src/data/scannable.rs:158:59: 158:69}`
help: try wrapping the expression in `Some`
|
158 ~ let stream = futures::stream::unfold(rx, |mut rx| Some(async move {
159 | rx.recv().await.map(|batch| (batch, rx))
160 ~ }))
|
error[E0599]: the method `fuse` exists for struct `Unfold<Receiver<Result<RecordBatch, Error>>, ..., _>`, but its trait bounds were not satisfied
--> rust/lancedb/src/data/scannable.rs:161:10
|
158 | let stream = futures::stream::unfold(rx, |mut rx| async move {
| ______________________-
159 | | rx.recv().await.map(|batch| (batch, rx))
160 | | })
161 | | .fuse();
| | -^^^^ method cannot be called due to unsatisfied trait bounds
| |_________|
error[E0308]: mismatched types
--> rust/lancedb/src/data/scannable.rs:178:26
|
178 | stream: once(async {
| _____________________----_^
| | |
| | arguments to this function are incorrect
179 | | Err(Error::InvalidInput {
180 | | message: "Stream has already been consumed".to_string(),
181 | | })
182 | | }),
| |_____________^ expected `Result<_, _>`, found `async` block
|
= note: expected enum `std::result::Result<_, _>`
found `async` block `{async block@rust/lancedb/src/data/scannable.rs:178:26: 178:31}`
note: function defined here
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/once.rs:20:8
|
20 | pub fn once<T, E>(item: Result<T, E>) -> Once<T, E> {
| ^^^^
help: try wrapping the expression in a variant of `std::result::Result`
|
178 ~ stream: once(Ok(async {
179 | Err(Error::InvalidInput {
180 | message: "Stream has already been consumed".to_string(),
181 | })
182 ~ })),
|
178 ~ stream: once(Err(async {
179 | Err(Error::InvalidInput {
180 | message: "Stream has already been consumed".to_string(),
181 | })
182 ~ })),
|
error[E0308]: mismatched types
--> rust/lancedb/src/data/scannable.rs:474:53
|
474 | let prepend = futures::stream::once(std::future::ready(Ok(batch)));
| --------------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `Result<_, _>`, found `Ready<Result<RecordBatch, _>>`
| |
| arguments to this function are incorrect
|
= note: expected enum `std::result::Result<_, _>`
found struct `std::future::Ready<std::result::Result<arrow_array::RecordBatch, _>>`
note: function defined here
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/once.rs:20:8
|
20 | pub fn once<T, E>(item: Result<T, E>) -> Once<T, E> {
| ^^^^
help: try wrapping the expression in a variant of `std::result::Result`
|
474 | let prepend = futures::stream::once(Ok(std::future::ready(Ok(batch))));
| +++ +
474 | let prepend = futures::stream::once(Err(std::future::ready(Ok(batch))));
| ++++ +
error[E0599]: the method `chain` exists for struct `futures::stream::Once<_, _>`, but its trait bounds were not satisfied
--> rust/lancedb/src/data/scannable.rs:477:37
|
477 | stream: prepend.chain(rest),
| ^^^^^ method cannot be called on `futures::stream::Once<_, _>` due to unsatisfied trait bounds
error[E0308]: mismatched types
--> rust/lancedb/src/data/scannable.rs:482:47
|
482 | stream: futures::stream::once(std::future::ready(Ok(batch))),
| --------------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `Result<_, _>`, found `Ready<Result<RecordBatch, _>>`
| |
| arguments to this function are incorrect
|
= note: expected enum `std::result::Result<_, _>`
found struct `std::future::Ready<std::result::Result<arrow_array::RecordBatch, _>>`
note: function defined here
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/once.rs:20:8
|
20 | pub fn once<T, E>(item: Result<T, E>) -> Once<T, E> {
| ^^^^
help: try wrapping the expression in a variant of `std::result::Result`
|
482 | stream: futures::stream::once(Ok(std::future::ready(Ok(batch)))),
| +++ +
482 | stream: futures::stream::once(Err(std::future::ready(Ok(batch)))),
| ++++ +
error[E0308]: mismatched types
--> rust/lancedb/src/data/scannable.rs:486:56
|
486 | let stream = futures::stream::once(std::future::ready(err));
| --------------------- ^^^^^^^^^^^^^^^^^^^^^^^ expected `Result<_, _>`, found `Ready<Result<_, Error>>`
| |
| arguments to this function are incorrect
|
= note: expected enum `std::result::Result<_, _>`
found struct `std::future::Ready<std::result::Result<_, error::Error>>`
note: function defined here
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/once.rs:20:8
|
20 | pub fn once<T, E>(item: Result<T, E>) -> Once<T, E> {
| ^^^^
help: try wrapping the expression in a variant of `std::result::Result`
|
486 | let stream = futures::stream::once(Ok(std::future::ready(err)));
| +++ +
486 | let stream = futures::stream::once(Err(std::future::ready(err)));
| ++++ +
error[E0599]: no method named `and_then` found for struct `Pin<Box<dyn Future<Output = Result<(), Error>> + Send>>` in the current scope
--> rust/lancedb/src/io/object_store.rs:153:32
|
153 | Box::pin(put_secondary.and_then(|_| put_primary))
| ^^^^^^^^
error[E0271]: expected `IntoIter<Result<RecordBatch, _>, 1>` to be an iterator that yields `Result<Result<RecordBatch, Error>, _>`, but it yields `Result<RecordBatch, _>`
--> rust/lancedb/src/query.rs:1465:25
|
1465 | return Box::pin(SimpleRecordBatchStream::new(
| ^^^^^^^^^^^^^^^^^^^^^^^ expected `Result<Result<RecordBatch, Error>, _>`, found `Result<RecordBatch, _>`
error[E0271]: expected `IntoIter<Result<RecordBatch, _>>` to be an iterator that yields `Result<Result<RecordBatch, Error>, _>`, but it yields `Result<RecordBatch, _>`
--> rust/lancedb/src/query.rs:1478:14
|
1478 | Box::pin(SimpleRecordBatchStream::new(stream::iter(batches), schema))
| ^^^^^^^^^^^^^^^^^^^^^^^ expected `Result<Result<RecordBatch, Error>, _>`, found `Result<RecordBatch, _>`
error[E0308]: mismatched types
--> rust/lancedb/src/remote/table/insert.rs:626:44
|
626 | let stream = futures::stream::once(async move {
| ______________________---------------------_^
| | |
| | arguments to this function are incorrect
... |
791 | | Ok::<_, DataFusionError>(batch)
792 | | });
| |_________^ expected `Result<_, _>`, found `async` block
|
= note: expected enum `std::result::Result<_, _>`
found `async` block `{async block@rust/lancedb/src/remote/table/insert.rs:626:44: 626:54}`
note: function defined here
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/once.rs:20:8
|
20 | pub fn once<T, E>(item: Result<T, E>) -> Once<T, E> {
| ^^^^
help: try wrapping the expression in a variant of `std::result::Result`
|
626 ~ let stream = futures::stream::once(Ok(async move {
627 | // Multipart writes with a byte budget split the partition into
...
791 | Ok::<_, DataFusionError>(batch)
792 ~ }));
|
626 ~ let stream = futures::stream::once(Err(async move {
627 | // Multipart writes with a byte budget split the partition into
...
791 | Ok::<_, DataFusionError>(batch)
792 ~ }));
|
error[E0277]: the trait bound `futures::stream::Once<_, _>: futures_core::stream::Stream` is not satisfied
--> rust/lancedb/src/remote/table/insert.rs:794:12
|
794 | Ok(Box::pin(RecordBatchStreamAdapter::new(
| ____________^
795 | | COUNT_SCHEMA.clone(),
796 | | stream,
797 | | )))
| |__________^ the trait `futures_core::stream::Stream` is not implemented for `futures::stream::Once<_, _>`
error[E0599]: no method named `try_collect` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/remote/table.rs:2442:49
|
2442 | let result: Result<Vec<_>> = stream.try_collect().await.map_err(Error::from);
| ^^^^^^^^^^^
error[E0277]: the trait bound `impl Stream<Item = Result<Bytes, Error>>: TryStream` is not satisfied
--> rust/lancedb/src/remote/util.rs:47:35
|
47 | Ok(reqwest::Body::wrap_stream(stream))
| -------------------------- ^^^^^^ unsatisfied trait bound
| |
| required by a bound introduced by this call
error[E0599]: no method named `map_ok` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/table/datafusion/insert.rs:200:30
|
200 | input_stream.map_ok(move |batch| {
| -------------^^^^^^ method not found in `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>`
error[E0308]: mismatched types
--> rust/lancedb/src/table/datafusion/insert.rs:208:44
|
208 | let stream = futures::stream::once(async move {
| ______________________---------------------_^
| | |
| | arguments to this function are incorrect
209 | | if let Some(tracker) = tracker
210 | | && write_params.write_progress.is_none()
... |
255 | | )?)
256 | | });
| |_________^ expected `Result<_, _>`, found `async` block
|
= note: expected enum `std::result::Result<_, _>`
found `async` block `{async block@rust/lancedb/src/table/datafusion/insert.rs:208:44: 208:54}`
note: function defined here
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/once.rs:20:8
|
20 | pub fn once<T, E>(item: Result<T, E>) -> Once<T, E> {
| ^^^^
help: try wrapping the expression in a variant of `std::result::Result`
|
208 ~ let stream = futures::stream::once(Ok(async move {
209 | if let Some(tracker) = tracker
...
255 | )?)
256 ~ }));
|
208 ~ let stream = futures::stream::once(Err(async move {
209 | if let Some(tracker) = tracker
...
255 | )?)
256 ~ }));
|
error[E0277]: the trait bound `futures::stream::Once<_, _>: futures_core::stream::Stream` is not satisfied
--> rust/lancedb/src/table/datafusion/insert.rs:258:12
|
258 | Ok(Box::pin(RecordBatchStreamAdapter::new(
| ____________^
259 | | COUNT_SCHEMA.clone(),
260 | | stream,
261 | | )))
| |__________^ the trait `futures_core::stream::Stream` is not implemented for `futures::stream::Once<_, _>`
error[E0599]: no method named `map_ok` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/table/datafusion.rs:128:29
|
128 | let stream = stream.map_ok(move |batch| {
| -------^^^^^^ method not found in `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>`
error[E0599]: no method named `map_err` found for struct `Pin<Box<dyn Future<Output = Result<Arc<...>, ...>> + Send>>` in the current scope
--> rust/lancedb/src/table/datafusion.rs:245:14
|
242 | let plan = self
| ____________________-
243 | | .table
244 | | .create_plan(&AnyQuery::Query(query), options)
245 | | .map_err(|err| DataFusionError::External(err.into()))
| | -^^^^^^^ method not found in `Pin<Box<dyn Future<Output = Result<Arc<...>, ...>> + Send>>`
| |_____________|
error[E0599]: no method named `next` found for struct `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>` in the current scope
--> rust/lancedb/src/table.rs:3048:48
|
3048 | while let Some(batch) = stream.next().await {
| ^^^^ method not found in `Pin<Box<dyn datafusion_physical_plan::RecordBatchStream + std::marker::Send>>`
error[E0277]: the trait bound `JoinHandle<Result<(), Error>>: Future` is not satisfied
--> rust/lancedb/src/table.rs:3038:23
|
3038 | let handles = FuturesUnordered::new();
| ^^^^^^^^^^^^^^^^^^^^^^^ the trait `futures::Future` is not implemented for `tokio::task::JoinHandle<std::result::Result<(), error::Error>>`
error[E0277]: `FuturesUnordered<JoinHandle<Result<(), Error>>>` is not an iterator
--> rust/lancedb/src/table.rs:3054:23
|
3054 | for handle in handles {
| ^^^^^^^ `FuturesUnordered<JoinHandle<Result<(), Error>>>` is not an iterator
error[E0277]: the trait bound `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}: futures::IntoFuture` is not satisfied
--> rust/lancedb/src/table.rs:3450:13
|
3449 | let mut sorted_sizes = join_all(
| -------- required by a bound introduced by this call
3450 | / frags
3451 | | .iter()
3452 | | .map(|frag| async move { frag.physical_rows().await.unwrap_or(0) }),
| |___________________________________________________________________________________^ the trait `futures::Future` is not implemented for `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}`
|
= note: `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}` implements similarly named trait `std::future::Future`, but not `futures::Future`
= help: the following other types implement trait `futures::Future`:
&'a mut F
AssertUnwindSafe<F>
BiLockAcquire<T>
Box<F>
Concat2<S>
Either<A, B>
Finished<T, E>
Fold<S, F, Fut, T>
and 43 others
= note: required for `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}` to implement `futures::IntoFuture`
note: required by a bound in `join_all`
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/future/join_all.rs:78:20
|
76 | pub fn join_all<I>(i: I) -> JoinAll<I>
| -------- required by a bound in this function
77 | where I: IntoIterator,
78 | I::Item: IntoFuture,
| ^^^^^^^^^^ required by this bound in `join_all`
error[E0277]: the trait bound `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}: futures::Future` is not satisfied
--> rust/lancedb/src/table.rs:3449:32
|
3449 | let mut sorted_sizes = join_all(
| ________________________________^
3450 | | frags
3451 | | .iter()
3452 | | .map(|frag| async move { frag.physical_rows().await.unwrap_or(0) }),
3453 | | )
| |_________^ the trait `futures::Future` is not implemented for `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}`
|
= note: `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}` implements similarly named trait `std::future::Future`, but not `futures::Future`
= help: the following other types implement trait `futures::Future`:
&'a mut F
AssertUnwindSafe<F>
BiLockAcquire<T>
Box<F>
Concat2<S>
Either<A, B>
Finished<T, E>
Fold<S, F, Fut, T>
and 43 others
= note: required for `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}` to implement `futures::IntoFuture`
note: required by a bound in `JoinAll`
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/future/join_all.rs:24:20
|
22 | pub struct JoinAll<I>
| ------- required by a bound in this struct
23 | where I: IntoIterator,
24 | I::Item: IntoFuture,
| ^^^^^^^^^^ required by this bound in `JoinAll`
error[E0277]: `JoinAll<Map<Iter<'_, FileFragment>, {closure@...}>>` is not a future
--> rust/lancedb/src/table.rs:3454:10
|
3449 | let mut sorted_sizes = join_all(
| ________________________________-
3450 | | frags
3451 | | .iter()
3452 | | .map(|frag| async move { frag.physical_rows().await.unwrap_or(0) }),
3453 | | )
| |_________- this call returns `JoinAll<std::iter::Map<std::slice::Iter<'_, FileFragment>, {closure@rust/lancedb/src/table.rs:3452:22: 3452:28}>>`
3454 | .await;
| ^^^^^ `JoinAll<Map<Iter<'_, FileFragment>, {closure@...}>>` is not a future
error[E0277]: the trait bound `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}: futures::Future` is not satisfied
--> rust/lancedb/src/table.rs:3454:10
|
3454 | .await;
| ^^^^^ the trait `futures::Future` is not implemented for `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}`
|
= note: `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}` implements similarly named trait `std::future::Future`, but not `futures::Future`
= help: the following other types implement trait `futures::Future`:
&'a mut F
AssertUnwindSafe<F>
BiLockAcquire<T>
Box<F>
Concat2<S>
Either<A, B>
Finished<T, E>
Fold<S, F, Fut, T>
and 43 others
= note: required for `{async block@rust/lancedb/src/table.rs:3452:29: 3452:39}` to implement `futures::IntoFuture`
note: required by a bound in `JoinAll`
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/future/join_all.rs:24:20
|
22 | pub struct JoinAll<I>
| ------- required by a bound in this struct
23 | where I: IntoIterator,
24 | I::Item: IntoFuture,
| ^^^^^^^^^^ required by this bound in `JoinAll`
error[E0282]: type annotations needed
--> rust/lancedb/src/utils/background_cache.rs:119:40
|
119 | inner: Arc::new(Mutex::new(CacheInner {
| ________________________________________^
120 | | state: State::Empty,
121 | | generation: 0,
122 | | })),
| |_____________^ cannot infer type of the type parameter `E` declared on the struct `CacheInner`
|
help: consider specifying the generic arguments
|
119 | inner: Arc::new(Mutex::new(CacheInner::<V, E> {
| ++++++++
error[E0282]: type annotations needed
--> rust/lancedb/src/utils/background_cache.rs:134:9
|
134 | cache.state.fresh_value(self.ttl, self.refresh_window)
| ^^^^^^^^^^^ cannot infer type for type parameter `E`
error[E0282]: type annotations needed
--> rust/lancedb/src/utils/background_cache.rs:173:23
|
173 | cache.state = State::Current(value, clock::now());
| ^^^^^^^^^^^^^^ cannot infer type of the type parameter `E` declared on the enum `State`
|
help: consider specifying the generic arguments
|
173 | cache.state = State::<V, E>::Current(value, clock::now());
| ++++++++
error[E0282]: type annotations needed
--> rust/lancedb/src/utils/background_cache.rs:182:23
|
182 | cache.state = State::Empty;
| ^^^^^^^^^^^^ cannot infer type of the type parameter `E` declared on the enum `State`
|
help: consider specifying the generic arguments
|
182 | cache.state = State::<V, E>::Empty;
| ++++++++
error[E0599]: no method named `boxed` found for `async` block `{async block@rust/lancedb/src/utils/background_cache.rs:269:22: 269:32}` in the current scope
--> rust/lancedb/src/utils/background_cache.rs:270:14
|
269 | let shared = async move { (fetch)().await.map_err(Arc::new) }
| ______________________-
270 | | .boxed()
| | -^^^^^ method not found in `{async block@rust/lancedb/src/utils/background_cache.rs:269:22: 269:32}`
| |_____________|
error[E0277]: the trait bound `TimeoutStream: futures_core::stream::Stream` is not satisfied
--> rust/lancedb/src/utils/mod.rs:345:9
|
345 | Box::pin(Self::new(inner, timeout))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ unsatisfied trait bound
error[E0599]: no method named `poll_next` found for struct `Pin<&mut TimeoutStream>` in the current scope
--> rust/lancedb/src/utils/mod.rs:376:22
|
376 | self.poll_next(cx)
| ^^^^^^^^^
|
= help: items from traits can only be used if the trait is implemented and in scope
= note: the following traits define an item `poll_next`, perhaps you need to implement one of them:
candidate #1: `futures_core::stream::Stream`
candidate #2: `sorts::stream::PartitionedStream`
help: there is a method `collect` with a similar name, but with different arguments
--> $CARGO_HOME/registry/src/index.crates.io-1949cf8c6b5b557f/futures-0.1.31/src/stream/mod.rs:563:5
|
563 | / fn collect(self) -> Collect<Self>
564 | | where Self: Sized
| |_________________________^
error[E0599]: no method named `poll_unpin` found for mutable reference `&mut Pin<Box<Sleep>>` in the current scope
--> rust/lancedb/src/utils/mod.rs:378:75
|
378 | TimeoutState::Started { deadline, timeout } => match deadline.poll_unpin(cx) {
| ^^^^^^^^^^ method not found in `&mut Pin<Box<Sleep>>`
error[E0599]: no method named `poll_next` found for struct `Pin<&mut Pin<Box<dyn RecordBatchStream + Send>>>` in the current scope
--> rust/lancedb/src/utils/mod.rs:386:27
|
386 | inner.poll_next(cx)
| ^^^^^^^^^
error[E0277]: the trait bound `MaxBatchLengthStream: futures_core::stream::Stream` is not satisfied
--> rust/lancedb/src/utils/mod.rs:419:13
|
419 | Box::pin(Self::new(inner, max_batch_length))
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ unsatisfied trait bound
error[E0599]: no method named `poll_next` found for struct `Pin<&mut Pin<Box<dyn RecordBatchStream + Send>>>` in the current scope
--> rust/lancedb/src/utils/mod.rs:439:50
|
439 | return Pin::new(&mut self.inner).poll_next(cx);
| ^^^^^^^^^
error[E0599]: no method named `poll_next` found for struct `Pin<&mut Pin<Box<dyn RecordBatchStream + Send>>>` in the current scope
--> rust/lancedb/src/utils/mod.rs:459:45
|
459 | match Pin::new(&mut self.inner).poll_next(cx) {
| ^^^^^^^^^
Some errors have detailed explanations: E0046, E0107, E0191, E0271, E0277, E0282, E0308, E0407, E0425...
For more information about an error, try `rustc --explain E0046`.
error: could not compile `lancedb` (lib) due to 118 previous errors
```
The docs have no link checking at all, so external links rot silently: a
trial run already found `docs/src/python/python.md` pointing at
`lancedb.github.io/lance-namespace`, which returns 404 since the
repository moved to the lance-format org.
Checking external links on the blocking path would be the wrong trade:
third-party hosts rate-limit automated clients, reject non-browser user
agents, and go down temporarily, so any of them having a bad minute
would turn unrelated PRs red. Following lance-format/lance#8315, this
adds a daily `lychee` run that reports broken links into a single
tracking issue, rewritten in place on each run and closed automatically
once every link resolves. The scan job runs the downloaded lychee binary
with a read-only token; everything that writes lives in a separate
report job, and a non-verdict lychee exit fails the run instead of
publishing a bogus report.
The check is restricted to http(s) links because much of `docs/src` is
generated API reference (the `js/` tree comes from `npm run docs`) and
the hand-written pages use mkdocstrings cross-references and
nav-relative paths that only resolve in the site mkdocs builds, so
relative links would be reported as broken on every run.
The one broken link the trial run surfaced is fixed here; after the fix,
a local run over all 154 files reports 0 errors across 216 unique links.
## Summary
- add regression coverage for repeated table opens through one database
connection
- assert that each open reuses the connection object-store client
without another registry miss
- exercise the table after every open so the test covers the complete
dataset-loading path
## Root cause
At the commit reported in #1600, opening a table constructed a separate
object-store client rather than reusing the client that had already
connected to the database. On S3 this repeated credential discovery,
which could fail intermittently in AWS Lambda and surface as
TableNotFound. The connection-owned Session reuse added later fixed the
runtime path, but no focused test protected the open-table invariant.
## Fix
Add a regression test backed by ObjectStoreRegistry statistics. Three
successive opens must add cache hits while leaving the miss count
unchanged, proving that open_table uses the connection Session and its
authenticated object-store client.
## Validation
- cargo fmt --all
- cargo test --quiet --features remote -p lancedb
database::listing::tests::test_open_table_reuses_connection_object_store
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples
- cargo test --quiet --features remote --tests
Fixes#1600
<!-- lance-gatekeeper-fix:v1 agent=974491978c3e42840f32dbc35492d856
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- convert PyArrow scalar values through their Python representation
before SQL literal rendering
- add an end-to-end regression for updating a fixed-size-list vector
from a queried FixedSizeListScalar
## Root cause
Python update literal conversion used single dispatch for native Python
and NumPy values but had no PyArrow Scalar registration. A
FixedSizeListScalar returned by a query therefore reached the
unsupported generic conversion instead of the existing recursive list
converter.
## Validation
- uv run --extra tests pytest python/tests/test_table.py::test_update
python/tests/test_table.py::test_update_with_arrow_scalar
python/tests/test_table.py::test_update_types -q
- uv run --extra tests pytest python/tests/test_util.py -q
- uv run --project python --extra tests --extra dev ruff format --check
python/python/lancedb/util.py python/python/tests/test_table.py
- uv run --project python --extra tests --extra dev ruff check .
Fixes#1228
<!-- lance-gatekeeper-fix:v1 agent=950dd892194e53b61c203d5e3715cac7
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary\n\n- add a create-table regression for a named database\n-
assert that the derived table URI uses URL separators\n- restore the
four query tests that were moved to temporary files for #1051\n\n## Root
cause\n\n historically joined table names with . On Windows this
inserted a backslash into , so Lance interpreted the URI as an invalid
local filename. The production URI builder now preserves forward slashes
for URI schemes; this change restores the issue-specific tests and adds
direct regression coverage for table creation and the derived URI.\n\n##
Validation\n\n- \n- \n- (passes with four pre-existing warnings in
unrelated remote-table code)\n-
running 814 tests
.......................................................................................
87/814
.....................................i.................................................
174/814
.......................................................................................
261/814
.......................................................................................
348/814
.......................................................................................
435/814
.......................................................................................
522/814
.......................................................................................
609/814
.......................................................................................
696/814
.......................................................................................
783/814
...............................
test result: ok. 813 passed; 0 failed; 1 ignored; 0 measured; 0 filtered
out; finished in 7.76s
running 39 tests
.......................................
test result: ok. 39 passed; 0 failed; 0 ignored; 0 measured; 0 filtered
out; finished in 0.23s
running 6 tests
......
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered
out; finished in 0.03s
running 5 tests
.....
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered
out; finished in 0.10s
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered
out; finished in 0.00s
running 2 tests
..
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered
out; finished in 0.00s
running 2 tests
..
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered
out; finished in 0.00s (867 passed, 1 ignored)\n- focused named-memory
create and restored query tests\n\nFixes #1051\n\n<!--
lance-gatekeeper-fix:v1 agent=5ddf7a9520292b4cbaa58b9ea5a1fe76
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Nothing validates index names, so a `/` in one is reachable, and the
remote client interpolates it straight into the URL, splitting the path
so the router 404s. The index then reads back as missing and cannot be
dropped, while `create_index` keeps succeeding because it sends the name
in the body.
Encode at the three affected sites, mirroring `fetch_blob_files`. The
shared Rust client covers all bindings.
## Summary
- add a public Node API regression test for JSON server errors from
remote table operations
- verify countRows reports the server message instead of an ArrayBuffer
decoding TypeError
## Root cause and fix
The former TypeScript remote HTTP client passed an Axios-decoded JSON
error object to TextDecoder, which masked the server response with an
ArrayBuffer TypeError. The current Rust-backed remote client consumes
non-success response bodies as text and propagates them through the Node
error chain. This test exercises that corrected path through countRows
and prevents the original failure from regressing.
## Validation
- pnpm build
- pnpm lint-ci
- pnpm test --runInBand __test__/remote.test.ts
- pnpm run docs
Fixes#825
<!-- lance-gatekeeper-fix:v1 agent=91591c3d6b065796e6166664ef638aa7
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- treat `NotFound` from the mirrored secondary copy as a cache miss
while preserving every other secondary error
- perform the durable primary copy after either a successful secondary
copy or a secondary cache miss
- cover both an initially missing secondary manifest and eviction
immediately before the secondary copy
## Root cause
Readers can use process-local secondary stores that do not contain a
staging manifest written by another process, or that evict it before
finalization. `MirroringObjectStore::copy_opts` propagated that
secondary `NotFound`, so older object_store versions could loop
indefinitely and the locked version aborted before performing the
durable primary copy.
## Validation
- `cargo fmt --all -- --check`
- `cargo test --quiet --features remote -p lancedb
io::object_store::test::test_copy_when -- --nocapture`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
Fixes#1176
<!-- lance-gatekeeper-fix:v1 agent=636210af9dcd25b6dceadebd2fcafc6f
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add an end-to-end regression for schemas created by a different Apache
Arrow package instance
- cover seeded table creation, filtered scanning, and Float64 vector
search across Arrow 15–18
## Root cause
Apache Arrow's runtime identity checks historically rejected schemas
created by another installed Arrow instance, producing the constructor
failures reported in the issue. LanceDB's peer dependency and
foreign-schema sanitization now handle that boundary, but the complete
reported workflow was only covered by separate unit tests. This
regression keeps the repaired behavior protected end to end.
## Validation
- `pnpm exec jest --runInBand __test__/table.test.ts` (281 passed)
- `pnpm lint-ci`
- `pnpm build`
- `pnpm run docs`
Fixes#882
<!-- lance-gatekeeper-fix:v1 agent=43b19dea581cfbc83ee1e9ed21a335a6
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- validate the generated LanceDB Cloud hostname during connection setup
- return a clear invalid-input error for empty, overlong, or oversized
DNS names before network resolution
- add Rust and Python regression coverage for malformed `db://`
authorities
## Root cause
The `db://` authority and region were interpolated into the Cloud API
hostname without DNS length validation. Empty or overlong labels
therefore reached the resolver and surfaced as an opaque IDNA
`UnicodeError` instead of a useful connection error.
## Validation
- `cargo test --quiet --features remote -p lancedb
test_rejects_invalid_cloud_dns_hostname --lib`
- `cargo check --quiet --features remote --tests --examples`
- `uv run --no-sync --extra tests pytest
python/tests/test_remote_db.py::test_async_remote_db
python/tests/test_remote_db.py::test_connect_rejects_invalid_cloud_dns_hostname
-q`
- `cargo fmt --all -- --check`
- `ruff check .`
- `ruff format --check python/python/tests/test_remote_db.py`
Fixes#799
<!-- lance-gatekeeper-fix:v1 agent=4d1597b3d244b58f0603ed40a8a59cf9
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- Adds a LanceDB regression for stable row IDs, scattered deletes,
IVF_RQ, and default index optimization.
- Verifies optimization completes and preserves the expected live-row
count.
## Root cause
Lance 3.0.1 built the stable-row-ID address list by dropping deleted IDs
while retaining the original ID list. The subsequent positional zip
misaligned IDs and addresses, so vector partition joins requested
deleted rows and failed with batch.num_rows() != chunk.len(). Lance PR
https://github.com/lance-format/lance/pull/7704 corrected the generic
filter, and the LanceDB dependency currently pinned on main contains
that correction.
## Fix
Add regression coverage at the Rust Table optimize surface using the
IVF_RQ configuration from the report. This locks the upstream correction
into the LanceDB workflow that originally crashed.
## Validation
- cargo fmt --all -- --check
- cargo test --quiet --features remote -p lancedb table::optimize::tests
(14 passed)
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples -p lancedb
Fixes#3330
<!-- lance-gatekeeper-fix:v1 agent=4c2c25373942aab9ba9f7444977de7e3
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a Python regression test for two partial-schema merge inserts
against the same BTree-indexed rows
- verify repeated updates retain one copy of every row and the final
update values
## Root cause
Lance 4.0, used by LanceDB 0.30.2, removed a rewritten fragment from the
index bitmap while stale BTree entries for that fragment remained
searchable. The next merge found each target through both the stale
index and the unindexed-fragment scan, producing the ambiguous-match
error. Lance fixed the root cause in lance-format/lance#6563 by applying
the fragment-bitmap allow-list to index results, and the Lance release
pinned by current LanceDB includes that fix. This test preserves the
corrected behavior through the Python API.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_table.py
-k merge_insert -q` (9 passed)
- `cd python && uv run --extra tests --extra dev ruff format --check
python/tests/test_table.py`
- `cd python && uv run --extra tests --extra dev ruff check
python/tests/test_table.py`
Repository-wide Ruff also reports 20 pre-existing violations in
untouched CI and plugin scripts.
Fixes#3280
<!-- lance-gatekeeper-fix:v1 agent=ee6b9565f9780712026076930566f116
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a minimized regression for mostly-null `list<float32>` data at the
v2.2 structural page boundary
- verify scans preserve all 64,885 rows, including 64,668 null list
values
## Root cause
Lance 3.0.0 sliced repetition/definition state using top-level row
offsets in the complex all-null decoder. At this page boundary, the list
and validity children were materialized at different lengths. The
current Lance dependency contains the upstream decoder repair; this test
locks that behavior into the LanceDB Python suite without duplicating
decoder logic.
## Validation
- reproduced the attached 1,892,466-row case on `lancedb==0.30.0` with
`expected 1024 got 285`
- verified the full attachment reads on the current branch
- `python/.venv/bin/ruff format --check
python/python/tests/test_table.py`
- `python/.venv/bin/ruff check .`
- `cd python && uv run --extra tests pytest
python/tests/test_table.py::test_read_mostly_null_list_v2_2_page_boundary
-q`
- `cd python && uv run --extra tests pytest python/tests/test_table.py
-q` (137 passed)
Fixes#3194
<!-- lance-gatekeeper-fix:v1 agent=0445adc5303a3302152cea3d2110bed1
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a LanceDB regression for `merge_insert` with a non-nullable
`FixedSizeBinary` column
- exercise matched updates, unmatched inserts, and source-missing
deletes
- assert the exact merge statistics and final row count
## Root cause
The Arrow `take` kernel previously ignored nulls in the index array for
`FixedSizeBinary`. DataFusion uses that kernel while constructing
outer-join results, so the join behind
`when_not_matched_by_source_delete` could place invalid values into
non-nullable columns. The current Arrow dependency contains the upstream
fix; this test locks the corrected behavior at the LanceDB API boundary.
## Validation
- `cargo fmt --all -- --check`
- `cargo test --quiet --features remote --tests`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
Fixes#2869
<!-- lance-gatekeeper-fix:v1 agent=e275446044185ef4e8cf88da6af3e70b
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add deterministic regression coverage that `Table.add()` releases
backing Arrow buffers without cyclic garbage collection
- track the foreign buffer owner rather than RSS, separating live input
retention from allocator high-water behavior
- preserve the bounded-lifetime behavior of the Scannable writer that
superseded the historical preprocessing path
## Root cause
The historical Python preprocessing/write path produced a high allocator
RSS while ingesting very wide IPC batches. The current Scannable writer
releases each input buffer when `Table.add()` completes; remaining RSS
is allocator high-water rather than a live Arrow reference. The resolved
behavior had no regression coverage, so a future native lifetime
regression could silently reintroduce the original failure mode.
## Validation
- `uv run --extra tests --extra dev maturin develop`
- `uv run --project python --extra tests pytest
python/python/tests/test_table.py::test_add
python/python/tests/test_table.py::test_add_releases_arrow_buffers_without_gc
-q`
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_table.py`
- `uv run --project python --extra dev ruff check .`
Fixes#2512
<!-- lance-gatekeeper-fix:v1 agent=29226408a8d07da592daf341d5384e37
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add an end-to-end Python regression for pandas DataFrame inputs merged
into a table created from a Pydantic model
- verify reordered, nullable Arrow source fields can update and insert
into a non-nullable target schema when the values contain no nulls
## Root cause
Lance merge_insert previously compared source schema nullability with
the target, unlike add. The upstream fix now pinned by LanceDB ignores
declared nullability during schema compatibility and validates actual
null values at write time. LanceDB lacked regression coverage for the
full pandas-to-Pydantic path, so this test locks in the correct behavior
without falsifying the input schema nullability.
## Validation
- 5 focused merge-insert tests passed
- Ruff lint passed for the repository
- Ruff format check passed for the changed file
- git diff --check passed
Fixes#2366
<!-- lance-gatekeeper-fix:v1 agent=f897fccfa206620c8a2acdc3bcd1c21f
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- capture the stale-index state behind the reported fixed-size-binary
panic: the vector and FTS indices cover newer fragments while the BTree
prefilter does not
- verify vector, FTS, and hybrid searches return matches from both
scalar-indexed and unindexed fragments without panicking
- preserve binding-level coverage for the Lance fix in
https://github.com/lance-format/lance/pull/3768, which restricts
incomplete scalar prefilters when search indices are further ahead
The production root cause is in Lance and the current LanceDB dependency
already contains that fix, so this change adds the missing LanceDB
Python regression coverage.
## Validation
- `cd python && uv run --no-sync pytest
python/tests/test_hybrid_query.py::test_hybrid_query_with_stale_fixed_size_binary_prefilter
-q`
- `cd python && uv run --no-sync pytest
python/tests/test_hybrid_query.py -q`
- `python/.venv/bin/ruff check .`
- `python/.venv/bin/ruff format --check
python/python/tests/test_hybrid_query.py`
Fixes#2370
<!-- lance-gatekeeper-fix:v1 agent=5d16e59b9e513fd9247e0698732fa283
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- cover explicit FixedSizeList schemas populated from Float32Array
values
- verify the original vector.0 failure stays fixed across Arrow 15, 16,
17, and 18
## Root cause and fix
In v0.16, schema subset inference treated typed-array vectors as nested
objects and looked up numeric paths such as vector.0, which do not exist
in a FixedSizeList schema. Current typed-array handling correctly
recognizes ArrayBuffer views as vector values instead of traversing
their elements. This change adds the missing regression coverage for the
reported explicit-schema path so that behavior cannot regress unnoticed.
## Validation
- pnpm test __test__/arrow.test.ts --runInBand
- pnpm lint
- pnpm build
- pnpm run docs
- pnpm test --runInBand (681 passed, 5 skipped)
Fixes#2134
<!-- lance-gatekeeper-fix:v1 agent=1d548cb70f6df110ce0a5b119395b52a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add fast regression coverage for VoyageAI `voyage-3` source embeddings
- verify table text uses `client.embed` and never
`client.multimodal_embed`
## Root cause
The original VoyageAI source-embedding path treated table source values
as images and always invoked the multimodal API. Production routing was
corrected by later merged changes, but the table regression was covered
only by API-gated slow tests. This test locks the corrected text routing
into the regular unit suite.
## Validation
- `cd python && uv run --extra tests pytest
python/tests/test_voyageai_embeddings.py -q`
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/tests/test_voyageai_embeddings.py`
- `uv run --project python --extra tests --extra dev ruff check .`
Fixes#2059
<!-- lance-gatekeeper-fix:v1 agent=49b9e2daeed95a78ce827e2bf90abda0
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- pass an Instructor-compatible `[instruction, text]` pair when
detecting embedding dimensions
- add a regression test that verifies the dimension probe uses the
configured source instruction
## Root cause
`InstructorEmbeddingFunction.ndims()` encoded a bare string even though
Instructor models require instruction/text pairs. With affected
`sentence-transformers` versions, the bare input omitted
`instruction_mask` and raised `KeyError` while defining the LanceDB
schema.
## Validation
- `uv run --extra tests pytest python/tests/test_embeddings.py -q` (`14
passed, 9 skipped`)
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/lancedb/embeddings/instructor.py
python/python/tests/test_embeddings.py`
- `uv run --project python --extra tests --extra dev ruff check .`
Fixes#2041
<!-- lance-gatekeeper-fix:v1 agent=4b05e0d9f3eef17bccfb446e788294f4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a Python regression for vector search over a sliced Arrow table
with nullable scalar columns
- verify the nearest row retains its non-null score values after the
table is written
## Root cause
Lance 0.19.2 deep-copied a validity bitmap without preserving its
non-zero bit offset. For a sliced nullable table, scalar values and
vectors began at the slice while the copied validity bitmap began at the
parent table's first row. That made valid score values appear null even
though the corresponding vector stayed intact. The upstream Lance repair
is already present in the current dependency; this adds a LanceDB-level
guard for the reported create/search path.
## Validation
- reproduced on Python 3.12 with LanceDB 0.16.0, pylance 0.19.2, PyArrow
18.0.0, and Polars 1.14.0
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_table.py`
- `uv run --project python --extra dev ruff check .`
- `cd python && uv run --extra tests pytest
python/tests/test_table.py::test_search_preserves_nulls_from_sliced_arrow_table
-q`
Fixes#1879
<!-- lance-gatekeeper-fix:v1 agent=bfa0551793f8e3cf3980cf64ad89908a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a LanceDB core regression for compaction overlapping appends
through separate table handles
- verify concurrent commits preserve fragment ID order on an indexed
table
- run the follow-up compaction that exposed the original row-ID ordering
failure and verify all rows remain
## Root cause
Older Lance versions could reserve fragment IDs for compaction, allow
concurrent appends to commit later IDs, and then commit the reserved
compaction fragments at the end of the manifest. A later compaction
could consequently receive row IDs out of order. Current Lance sorts
fragments at the transaction boundary; this adds the missing
LanceDB-level regression coverage for the Node-visible concurrency
contract.
## Validation
- `cargo fmt --all`
- focused regression passed once with output and 20 repeated runs
- `cargo test --quiet --features remote -p lancedb
table::optimize::tests` (14 passed)
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests` (867 passed, 1 ignored)
Fixes#1498
<!-- lance-gatekeeper-fix:v1 agent=93aaefb15507dca52d064e15388773d7
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- make the existing #1968 regression explicitly assert that schema-only
table creation succeeds
- verify the new table has zero rows and preserves the requested
fixed-size vector schema before accepting subsequent data
## Root cause
In v0.16.0, schema-only table creation sent an empty table through
vector sanitization, which calculated a remainder using `len(data)` and
raised `ZeroDivisionError`. Later refactors removed that runtime path,
but the issue-specific regression only asserted the final row count
after a subsequent add. This change makes the reported operation and its
expected empty-table state explicit so the original defect remains
directly covered.
## Validation
- `uv run --extra tests pytest
python/tests/test_table.py::test_create_table_without_data_with_vector_schema
-q`
- `uv --project python run --extra tests --extra dev ruff format --check
python/python/tests/test_table.py`
- `uv --project python run --extra tests --extra dev ruff check .`
- `git diff --check`
Fixes#1968
<!-- lance-gatekeeper-fix:v1 agent=b8ec6f40f4bba2f9beeaaae12233e5c4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- select rustls with native certificate roots explicitly for LanceDB's
remote HTTP client
- add a Linux regression test that rejects `libssl` or `libcrypto`
dependencies in the built Python extension
## Root cause
The Python remote client originally enabled reqwest's native TLS
backend. During manylinux wheel repair, that caused OpenSSL 1.1
libraries to be bundled into the wheel. Loading those libraries on RHEL
9 with FIPS enabled aborts during the OpenSSL self-test before `import
lancedb` can complete.
LanceDB has since moved away from native TLS, but its own reqwest
dependency relied on transitive rustls feature selection and the built
extension had no regression guard. This change makes rustls selection
explicit and tests the produced Linux native module's dynamic
dependencies.
## Validation
- `uv run --no-sync pytest python/tests/test_import.py -q`
- `ruff format --check python`
- `ruff check .`
- `cargo fmt --all -- --check`
- `cargo check --quiet --features remote --tests --examples`
- `ldd python/lancedb/_lancedb.abi3.so` (no `libssl` or `libcrypto`
dependency)
- verified the resolved Python Rust dependency graph contains rustls and
no `openssl-sys` or `native-tls`
Fixes#1884
<!-- lance-gatekeeper-fix:v1 agent=31f916c7ac5c072bbbd54f3539d24f71
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- align the PyO3 runtime and build ABI floor with the declared Python
3.10 minimum
- add a regression test that keeps both ABI features synchronized with
`requires-python`
## Root cause
The Python 3.10 support-floor update originally changed PyO3 to
`abi3-py310`, but a later dependency update reverted both PyO3 features
to `abi3-py39`. Published Windows wheels were consequently tagged
`cp39-abi3` while importing `PyCMethod_New`, a stable-ABI procedure
absent from CPython 3.9.0 and 3.9.1. Windows reports that mismatch as
“The specified procedure could not be found” while loading `_lancedb`.
Restoring `abi3-py310` makes the wheel tag and native imports agree with
the package metadata and prevents future wheels from advertising
unsupported Python 3.9 compatibility.
## Validation
- `uv run --extra tests pytest python/tests/test_package_metadata.py -q`
- `uv run --extra tests --extra dev ruff format --check .`
- `uv run --extra tests --extra dev ruff check .`
- `cargo fmt --all`
- `cargo check --quiet -p lancedb-python`
- `uvx --from maturin==1.12.4 maturin build --profile ci` (built
`lancedb-0.37.1b0-cp310-abi3-manylinux_2_34_x86_64.whl`)
Fixes#2051
<!-- lance-gatekeeper-fix:v1 agent=d66c984498190d2207d1c5126cba5047
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- expand the synchronous debugger regression to enumerate every exposed
connection attribute while the Python background loop is unavailable
- retain direct representation checks for connections and tables
## Root cause
VS Code debugpy suspends Python threads at a breakpoint and inspects
local variables. Connection representation and property access
previously dispatched asynchronous work to LanceDBBackgroundEventLoop
and waited for the suspended loop thread, deadlocking the debugger. The
production safeguards landed in #3620 and #3788; this regression
exercises debugger-style whole-object expansion so a newly exposed
property cannot reintroduce the original failure.
## Validation
- uv run --no-sync pytest
python/tests/test_db.py::test_sync_debugger_inspection_does_not_use_background_loop
python/tests/test_db.py::test_read_consistency_interval_does_not_use_background_loop
-q (2 passed)
- uv run --no-sync pytest python/tests/test_db.py -q (48 passed)
- python/.venv/bin/ruff format --check python/python/tests/test_db.py
- python/.venv/bin/ruff check .
- git diff --check
Fixes#3611
<!-- lance-gatekeeper-fix:v1 agent=cdf4b39b2ce2ccb3eb5fe501acae77bb
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- replace the synthetic registry-variable metadata test with the OpenAI
embedding function reported in #2387
- verify the resolved API key survives table metadata reconstruction
- assert the OpenAI client receives the resolved key while serialized
metadata retains the variable reference
## Root cause
LanceDB 0.22.0 reconstructed embedding functions from table metadata
with the model constructor, bypassing EmbeddingFunction.create and
leaving the literal $var:api_key placeholder in OpenAI configuration.
The production path was corrected for duplicate #2181 by #2640; this
change gives that fix direct, network-free OpenAI regression coverage
for #2387.
## Validation
- uv run --extra tests pytest python/tests/test_embeddings.py -q (13
passed, 9 skipped)
- uv run --project python --extra dev ruff check .
- uv run --project python --extra dev ruff format --check
python/python/tests/test_embeddings.py
- git diff --check
Fixes#2387
<!-- lance-gatekeeper-fix:v1 agent=d453b1b9b2a298a776f2e4ea1b1449b5
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add Python regression coverage for an IVF build that cannot form all
requested non-empty partitions
- verify hierarchical k-means returns an actionable RuntimeError instead
of panicking or silently creating a degenerate index
- exercise the current Lance v10.1.0-beta.1 dependency, which contains
the upstream error-return fix
## Root cause
Hierarchical k-means previously guarded a shortfall in generated
clusters with only a debug assertion. Debug builds panicked, while
release builds could silently publish an index with many empty
partitions. The upstream Lance fix now returns a descriptive error and
is already included in the dependency pinned on main; this test locks in
propagation through the LanceDB Python API.
## Validation
- uv run --extra tests pytest python/tests/test_index.py -q (24 passed)
- uv run --extra tests pytest
python/tests/test_index.py::test_create_ivf_index_reports_unsplittable_partitions
-q (1 passed)
- python/.venv/bin/ruff format python/python/tests/test_index.py
- python/.venv/bin/ruff check .
- git diff --check
Fixes#3649
<!-- lance-gatekeeper-fix:v1 agent=a4d34448a9d350a3e2e659f33f5db6f2
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
<!-- lance-gatekeeper-fix:v1 agent=5c80c44c083b3b8ad0da595419d468fc
generation=1 -->
## Root cause
The legacy synchronous Python table called `delete` on a shared, mutable
`lance.Dataset`. Concurrent table operations could hold a PyO3 borrow
while delete requested an exclusive borrow, producing `RuntimeError:
Already borrowed`. The current async-backed binding fixes this by
cloning its thread-safe Rust table handle before awaiting, but that
concurrency contract had no regression coverage.
## Fix
- Document why delete must clone the Rust table handle before entering
its async future.
- Add a barrier-synchronized regression test that deletes distinct rows
through one shared table from eight Python threads.
- Verify every delete commits exactly one row, every commit gets a
distinct version, and no rows remain.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo fmt --all -- --check`
- `uv run --extra tests --extra dev ruff format --check
python/tests/test_table.py`
- `uv run --extra tests --extra dev ruff check
python/tests/test_table.py`
- `uv run --extra tests --extra dev pytest
python/tests/test_table.py::test_concurrent_deletes_are_thread_safe
python/tests/test_table.py::test_delete
python/tests/test_table.py::test_delete_expr
python/tests/test_table.py::test_delete_expr_async -q` (4 passed)
- Manual stress reproduction: 100 concurrent deletes on one table
completed at versions 2–101 with zero rows remaining.
Fixes#530
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- cache the immutable read consistency interval on synchronous
connection wrappers
- keep debugger property expansion from dispatching to the background
event loop
- cover direct connections and wrappers reconstructed from native
connections
## Root cause
The debugger expands connection variables by evaluating properties after
suspending all Python threads.
`LanceDBConnection.read_consistency_interval` dispatched a coroutine to
`LanceDBBackgroundEventLoop` and synchronously waited for it, but that
loop thread was also suspended, causing a deadlock.
## Validation
- `uv run --no-sync pytest python/tests/test_db.py -q` (48 passed)
- `ruff format --check python/python/lancedb/db.py
python/python/tests/test_db.py`
- `ruff check .`
- `git diff --check`
Fixes#3773
<!-- lance-gatekeeper-fix:v1 agent=e2e612236d722d926f64245d3f682bbc
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Description
`Table::optimize()` compacts through
`lance::dataset::optimize::compact_files`
(`rust/lancedb/src/table/optimize.rs:155`). Until
lance-format/lance#7965 that rewrite corrupted blob columns holding null
or empty values, which is what #3744 reports:
- **storage 2.0** (legacy v1 `lance-encoding:blob` descriptors): every
payload following a null or empty row in the same fragment was rewritten
as `{position: 0, size: 0}`, so it read back as `b""` and the new
fragment no longer referenced the bytes — silent payload loss,
unrecoverable once the pre-optimize versions are pruned.
- **storage 2.2** (blob v2): a valid empty value was rewritten as null,
destroying the null-vs-empty distinction.
Both manifestations share one root cause: `is_inline_null_blob`
classified any inline blob with `position == 0 && size == 0` as null,
which is also exactly what a *valid empty value* looks like. Such rows
were dropped from `blob_read_addrs`, misaligning every payload that
followed.
The behaviour is already correct on `main`: the vendored lance crate
first carried the fix at `v10.0.0-beta.3` (#3710) and is now
`v10.1.0-beta.1` (#3757). What was missing is coverage — nothing in this
repo exercised a blob column containing a null or empty value through
`optimize()`, which is why this shipped unnoticed. This PR adds that
guard.
## Tests
Two tests in `rust/lancedb/tests/blob_integration.rs`, reusing the
file's existing 64 KiB dedicated-blob helpers and a delete-triggered
fragment rewrite. After `id IN (1, 4)` is deleted the surviving rows are
`2` (null), `3` (valid empty), `5` and `6` (payloads) — payloads sit
immediately after the null/empty, which is where the misalignment
landed.
- `optimize_preserves_v1_blob_payloads_with_null_and_empty` — storage
2.0; asserts the **payload bytes** are unchanged across
`OptimizeAction::All` (what the Python/Node `optimize()` bindings
invoke). Payloads are read through `lance::Dataset::take_blobs`, since
`Table::fetch_blobs` rejects legacy v1 columns. The before/after
descriptors are reported on failure but deliberately *not* asserted:
compaction repacks the blob file, so they shift legitimately (id 5
`(131072, 65536)` → `(0, 65536)`, id 6 `(196608, 65536)` → `(65536,
65536)`). Note that a post-compaction `position: 0` is both the
legitimate first-payload offset and the bug's signature, so asserting
descriptors would be actively misleading.
- `optimize_preserves_blob_v2_null_and_empty_distinction` — storage >=
2.2; asserts a null stays null and a valid empty value stays non-null
empty.
Both assert the pre-optimize state first, so a setup change that stops
producing the null/empty/payload mix fails loudly instead of passing
vacuously.
Both also assert the returned `CompactionMetrics` show a fragment was
actually rewritten. These tests depend on `delete("id IN (1, 4)")`
pushing the fragment past lance's `materialize_deletions_threshold` (0.1
by default; 2 of 6 rows here). That coupling is invisible and unasserted
otherwise: against a forced no-op (`materialize_deletions_threshold:
1.5`) the metrics come back all zeroes and *every payload assertion
still passes*. Since the whole point of these tests is to survive
dependency changes, they check that the rewrite happened rather than
trusting the planner to keep selecting the fragment.
Guard verified against a pre-fix lance: with the published
`lancedb==0.36.0` wheel (vendors lance 9.0.0), `Table.optimize()` on the
same data rewrites the descriptors of the two rows following the
null/empty from `(131072, 65536)` and `(196608, 65536)` to `(0, 0)`, and
the payloads read back empty. Against the pinned `v10.1.0-beta.1`, all
39 tests in the file pass, adding roughly 10–20 ms to the file's
runtime.
## Not addressed here
- **No released artifact has the fix yet.** PyPI `lancedb` 0.36.0
(2026-07-29) vendors lance 9.0.0; npm `@lancedb/lancedb` 0.37.1-beta.0
predates the bump. No 9.x lance tag carries the fix: `v10.0.0-beta.3` is
the first tag containing it, every `v9.1.0-beta.1`…`beta.8` is behind
it, and `v9.0.0` / `v9.0.1-rc.1` sit on a diverged branch without it. A
stable lancedb release needs a stable lance >= 10.
- **The version skew #3744 flagged is still live.**
`python/pyproject.toml` pins `pylance==9.0.0rc1` for the `tests` extra
against a vendored `10.1.0-beta.1`, so Python CI still cannot observe
this class of divergence.
- **Only the single-fragment rewrite shape is covered.** Both tests
rewrite one fragment by materializing deletions. lance's own
`test_compact_blob_v1/v2_preserves_null_empty_and_payload_order` cover
the multi-fragment merge shape (3 fragments → 1) at unit level, so this
PR is complementary rather than redundant — it covers the binding-level
path through `Table::optimize` — but it would not catch a regression
that only appears when *merging* fragments.
`multi_fragment_dedicated_blob_table` in the same file makes that a
cheap follow-up.
Closes#3744🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Table::add_columns now takes no arguments and returns AddColumnsBuilder,
so calls become .add_columns().transform(t).execute().
read_columns was the second positional argument but reaches only one of
the five transform variants. In lance's add_columns_to_fragments only
BatchUDF receives the caller's value: SqlExpressions replaces it with
the columns its expressions reference, Stream and Reader pass None, and
AllNulls reads nothing. So it was mandatory on every call -- all
eighteen call sites here passed None -- and silently discarded four
times out of five. As a builder method it is optional, and setting it
where lance would discard it is now an error, which does reject a call
that previously succeeded while ignoring the argument.
Matches the builders add, update, and merge_insert already use.
## Summary
`LanceMergeInsertBuilder.when_not_matched_by_source_delete()` didn't
clear a previously-set condition when called again with no argument (or
a different condition type). Per the docstring, `condition=None` means
"delete all unmatched rows," but if the builder had already been
configured with a string/Expr condition, a later no-arg call left the
stale condition in place instead of widening the delete to
unconditional.
Fixes#3767
## Change
Each call now unconditionally sets both
`_when_not_matched_by_source_condition` and
`_when_not_matched_by_source_condition_expr` (one to the new value, the
other to `None`), so the latest call always wins — consistent with every
other setter on this builder (e.g.
`when_matched_update_all(where=...)`).
## Test plan
- [x] New regression test
`test_merge_insert_by_source_delete_reconfigure` in
`python/python/tests/test_table.py`
- [x] `uv run --extra tests pytest
python/tests/test_table.py::test_merge_insert_by_source_delete_reconfigure
python/tests/test_table.py::test_merge_insert_by_source_delete_expr
python/tests/test_table.py::test_merge_insert_by_source_delete_expr_async
-vv` — 3 passed
- [x] `uv run --extra dev ruff format` / `ruff check` — clean
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
`LanceHybridQueryBuilder._create_query_builders()` checked
`self._minimum_nprobes` for truthiness instead of `is not None` — the
very next line correctly checks `is not None` for
`self._maximum_nprobes`. Since `0` is falsy in Python,
`.minimum_nprobes(0)` on a hybrid query silently dropped the value
instead of forwarding it to the vector sub-query, where it would raise
the same `ValueError` a plain vector query raises for the same input
(`minimum_nprobes must be greater than 0`, validated in
`rust/lancedb/src/query.rs` and covered for the plain-query path by
`test_invalid_nprobes_sync`).
Fixes#3766
## Change
One-line fix: `if self._minimum_nprobes:` → `if self._minimum_nprobes is
not None:`, matching the existing `maximum_nprobes` check right below
it.
## Test plan
- [x] New regression test
`test_hybrid_query_minimum_nprobes_zero_raises` in
`python/python/tests/test_hybrid_query.py`
- [x] `uv run --extra tests pytest python/tests/test_hybrid_query.py
-vv` — 13 passed
- [x] `uv run --extra dev ruff format` / `ruff check` — clean
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
- Implements Cloud `fetch_blob_files`: returns real seekable `BlobFile`
handles over HTTP Range instead of `NotSupported`.
- Completes the second Cloud blob read verb after #3684 (`fetch_blobs` =
eager whole bytes; this = lazy / partial / sequential reads).
- Same public handle API as local (`read_range`, `read_up_to`, `seek`,
`tell`, `close`), so one code path works for local and Cloud.
Large blobs (video, audio, PDFs) should not require downloading the
whole object to inspect a header or stream a slice. After search,
callers open a handle and read only what they need:
```python
hits = table.search(vec).select(["id", "video"]).limit(5).to_arrow()
with table.fetch_blob_files("video", hits)[0] as f:
header = f.read_range(0, 256)
f.seek(keyframe_offset)
chunk = f.read_up_to(1 << 20)
```
### Behavior
- Handle creation probes size with `bytes=0-0` (bounded concurrency,
input order preserved).
- `204` → null (`None`); `416` with `bytes */0` → valid empty blob;
other `416` → error.
- `read_range` validates `Content-Range` and body length; OOB ranges
fail with `invalid_input` before the request (aligned with Lance).
- `read_up_to` reuses one open-ended Range response across sequential
reads; `seek` drops it.
- Servers older than 0.5.0 get a clear `NotSupported` (does not suggest
`fetch_blobs`, which they also lack).
## Testing
- `cargo test --features remote -p lancedb remote_blob`
- `cargo test --features remote -p lancedb test_blob`
- `cargo clippy --features remote --tests --examples` (no new warnings
from this change)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Bumps the pinned Rust toolchain from 1.95.0 to the latest stable
(1.97.0).
Rust 1.97's clippy adds `useless_borrows_in_formatting`, which flags a
redundant `&` in `format!`/`debug!` arguments in a few places. This PR
removes those to keep `cargo clippy` clean.
No behavior change; the MSRV (`rust-version = "1.91.0"`) is unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updates the Lance Rust workspace dependencies and Java lance-core
version to v10.1.0-beta.1.
Includes a compatibility fix for the Lance file writer API by using the
explicit V2_1 writer creation path for permutation shuffle spill files.
Triggered by
https://github.com/lance-format/lance/releases/tag/v10.1.0-beta.1
Adds job operations to the connection surface, building on the Job
handle from #3742: job(id), list_jobs, get_job, cancel_job, and
job_history, plus a non-blocking Job.status(). Implemented on the
Database trait (defaulting to NotSupported), the remote backend
(/v1/jobs), and the Python and Node bindings; job_history returns Arrow
batches.
errors() and progress() are not included.
Tested with mocked endpoints in all three languages.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
IndexBuilder::execute now returns a Job with wait and cancel methods.
Local tables build the index synchronously and return an already-done
job. Remote tables read the job id the server returns from create_index
and track it through the /v1/jobs API: wait polls describe until the job
reaches a terminal state and cancel posts a cancellation. Servers that
return no job id yield a done job, so behavior against older servers is
unchanged. The job id is not exposed on the handle.
The Python and TypeScript bindings keep their current signatures and
discard the handle; exposing Job there is left to follow-ups.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`RemoteDBConnection.open_table` accepts `storage_options` and never uses
it:
```python
def open_table(
self,
name: str,
*,
namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None,
index_cache_size: Optional[int] = None,
...
) -> Table:
...
if index_cache_size is not None:
logging.info("index_cache_size is ignored in LanceDb Cloud ...")
table = LOOP.run(self._conn.open_table(name, namespace_path=namespace_path))
```
The value is never passed down and never mentioned. `index_cache_size`
is ignored on Cloud in the
same way, but it says so.
I checked this at runtime on 0.34.0, not just by reading it: swapping
the inner connection for a
recorder, `open_table("t", storage_options={...})` hands the layer below
`['namespace_path']` and
nothing else, no log record is emitted, and the same probe shows
`index_cache_size` producing its
message as expected.
This adds the matching log line, so the two ignored parameters behave
the same way. `ruff check` and
`ruff format --check` are clean on the file.
A note on severity. This is not a security hole and nothing is exposed.
Someone passing credentials
there gets silence instead of an error, and finds out later.
One thing I am unsure about, and it changes the fix. I have assumed
per-table storage options are
meaningless on Cloud, which is what the `index_cache_size` line next to
it implies about managed
storage. If they are supposed to work, then the right change is to pass
them through to
`self._conn.open_table` instead and this patch is the wrong one. Happy
to redo it that way.
I did not check whether `create_table` or the async connection have the
same gap.
`docs/src/python/python.md` is the whole Python API reference, but it is
maintained by hand and had drifted from the public API. Anything not
listed there simply doesn't get rendered, so a number of public,
documented, tested APIs were invisible to users — most notably branch
management, where `diff` and `merge` live.
I audited every public symbol reachable from `lancedb` and its
subpackages against the `:::` directives on the page. This adds the
missing ones:
- **Branching** — `Branches`, `AsyncBranches` (`list` / `create` /
`checkout` / `delete` / `diff` / `merge`)
- **Tables** — `TableStatistics` (returned by `Table.stats()`; the
fragment-level stats classes were already listed)
- **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`,
`BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`,
`Occur`
- **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`,
`AsyncTakeQuery`
- **Indices** — `Fm` (the FM-index for substring search), `IndexConfig`
- **Blobs** — `blob`, `BlobType`, `BlobFile`
- **Namespaces** — `connect_namespace`, `connect_namespace_async`, and
both namespace connection classes
- **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`,
`OAuthFlowType`
- **Rerankers** — the `Reranker` base class plus `JinaReranker`,
`RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`,
`VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed)
- **Embeddings** — `get_registry`, `register`, and the 14 embedding
functions that were missing (3 of 17 were listed)
- **PyTorch** — `StreamingDataset` and the permutation API it is built
on
- **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`,
`pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two
exception types
It also repairs cross-references in docstrings that no longer resolve:
links into guide pages that have since moved to lancedb.com
(`querying-an-ann-index`, `experimental-full-text-search`),
`lance.dataset` references with no inventory behind them, and the
relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`.
Deliberately left out: concrete implementation classes reached through
their abstract base (`LanceTable`, `LanceDBConnection`,
`RemoteDBConnection`), query base classes already covered by
`inherited_members: true`, and internal plumbing such as
`FullTextSearchQuery` and `ColumnOrdering`.
## Testing
The docs job only runs on pushes to `main`, so I built the site locally
and compared against a build of `upstream/main`: every added entry
resolves, and no symbol that was rendered before stopped being rendered
when the four packages moved to automodule. `mkdocs build --strict`
exits 0 on this branch, against 61 warnings on `main`.
## Also in this PR
`lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and
`lancedb.rerankers` are now rendered by a single mkdocstrings directive
each, driven by the module's `__all__`, rather than a hand-maintained
list. These four are where most of the drift was, and `__all__` is
harder to forget than a docs page. `lancedb.embeddings` had no
`__all__`; without one mkdocstrings renders no members at all for a
re-export package, so one is added. AGENTS.md gains a section on how the
page is wired up and how to build the docs locally.
Rendering all that code for the first time surfaced ~100 more build
warnings, which would have made #3707 (turning on `mkdocs build
--strict`) harder to land, so the warning backlog is cleared here too.
97 of the 158 warnings were one systematic false positive — griffe
cannot see the generated `__init__` of a pydantic dataclass, so every
documented parameter looks unknown — switched off via
`warn_unknown_params`. The remaining 61 came from 15 docstrings with
real bugs: prose trailing a `Parameters` section (we were rendering
parameters called `The`, `you` and `To`), types dropped because numpydoc
needs spaces around the colon, `num_partitions, default sqrt(num_rows)`
parsing as a list of names and inventing a `default` parameter, and one
parameter indented five spaces. `mkdocs build --strict` now exits 0.
---
#3747 (the coverage test that keeps this from happening again) is
stacked on this branch, so review it after this one.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
In the current LanceDB usage implementation, there is no way to check
whether a table or namespace already exists. This PR introduces the
namespace_exists and table_exists methods to determine the existence of
tables and namespaces.
useage like this:
```
# check table exists
db.table_exists(table_id=['xxx'])
# check namespace exists
db.namespace_exists(namespace_id=['xxx'])
```
fixes: #3419
---------
Signed-off-by: farmer <farmerchillax@outlook.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Remote half of the blob read path. #3578 did local Python. This makes
`RemoteTable` hit the server.
- `fetch_blobs(column, row_ids or hits)` → bytes over `POST
/v1/table/{id}/fetch_blobs/`
- `blob_columns()` from the cached schema (describe already has the
metadata, no extra route)
- search then `fetch_blobs` works. row identity rides inside the blob
descriptor so you do not need a public `_rowid`
- `fetch_blob_files` still `NotSupported` on remote. use `fetch_blobs`
for full bytes for now. Range is a follow up
Accepts Binary / LargeBinary / BinaryView on the way back. Empty
`row_ids` short-circuits. Version + branch go in the request body same
as other read calls.
### Example
```python
db = lancedb.connect(uri="db://my-project", api_key=...)
table = db.open_table("clips")
hits = table.search(query_vec).select(["id", "video"]).limit(10).to_arrow()
# hits is just id + video. row ids are stashed on the descriptor
blobs = table.fetch_blobs("video", hits) # null-aligned, same length as hits
```
Or pass ids yourself:
```python
blobs = table.fetch_blobs("video", [10, 20, 30])
```
### Testing
- `cargo test -p lancedb --features remote --lib`
- `cargo test -p lancedb --features remote --test blob_integration`
- `pytest python/tests/test_remote_db.py -k remote_blob`
- live e2e against a local 0.5.0 remote server (search → fetch, nulls,
nested path, old server gate)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`table_names()` lists any `*.lance` directory, but `open_table()` maps
every `DatasetNotFound` to `TableNotFound`, so a corrupt or
partially-written table looks identical to one that never existed
(#3127). This takes the issue's Option 2: on `DatasetNotFound`, check
the parent listing for the table's `.lance` entry — the same predicate
`table_names()` uses — and return a new `TableCorrupted` error when the
directory is present. The check runs only on the error path, and any
failure in the recheck falls back to the previous `TableNotFound`
behavior.
Tests cover the reporter's empty-dir repro, a deleted-manifest case,
true absence (still `TableNotFound`), and an end-to-end list-then-open
assertion; the three new corrupt-case tests fail without the src change.
`cargo test -p lancedb --lib` 732 passed, clippy/fmt clean, `cargo check
--workspace --all-targets` clean (both language bindings end in wildcard
error arms).
Two notes for review: `Error` isn't `#[non_exhaustive]`, so the new
variant is technically semver-breaking for exhaustive matchers (pre-1.0,
and the alternative — changing `TableNotFound`'s shape — breaks more);
and on the Python side corrupt tables now surface as `RuntimeError`
rather than `ValueError`, which is the intended distinction but worth a
maintainer's eye. `open_from_namespace` was left unchanged since
namespace listings come from a server-side registry, not directory
globbing.
Closes#3127
## Summary
Fixes#2339. `merge_insert()` on the remote client could mask the real
cause of a mid-stream input error, reporting only:
> stream error sent by user: unexpected internal error
## Root cause
There were two divergent streaming-write code paths in the remote
client:
- `add()` uses `RemoteInsertExec`, which streams the request body
through a `tokio::sync::oneshot` error side-channel and drains it before
reporting the HTTP result. If the input stream errors mid-body, the
original error is recovered.
- `merge_insert()` used a legacy path (`send_streaming` ->
`reader_as_body`) that piped arrow `Some(Err(e))` straight into the
HTTP2 request body. Hyper swallows body-stream errors under HTTP2 (see
hyperium/hyper#2547), so the original error was lost and only the
generic transport error surfaced.
## Fix
Consolidate both write paths onto the side-channel mechanism instead of
patching the legacy path:
- Generalize `RemoteInsertExec` into `RemoteWriteExec`, carrying a
`WriteOp` enum (`Insert { overwrite }` | `MergeInsert { query, timeout
}`) that selects the endpoint, query params, request-timeout header, and
response parsing. The executor returns a `WriteResult` enum (`Add` |
`Merge`) with typed accessors, and `with_new_children` still resets the
result so the rescannable retry loop is unaffected.
- Route `merge_insert()` through `RemoteWriteExec`. The public API only
accepts a `RecordBatchReader` (not rescannable), so the reader is
buffered into a `Vec<RecordBatch>` before the retry loop to preserve the
previous retry-on-retryable-status behaviour. This mirrors what the old
`send_streaming(with_retry=true)` path already did.
- Remove the now-unused `send_streaming` / `reader_as_body` /
`buffer_reader` / `make_reader` helpers. Multipart stays insert-only
(the server has no multipart merge_insert endpoint), so that hot path is
behaviorally unchanged.
## Testing
- Added `test_merge_insert_input_error_surfaces_original`, which drives
an erroring input through the single-request `merge_insert` path and
asserts the original error (`boom`) is surfaced rather than the masked
HTTP error. Confirmed it fails without the side-channel drain (it then
reports a masked `500 ... request or response body error`).
- Full suite green: `cargo test -p lancedb --lib --features remote` ->
694 passed, 0 failed. Includes the existing
`test_merge_insert_retries_on_409`, confirming retry behaviour is
preserved.
## Summary
- keep the existing synchronous `connect()` path unchanged
- make `LanceDBConnection.__repr__` and `LanceTable.__repr__`
side-effect-free
- add a regression test that verifies sync reprs do not call the Python
background loop
## Root cause
The freeze is caused by debugger rendering, not by `connect()` itself:
1. debugpy stops at a breakpoint and suspends all Python threads.
2. The debugger renders the new `db_connection` local by calling
`repr()`.
3. `LanceDBConnection.__repr__` reads `read_consistency_interval`.
4. That property calls `LOOP.run(...).result()`.
5. The `LanceDBBackgroundEventLoop` thread is suspended by the debugger,
so `repr()` waits for a thread that cannot run.
This explains why the symptom appears immediately after `connect()`: it
is the first point where a connection object exists in locals and is
automatically rendered. `LanceTable.__repr__` had the same problem
because it also read the connection's consistency interval.
This follows the same principle as #3411: `__repr__` must not trigger
async work or I/O that a debugger assumes is lightweight.
## Evidence
I reproduced the behavior with the real LanceDB classes and debugpy
1.8.21 using a DAP client:
- latest `main` (`ff6ff099`): the debugger reported `allThreadsStopped:
true`, and evaluating `repr(db_connection)` timed out
- this branch (`5755a5ba`): the same evaluation returned
`LanceDBConnection(uri='/tmp/lancedb-debug-repro')` immediately
- setting `PYDEVD_UNBLOCK_THREADS_TIMEOUT=0` also allowed the original
repr path to complete, independently confirming that it was waiting on a
suspended thread
The regression test creates a connection and table, replaces `LOOP.run`
with a function that fails, and verifies that both reprs still work.
## Validation
- `maturin develop --manifest-path python/Cargo.toml`
- `python -m pytest
python/python/tests/test_db.py::test_sync_repr_does_not_use_background_loop
python/python/tests/test_table.py::test_consistency -q` (`4 passed`)
- `ruff check .`
- `ruff format --check python/python/lancedb/db.py
python/python/lancedb/table.py python/python/tests/test_db.py
python/python/tests/test_table.py`
- `git diff --check`
Refs #3611.
`test_read_consistency_interval` asserted that a table opened with a
100ms `read_consistency_interval` still read stale data immediately
after a concurrent write. The cache timestamp is set when the table is
opened and reads within the interval do not refresh it, so that
assertion only held if the intervening open/count/commit/count sequence
finished within 100ms of real wall-clock time. On a loaded CI runner it
did not: the TTL expired, `count_rows` refreshed synchronously, and the
test failed with `left: 1, right: 0`. This broke the Rust workflow on
`main` at 0bc08160 (a Python-only commit).
This pins the `background_cache` mock clock once `table2` has seeded its
cache, and advances it explicitly in place of `tokio::time::sleep`, so
the test controls when the interval elapses. Same approach as #3547.
With the clock pinned there is no real sleep left to be imprecise, so
the `cfg(not(target_os = "windows"))` guard is dropped and the test now
runs on Windows too.
Verified by inserting a stall before the write: 120ms reproduces the
original failure deterministically, and with this change the test still
passes with a 500ms stall.
Fixes#3712
## What
Expose custom FTS stop-word lists in the Python and TypeScript public
APIs, including their standalone tokenize helpers and remote index
creation.
This PR supports concrete string lists only. It does not add file or
LanceDB-table stop-word sources.
## Why
Rust already exposes Lance's custom stop-word list option. The Python
and TypeScript APIs did not pass it through, and local index details did
not retain the full tokenizer parameters needed by index-backed
tokenization after reopening a table.
## How
- Add `custom_stop_words` / `customStopWords` to the Python and
TypeScript FTS and tokenize options.
- Preserve `None` / `undefined`, empty lists, and list contents without
normalization.
- Load the persisted FTS segment parameters when returning local index
details.
- Serialize the concrete list in remote create-index requests.
- Keep Python and TypeScript tests thin; behavior, persistence, query
tokenization, and remote JSON coverage live primarily in Rust.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
- Python extension rebuild with `uv` and `maturin`
- Targeted Python tests: 4 passed
- Python `ruff format --check` and `ruff check`
- TypeScript build, typecheck, Biome lint, generated docs, and targeted
tests
---------
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
## Summary
- add a keyword-only `transform_parallelism` option to
`StreamingDataset`
- preserve CPU auto-detection by default and fall back to one worker
when unavailable
- apply the configured limit to both the transform executor and
concurrency semaphore
- document and test explicit, default, fallback, and invalid values
## Testing
- `uv run --extra tests --with torch pytest
python/tests/test_elastic_dataloader.py -q` (`136 passed`)
- `uvx ruff check python/lancedb/streaming.py
python/tests/test_elastic_dataloader.py`
- `uvx ruff format --check python/lancedb/streaming.py
python/tests/test_elastic_dataloader.py`
- `git diff --check origin/main...HEAD`
Closes#3695
Co-authored-by: buduoqiu <yaodong-shen@users.noreply.github.com>
Standard GitHub-hosted runners are free on public repos, so all Actions
spend here is on the `*-8x-*` / `4x` larger runners. Measured over 30
days at current (post-Jan-2026) larger-runner rates, that is ~$1,400/mo,
and `npm-publish` is ~70% of it.
## Changes
**Fat LTO was forcing builds onto large runners.** `[profile.release]`
in `.cargo/config.toml` sets `lto = "fat"` with `codegen-units = 1`,
which is single-threaded and the peak-memory step. The macOS
`npm-publish` build was 111 of its 113 minutes in one `napi build` step,
making it the critical path of the whole publish pipeline. The ThinLTO
override already applied to Windows now covers macOS too, and both
Windows builds move from `windows-2025-8x-x64` to the free standard
`windows-2025`.
**The npm-publish cargo cache never existed.** There are zero caches
with its key prefix. The key was static, so `actions/cache` (which only
writes on a miss) could never refresh it, and a multi-GB release
`target/` per target could never fit the repo's 10 GB budget anyway. Now
caches only the crate registry, keyed on `Cargo.lock`. The docker builds
also mounted `.cargo/registry/*` while the cache saved `.cargo-cache`,
so containers re-downloaded the registry every run.
**Cache eviction thrash.** Repo cache usage is 10.4 GB against GitHub's
10 GB cap, so every PR run evicted main's warm entries. `rust.yml` and
`nodejs.yml` now restore everywhere but only save from `main`.
**npm-publish moves to nightly + tags** instead of every push to main
(~90/month). The cross-compiled targets do need watching, so
`report-failure` now fires on scheduled runs, and dedupes onto an
existing open issue rather than filing one per night.
**rust.yml aarch64-pc-windows-msvc** cross-compiled its tests and then
skipped them, paying full codegen and link cost for a compile check.
`windows-11-arm` is now GA and free on public repos, so it builds and
tests natively. Its test step also passes `--target` — without it cargo
used `target/ci/` rather than `target/<triple>/ci/` and rebuilt the
entire dependency graph a second time.
**pypi-publish.yml had no concurrency group**, so force-pushes left a
~74 minute Windows job running.
## What is cost vs. wall-clock
| Change | Cost | Wall-clock |
|---|---|---|
| Windows npm-publish → free runners | **−$570/mo** | slower per job
(8→4 cores) |
| npm-publish nightly | **−$125/mo** | — |
| pypi-publish concurrency | small | — |
| macOS ThinLTO | $0 (already free) | **−~50 min** per release |
| rust aarch64 Windows native | $0 (already free) | **−~25 min** |
| rust `--target` on test step | $0 | large, avoids a second full build
|
| rust-cache `save-if` | small | faster via real cache hits |
## Risks
- The two Windows builds now have 4 cores instead of 8 and ~14 GB of
free disk. If they fail, it is most likely disk rather than memory;
fallback is `windows-2025-4x-x64`, which still halves that line.
- `windows-11-arm` has a thinner toolset (choco/vcpkg/protoc under
emulation) and this enables a test step that has never run, so it may
surface real aarch64 failures. That is the point, but it is the change
most likely to need iteration.
- ThinLTO applies to published macOS and Windows binaries, typically
within a few percent of fat LTO. Linux release builds are untouched.
## Follow-ups
- `python.yml` `pydantic1x` (37 min) and `Doctest` (33 min) each rebuild
the extension from source via `pip install -e .` with no Rust cache;
they should consume the wheel the `linux` job already builds. Worth
~$235/mo and ~70 min of compute per run. Separate PR.
- The three `ubuntu-2404-8x-x64` npm-publish builds (~$420/mo at the old
cadence) are the remaining large-runner spend;
`aarch64-unknown-linux-gnu` could run natively on free
`ubuntu-24.04-arm`. Worth doing after this lands so the ThinLTO change
can be validated first.
- The wheel composite actions declare `python-minor-version` as required
but never use it, and every caller omits it (actionlint warns).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`list_versions()` against a remote table on a server that uses
lance-namespace was failing. The server was returning
`timestamp_millis`, while db-catalog deployments were returning
`timestamp`, and the client was only accepting `timestamp`. So, updated
the client to accept both. (assuming we're migrating over time;
eventually we can turn off the `timestamp` code path I suppose.)
## Summary
Lance can now plan multiple byte ranges for the same blob in one
`read_blob_ranges` operation, but LanceDB users currently cannot expose
a complete set of logical ranges to that planner.
This complements `BlobFile`: file-like consumers such as PyAV can
continue to discover ranges dynamically, while callers that already know
the ranges for a batch can submit them together.
## Motivating example
A training table may store a large video blob together with a small
application-level clip index:
```text
video: blob
clips: [{offset, length}, ...]
```
The caller can select the videos and clips for a batch, obtain their row
IDs from the query, and read all of the selected windows together:
```python
rows = (
table.search()
.select(["clips"])
.with_row_id(True)
.limit(64)
.to_arrow()
.to_pylist()
)
requests = []
for row in rows:
clip = sample_clip(row["clips"])
requests.append(
(row["_rowid"], clip["offset"], clip["length"])
)
chunks = table.fetch_blob_ranges("video", requests)
```
Here, `_rowid` comes from the LanceDB query, while `offset` and `length`
come from the application's clip index and are relative to that row's
video blob. The caller describes only the logical reads; Lance still
handles validation, source grouping, coalescing, scheduling, and byte
backpressure.
Lance v10.0.0-beta.5 returns one logical result per blob selector or
range request and explicitly distinguishes null blobs from valid empty
values. LanceDB consumes that aligned result contract directly and only
adds a cardinality check for unresolved row IDs.
This PR exposes batched blob-range reads on local Rust and Python
tables. Results preserve request identity, duplicates, null slots, and
valid empty ranges while allowing Lance to execute the physical reads
out of order. Scheduler buffer sizing remains an internal Lance concern,
so the LanceDB API does not expose `io_buffer_size`.
Cloud tables continue to report this operation as unsupported until
there is a corresponding remote API.
SELECT COUNT(*) FROM t WHERE <predicate> — and any query that plans an
empty-projection scan — panics the executing query task:
InvalidArgumentError("must either specify a row count or at least one
column")
Root cause
MetadataEraserExec wraps every LanceDB table scan to strip schema-level
metadata, rebuilding each batch in execute():
RecordBatch::try_new(schema.clone(), batch.columns().to_vec()).unwrap()
RecordBatch::try_new infers the row count from the columns. COUNT(*)
with a filter is planned with an empty projection, so the scan emits
zero-column batches — there are no columns to infer a length from,
try_new returns Err, and the .unwrap() panics.
(This is specific to the empty-projection case: COUNT(*) with no filter
is answered from statistics and never scans, and COUNT(<col>) projects a
column — both already work.)
Updates the Rust workspace Lance dependencies and Java lance-core
dependency to v10.0.0-beta.5. No compatibility fixes were required;
full-workspace Clippy passes with warnings denied. Lance tag:
https://github.com/lance-format/lance/releases/tag/v10.0.0-beta.5
## What
MemWAL LSM **read** support. When a table has an LSM write spec
(`set_lsm_write_spec`), `merge_insert` upserts live in the MemWAL
active/frozen memtables and flushed SSTables until an external
compaction merges them into the base table, so a normal scan returns
**stale** data. This routes reads through Lance's `LsmScanner` so
queries also surface that in-flight data, deduplicated by primary key
(newest generation wins).
## How
- Adds a **`use_lsm: Option<bool>`** query flag, symmetric with the
`merge_insert` flag:
- **unset** — auto-route through the LSM scanner when the table carries
a write spec
- **`use_lsm(true)`** — force the LSM path; error if there is no spec
- **`use_lsm(false)`** — read the base table only (the escape hatch)
- Plain scan, single-column full-text search, and single-vector ANN all
run through one `LsmScanner` (assembled from on-disk shard manifests
plus the cached writer's in-memory memtables), so a `where` predicate is
honored as a **prefilter** uniformly — including for vector search.
- **Compaction-aware snapshots:** an SSTable generation is dropped only
once it is both compacted into the base table and covered by the arm's
base-index catch-up (`index_catchup`); plain scans use the compaction
watermark alone.
- Query shapes the scanner cannot honor hard-error with guidance to set
`use_lsm(false)`: hybrid, multi/binary vectors, `with_row_id`,
reranking, `order_by`, dynamic/Substrait projection or filters,
`distance_range`, `use_index(false)`, postfilter, take-by-row-id/offset,
reads from a time-traveled version, and an unmaintained or ambiguous
(multiple) FTS/vector index. Namespace-pushdown queries fall back to
local execution when a spec is present; WAL-only writers are handled.
- Exposed across the Rust core and the Python (`use_lsm`) and TypeScript
(`useLsm`) bindings, including `TakeQuery`.
Rebased from Lance `7.2.0-beta.3` to `10.0.0-beta.3`.
Python was versioned and tagged separately from the Rust, Java, and
Node.js SDKs, and had drifted three minor versions ahead (0.36 vs 0.33).
Users had no way to tell which Python version corresponded to which Rust
or Node release, and the gap had no meaning behind it.
This unifies the two tracks so there is one version and one tag for all
four SDKs.
## Version
The shared version is set to `0.37.0-beta.0`. Python continues its own
sequence (highest published: 0.36 → 0.37) while Rust, Java, and Node.js
jump 0.33 → 0.37 to meet it. Picking Python's next minor means Python
users see no discontinuity at all, and only the other SDKs skip forward.
Note that `main` trails the `release/v0.32` branch on both lines (main
is at 0.32.0-beta.3 / 0.35.0-beta.3; the release branch carries
0.33.0-beta.0 / 0.36.0-beta.0), so 0.37 is chosen to clear the highest
tag on either branch. Every index stays monotonic:
| index | publishes | last published | next |
|---|---|---|---|
| PyPI | stable only | 0.34.0 | 0.37.0 |
| Fury | previews | 0.36.0b0 | 0.37.0-beta.1 |
| npm | both | 0.33.0-beta.0 | 0.37.0-beta.1 |
| crates.io | stable only | 0.31.0 | 0.37.0 |
| Maven | both | 0.33.0-beta.0 | 0.37.0-beta.1 |
A one-time jump for three SDKs, versus explaining the offset
indefinitely.
## Mechanism
* `python/.bumpversion.toml` is removed. `python/Cargo.toml` — the
source of the Python package version, since `pyproject.toml` declares
`dynamic = ["version"]` — becomes a tracked file of the root config. Its
`cargo update -p lancedb-python` pre-commit hook is dropped as
redundant: `ci/update_lockfiles.sh` already refreshes every workspace
member version in `Cargo.lock`.
* `pypi-publish.yml` triggers on `v*` instead of `python-v*`, so one tag
releases all four packages. `ci/bump_version.sh` and
`make-release-commit.yml` lose their now-dead tag-prefix and
per-language plumbing, including the `python` / `other` dispatch inputs.
* The two byte-identical GH release jobs in `npm-publish.yml` and
`pypi-publish.yml` are replaced by a single `gh-release.yml`. One
release per tag, named `LanceDB vX.Y.Z`, instead of separate "Python
LanceDB" and "Node/Rust LanceDB" releases for the same commit.
The trade-off: there is no longer a way to ship a Python-only patch
without also releasing crates.io, Maven, and npm. That is the cost of
making drift structurally impossible.
## Beta releases marked "Latest" (#3666)
Both GH release jobs used:
```yaml
prerelease: ${{ contains('beta', github.ref) }}
```
The arguments are reversed. `contains(search, item)` asks whether
*`search`* contains *`item`*, so this evaluated "does the literal string
`'beta'` contain `refs/tags/python-v0.35.0-beta.2`?" — always `false`.
Every beta was published as a full release, and GitHub awards "Latest"
to the newest non-prerelease.
The new workflow derives the flag from the parsed version rather than
the raw ref, and sets `make_latest` explicitly:
```yaml
prerelease: ${{ steps.extract_version.outputs.prerelease }}
make_latest: ${{ steps.extract_version.outputs.prerelease == 'false' }}
```
npm was never affected (`--tag preview` uses correct bash), and PyPI
already excludes pre-releases from resolution.
This only fixes releases published from here on. Already-published betas
need a one-time backfill:
```shell
gh api --paginate /repos/lancedb/lancedb/releases \
--jq '.[] | select(.prerelease == false) | select(.tag_name | test("beta")) | .id' \
| xargs -I{} gh api -X PATCH /repos/lancedb/lancedb/releases/{} -F prerelease=true
```
## Verification
Ran `ci/bump_version.sh` end-to-end against this branch with the release
tooling installed:
* `preview` → tags `v0.37.0-beta.1` (previous tag `v0.33.0-beta.0`
detected, `pre_n` bump)
* `stable` → tags `v0.37.0`
* Both paths update `.bumpversion.toml`, `rust/lancedb/Cargo.toml`,
`nodejs/Cargo.toml`, `python/Cargo.toml`, `nodejs/package.json`, the 7
`nodejs/npm/*/package.json` files, both Java poms, and
`docs/src/java/java.md` together
* `check_breaking_changes.py` resolves the last stable as `v0.31.0`, so
the minor-version gate passes
All five touched workflows parse as valid YAML and the pre-commit hooks
pass.
## Notes for review
* This targets `main` only, so it takes effect at the next
release-branch cut. The in-flight `release/v0.32` branch still carries
`v0.33.0-beta.0` / `python-v0.36.0-beta.0`; if we want the imminent
stable to be 0.37.0, this needs to be applied there too.
* Historical `python-v*` tags are left alone. The changelog builder
scans `^v`, which does not match them, so the first unified release's
notes will compute `fromTag` from the Rust/Node line only — a one-time
gap in the Python-side changelog.
* Pre-existing and not addressed here: `ci/update_lockfiles.sh --amend`
amends the commit that `bump-my-version` has already tagged, so the
lockfile update lands outside the tag on stable releases.
Fixes#3666
## What changed
- add `block_size` to Python FTS configuration and the deprecated
local/remote helpers
- add `blockSize` to the TypeScript FTS options and propagate it through
the NAPI binding
- serialize the value as `block_size` for remote index creation
- document the existing Rust builder API and generate the TypeScript API
reference
- add local, remote, metadata, search, and invalid-value regression
coverage
## Why
Lance supports configuring the number of documents per compressed FTS
posting block, but LanceDB's Python and TypeScript APIs did not expose
the setting. This made the experimental FTS V3 layout unavailable
through those clients and allowed the value to be dropped before index
creation.
## How it works
The default remains `128`. Supported values are `128` and `256`;
selecting `256` uses the experimental FTS V3 format. Invalid values are
rejected by the Lance builder and surfaced as Python or JavaScript
errors.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo +1.94.0 clippy --quiet --features remote --tests --examples --
-D warnings`
- targeted Rust local and remote index tests
- Rust doctests: 34 passed
- Python Ruff checks, doctest, and targeted local/remote tests: 5 passed
- TypeScript build, Biome lint, generated docs, and targeted Jest tests:
9 passed
- `git diff --check`
## Limitations
The Java client remains unchanged because its external remote REST model
does not currently expose `block_size`.
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
## What
`AnswerdotaiRerankers(return_score="all").rerank_hybrid(...)` (and
`ColbertReranker`, which subclasses it without overriding
`rerank_hybrid`) raises:
```
pyarrow.lib.ArrowInvalid: Invalid sort key column: No match for FieldRef.Name(_relevance_score) in _rowid: int64 ...
```
## Why
```python
combined_results = self.merge_results(vector_results, fts_results)
combined_results = self._rerank(combined_results, query)
if self.score == "relevance":
combined_results = self._keep_relevance_score(combined_results)
elif self.score == "all":
combined_results = self._merge_and_keep_scores(vector_results, fts_results)
```
When `score == "all"`, `combined_results` is unconditionally overwritten
by `_merge_and_keep_scores(vector_results, fts_results)` **after**
`_rerank()` already computed and appended `_relevance_score` —
discarding it. The following `sort_by("_relevance_score", ...)` then has
nothing to sort on.
Every sibling reranker that supports `return_score="all"`
(`cross_encoder`, `openai`, `cohere`, `jinaai`, `voyageai`, `watsonx`)
instead calls `_merge_and_keep_scores()` **before** `_rerank()`. This
file is the one place the ordering got inverted when `"all"` support was
added (#2509) — a copy/paste inconsistency across the six files that PR
touched. Fix mirrors the pattern already used (and tested) by the other
five rerankers.
Also drops the now-stale `"Only 'relevance' is supported for now"`
docstring line on both classes, left over from before `"all"` support
existed.
## Testing
Added `test_answerdotai_reranker_return_all`, mirroring the existing
`test_cross_encoder_reranker_return_all`. Verified locally with the real
built Rust extension: red (reproduces the exact `ArrowInvalid` above) →
green, using the actual `rerank_hybrid`/`_rerank`/`base.py` code path
with the model call mocked out — my local environment's
`rerankers==0.10.0` fails to load the real ColBERT model against the
available `transformers` version (`AttributeError: 'ColBERTModel' object
has no attribute 'all_tied_weights_keys'`), which I confirmed also
breaks the **pre-existing**, unmodified
`test_colbert_reranker`/`test_answerdotai_reranker` baseline tests
identically — an unrelated local dependency-version issue, not a
regression from this change. `ruff check`/`ruff format` clean; full
`test_rerankers.py` run: 9 passed / 8 skipped / 3 failed (the 3 failures
are exactly those two pre-existing tests plus my new one, all failing at
model-loading time for the same unrelated reason before reaching the
changed code).
---
Disclosure: this PR was drafted with AI assistance (Claude); I reviewed,
tested, and take responsibility for the change.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
This updates the Java API reference to close the documentation gaps that
can be fixed from the current Java source and generated namespace API.
The patch adds an empty table example, shows how to wrap returned Arrow
IPC query bytes in a reusable `ArrowFileReader` helper, and documents
the Java index operations that are currently exposed by the namespace
client: vector indexes, scalar indexes, full text search indexes, and
listing indexes.
## Issue Links
Fixes https://github.com/lancedb/docs/issues/157
Fixes https://github.com/lancedb/docs/issues/160
Partially addresses https://github.com/lancedb/docs/issues/159 by
documenting the index parameters currently exposed by Java. The
requested `num_partitions` example is still blocked because
`CreateTableIndexRequest` does not expose IVF training parameters yet.
Not included: https://github.com/lancedb/docs/issues/158. The current
Java docs and source remain remote namespace oriented, so local DB
connection documentation should wait until the Java local DB API is
available and can be verified.
## Validation
- Built the Java core module with OpenJDK 17:
`./mvnw -pl lancedb-core -am -DskipTests compile`
- Checked the Markdown diff:
`git diff --check -- docs/src/java/java.md`
The Java build succeeds. It still reports pre-existing checkstyle
warnings in the namespace client builder, but the Maven build is green.
## Summary
- reconstruct foreign Arrow Map schemas from their single sanitized
entries field
- reject malformed Map types with anything other than one child
- preserve the complete Map schema and `keysSorted` value through
empty-table creation and IPC round trips across Arrow 15–18
## Testing
- `./node_modules/.bin/jest --runInBand __test__/arrow.test.ts
__test__/sanitize.test.ts`
- `pnpm lint`
- `pnpm build`
- `pnpm run docs`
Fixes#2337
## What
- Replace legacy model names in `WatsonxEmbeddings` with the current
supported set:
- `ibm/granite-embedding-278m-multilingual` (new default, 768-dim)
- `ibm/slate-125m-english-rtrvr-v2` (768-dim)
- `ibm/slate-30m-english-rtrvr-v2` (384-dim)
- `intfloat/multilingual-e5-large` (1024-dim)
- `sentence-transformers/all-minilm-l6-v2` (384-dim)
- Add `space_id` field — mutually exclusive with `project_id`, mirrors
the
existing pattern in `WatsonxReranker`
- `project_id` / `space_id` resolution now falls back to
`WATSONX_PROJECT_ID` /
`WATSONX_SPACE_ID` env vars; exactly one must be supplied
## Why
The previously hardcoded models (`ibm/slate-125m-english-rtrvr`,
`sentence-transformers/all-minilm-l12-v2`) are legacy and no longer
listed as
supported by the watsonx.ai platform. `space_id` scoping was already
supported
by `WatsonxReranker` but was missing from the embeddings counterpart.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Updates the Rust workspace Lance dependencies and Java lance-core from
v9.1.0-beta.5 to v9.1.0-beta.7, including the generated Cargo lockfile.
No LanceDB compatibility changes were required for this release. See the
[Lance v9.1.0-beta.7
release](https://github.com/lance-format/lance/releases/tag/v9.1.0-beta.7).
This PR adds some support for `diff` / `merge` in the remote client as
for local tables we stay `NotSupported` until
https://github.com/lance-format/lance/issues/7263.
This wires the two review-and-land calls against the remote REST API:
- `POST /v1/table/{id}/branches/diff`
- `POST /v1/table/{id}/branches/merge`
Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python
returns the wire JSON, same shape as the REST response.
Merge here means promoting a branch's added columns onto `main`.
### Behavior
- Remote only. Local raises `NotSupported`.
- A rejected merge is not an exception. HTTP 409 still returns `Ok` / a
dict with `status="rejected"` and blockers in `diff.mergeBlockers`.
- Unknown blocker / status codes parse as `Unknown` so a newer server
does not break older clients.
- `MergePreview` tolerates missing fields for the same reason.
- Merge requests are not retried. 409 is final and carries the body you
need.
### Example
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.merge("exp", dry_run=True)
result = table.branches.merge("exp", dry_run=False)
if result["status"] == "merged":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "rejected":
print(result["diff"]["mergeBlockers"])
```
### Testing
cargo test -p lancedb --features remote diff_branch
cargo test -p lancedb --features remote merge_branch
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Problem
On the remote (LanceDB Cloud) write path, each write partition is
uploaded as a **single** `/insert?upload_id=...` request that stays open
until the whole partition has been streamed and the server has written
it to object storage. For large bulk ingests a partition can be many GB,
so a single request can run longer than the client read timeout (default
300s), surfacing as:
```
lancedb.remote.errors.HttpError: operation timed out
```
The server already supports staging **multiple** parts under one
`upload_id` (each `/insert` writes a separate transaction that
`complete` merges atomically), but the client never used that — it sent
one part per partition.
## Change
Split each partition into multiple parts of at most
`max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded
as its own `/insert?upload_id=...&upload_part_id=...` request. This
bounds how long any single request stays open, independent of total data
size or write parallelism.
Key properties:
- **Still streamed, not buffered.** Each part's body is driven through a
bounded channel while the request is in flight (`futures::join!` of a
producer + the send), so peak memory stays at a couple of batches per
partition regardless of the part size. Backpressure from a
slow/throttled server still propagates upstream.
- **Correct part accounting.** An empty partition still sends exactly
one (schema-only) part so `complete` has a transaction to commit; a size
cut landing exactly on the end of input does not emit a trailing empty
part.
- **Multipart only.** The single-request (non-multipart) path is
unchanged.
## Config
New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable
via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable.
**Default 1 GiB** (`Some(0)` disables splitting → one request per
partition). Python users pick up the default/env automatically through
the remote client.
## Tests
- `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each
batch in its own part → N requests, each carrying the shared `upload_id`
and a distinct `upload_part_id`.
- `test_multipart_single_part_when_under_budget`: a large budget keeps
the partition in a single request.
- Verified end-to-end against a live remote table: a forced-chunked
multipart add (many parts) assembles to the correct row count.
Related to ENT-1883.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Fixes#1653.
`infer_vector_column_name` in `util.py` could silently return `None`
when `query is None` and `query_type` is not `"fts"` or `"hybrid"`. This
`None` then propagated into downstream code, causing a cryptic
`TypeError: expected bytes, NoneType found` rather than a clear error
message.
## Changes
- **Removes the no-op `try/except Exception as e: raise e`** around
`inf_vector_column_query` (it was catching and immediately re-raising
without adding any value)
- - **Adds a `None` guard** after the inference block: if
`vector_column_name` is still `None` at this point, raise a clear
`ValueError` pointing the user to pass `vector_column_name` explicitly
## Before / After
**Before:** cryptic `TypeError: expected bytes, NoneType found` deep in
schema lookup code
**After:**
```
ValueError: No vector column found in the schema. Please specify the vector column name explicitly via the `vector_column_name` parameter.
```
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Some additions to our lancedb skill to enable agents to use the jobs
methods that we recently added. Eval tests (below, with and without
these additions to the skill) suggest that they're helping, mostly to
find the right method calls. These are a little unusual because they
require REST server connection, they're not yet implemented in the SDKs.
```
┌─────────────────────┬───────────┬────────────┬─────────────┬──────────┬───────────┬──────────┬───────────┐
│ eval │ grade w/o │ grade with │ improvement │ time w/o │ time with │ cost w/o │ cost with │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ 8-list-running-jobs │ 2.5/3 │ 3/3 │ +0.5 │ 123s │ 29s │ $0.58 │ $0.18 │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ 9-describe-job │ 1/5 │ 5/5 │ +4.0 │ 159s │ 52s │ $0.62 │ $0.25 │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ 10-cancel-job │ 3/3 │ 3/3 │ +0.0 │ 99s │ 35s │ $0.55 │ $0.21 │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ TOTAL │ 6.5/11 │ 11/11 │ +4.5 │ 381s │ 116s │ $1.75 │ $0.65 │
└─────────────────────┴───────────┴────────────┴─────────────┴──────────┴───────────┴──────────┴───────────┘
```
Failure reasons are because the agent didn't know the right method to
call, spent all its turns guessing REST calls, tried to inspect lancedb
code, but didn't find the answer in here.
## Problem
`table.add(dataset)` with a `pyarrow.dataset.Dataset` OOMs the client
during bulk ingestion of wide rows (e.g. embedding columns), even
against a remote table where the upload itself is streaming.
The cause is in `to_scannable`: a `Dataset` is scanned with pyarrow's
default scanner settings (`batch_size=131072` rows,
`batch_readahead=16`, `fragment_readahead=4`). pyarrow's internal
threads prefetch that read-ahead window independently of LanceDB's
backpressure, so for wide rows a large fraction of the dataset is held
in memory. On the remote path this is then multiplied across the
multipart write partitions (one in-flight batch per partition, up to
CPU-core count).
Reproduced on a 10 GB / 1.55M-row dataset with two 768-dim float32
embeddings: peak client RSS ~11.7 GB for the scan alone (6.8 GB after
consuming a *single* batch), ~15.4 GB for the full remote `add()`.
## Fix
`to_scannable` now sizes the scanner from an estimate of bytes-per-row
derived from the schema:
- **Narrow datasets keep pyarrow's defaults** (empty scanner kwargs) —
no throughput regression. The bound only engages above ~410 bytes/row.
- **Wide rows** get a smaller `batch_size` (~16 MiB/batch) and reduced
read-ahead (`batch_readahead=2`, `fragment_readahead=1`) so peak
in-flight memory stays near a ~1 GiB budget. Read-ahead (not just batch
size) has to drop, because pyarrow pins whole row-group buffers.
On the 10 GB dataset this drops peak client RSS to ~1.4 GB, and it stays
flat as the dataset grows. The `Dataset`/`LanceDataset` scannables
remain rescannable (retry-safe).
## Also: expose `write_parallelism` on `add()`
`AddDataBuilder::write_parallelism` already existed in Rust but was not
exposed in Python. This PR forwards it through the async, sync, and
remote `add()` methods, so users can cap the number of parallel write
partitions (each buffers data in flight) to trade throughput for memory
on large uploads.
## Tests
- `test_scannable.py`: bytes-per-row estimation; narrow → defaults; wide
→ bounded; `Dataset` reader streams bounded batches and stays
rescannable.
- `test_table.py`: `write_parallelism` on sync and async `add()`, and
that `write_parallelism=0` is rejected.
Fixes ENT-1883
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Moves the skill from .agents/skills/lancedb to
plugins/lancedb/skills/lancedb, as recommended by codex and claude.
Install path now is:
### Codex/ChatGPT app
Codex: Plugins -> "Create" arrow -> Add plugin marketplace
search for lancedb plugin and install
### Codex CLI
```
codex plugin marketplace add lancedb/lancedb
codex plugin install lancedb@lancedb # name@marketplace
```
### Claude app
Settings -> Plugins -> Add -> Add marketplace
search for lancedb plugin and install
### Claude CLI
```
claude plugin marketplace add lancedb/lancedb
claude plugin install lancedb@lancedb
```
Here's how it looks on ChatGPT/Codex app:
(the main icon has light and dark modes; the smaller one on the skill
doesn't so I made it gray 🤷 )
<img width="764" height="560" alt="Screenshot 2026-07-16 at 2 49 24 PM"
src="https://github.com/user-attachments/assets/b82cda16-3392-4740-ac47-b2f187cb2655"
/>
Hi, and thank you for LanceDB.
Small CI supply-chain hardening. In `make-release-commit.yml`, the
release job checks out with `LANCEDB_RELEASE_TOKEN` (a push-capable PAT)
and its final step pushes the version tag using a third-party action
pinned to a **mutable branch**:
```yaml
- name: Push new version tag
uses: ad-m/github-push-action@master
with:
github_token: ${{ secrets.LANCEDB_RELEASE_TOKEN }}
```
`@master` can move after review; whatever it points at then runs with
that release token in scope. This PR pins it to the commit behind the
current release (`v1.3.0` → `881a6320…`), keeping the version visible as
a comment. Behavior today is unchanged.
For transparency: I used AI assistance to spot and draft this; I
verified the workflow and resolved the SHA myself.
Tracks #3324. On x86_64 CPUs without AVX2 (Sandy Bridge / Ivy Bridge /
Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD), `import
lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled
function. Per [westonpace's
review](https://github.com/lancedb/lancedb/issues/3324#issuecomment-4328944354),
the default `lancedb` wheel stays fast; pre-Haswell users get a
separately-published `lancedb-compat` wheel.
## Summary
- Adds a `lancedb-compat` matrix entry to `pypi-publish.yml` that builds
with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (Nehalem-class baseline).
Same Python API (`import lancedb` works) — files install to the same
namespace, so the two wheels conflict at install time and users pick
one. Same pattern as `psycopg2` / `psycopg2-binary` and `tensorflow` /
`tensorflow-cpu`.
- Generalizes `build_linux_wheel` and `upload_wheel` composites with
optional `package-name` and `rustflags` inputs (defaults preserve the
existing 4 `lancedb` matrix entries verbatim).
- Documents the choice in `python/README.md`: `pip install
lancedb-compat` for pre-Haswell hosts.
The default `.cargo/config.toml` baseline is unchanged.
## Sequencing
1. ~~lance-format/lance#6630 merges → runtime SIMD dispatch lands in
lance.~~ **Done — merged.**
2. lancedb's lance dep is bumped to a release that includes it (separate
PR / normal cadence).
3. This PR's `lancedb-compat` wheel build path starts producing a wheel
that runs on pre-Haswell hardware. **Maintainer setup**: register
`lancedb-compat` on PyPI and configure trusted publishing.
## Verified end-to-end on Sandy Bridge Xeon E5-2609
Verification was done locally against a fork-pinned lance dep that
includes the runtime dispatch implementation, using the same
`RUSTFLAGS="-C target-cpu=x86-64-v2"` flags this PR uses in CI:
```
$ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
$ pip install ./target/wheels/lancedb-*.whl
$ python verify.py
PASS: import + simd dispatch + table create + vector search all work.
```
Pre-fix on the same CPU (default `pip install lancedb`): `Illegal
instruction (core dumped)`. Full reproducer (deps + clone + build +
verification):
https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df.
Fork-internal verification PR with the dep bump and full logs:
[`tobocop2/lancedb#2`](https://github.com/tobocop2/lancedb/pull/2).
## Benchmarks — no regressions on modern CPUs from the lance-side change
These are the numbers I ran for the lance PR, confirming the runtime
dispatch doesn't slow down the default (`target-cpu=haswell`) wheel that
existing users install. Criterion, one machine, one session, base → PR,
no `RUSTFLAGS` override. Full methodology, null experiments, and logs:
[lance-format/lance#6630 benchmark
comment](https://github.com/lance-format/lance/pull/6630#issuecomment-4933063394)
and the [logs
gist](https://gist.github.com/tobocop2/3c6d0f449cbd736aa2501f89a7fe56a2).
| benchmark | EPYC 7B13 (`avx2`, `fma`, no `avx512f`) | Xeon Cascade
Lake (`avx512f`) |
|---|---|---|
| `Cosine(f32, scalar)` *(control)* | +0.04% | +0.09% |
| `Cosine(f64, scalar)` | −0.34% | −1.94% |
| `Cosine(u8, SIMD)` | +2.30% | +3.63% |
| `Dot(f16, SIMD)` | −0.58% | +0.61% |
| `Dot(f32, SIMD)` | +0.34% | **−6.08%** |
| `Dot(f32, arrow_arity)` | +0.02% | −0.00% |
| `L2(f32, scalar)` | −0.10% | −0.02% |
| `L2(f32, simd)` (dim 1024) | +2.63% | −0.53% |
| **`L2(simd,f32x8)` (dim 8)** | **−45.9%** | **−25.1%** |
| `L2(u8, SIMD)` | +0.42% | −3.11% |
| `NormL2(f32, SIMD)` | −1.02% | −4.17% |
| `NormL2(f64, SIMD)` | +3.51% | −0.58% |
Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector
width — improves 25–46%.
---
To be transparent: this isn't my domain of expertise and the lance-side
implementation is AI-generated. I verified it works end-to-end on the
failing hardware. Happy to roll in feedback.
Routes local sync child-namespace operations through the Rust-backed
connection instead of the Python namespace-client fallback.
Also keeps lazy namespace-client construction for table-to-Lance
conversion and preserves public namespace error mappings.
Validated locally with ruff format/check and targeted namespace pytest.
BREAKING CHANGE: splits generated by the permutation data loader will
not be the same, due to a change in hash function.
Updates the Lance dependencies and Java lance-core to
[v9.0.0-rc.1](https://github.com/lance-format/lance/releases/tag/v9.0.0-rc.1).
Includes the required DataFusion 54 and Lance file-reader compatibility
updates.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds client-side support for analyze_plan distributed metrics modes
across Rust, Python, and TypeScript clients. Defaults to aggregate for
backward compatibility and sends the remote distributed_metrics
parameter only when a non-default mode is requested.
Fixes#3174
Also fixes#3645
Empty record batches now append correctly typed empty embedding arrays
without invoking embedding providers. This avoids OpenAI requests with
an invalid empty input while preserving source-column validation and
the non-empty execution paths.
As a small cleanup, the single- and multi-embedding code paths now share
a single upfront lookup of their source columns ("input_columns")
instead
of each path looking them up independently. Also moves `lance-testing`
from regular dependencies to dev-dependencies where it belongs.
Tests run:
- `cargo fmt --all -- --check`
- `cargo test --quiet -p lancedb --lib
empty_batch_skips_embedding_functions`
- `cargo test --quiet -p lancedb --lib
empty_batch_still_validates_source_column`
- `cargo test --quiet -p lancedb --lib
test_create_empty_table_with_embeddings`
- `cargo check --quiet -p lancedb --features remote --tests --examples`
- `cargo clippy --quiet -p lancedb --features remote --tests --examples`
- `cargo test --quiet -p lancedb --lib`
- `cargo test --quiet --features remote --tests`
## Summary
Fix `on_bad_vectors="fill"` so it replaces only invalid or missing
vector values instead of replacing the entire vector row.
Fixes#3026.
## Reasoning
The old Python sanitizer detected whether a vector row was bad at row
granularity. For `fill`, it then used that row-level flag to replace the
whole vector with `[fill_value] * dim`. That meant an input like `[1.0,
NaN, 3.0]` became `[0.0, 0.0, 0.0]`, even though the documented and more
useful behavior is to preserve valid values and fill only the bad
element.
I checked whether this should be a Rust-side fix so TypeScript users
would benefit too. Today, Rust core exposes `NaNVectorBehavior::{Error,
Keep}` for rejecting or keeping NaN vectors, while the Python
`on_bad_vectors` API (`error`, `drop`, `fill`, `null`) is implemented in
the Python ingestion sanitizer before data reaches Rust. TypeScript does
not expose the Python `on_bad_vectors="fill"` behavior today. Moving
this exact behavior to Rust would be a broader cross-language API
change, so this PR keeps the fix scoped to the currently affected Python
API.
## What changed
- Added a small helper that fills bad vector rows by preserving valid
elements, replacing NaN elements with `fill_value`, truncating vectors
longer than the expected dimension, and padding short vectors with
`fill_value`.
- Kept the existing fast path unchanged: the helper only runs after bad
vectors are detected and `on_bad_vectors="fill"` is selected.
- Updated sanitizer and table tests to assert element-wise NaN
replacement and short-vector padding for both `create_table` and `add`.
## Validation
- `uv run ruff format .`
- `uv run ruff check .`
- `cd python && uv run --no-sync pytest
python/tests/test_util.py::test_handle_bad_vectors_jagged
python/tests/test_util.py::test_handle_bad_vectors_nan
python/tests/test_table.py::test_create_with_nans
python/tests/test_table.py::test_add_with_nans -vv`
Targeted pytest result: `10 passed`.
## Why this fix is Python-side (and not Rust)
The problematic behavior lives in Python’s `on_bad_vectors` sanitizer,
before data is handed off to Rust. Rust currently only exposes
`NaNVectorBehavior::{Error, Keep}` for add operations, while Python has
the richer `on_bad_vectors={"error","drop","fill","null"}` API.
TypeScript does not currently expose the Python-style fill behavior, so
moving this exact fix into Rust would require designing a broader
cross-language bad-vector handling API.
This PR keeps the change scoped to the existing affected surface:
Python’s `on_bad_vectors="fill"` path. This way, Python users
immediately benefit.
## What the new agent skill covers
We want to help users _easily_ write LanceDB pipelines to bring their
data in from other places, no matter whether they use LanceDB OSS or
Enterprise.
The `lancedb` set of skills contains guidance for agents on the
following:
- Distinguishes local and remote table capabilities.
- Promotes bounded reads using `select()` and `limit()`.
- Prevents accidental full-table materialization.
- Documents correct Python sync/async scan APIs.
- Recommends validated Python schemas and batched ingestion.
- Provides indexing, query-tuning, diagnostics, and maintenance
guidance.
- Documents the Enterprise table-name cache issue: avoid immediately
reusing a dropped or overwritten table name; write to a fresh name and
rename after propagation.
- Adds Python and TypeScript API, pattern, and performance references.
- Adds a heuristic scanner for potentially unsafe Python and TypeScript
materialization patterns.
This change only adds agent documentation and tooling: no LanceDB
runtime code, Rust code, SDK APIs, dependencies, or CI configuration are
modified.
## Context
The LanceDB agent skill was accidentally pushed directly to `main` in
`8ea78e3fbcb26718112ab4ddec55a91804b869d3`, bypassing the normal review
workflow. That commit was reverted on `main` by `c12a6dce` so the
protected branch is back to its prior content.
## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage
`Dataset::index_statistics()` loads index files and does meaningful CPU
work to serialize low-level info. Most fields
`NativeTable::index_stats()` needs are available from manifest metadata
via `Dataset::describe_indices()`, which is much cheaper.
`NativeTable::index_stats()` now:
- Calls `describe_indices()` filtered by name; returns `Ok(None)` if no
match.
- Parses `distance_type` from `description.details()` JSON (the
`VectorIndexDetails` proto stored in the manifest by recent Lance
versions).
- Falls back to `index_statistics()` only for vector indices where
`details()` returns no `distance_type` — this handles older Lance
datasets that didn't write `VectorIndexDetails`.
- `Unknown` index types (e.g. Lance's internal `FragReuseIndex`) are
explicitly filtered out of `list_indices` rather than erroring.
## Test plan
- [x] `test_create_scalar_index` — asserts `index_type`,
`distance_type`, and `num_unindexed_rows > 0` after adding rows
post-index
- [x] `test_create_fm_index`, `test_create_bitmap_index`,
`test_create_label_list_index` — added `index_stats` assertions
- [x] IvfPq, IvfHnswPq, IvfHnswSq, IvfHnswFlat tests assert
`distance_type == Some(L2)`
- [x] `test_list_indices_skip_frag_reuse` — FragReuseIndex is filtered
by the Unknown guard in `list_indices`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- serialize sync phrase queries consistently for execution and query
plans
- restore the documented no-argument hybrid `phrase_query()` behavior
- keep reranker input as the original user text without mutating the
builder
Fixes#3653.
## Testing
- `python/.venv/bin/python -m pytest <8 focused test nodes> -q` (`8
passed`)
- `python/.venv/bin/python -m ruff format --check
python/python/lancedb/query.py python/python/tests/test_fts.py
python/python/tests/test_hybrid_query.py`
- `python/.venv/bin/python -m ruff check .`
- `git diff --check origin/main...HEAD`
The complete hybrid module and the real native FTS phrase test were not
completed
in the current PyO3 runtime environment: both stalled in the native
`lancedb.connect()` fixture and were interrupted without an assertion
failure.
The CODEOWNERS file added in #3312 automatically requests reviewers on
every PR — the `*` default owner routes all changes to two reviewers.
This is mostly noise for contributors, and we prefer a single requested
reviewer per PR.
Remove the file.
Reverts #3312.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Carrying over from #2915, this patch introduces:
* Single-API call batching support for Gemini embeddings (up to 100 at a
time, the API limit)
* A versioned user agent header for Gemini API calls
* Support for [variable embedding dimension
size](https://ai.google.dev/gemini-api/docs/embeddings#control-embedding-size)
(Gemini is MRL trained)
## Summary
- preserve explicit `0.0` distance bounds in synchronous hybrid search
- distinguish omitted `None` endpoints from zero-valued endpoints when
configuring the vector child query
- add a public end-to-end regression test for a zero upper bound
## Testing
- `cd python && uv run --extra tests pytest
python/tests/test_hybrid_query.py -q`
- `uv run --project python ruff format --check
python/python/lancedb/query.py python/python/tests/test_hybrid_query.py`
- `uv run --project python ruff check .`
Fixes#3651
The `build - aarch64-pc-windows-msvc` node build job (and, marginally,
the x86_64 one) had started hitting `rustc-LLVM ERROR: out of memory`
while linking the `lancedb-nodejs` cdylib — most recently surfaced by
#3526, which adds the goosefs backend (and its tonic/prost gRPC subtree)
to the default node binary.
The peak-memory step is the fat-LTO codegen (`lto=fat`,
`codegen-units=1` from `.cargo/config.toml`), which merges the whole
crate graph into a single LLVM module and runs single-threaded. It
therefore neither parallelizes across cores nor fits in the 16 GB of the
standard `windows-latest` runner as the dependency graph grows.
This PR:
- Moves both `*-pc-windows-msvc` node build jobs to
`windows-2025-8x-x64` (more memory + cores).
- Overrides the release profile to ThinLTO for just these jobs, via
`CARGO_PROFILE_RELEASE_LTO=thin` /
`CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16` in `pre_build`. ThinLTO
parallelizes the cross-module optimization across the runner's cores and
keeps peak memory well under the limit. Scoped so Python wheels and Rust
release builds keep fat LTO.
The larger runner alone would clear the OOM but waste the added cores on
the single-threaded fat-LTO tail; ThinLTO is what makes the extra cores
actually reduce wall-clock and gives durable memory headroom for future
dependency growth.
Tradeoff: ThinLTO can leave a small runtime-perf gap vs fat LTO for the
node native binary, but it recovers most of it and is a common release
configuration.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.
## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.
## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.
## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.
Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.
## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps Lance to v9.0.0-beta.19, which includes lance-format/lance#7687
for side-effect-free DirectoryNamespace read paths.
This fixes root-level read-only table opens that previously could
trigger `__manifest` creation through directory namespace construction,
including Hugging Face bucket reads with read-only tokens. A LanceDB
regression test now covers root listing operations without creating
`__manifest`.
Fixes#3633.
### Summary
`flatten_columns` raises `ValueError` when called with `flatten=False`,
even though `False` should mean "do not flatten". This is reachable from
the public API — `Query.to_pandas(flatten=...)` and
`to_batches(flatten=...)` type their `flatten` param as
`Optional[Union[int, bool]]` and pass it straight to `flatten_columns`.
### Cause
`bool` is a subclass of `int`, so `isinstance(False, int)` is `True`.
`flatten=False` skips the `flatten is True` check, falls into the
integer branch, and `False <= 0` evaluates to `True`, raising:
```
ValueError: Please specify a positive integer for flatten or the boolean value `True`
```
### Reproduction
```python
import lancedb
db = lancedb.connect("/tmp/db")
t = db.create_table("t", data=[{"id": 1, "vector": [0.1, 0.2]}])
t.search([0.1, 0.2]).to_pandas(flatten=False) # -> ValueError
```
### Fix
Guard the integer branch with `not isinstance(flatten, bool)` so that
`flatten=False` (and `None`) mean "do not flatten". Behavior is
otherwise unchanged:
- `flatten=True` → flatten all nested levels
- positive `int` → flatten to that depth
- non-positive `int` (e.g. `0`) → still rejected with `ValueError`
Added a regression test in `tests/test_util.py` covering `None`,
`False`, `True`, a positive depth, and `0`.
This PR fixes a serialization error when using Ollama embeddings in
`create_table`.
The use of `@cached_property` for the Ollama client was causing issues
during serialization/pickling, which is required by certain LanceDB
operations (like when using multiprocessing or certain storage
backends). Switching to a standard `@property` ensures the client is
instantiated when needed without being stored in a way that breaks
serialization.
Verified with the following script:
```python
import lancedb
from lancedb.embeddings import get_registry
import pickle
registry = get_registry().get(\"ollama\")
model = registry(name=\"llama3\")
# This would fail before the fix
pickled = pickle.dumps(model)
unpickled = pickle.loads(pickled)
```
Fixes#2629 (or similar serialization issues reported).
---------
Co-authored-by: Unmilan Mukherjee <Missing-Identity@users.noreply.github.com>
## Summary
Closes#3525
This PR wires up two new optional object-store backends at the LanceDB
layer, exposing capabilities that already exist upstream in `lance` /
`lance-io`:
| Backend | Cargo feature | Default in Rust crate | Default in Python
wheel | Default in Node binding |
| --- | --- | --- | --- | --- |
| **Tencent COS** | `cos` | ❌ off | ✅ on | ❌ off |
| **GooseFS** | `goosefs` | ❌ off | ✅ on | ✅ on |
Both backends are additive and do not affect existing users who don't
opt in.
## Motivation
- **Tencent COS** is the dominant object storage in the China region.
Tencent Cloud users currently need an S3-compatible proxy or a private
fork to use LanceDB against COS buckets.
- **GooseFS** is Tencent Cloud's distributed cache acceleration layer
that sits in front of COS/S3, a common pattern for vector search / AI
training where the same hot dataset is read repeatedly.
- This brings COS / GooseFS to feature parity with the existing
first-class backends (`aws`, `gcs`, `azure`, `oss`, `huggingface`).
See the linked issue #3525 for the full discussion.
## Changes
### `rust/lancedb/Cargo.toml`
Add two new optional features that pull through the corresponding
upstream feature flags:
```toml
cos = ["lance/tencent", "lance-io/tencent"]
goosefs = [
"lance/goosefs",
"lance-io/goosefs",
"lance-namespace-impls/dir-goosefs",
]
```
### `python/Cargo.toml`
Enable both `cos` and `goosefs` by default for the Python wheels, so
`pip install lancedb` works against COS / GooseFS out of the box
(consistent with how `aws` / `gcs` / `azure` / `oss` are bundled today):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/cos", "lancedb/goosefs"]
```
### `nodejs/Cargo.toml`
Enable `goosefs` by default for the Node binding (COS kept opt-in to
limit the default native binary size; can be revisited based on demand):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/goosefs"]
```
### `Cargo.lock`
Regenerated to reflect the transitive dependencies brought in by the new
upstream features. No manual edits.
## Example Usage
### Rust
```toml
# Cargo.toml
lancedb = { version = "0.30", features = ["cos", "goosefs"] }
```
```rust
// Tencent COS
let db = lancedb::connect("cos://my-bucket/my-db").execute().await?;
// GooseFS
let db = lancedb::connect("goosefs://my-namespace/my-db").execute().await?;
```
### Python
```python
import lancedb
db = lancedb.connect(
"cos://my-bucket/my-db",
storage_options={
"secret_id": "...",
"secret_key": "...",
"region": "ap-guangzhou",
},
)
```
## Backwards Compatibility
- All new features are **opt-in** at the Rust crate level (`default =
[]` for `lancedb` itself is unchanged).
- The Python wheel gains both backends by default, increasing wheel size
slightly but matching the existing pattern of bundling all major cloud
backends.
- Node binding only adds `goosefs` to defaults; existing users see no
behavior change.
## Testing
- `cargo check --all-features` ✅
- `cargo check -p lancedb --features cos` ✅
- `cargo check -p lancedb --features goosefs` ✅
- End-to-end COS / GooseFS smoke tests require Tencent Cloud credentials
and are intentionally not added to CI in this PR (same approach used for
`s3-test`). Happy to add a gated test feature in a follow-up if
reviewers prefer.
## Checklist
- [x] Added `cos` and `goosefs` features to `rust/lancedb/Cargo.toml`
- [x] Updated `python/Cargo.toml` default features
- [x] Updated `nodejs/Cargo.toml` default features
- [x] Regenerated `Cargo.lock`
- [x] Verified build with `--all-features`
- [ ] Documentation update (can be done in a follow-up PR once API
stabilizes)
## Related
- Issue: #3525
- Upstream support:
[`lance/tencent`](https://github.com/lance-format/lance),
[`lance/goosefs`](https://github.com/lance-format/lance)
## Summary
Adds `Table::get_lsm_write_spec` returning `Option<LsmWriteSpec>` — the
read counterpart to the existing `set_lsm_write_spec` /
`unset_lsm_write_spec`. Returns `None` when the MemWAL LSM write path is
not enabled; otherwise reconstructs the spec (mode, shard column,
`num_buckets`, `maintained_indexes`, `writer_config_defaults`) exactly
as installed.
## Changes
- **Rust core (`NativeTable`)** — reconstructs the spec from
`mem_wal_index_details()`, resolving the shard column from its Lance
field id via the dataset schema. This is a raw metadata read, so it is
unaffected by `describe_indices` system-index filtering.
- **Remote (`RemoteTable`)** — reads the `__lance_mem_wal` system index
through `index/list` with `include_system: true` (so the curated
`list_indices` surface stays unchanged), then parses the index `details`
JSON. It matches the index by name and ignores `index_type`, so no
client `IndexType` variant is needed. It uses the **server-resolved
`column` name** from the details (Lance field ids do not travel to the
remote client).
- **Python + TypeScript bindings** — sync and async, mirroring
`set`/`unset`, with round-trip tests (bucket / identity / unsharded,
plus `None` when unset).
## Tests
- Rust: native round-trip unit test + remote mock-endpoint tests
(present + absent). All green (`cargo test --features remote -p
lancedb`).
- Python/TS: round-trip tests added; binding-runtime execution runs in
CI.
## Dependencies for the remote path
The remote path is complete on the client side but depends on two
out-of-repo pieces to work end-to-end:
1. **lance** — emit the server-resolved shard **`column`** name in the
MemWAL index `details` JSON (field ids can't reach the client). See
lance-format/lance#7667.
2. **server** — honor `include_system` on `index/list` so the
`__lance_mem_wal` entry is returned for this read.
Against an older server (no `include_system`), the remote getter
degrades gracefully to `Ok(None)` rather than erroring.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Updates LanceDB's Lance dependencies to v9.0.0-beta.18.\n\nThis
refreshes the Rust workspace lockfile and Java lance-core version using
the repository update script. Triggering Lance tag:
https://github.com/lancedb/lance/releases/tag/v9.0.0-beta.18
Closes#3245.
> **BREAKING CHANGE:** `with_format("torch")` no longer returns a list
of stacked row tensors. It now returns per-row dicts so PyTorch's
default `DataLoader` collate stacks them into `{col: tensor(B,)}`.
Switch to `with_format("torch_row")` to keep the old shape.
### What changed
`"torch"` now returns a list of per-row dicts (`[{col: tensor}, ...]`)
at every indexed access path. The default `DataLoader` collate stacks
them into a column-keyed batched dict, no custom `collate_fn` needed.
The old shape is preserved under a new `"torch_row"` literal.
`"torch_col"` is unchanged.
The unbatching lives inside the transform (`batch_to_tensor_dict`), not
`__getitems__`, so the shape survives pickling and works under
`DataLoader(num_workers>0, multiprocessing_context="spawn")`.
### Format comparison
| Format | `iter(batch_size=N)` | `__getitems__([0,1,2])` | `DataLoader`
default collate |
|---|---|---|---|
| `"torch"` (new) | `list[{col: tensor}]` length N | `list[{col:
tensor}]` length 3 | `{col: tensor(B,)}` |
| `"torch_row"` (old `"torch"` behavior) | `list[tensor(n_cols,)]`
length N | `list[tensor(n_cols,)]` length 3 | `tensor(B, n_cols)` |
| `"torch_col"` (unchanged) | `tensor(n_cols, N)` | `tensor(n_cols, 3)`
| needs `collate_fn=lambda x: x` |
Output matches HuggingFace `Dataset.set_format("torch")` on container
shape, keys, and values at every access path. The only divergence:
HuggingFace downcasts `float64` to `torch.float32` by default, LanceDB
preserves dtype. Verified by `scripts/verify_torch_format.py`.
### Migration
```python
# Old default — column names lost, shape was tensor(B, n_cols)
DataLoader(Permutation.identity(table).with_format("torch"))
# New default — column names preserved
DataLoader(Permutation.identity(table).with_format("torch")) # {col: tensor(B,)}
# Keep old behavior
DataLoader(Permutation.identity(table).with_format("torch_row")) # tensor(B, n_cols)
```
Fixes#3296
## Problem
The repository has no `CODEOWNERS` file, so there is no enforced review
routing for sensitive areas such as release workflows, auth code, and
FFI boundaries. This means changes to critical paths can be merged
without an explicit codeowner review.
## Solution
Add `.github/CODEOWNERS` covering:
- `/.github/workflows/` — release/publish workflows (supply chain risk)
- `/rust/lancedb/src/remote/` — remote client & auth code
- `/python/src/` and `/nodejs/src/` — FFI language boundaries
The listed owners (`@jackye1995`, `@wjones127`, `@Xuanwo`, `@AyushExel`)
are based on recent merge activity. Feel free to adjust to match the
actual team structure or replace with GitHub team handles if preferred.
## Testing
No code change — only adds a metadata file. GitHub will start routing
review requests automatically once this is merged and branch protection
is configured to require codeowner approval.
Co-authored-by: octo-patch <octo-patch@github.com>
### **Summary**
Closes#3212
Extends the Python `lit()` helper to natively support three additional
types (`date`, `datetime`, and `Decimal`) and implements reflexive
operators for the `Expr` class.
This implementation specifically addresses the blocking feedback
regarding precision loss, CI discovery, and query engine limitations:
* **Logic Refactoring**: Simplified `lit()` by combining `date` and
`datetime` normalization into ISO-8601 strings, ensuring stable SQL
parsing across different engine locales.
* **Precision Preservation**: `decimal.Decimal` objects are now passed
as high-precision strings to the Rust bridge, bypassing intermediate
float conversions and preserving full 128-bit decimal precision for
DataFusion.
* **Averted CI Failures**: Temporarily deferred `bytes` literal support
to a future PR to resolve a known DataFusion `expr_to_sql` limitation
that was crashing the `Doctest` runner.
* **Reflexive Operators**: Added support for "literal-first" arithmetic
and logical operations (e.g., `10 + col('a')` or `True &
col('active')`). Redundant reflexive comparisons (e.g., `__rlt__`) were
pruned as Python's data model handles them automatically.
* **Integration Verification**: Added dedicated integration tests in the
official test directory to ensure the query engine correctly handles the
new types and preserves bit-perfect fidelity.
### **Changes**
####
[python/python/lancedb/expr.py](file:///c:/Users/Laksh/Documents/lancedb/python/python/lancedb/expr.py)
* Updated `lit()` to handle `date`, `datetime`, and `Decimal` natively.
* Implemented reflexive operators (`__radd__`, `__rand__`, `__rmul__`,
etc.) to support literals on the left-hand side.
* Removed the problematic `bytes` doctest example and `lit()` type
support to unblock CI.
####
[python/src/expr.rs](file:///c:/Users/Laksh/Documents/lancedb/python/src/expr.rs)
* Modified the Rust FFI bridge to extract `Decimal` objects as strings.
* Ensured the `expr_lit` handler is ready to receive normalized temporal
strings.
* Consolidated imports and added missing operator documentation.
####
[python/python/lancedb/_lancedb.pyi](file:///c:/Users/Laksh/Documents/lancedb/python/python/lancedb/_lancedb.pyi)
* Updated type stubs for `expr_lit` to include `Any` (allowing for
`Decimal`).
### **Testing**
Added several new advanced test cases in
[python/python/tests/test_expr.py](file:///c:/Users/Laksh/Documents/lancedb/python/python/tests/test_expr.py)
covering:
* **High-precision Decimal preservation**: Verified against 128-bit
boundaries with a "one point off" test case (`1.234567890123456789 <
1.234567890123456790`).
* **Reflexive operator positioning**: Verified successful query
construction with literals on the left.
* **Timezone-aware normalization**: Confirmed stable behavior for
`datetime` objects.
* **Integration Testing**: Confirmed Date32 and Decimal columns return
the correct Python types and values from the engine during `.to_arrow()`
calls.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
# Elastic Streaming Dataloader
## Motivation
Training large models on LanceDB tables today requires loading the
entire dataset
into memory or writing bespoke batching logic. This PR introduces
`StreamingDataset`, a PyTorch `IterableDataset` that streams directly
from a
LanceDB table with two hard guarantees that are difficult to achieve
together:
**elastic determinism** and **resumability**.
## Goals
### Elastic determinism
The dataset partitions the table into a fixed number of *splits*
(controlled by
`num_splits`, `shuffle_seed`, and `epoch`). Samples are yielded by
round-robining
over splits one sample per split per cycle. Because the split structure
is fixed,
the set of samples that makes up each global training step is identical
regardless
of `world_size` or `num_workers`. You can scale your cluster up or down
between
runs and the model sees the same data in the same order — no
re-sharding, no
gradient variance from topology changes.
### Resumability
`state_dict()` / `load_state_dict()` capture how many samples each split
has
consumed. Because all splits are the same size and the round-robin
design keeps
them in lockstep, the state reduces to a single scalar
(`samples_consumed_per_split`)
that is topology-independent. A checkpoint saved with 8 GPUs can resume
correctly
on 4 GPUs or 16 GPUs without any adjustment.
### PyTorch `IterableDataset` / streaming
`StreamingDataset` implements the standard PyTorch `IterableDataset`
interface, so
it drops into any existing `DataLoader` pipeline without modification.
Data is
fetched lazily from Lance in chunks — only the rows needed for the
current batch are
ever in memory.
Compared to the map dataset this takes more work from pytorch and puts
it into the dataset itself (e.g. shuffling, filtering, etc.). We do this
because we cannot achieve things like elastic determinism or
prefiltering otherwise.
### Multi-worker support
DataLoader workers are automatically assigned contiguous sub-blocks of
splits (the
rank's splits are divided evenly across workers). Each worker is
independent:
no shared state, no inter-process coordination. The only constraint is
that
`num_splits` must be divisible by `world_size * num_workers`.
That being said, multi-worker is highly discouraged as it relies on
multiprocessing which is inefficient. Still, we want to support it.
### Filters as prefilters
Filters are applied at *permutation-build time* via
`PermutationBuilder.filter()`,
not re-evaluated on every fetch. The filtered row IDs are stored in the
permutation
table so that subsequent reads see only the matching rows. This allows
us to avoid loading rows that don't match the filter (which is the
default pytorch behavior)
### Prefetching
Two parameters control the I/O pipeline:
- `read_batch_size` (default 64) — number of rows fetched per
`take_offsets` call.
Larger values amortise per-request overhead, which is critical on object
storage
where a single round-trip can cost ~100 ms.
- `prefetch_batches` (default 4) — number of batches prefetched in
parallel per
split via a `ThreadPoolExecutor`. While the model processes the current
batch,
the next several batches are already in flight, hiding storage latency
behind
compute.
If set correctly then you can get good performance even with
num_workers=0 (unless you are bottlenecked on transform).
### Transform parallelism
The underlying `Permutation` API supports a `with_transform()` callback
for
decoding, augmentation, and format conversion. Unfortunately, this is
not parallelized. Pytorch typically parallelizes this with num_workers
which is multiprocessing which is highly inefficient. For simple
transforms we should be able to utilize multithreading and Rust based
UDFs. For complex python UDFs we could have a dedicated multiprocessing
pipeline for just the transform. Or we could just utilize
multithreading. In both cases we would exclude the I/O stage from the
multiprocessing because that ends up being very memory hungry and
inefficient.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
quick-xml < 0.41.0 has two DoS advisories (quadratic attribute-name
check and unbounded namespace allocation in NsReader). All three
versions in our lockfile (0.26.0, 0.38.4, 0.39.4) are below the patched
threshold.
These are pulled in transitively by inferno (dev-only flame-graph dep),
lance-namespace-impls (git dep from lance), and opendal/reqsign (cloud
storage XML parsing). None of these paths expose attacker- controlled
XML; clearing them requires upstream to upgrade to quick-xml >= 0.41.0.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- When an embedding function returns an empty list (e.g. `[]`) for an
input row — as can happen when a model produces no output for a blank
string — `_append_vector_columns` crashed with `ArrowInvalid: Length of
item not correct: expected N but got array of size 0` because PyArrow
cannot fit a zero-length value into a fixed-size list element.
- The fix adds a validation step in `gen()`, inside
`_append_vector_columns`, that replaces any vector whose length does not
match the expected `ndims` (including empty lists and `None`) with
`None` before `pa.array()` is called.
- `None` is a valid null in a PyArrow fixed-size list array, so the bad
entry flows into `_handle_bad_vectors` and is handled according to the
caller-supplied `on_bad_vectors` policy (`error` / `drop` / `fill` /
`null`) instead of causing an unconditional crash.
## Test plan
- [ ] Added `test_embedding_with_empty_output_vectors` in
`python/python/tests/test_embeddings.py` that uses an embedding function
returning `[]` for empty-string inputs, calls `table.add(...,
on_bad_vectors="drop")`, and asserts no crash and that bad rows are
correctly dropped.
- [ ] Existing `test_embedding_with_bad_results` continues to pass (NaN
vectors still handled correctly).
- [ ] Verified manually that `pa.array([[1.,2.,3.,4.], []],
type=pa.list_(pa.float32(), 4))` raises `ArrowInvalid` without the fix,
and succeeds with `None` in place of `[]`.
Fixes#1672
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## What
`MRRReranker.rerank_multivector` averages each document's reciprocal
ranks over the wrong denominator. It divides by the number of rankings
the document *happens to appear in*, instead of the total number of
rankings being fused.
```python
# python/python/lancedb/rerankers/mrr.py
for result_id, reciprocal_ranks in mrr_score_map.items():
mean_rr = np.mean(reciprocal_ranks) # divides by len(present systems)
```
`mrr_score_map[doc]` only accumulates a reciprocal rank for the systems
in which the document was returned, so `np.mean` never accounts for the
systems that missed it.
## Why it's wrong
Mean Reciprocal Rank fusion treats a system that didn't return a
document as a reciprocal rank of `0` and averages across **all**
systems. That's the exact mechanism by which it rewards cross-system
consensus. Dividing by the appearance count removes that, so a document
liked by a single ranking can beat one ranked highly by every ranking.
Concretely, fusing 3 vector rankings:
| Doc | Ranks | Current score | Correct score |
|-----|-------|---------------|---------------|
| A | #1 in 1 system only | `mean([1.0]) = 1.000` | `1.0 / 3 = 0.333` |
| B | #1, #1, #2 across all 3 | `mean([1, 1, .5]) = 0.833` | `2.5 / 3 =
0.833` |
The current code ranks **A above B** - a document two of three rankings
ignored outranks one all three ranked at or near the top.
This also makes `rerank_multivector` inconsistent with `rerank_hybrid`
in the same file, which already treats a missing system as `0`
(`vector_rr = 0.0` / `fts_rr = 0.0`), and with the class docstring
("average of reciprocal ranks across different search results").
## Fix
Divide the summed reciprocal ranks by the total number of rankings:
```python
num_systems = len(vector_results)
...
mean_rr = float(np.sum(reciprocal_ranks)) / num_systems
```
## Tests
Adds `test_mrr_multivector_rewards_consensus`, which asserts the exact
MRR scores and that the consensus document ranks first. It fails on
`main` and passes with this change. Existing reranker tests are
unaffected.
BREAKING CHANGE: When passing multiple where clauses to a query, they
now stack instead of replacing the previous filter.
Previously, calling `where`/`only_if` more than once on a query silently
replaced the previous filter, so only the last filter was applied. This
was
surprising and could return rows that an earlier filter should have
excluded.
This implements the alternative suggested in
https://github.com/lancedb/lancedb/pull/3514#issuecomment-4664901580:
instead of
rejecting a second filter, repeated filters are combined with a logical
AND
(`(previous) AND (new)`).
The combination happens in the Rust core (`QueryBase::only_if` and
`only_if_expr`), so it applies to all SDKs at once (Rust, Python async,
and
TypeScript). The Python sync query builder keeps its own filter state,
so it
combines filters in the binding layer as well.
SQL string and expression filters are combined within their own
representation.
When the two representations are mixed, the expression is lowered to SQL
(via
`expr_to_sql_string`) and the filters are combined as SQL strings, so
chaining
`where` works regardless of which form each filter takes.
Fixes#2649
## Tests
- Rust: `cargo test --features remote -p lancedb --lib query`
- Python: `uv run --extra tests pytest python/tests/test_query.py`
- TypeScript: `pnpm test __test__/query.test.ts`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lancedb's public API forces downstream crates to construct foreign types
— `RecordBatch`/arrays/builders for `Table::add(...)` (arrow), and
`datafusion_expr::Expr` for `only_if_expr`/`expr_projection`/merge
filters. The required version must exactly match lancedb's internal
arrow/datafusion line, but nothing on the API surface makes that
visible. Drift surfaces only as confusing trait/type errors:
```text
error[E0277]: the trait bound `RecordBatch: Scannable` is not satisfied
= note: there are multiple different versions of crate `arrow_array` in the dependency graph
```
This re-exports the crates lancedb already pins, so consumers can rely
on a single, guaranteed-matching line via a discoverable import path
instead of declaring their own (potentially mismatched) direct
dependency.
- `lancedb::arrow::{arrow, arrow_array, arrow_buffer, arrow_cast,
arrow_data, arrow_ipc, arrow_ord, arrow_schema, arrow_select}` —
previously only `arrow_schema` was re-exported. `arrow-buffer` is
promoted from a transitive to a direct dependency.
- `lancedb::datafusion` — `Expr` is a first-class part of the query and
merge APIs (`only_if_expr`, `expr_projection`,
`QueryFilter::Datafusion`, `when_matched_update_all_expr`), and
`ExecutionPlan` is returned from `create_plan`.
This follows DataFusion's own precedent of re-exporting `arrow`. The
coupling already exists via the trait/impl bounds — this surfaces it
rather than hiding it behind an `E0277`.
Closes#3575🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary:
- Route built-in async namespace-backed connections through the Rust
namespace connector.
- Delegate async namespace/table management methods to the inner
AsyncConnection while keeping the custom implementation Python-client
fallback.
- Add regressions for the native async dir path and lazy
namespace_client() construction.
Validated locally with targeted namespace/db/table pytest, full
test_namespace.py, ruff, cargo fmt/check/clippy, and cargo test -p
lancedb-python.
Summary:
- Route built-in sync namespace connections through the Rust namespace
connector.
- Keep custom namespace clients on the existing Python fallback.
- Preserve namespace-backed to_lance compatibility with lazy Python
client construction and add regressions.
## Summary
Adds per-session monotonic reads for remote (LanceDB Cloud/Enterprise)
tables, preventing successive reads on a handle from moving *backward*
in dataset version when a load balancer routes them to query nodes with
differently-cached views.
Each `RemoteTable` handle tracks the highest dataset version it has
observed in a read response — surfaced by the server via a new
`x-lancedb-version` response header — and sends it back as
`x-lancedb-min-read-version` on subsequent reads (`count_rows`,
`query`). A query node whose cache is behind that version refreshes
before serving; a node already at/beyond it serves from cache at no
extra cost.
The watermark is sourced only from reads (always committed dataset
versions), so unlike the retired `x-lancedb-min-version` it is
unaffected by WAL writes returning WAL entry ids. It is reset on
`checkout_latest()`. Both headers are optional and ignored by older
peers.
Server-side enforcement lives in LanceDB Enterprise. Targets the
`codex/update-lance-9-0-0-beta-8` integration branch to match the
Enterprise submodule pin.
This PR is part cleanup, part feature, part example.
It removes `IntoArrow` and `IntoArrowStream`. There was only one
redundant call site between the two. Once we moved everything to
`Scannable` these traits no longer serve any purpose.
It adds a `Scannable` impl for a polars DataFrame. We used to have this
at one point for `IntoArrow` so this is more like a regression fix than
anything.
It adds an example (and unit test) which ensures we can ingest from a
Polars DataFrame and export to one. LazyFrame support would be a
follow-up (though a pretty straightforward one) but we've never had
proper LazyFrame support before.
Agents seemed to have trouble finding the right calls to work with
branches (create, list, delete) and passing the right params to get it
to work. We probably don't need a big skill to get it on the right track
but a little nudge seems helpful. Doing a couple simple tasks, it saved
about half the time and tokens, so feels worthwhile. Created with the
Claude skills creator, hence the "skill.md in a bare folder"
organization - happy to move it if that's not the standard anymore.
```
Benchmark results (3 evals, with-skill vs baseline):
┌────────────────┬────────────┬────────────────────┐
│ Metric │ With skill │ Without skill │
├────────────────┼────────────┼────────────────────┤
│ Pass rate │ 3/3 (100%) │ 3/3 (100%) │
├────────────────┼────────────┼────────────────────┤
│ Avg time │ 51s │ 142s (2.8× slower) │
├────────────────┼────────────┼────────────────────┤
│ Avg tokens │ 19,305 │ 36,513 (47% more) │
├────────────────┼────────────┼────────────────────┤
│ Avg tool calls │ 5.7 │ 26 (4.5× more) │
└────────────────┴────────────┴────────────────────┘
```
Expose the merged Rust OAuth header provider through the Node/TypeScript
connection path.
Includes:
- Native OAuthConfig conversion for napi-rs
- ConnectionOptions.oauthConfig plumbing
- Public TypeScript OAuthConfig and OAuthFlowType exports
- Generated TypeScript API docs for the new config surface
- input-validation and debug-redaction coverage in the Rust binding
layer
Local validation: cargo fmt --all; git diff --check.
Fixes#3589
## Problem
Multiple `warnings.warn()` calls across the Python client are missing
the `stacklevel=2` parameter. This causes warning messages to point to
lancedb internal code instead of the user's code that triggered the
warning, making debugging difficult.
## Solution
Add `stacklevel=2` to 7 `warnings.warn()` calls across 4 files:
| File | Warnings Fixed |
|------|---------------|
| `remote/db.py` | `request_thread_pool`, `connection_timeout`,
`read_timeout` deprecation warnings |
| `remote/table.py` | `cleanup_old_versions`, `compact_files`,
`optimize` no-op warnings |
| `table.py` | `data_storage_version`, `enable_v2_manifest_paths`,
`retrain` deprecation warnings |
| `embeddings/colpali.py` | `use_token_pooling` deprecation warning |
## Verification
- All 4 modified files pass `ast.parse()` syntax check
- Only `stacklevel=2` added — no other changes
## Changelog
| Date | Change | Author |
|------|--------|--------|
| 2026-06-27 | Add missing stacklevel=2 to warnings.warn() calls |
rtmalikian |
### Files Changed
- `python/python/lancedb/remote/db.py` — Add stacklevel=2 to 3
deprecation warnings
- `python/python/lancedb/remote/table.py` — Add stacklevel=2 to 3 no-op
warnings
- `python/python/lancedb/table.py` — Add stacklevel=2 to 3 deprecation
warnings
- `python/python/lancedb/embeddings/colpali.py` — Add stacklevel=2 to 1
deprecation warning
### Verification
- Syntax check passed on all modified files
---
**About the Author:** Raphael Malikian — Clinical AI Solutions
Architect. I specialise in building and fixing AI/ML systems for
healthcare, including vector databases, RAG pipelines, and clinical NLP.
If you need help with your project or think I can add value to your
organisation, feel free to reach out — I'd love to connect.
📧rtmalikian@gmail.com🔗 GitHub: https://github.com/rtmalikian🔗 LinkedIn:
http://www.linkedin.com/in/raphael-t-malikian-mbbs-bsc-hons-71075436a
---
**Disclosure:** This code was developed with assistance from
DeepSeek-V4-Pro (DeepSeek) via Hermes Agent (Nous Research). All changes
were reviewed, tested against the actual codebase, and verified for
correctness.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Fixes#2934
## Problem
Passing a `RemoteTable` to `permutation_builder()` raises a cryptic
`AttributeError`:
```
AttributeError: 'RemoteTable' object has no attribute '_inner'
```
This leaves users confused about what went wrong and why.
## Root Cause
`PermutationBuilder.__init__()` calls `async_permutation_builder(table)`
which accesses `table._inner` — the underlying Rust Lance table object.
`RemoteTable` connects to LanceDB Cloud/Enterprise and does not have a
local `_inner` attribute, making permutations fundamentally unsupported
on remote tables.
## Solution
Added an early check in `PermutationBuilder.__init__()` that verifies
the table has `_inner` before calling the Rust function, raising a clear
`TypeError` with an explanation of why permutations don't work on remote
tables.
## Verification
- Syntax validated with `ast.parse()`
- Structural verification: single call site (`permutation_builder()`),
guard placed before Rust FFI call
- Error message tested with mock: `MockRemoteTable()` correctly triggers
`TypeError`
## Changelog
| Date | Change | Author |
|------|--------|--------|
| 2026-06-28 | Added remote table guard in PermutationBuilder.__init__ |
rtmalikian |
### Files Changed
- python/python/lancedb/permutation.py — Added `hasattr(table,
"_inner")` check with clear error
---
**About the Author:** Raphael Malikian — Clinical AI Solutions
Architect. I specialise in building and fixing AI/ML systems for
healthcare, including vector databases, RAG pipelines, and clinical NLP.
If you need help with your project or think I can add value to your
organisation, feel free to reach out — I'd love to connect.
📧rtmalikian@gmail.com🔗 GitHub: https://github.com/rtmalikian🔗 LinkedIn:
http://www.linkedin.com/in/raphael-t-malikian-mbbs-bsc-hons-71075436a
---
**Disclosure:** This code was developed with assistance from
deepseek-v4-pro (DeepSeek) via Hermes Agent (Nous Research). All changes
were reviewed, tested against the actual codebase, and verified for
correctness.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Updates Lance Rust workspace dependencies and Java lance-core to
v9.0.0-beta.10.
No compatibility code changes were required; clippy and rustfmt passed
after installing the missing runner components.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.10
Expose the merged Rust OAuth header provider through the Python async
connection path.
Includes:
- Python OAuthConfig and OAuthFlowType public config objects
- PyO3 conversion into the Rust OAuthConfig
- connect_async(oauth_config=...) plumbing
- repr redaction coverage for client_secret
Local validation: cargo fmt --all; ruff format/check on touched Python
files.
## Summary
Add the Rust OAuth header provider for remote LanceDB connections.
This supports client credentials and Azure managed identity flows,
handles token caching and refresh, redacts secrets in Debug output, and
wires `ConnectBuilder::oauth_config()` into the remote client while
rejecting ambiguous API-key/header-provider combinations.
By default the read freshness provider was not included in the namespace
client, preventing the read freshness headers from being included in the
request. This prevents checkout_latest() from working as expected when
using the namespace client.
This fix ensures the provided is built into the client when the
namespace impl and properties are provided.
## Summary
Skip inserting the x-api-key header when the configured API key is
empty.
This lets bearer-token or other dynamic-header authentication avoid
sending an empty static API key header alongside the real auth header.
Fixes#3563
## Summary
- Add `stacklevel=2` to 10 `warnings.warn()` calls across 4 files
- Fix broken message concatenation in `table.py` where the second string
was incorrectly passed as the `category` parameter
## Problem
Multiple `warnings.warn()` calls in the `python/lancedb/` codebase were
missing the `stacklevel` parameter. Without `stacklevel=2`, warnings
point to library internals instead of the caller's code, making it
impossible for users to identify which of their function calls triggered
the warning.
Additionally, two calls in `table.py` (lines 3411 and 3420) had a more
serious bug: the deprecation message was split across two separate
string arguments, causing the second string to be passed as the
`category` parameter instead of being concatenated with the first
string. This would cause `TypeError` when the warning was triggered.
## Changes
| File | Fixes | Description |
|------|-------|-------------|
| `embeddings/colpali.py` | 1 | Add `stacklevel=2` to
`use_token_pooling` deprecation warning |
| `remote/db.py` | 3 | Add `stacklevel=2` to `request_thread_pool`,
`connection_timeout`, `read_timeout` deprecation warnings |
| `remote/table.py` | 3 | Add `stacklevel=2` to `cleanup_old_versions`,
`compact_files`, `optimize` no-op warnings |
| `table.py` | 3 | Fix broken message concatenation for
`data_storage_version` and `enable_v2_manifest_paths` deprecation
warnings + add `stacklevel=2` to `retrain` deprecation warning |
## Verification
```python
# All warnings.warn() calls now have stacklevel
python3 -c "import ast, os; ..."
# Result: All warnings.warn() calls now have stacklevel!
```
## Changelog
| Date | Change | Author |
|------|--------|--------|
| 2026-06-20 | Fix missing stacklevel=2 in 10 warnings.warn() calls +
fix broken message concatenation | rtmalikian |
### Files Changed
- `python/python/lancedb/embeddings/colpali.py` — Add stacklevel=2
- `python/python/lancedb/remote/db.py` — Add stacklevel=2 to 3
deprecation warnings
- `python/python/lancedb/remote/table.py` — Add stacklevel=2 to 3 no-op
warnings
- `python/python/lancedb/table.py` — Fix broken message concatenation +
add stacklevel=2
### Verification
- AST-based audit confirms all `warnings.warn()` calls now include
`stacklevel=2`
- Syntax check passes for all 4 modified files
---
**About the Author:** Raphael Malikian — Clinical AI Solutions
Architect. I specialise in building and fixing AI/ML systems for
healthcare, including vector databases, RAG pipelines, and clinical NLP.
If you need help with your project or think I can add value to your
organisation, feel free to reach out — I'd love to connect.
📧rtmalikian@gmail.com🔗 GitHub: https://github.com/rtmalikian🔗 LinkedIn:
http://www.linkedin.com/in/raphael-t-malikian-mbbs-bsc-hons-71075436a
---
**Disclosure:** This code was developed with assistance from **Hermes
Agent** (Nous Research). All changes were reviewed, tested against the
actual codebase, and verified for correctness.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Updates LanceDB's Lance dependencies to v9.0.0-beta.2 across the Rust
workspace and Java lance-core dependency.\n\nNo compatibility fixes were
required; clippy and formatting pass after installing the missing
toolchain components on the runner. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.2
This PR is for the Read path against blob v2. #3528 handles declare +
write, and this this adds materialization on local tables.
- blob_columns()
- fetch_blobs(column, row_ids) → bytes
- fetch_blob_files(column, row_ids) → lazy handles
- Pass _rowid from query().with_row_id(). Remote returns NotSupported.
(for now)
### Use cases
search, grab row ids, materialize images:
```rust
let row_ids = /* _rowid from hits */;
let images = table.fetch_blobs("image", &row_ids).await?;
```
Large blobs: open handles, read only what you need:
```rust
let handles = table.fetch_blob_files("image", &row_ids).await?;
let bytes = handles[0].as_ref().unwrap().read().await?;
```
Filter then batch fetch: collect ids from a filter, one call.
Multiple blob columns: image and thumbnail independently.
Row ids from before compact: still resolve.
### Alignment note
Lance `read_blobs` drops null rows. We descriptor-take first, read
non-null ids, re-expand to match input order. Null and zero-length blobs
come back null/None. Bytes path sets `preserve_order(true)`. So I added:
```
TODO(lance): expose selection_index or an aligned execute so we can drop the pre-read.
```
### Tests
`cargo test -p lancedb --test blob_integration`
- 30 tests covering nulls, reorder, dups, cross-fragment bytes + files,
compact, delete, legacy v1 errors.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The server now serializes an index's `created_at` as an RFC 3339 string
(e.g. `"2026-06-18T21:37:36.637Z"`), but the client deserializer only
accepted a unix timestamp in milliseconds. This caused `list_indices` to
fail with:
```
Failed to parse list_indices response: invalid type: string "2026-06-18T21:37:36.637Z", expected a unix timestamp in milliseconds
```
This PR replaces the fixed millisecond deserializer with a custom one
that accepts both an RFC 3339 string (current server) and a
unix-millisecond integer (legacy deployments), so the client works
against any server version.
It also improves the `IndexConfig` repr in the Python bindings.
Previously it printed only three fields (`Index(FTS, columns=["text"],
name="text_idx")`), hiding the metadata that `list_indices` returns. It
now renders every populated field, omitting any that are `None`. Each
value is valid Python — integer counts use `_` thousands separators and
`created_at` uses the `datetime` repr — so values round-trip. The real
repr is a single line; it's wrapped here for readability:
```python
>>> table.list_indices()
[IndexConfig(
name="text_idx",
index_type="FTS",
columns=["text"],
index_uuid="aefd3e00-2f95-4bdc-92ac-06de84442bf1",
type_url="/lance.table.InvertedIndexDetails",
created_at=datetime.datetime(2026, 6, 18, 21, 37, 36, 637000, tzinfo=datetime.timezone.utc),
num_indexed_rows=2,
size_bytes=3_669,
num_segments=1,
index_version=1,
index_details={
'lance_tokenizer': None,
'base_tokenizer': 'simple',
'language': 'English',
'with_position': False,
'max_token_length': 40,
'lower_case': True,
'stem': True,
'remove_stop_words': True,
'custom_stop_words': None,
'ascii_folding': True,
'min_ngram_length': 3,
'max_ngram_length': 3,
'prefix_only': False,
},
)]
```
Fixes#3556🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updates LanceDB's Lance dependencies from v8.0.0-beta.17 to
v8.0.0-beta.19.
This includes the Rust workspace Lance crates, Cargo.lock refresh, and
Java lance-core version bump. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.19
Updates the Lance Rust workspace dependencies and Java lance-core
dependency to v8.0.0-beta.17.
No LanceDB compatibility code changes were required; validation passed
with cargo clippy and cargo fmt. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.17
The "Create release commit" workflow (`make-release-commit.yml`) has
failed on its last two runs; no release tags have been created since
June 4. Since this workflow creates the tag that the cargo/npm/pypi/java
publish workflows trigger off of, all recent releases are effectively
blocked.
The workflow installs `bump-my-version` unpinned. Version `1.4.0` added
a check that refuses to run `pre_commit_hooks` containing shell syntax
(pipes, `&&`, `if`, variable expansion) unless `allow_shell_hooks =
true` is set. Both bumpversion configs use such hooks:
- `python/.bumpversion.toml` — updates `Cargo.lock` after the bump
(fails first)
- `.bumpversion.toml` — runs `mvn versions:set` for the Java packages
The job dies at the version-bump step with:
> Hook '…' contains shell syntax (pipes, redirects, or variable
expansion). Set `allow_shell_hooks = true` in your configuration to
enable shell execution…
This sets `allow_shell_hooks = true` in both configs to restore the
previous behavior.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#3506
## Problem
The Bedrock embedding compute path
(`rust/lancedb/src/embeddings/bedrock.rs`) panics instead of returning a
typed error in several places:
- `serde_json::to_vec(&request_body).unwrap()`: request serialization.
- `block_in_place(...).unwrap()`: the AWS `invoke_model` send result;
any API error terminates the worker instead of propagating.
- `v.as_f64().unwrap() as f32`: panics on non-numeric values in the
returned embedding array.
- `Handle::current()` + `block_in_place` assume a multi-threaded Tokio
runtime and panic when that assumption does not hold (no runtime, or a
current-thread runtime).
Malformed payloads, non-numeric embedding values, or an incompatible
runtime should surface as typed errors and never panic.
## Fix
- Serialize the request body before the blocking section so a
serialization failure returns `Error::Runtime` via `?`.
- Map the `invoke_model` send error to `Error::Runtime` instead of
`unwrap`.
- Add a `json_array_to_f32` helper that converts the response array to
`Vec<f32>`, returning `Error::Runtime` for a missing/non-array field or
a non-numeric element (used by both the Titan and Cohere paths).
- Add `current_multi_thread_handle()` (`Handle::try_current()` + a
`RuntimeFlavor::CurrentThread` guard) so an absent or incompatible
runtime returns a typed error rather than panicking in `block_in_place`.
Scope note: the sibling `openai.rs` provider uses the same
`block_in_place` + `block_on` bridge, so the bridge pattern itself is
kept; this change only removes the panic paths that are specific to the
Bedrock provider.
## Testing
Added 6 unit tests (no AWS credentials required):
- `json_array_to_f32`: valid numbers, non-array payload, and non-numeric
element.
- `current_multi_thread_handle`: errors with no runtime, errors on a
current-thread runtime, and succeeds on a multi-threaded runtime.
All pass; `cargo fmt` and `cargo clippy` clean. Build/test with
`--features bedrock,lance/protoc`.
## Summary
- clarify the Python error for passing a single dictionary to table
creation/add paths
- add a regression test for `create_table(..., data=dict)` so it points
users to a list of dictionaries
Fixes#409
## Testing
- `python -m pytest python/tests/test_table.py -q`
- `python -m ruff format python/lancedb/table.py
python/lancedb/scannable.py python/tests/test_table.py`
- `python -m ruff check python/lancedb/table.py
python/lancedb/scannable.py python/tests/test_table.py`
### Bug
`value_to_sql({...})` builds a DataFusion `named_struct(...)` literal
but interpolates the struct field names directly as `f"'{k}'"`. A field
name that contains a single quote therefore produces invalid SQL:
```python
>>> from lancedb.util import value_to_sql
>>> value_to_sql({"it's": 1})
"named_struct('it's', 1)" # invalid SQL — the quote terminates the literal
```
String *values* are already escaped (single quotes doubled) by the `str`
branch of `value_to_sql`, so keys and values were handled
inconsistently. This affects `Table.update(values={...})` /
`merge_insert` when a struct column has a field name containing `'`.
### Fix
Render the key through `value_to_sql(str(k))` so field names are escaped
exactly like string values:
```python
>>> value_to_sql({"it's": 1})
"named_struct('it''s', 1)"
```
Keys without special characters are unchanged (`'a'` stays `'a'`), so
existing behavior is preserved.
### Verification
```
$ pytest python/tests/test_util.py -k value_to_sql_dict
```
The new `test_value_to_sql_dict_key_escaping` covers quoted keys (incl.
nested structs) and fails on `main` (`named_struct('it's', 1)`), passes
with this change; the existing `test_value_to_sql_dict` still passes.
Co-authored-by: JSap0914 <JSap0914@users.noreply.github.com>
Fixes#3360.
This updates native table writes so local write progress uses Lance
writer byte stats instead of Arrow in-memory batch size once write bytes
are available. The change wires the existing `WriteProgressTracker` into
`InsertExec` for native `add` writes, installs a Lance `WriteProgressFn`
only when no lower-level callback is already configured, and keeps the
existing public `InsertExec::new` signature unchanged.
Validation:
- `cargo test -p lancedb --features remote
table::write_progress::tests::test_progress_uses_lance_write_bytes_for_local_tables
-- --nocapture` passed: 1 passed, 0 failed.
- `cargo test -p lancedb --features remote table::write_progress::tests
-- --nocapture` passed: 7 passed, 0 failed.
- `cargo check --quiet --features remote --tests --examples` passed.
- `cargo fmt --all --check` passed.
- `git diff --check` passed.
- `git diff | gitleaks stdin --no-banner --redact --timeout 30` passed:
no leaks found.
I did not run the full `cargo test --quiet --features remote --tests`
suite.
Co-authored-by: Ghxst <200635707+GHX5T-SOL@users.noreply.github.com>
Closes#3502
## Problem
A bare, unparameterised `typing.List` / `typing.Tuple` field crashes
`to_arrow_schema` with an opaque `AttributeError: __args__`:
```python
from typing import Tuple
from lancedb.pydantic import LanceModel
class Doc(LanceModel):
items: Tuple
Doc.to_arrow_schema() # AttributeError: __args__
```
In `_py_type_to_arrow_type`, the branch `elif getattr(py_type,
"__origin__", None) in (list, tuple)` is taken for a bare generic (its
`__origin__` is `list / tuple`), but the next line reads
`py_type.__args__[0]`, and a bare generic has no `__args__`. Other
unsupported types (e.g. `Dict[str, int]`) correctly raise a clear
`TypeError`, so this case is inconsistent.
Fix
Guard the element-type lookup with `getattr(py_type, "__args__", None)`
and raise a clear `TypeError` when it is missing, matching the existing
behavior for other unsupported types. Bare builtin list / tuple are
unaffected (their `__origin__` is `None`, so they already fall through
to the existing `TypeError`).
Testing
- Added `test_bare_generic_raises_type_error` covering both `List` and
`Tuple`.
- ruff format and ruff check clean.
This PR fixes a flaky test I hit on Windows test in #3528.
Looks like `test_eventual_consistency_background_refresh` was failing
with `v_cached` expected 1, got 2. There was a pr which swapped
`tokio::time::sleep(300ms)` for `clock::advance_by(300ms)`, which is
pretty much fine but the test necer pinned the clock so the first
`get()` locks the `cached_at` on wall time. Therefore, if our CI is
taking long enough the ttl expires before the value assertion in the
test.
So now we can add a `pin()` and call it first `get()`. After that we can
advance the clock manually with no problems.
Also, it's worth noting that I tried pinning in `BackgroundCache::new()`
first. That broke another test `test_reload_resets_consistency_timer`,
which uses real `tokio::time::sleep` and needs wall clock after
`clear_mock()`. So the pin stays in this test only. And this should
unblock us.
Failing instances:
-
https://github.com/lancedb/lancedb/actions/runs/27567527236/job/81495265474?pr=3528
-
https://github.com/lancedb/lancedb/actions/runs/27560366489/job/81470414928
### Description
`db://`-style connections that use the lance-namespace path
(`LanceNamespaceDatabase` → `NativeTable` + the lance-namespace REST
client) never sent a read-freshness signal. Against a server configured
to serve cached table metadata up to some staleness window, this allows
stale-read-after-write across handles and processes. The remote table
path already solved this (#3439). This brings the namespace path to
parity.
The namespace REST client doesn't let callers attach headers directly,
but it forwards a `DynamicContextProvider`'s `headers.*` context entries
as HTTP headers per request. So:
- A shared per-table baseline map is created before the namespace
client. I built and installed on the `ConnectBuilder` via a context
provider.
- On read operations the provider emits ·x-lancedb-min-timestamp =
max(baseline, now − read_consistency_interval)`
(RFC3339), keyed by the operation's `object_id`.
- Each table handle bumps its baseline (monotonically) on
`checkout_latest()`, `restore()`, and every data/schema write.
`checkout_latest()` is the primary hook: consumers refresh a handle
there after writing elsewhere, then poll.
Read operations that carry the floor: `describe_table`,
`list_table_versions`, `query_table`, `list_tables`.
`list_table_versions` is what resolves "latest" for managed-versioning
tables (`get_latest_version`), so it's the op that makes
`checkout_latest()` actually observe a prior write.
`describe_table_version` is excluded (pinned to an immutable version).
This mirrors #3439 (timestamp baseline, `max(baseline, now − interval)`,
monotonic); no `min_version` and no body channel, since the namespace
path has no version-returning write responses.
### Testing
- Unit tests for `compute_min_timestamp` / `next_freshness_baseline` and
the provider (header at/after a bumped baseline; nothing for an empty
baseline + no interval; interval floor applies; non-read ops emit
nothing; `list_tables` uses only the interval floor).
- Verified end-to-end against a local server that honors the header:
reads carry `x-lancedb-min-timestamp`, writes don't, and read-your-write
holds.
## Feature
### What is the new feature?
Adds Rust core API support for configuring vector query approximation
mode with `ApproxMode::{Fast, Normal, Accurate}`.
### Why do we need this feature?
Lance already exposes `lance_index::vector::ApproxMode` and scanner
support for controlling the speed/accuracy tradeoff for approximate
vector search. LanceDB Rust queries need to expose and pass this setting
through for local/native and remote vector searches.
### How does it work?
- Adds public `ApproxMode` in `rust/lancedb`, with lowercase serde,
`Default::Normal`, parse/display, and conversions to/from Lance's
`ApproxMode`.
- Adds `approx_mode: Option<ApproxMode>` to `VectorQueryRequest` and a
`VectorQuery::approx_mode(...)` builder.
- Applies the mode to native/local Lance scanners after `nearest(...)`
when explicitly set.
- Sends `approx_mode` in remote query JSON only when explicitly set;
default requests omit it.
## Validation
- `cargo fmt --all`
- `cargo test --quiet --features remote approx_mode`
- `cargo test --quiet --features remote
test_query_vector_default_values`
- `cargo check --quiet --features remote --tests --examples`
- `git diff --check`
## Feature
### What is the new feature?
Adds Rust core API support for configuring vector query approximation
mode with `ApproxMode::{Fast, Normal, Accurate}`.
### Why do we need this feature?
Lance already exposes `lance_index::vector::ApproxMode` and scanner
support for controlling the speed/accuracy tradeoff for approximate
vector search. LanceDB Rust queries need to expose and pass this setting
through for local/native and remote vector searches.
### How does it work?
- Adds public `ApproxMode` in `rust/lancedb`, with lowercase serde,
`Default::Normal`, parse/display, and conversions to/from Lance's
`ApproxMode`.
- Adds `approx_mode: Option<ApproxMode>` to `VectorQueryRequest` and a
`VectorQuery::approx_mode(...)` builder.
- Applies the mode to native/local Lance scanners after `nearest(...)`
when explicitly set.
- Sends `approx_mode` in remote query JSON only when explicitly set;
default requests omit it.
## Validation
- `cargo fmt --all`
- `cargo test --quiet --features remote approx_mode`
- `cargo test --quiet --features remote
test_query_vector_default_values`
- `cargo check --quiet --features remote --tests --examples`
- `git diff --check`
### Description
Adding branch support for RemoteTable by threading a branch selector
onto every operation the data plane accepts it on. Exposes the
currentBranch to nodejs and python through the bindings.
Matching the server handlers, the branch rides as:
- a `?branch=` query parameter for Arrow-body and query-only ops
(insert, merge_insert, multipart_*, version/list, drop_index)
- a `branch` field in the JSON body for everything else (count_rows,
query, update, delete, create_index, column ops, index list/stats,
stats, restore, describe, tags create/update)
A main-branch handle (`branch == None`) produces byte-identical requests
to before: no `branch` field and no `?branch=`
- Handle-per-branch: `create_branch` / `checkout_branch` return a new
handle with fresh caches and reset version/freshness state, mirroring
`NativeTable`.
- `create_branch` maps 409 to already-exists, 400 to invalid, and 404 to
not-found with source context, and sends without retry so the 409 stays
observable.
- `Ref` translation covers version, version-number (relative to the
handle's branch), and tag (resolved via the tags endpoint); `"main"` and
empty normalize to the main branch.
- Python branch handles persist their branch (and pinned version) across
pickle/fork, so a forked or pickled handle reopens on its branch rather
than silently reverting to main.
### Tests
- Rust mock tests per op category (query-param and body mechanisms,
branch CRUD, error paths, backward-compat).
- Python sync branch CRUD, `open_table(branch=)`, and a pickle
round-trip regression test.
Updates LanceDB's Lance dependencies to v8.0.0-beta.14.\n\nThis
refreshes the Rust workspace lockfile and Java lance-core version; no
compatibility code changes were required. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.14
`RemoteTable::list_indices` currently makes one `/index/list/` call plus
one `/index/{name}/stats/` call per index just to recover `index_type`.
When the server returns `index_type` directly in the `/index/list/`
response, all enriched fields are used and the per-index stats fan-out
is skipped entirely. When `index_type` is absent (legacy servers), the
existing stats fallback runs as before. This is content-based: no
version header required.
## Changes
- `RemoteTable::parse_index_list_response` replaces the old split
between enriched and legacy parsers. A single struct deserializes both
old and new response shapes, with all fields except `index_name` and
`columns` optional. `index_type` acts as the sentinel: present → use
enriched fields directly; absent → call `/index/{name}/stats/`.
## Tests
Added `test_list_indices_enriched` covering:
- All enriched fields populated correctly when `index_type` is in the
list response
- Optional fields absent from the response deserialize as `None`
- Stats endpoint is **not** called (panics if hit), verifying the
fan-out is eliminated
Existing `test_list_indices` and `test_list_indices_nested_field_paths`
exercise the legacy path unchanged.
## Depends on
- #3497 (expand `IndexConfig`) — already merged
- Server-side enriched response support
Closes#3494
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Two skills to help people connect and manage their column metadata using
a server that implements the [REST
API](https://lance.org/format/catalog/rest/)
lancedb-column-metadata was built using the [Claude skill
creator](https://claude.com/plugins/skill-creator); without the skill it
was usually calling at least one method that didn't exist and usually
not setting "replace": "false". So, while the base case is already
pretty good, adding this skill improves things somewhat.
lancedb-connect should help with most agentic workflows, because
"finding all the things you need to connect to your server" can be the
hardest part.
## Summary
- Extracts the `create_index` code cluster from `table.rs` into a new
`rust/lancedb/src/table/create_index.rs` submodule, continuing the work
from #2949.
- Moves 8 `NativeTable` inherent methods (`load_indices`,
`validate_index_type`, `build_ivf_params`, `get_num_sub_vectors`,
`get_vector_dimension`, `resolve_index_field`, `make_index_params`,
`get_index_type_for_field`) and 11 associated tests into the new module.
- Reduces `table.rs` from ~5009 to ~3804 lines (-1205 lines) with no
behavioral changes.
## Test plan
UT
## Summary
Surfaces the rich per-index metadata added in #3497 to the Python and
Node.js language bindings. Closes#3495.
New optional fields exposed on `IndexConfig` in both bindings:
- `index_uuid` / `indexUuid` — UUID of the first index segment
- `type_url` / `typeUrl` — protobuf type URL for the index
- `created_at` / `createdAt` — creation timestamp (milliseconds since
Unix epoch)
- `num_indexed_rows` / `numIndexedRows` — rows covered by the index
- `num_unindexed_rows` / `numUnindexedRows` — rows not yet indexed
- `size_bytes` / `sizeBytes` — total index file size in bytes
- `num_segments` / `numSegments` — number of index segments
- `index_version` / `indexVersion` — on-disk format version
- `index_details` / `indexDetails` — type-specific JSON details string
All fields are `None`/`undefined` for remote tables (which don't yet
surface this metadata through the server response).
## Changes
- `python/src/index.rs`: extend `IndexConfig` pyclass; update `From`
impl; update `__getitem__`
- `python/python/lancedb/_lancedb.pyi`: add type hints for new fields
- `python/python/tests/test_table.py`: new `test_index_config_fields`
test
- `nodejs/src/table.rs`: extend `IndexConfig` napi struct; update `From`
impl
- `nodejs/__test__/table.test.ts`: new test; update existing `toEqual`
assertions to `expect.objectContaining` to accommodate new fields
## Test plan
- [x] Python: `uv run --extra tests pytest
python/tests/test_table.py::test_index_config_fields`
- [x] Node.js: `pnpm test __test__/table.test.ts`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Closes#3412
Implements `rename_table` for `LanceNamespaceDatabase` (sync and async
Python) and the Rust `NamespaceDatabase` backend. Previously these
raised `NotImplementedError`; this PR delegates to the
`LanceNamespace.rename_table` method which is part of the
lance-namespace spec.
### Changes
- **`rust/lancedb/src/database/namespace.rs`**: Remove the
`NotImplementedError` stub for `rename_table`. Build a
`RenameTableRequest` (with `id`, `new_table_name`, and optionally
`new_namespace_id`) and call `self.namespace.rename_table(...)`,
mirroring the existing `drop_table` pattern.
- **`python/python/lancedb/namespace.py`**: Import `RenameTableRequest`
from `lance_namespace`. Replace the `raise NotImplementedError` in both
`LanceNamespaceDatabase.rename_table` (sync) and
`AsyncLanceNamespaceDatabase.rename_table` (async) with a call to
`self._namespace_client.rename_table(request)`.
- **`python/python/tests/test_namespace.py`**: Replace the
`test_rename_table_not_supported` test (which checked for
`NotImplementedError`) with `test_rename_table`, which:
1. Creates a table in a namespace
2. Calls `rename_table` with `cur_namespace_path` and
`new_namespace_path`
3. Asserts the old name is gone from `table_names()`
4. Asserts the new name appears in `table_names()`
5. Verifies the renamed table can be opened
## Test plan
- [ ] Existing namespace tests pass in CI (all rely on
`lance.namespace.DirectoryNamespace` which requires the full lance
package)
- [ ] `test_rename_table` exercises the full rename path: create →
rename → verify old gone → verify new present → open
- [ ] Rust build passes with the updated `namespace.rs` (requires Rust
toolchain in CI)
## Summary
Closes#3406
Add a regression matrix in `python/python/tests/test_nested_fields.py`
that exercises the full nested field index lifecycle for both the sync
and async Python table APIs. The tests will fail if any implementation
regresses to leaf-only field names in `list_indices`, `index_stats`,
search, or filter results.
## Test scenarios covered
**Index types:** BTree scalar, IvfPq vector, FTS
**Field-name edge cases (per acceptance criteria):**
- `rowId` — camelCase top-level field
- `` `row-id` `` — hyphenated top-level field (escaped)
- `parent.`\``leaf.name`\`` ` — struct leaf whose name contains a
literal dot
- `MetaData.userId` — mixed-case nested path
- `` `meta-data`.`user-id` `` — hyphenated struct with hyphenated leaf
**Lifecycle operations per index type:**
- `create_index` / `create_scalar_index` / `create_fts_index`
- `list_indices` → verify canonical full dotted path (not leaf name)
- `index_stats` → verify row count and index type
- Filtered scan (`WHERE nested.field = value`)
- Vector search via nested embedding column
- FTS search via nested text column
- `add` (append) then re-check index listing
- `optimize` then re-check index listing
**Both sync and async APIs** are covered in parallel test classes.
## Notes
Lance forbids top-level field names that contain a literal `.`, so the
`` `a.b` `` acceptance-criterion variant is exercised as a *struct leaf*
field (`parent.`\``leaf.name`\``) rather than a top-level column.
Another little pain point as I was working to integrate with
paperless-ngx. The read path of table.search() or table.query() already
accepted an Expr, but write paths Table.delete and
merge_insert(...).when_not_matched_by_source_delete did not. This PR
attempts to close that gap, so writes and reads can both use Expr,
instead of one side needing to build a string.
Updates Lance dependencies to v8.0.0-beta.11 and refreshes the Rust and
Java lock/config files. This also adapts namespace external manifest
store call sites to the new table-root-aware constructor required by
Lance. Triggering tag:
https://github.com/lancedb/lance/releases/tag/v8.0.0-beta.11
`IndexConfig` (returned by `Table::list_indices`) previously exposed
only `name`, `index_type`, and `columns`. Lance's `describe_indices`
provides richer per-index info cheaply (reads manifest metadata, often
cached), so this surfaces it.
Adds these `Option<T>` fields to `lancedb::index::IndexConfig`,
populated in `NativeTable::list_indices` from the `IndexDescription`:
- `index_uuid`: uuid of the first segment
- `type_url`: protobuf type URL (`IndexDescription::type_url`)
- `created_at`: minimum creation time across segments
- `num_indexed_rows`: approximate rows indexed across segments
- `num_unindexed_rows`: table row count minus `num_indexed_rows`
- `size_bytes`: total size of index files across segments
- `num_segments`: number of segments making up the index
- `index_version`: on-disk index format version (first segment)
- `index_details`: index-type-specific details as JSON
This field set mirrors the lance-namespace `IndexContent` contract
(lance-format/lance-namespace#348) so client and server agree on the
same shape. Note these are populated **locally** via `describe_indices`
— `NativeTable::list_indices` reads the dataset directly and does not
depend on the namespace spec change.
`RemoteTable` leaves the new fields `None` until a follow-up wires them
through the server response (#3494). Bindings exposure will also be a
follow up: #3495
Existing `list_indices` tests in `rust/lancedb/src/table.rs` are
extended to assert the new fields.
Fixes#3492🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The `Expr` build already includes a lot of useful filtering options,
`eq, ne, gt/gte, lt/lte, and_, or_, contains, cast`, but is was missing
a membership like `isin`. This PR adds that support, as minimally as
possible, allowing easy filtering for membership in a list, without
needing to be a series of `where` expressions.
I didn't see anything in CONTRIBUTING.md about needing a feature request
or issue first, so I just made the change. My apologies if I missed that
somewhere.
Thanks for the vector store, we're using it now in paperless-ngx.
Adds an FM-Index — a scalar index over string and binary columns that
accelerates substring search (`contains(col, 'needle')`), distinct from
the tokenized `FTS` index — across the Rust core and the Python and
TypeScript bindings.
## Rust
- `Index::Fm(FmIndexBuilder)` and `IndexType::Fm`.
- `make_index_params` maps `Index::Fm` to Lance's
`ScalarIndexParams::for_builtin(BuiltinIndexType::Fm)`.
- `supported_fm_data_type` validates
`Utf8`/`LargeUtf8`/`Binary`/`LargeBinary` columns.
- `list_indices` round-trips the type (`"Fm"` → `IndexType::Fm`); the
remote wire type is `"FM"`.
## Python
Adds `lancedb.index.Fm`, accepted by `create_index`:
```python
from lancedb.index import Fm
await tbl.create_index("text", config=Fm())
```
## TypeScript
Adds the `Index.fm()` factory:
```ts
await tbl.createIndex("text", { config: Index.fm() });
```
## Summary
This PR extends nested-field regression coverage across Rust
local/remote, Python sync/async, and Node so canonical escaped paths
stay consistent across scalar, vector, and FTS index lifecycle behavior.
It also aligns LanceDB's LabelList type gate with Lance by accepting
`LargeList<primitive>` columns while keeping `List<Struct<...>>`
unsupported until Lance defines stable membership semantics for struct
labels.
Part of #3406.
## Summary
Fixes the `NAPI_RS_FORCE_WASI=false` issue by upgrading `@napi-rs/cli`
from `3.5.1` to `3.7.0`.
Closes#3267
## Root Cause
In the `native.js` loader generated by `napi build`, the check was:
```js
if (!nativeBinding || process.env.NAPI_RS_FORCE_WASI) {
```
In JavaScript, any non-empty string is truthy, so
`NAPI_RS_FORCE_WASI=false` (a non-empty string) inadvertently triggered
the WASI fallback path. This caused an `ENOENT` error when
`lancedb.wasi.cjs` was not present.
## Fix
`@napi-rs/cli@3.7.0`
([napi-rs/napi-rs#3236](https://github.com/napi-rs/napi-rs/pull/3236))
introduced a tri-state check in the template that generates `native.js`:
**Before (generated by @napi-rs/cli@3.5.1):**
```js
if (!nativeBinding || process.env.NAPI_RS_FORCE_WASI) {
```
**After (generated by @napi-rs/cli@3.7.0):**
```js
const forceWasi =
process.env.NAPI_RS_FORCE_WASI === 'true' || process.env.NAPI_RS_FORCE_WASI === 'error'
if (!nativeBinding || forceWasi) {
```
Only the literal string `'true'` (or `'error'` for strict mode) now
activates the WASI path. All other values, including `'false'`, `'0'`,
or an unset variable, behave as if WASI is not forced.
## Changes
- `nodejs/package.json`: bump `@napi-rs/cli` from `3.5.1` to `3.7.0`
- `nodejs/package-lock.json` / `nodejs/pnpm-lock.yaml`: update lock
files to match
The fix is in the upstream napi-rs tool; the generated `native.js` is
not committed to this repository and is produced at build time by `napi
build`.
## What's broken
`Table.update(values={...})` raises `NotImplementedError: SQL conversion
is not implemented for this type` when a value is a numpy scalar such as
`np.int64`, `np.int32`, `np.float32`, or `np.bool_`. These arise
naturally from indexing an ndarray or a pandas int/bool column.
`np.float64` happens to work (it subclasses `float`), which makes the
failure inconsistent and surprising.
```python
df = pd.DataFrame({"id": np.array([10, 20], dtype="int32")})
t.update(where="id = 1", values={"id": df["id"].iloc[0]}) # np.int32
# -> NotImplementedError: SQL conversion is not implemented for this type
```
## Why it happens
`value_to_sql` is a `singledispatch` with handlers only for native
Python types and `np.ndarray`; numpy `integer`/`floating`/`bool_`
scalars aren't Python subclasses, so they fall through to the
`NotImplementedError` base.
## Fix
Register handlers for `np.bool_`, `np.integer`, and `np.floating` that
delegate to the existing native handlers.
## Test
`value_to_sql` on `np.int32/int64/float32/float64/bool_` all convert;
`np.int32` raised before.
Co-authored-by: Ishaan Samantray <ishaansamantray@Ishaans-MacBook-Pro.local>
### Description
Stacked on #3490. Adds an optional version to branch checkout across the
Rust core and the Python and TypeScript SDKs, so you can open a specific
version on a branch ("version V of branch B"), not just the branch's
latest version
Rust
```rust
// Open version 3 of branch "exp" (a read-only view): check out from an
// existing table, or open it directly from the connection.
let exp_v3 = table.checkout_branch("exp", Some(3)).await?;
let exp_v3 = db.open_table("items").branch("exp").version(3).execute().await?;
// checkout_latest re-attaches to the branch's writable HEAD.
exp_v3.checkout_latest().await?;
// With no branch, a version opens main at that version.
let main_v3 = db.open_table("items").version(3).execute().await?;
```
Python
```python
# Open version 3 of branch "exp" (a read-only view): check out from an
# existing table, or open it directly from the connection.
branch_v3 = await table.branches.checkout("exp", version=3)
branch_v3 = await db.open_table("items", branch="exp", version=3)
# checkout_latest re-attaches to the branch's writable HEAD.
await branch_v3.checkout_latest()
# With no branch, a version opens main at that version.
main_v3 = await db.open_table("items", version=3)
```
TypeScript
```typescript
// Open version 3 of branch "exp" (a read-only view): check out from an
// existing table, or open it directly from the connection.
const branchV3 = await (await table.branches()).checkout("exp", 3);
const opened = await db.openTable("items", undefined, { branch: "exp", version: 3 });
// checkoutLatest re-attaches to the branch's writable HEAD.
await branchV3.checkoutLatest();
// With no branch, a version opens main at that version.
const mainV3 = await db.openTable("items", undefined, { version: 3 });
```
### Testing
- Added unit tests (Rust, Python sync + async, TypeScript):
branch-scoped resolution at a version number shared with `main` and with
another branch, read-only enforcement on a pinned handle,
`checkout_latest` recovery to the branch's HEAD, fork-point reads, and
the nonexistent-version/branch error paths.
- Ran smoke tests against the Python and TypeScript SDKs on local
machine.
### Description
Adds first-class support for table branches across the Rust core and the
Python and TypeScript SDKs.
Rust
```rust
use lance::dataset::refs::Ref;
// Create a branch from main and write to it — main is untouched.
let exp = table.create_branch("exp", Ref::Version(None, None)).await?;
exp.add(batches).await?;
// Reopen the branch later: check out from a table, or open it directly.
let exp = table.checkout_branch("exp").await?;
let exp = db.open_table("items").branch("exp").execute().await?;
let branches = table.list_branches().await?;
table.delete_branch("exp").await?;
```
Python
```python
# Create a branch from main and write to it
branch = await table.branches.create("exp", from_ref="main")
await branch.add(data)
# Reopen the branch later: check out from a table, or open it directly.
branch = await table.branches.checkout("exp")
branch = await db.open_table("items", branch="exp")
await table.branches.list()
await table.branches.delete("exp")
```
TypeScript
```typescript
const branches = await table.branches();
// Create a branch from main and write to it
const branch = await branches.create("exp");
await branch.add(data);
// Reopen the branch later: check out from a table, or open it directly.
const checkedOut = await branches.checkout("exp");
const opened = await db.openTable("items", undefined, { branch: "exp" });
await branches.list();
await branches.delete("exp");
```
### Testing
- Added unit tests
- ran smoke tests against python and typescript sdks on local machine
### Next steps
- Add RemoteTable support
- Add Branch Comparison support
- Merge Branching support
## Bug Fix
### What is the bug?
Namespace-backed `LanceTable.to_arrow()` full-table reads bypassed the
existing `QueryTable` server-side query path and called the lower-level
table `to_arrow()` implementation directly. In Geneva/Sophon this could
fail while parsing the Arrow IPC response for
`hist.get_table().to_arrow()` / `to_pandas()`, even though
`hist.get_table().search().to_arrow()` worked.
### What issues or incorrect behavior does the bug cause?
Full-table reads on namespace-backed tables with `QueryTable` pushdown
could fail with Arrow IPC parse errors, while query/search reads on the
same table succeeded. Since `to_pandas()` delegates through `to_arrow()`
for non-blob/native cases, pandas export was affected too.
### How does this PR fix the problem?
When `QueryTable` pushdown is enabled, sync and async table `to_arrow()`
now construct a plain no-filter, no-limit, all-columns query and execute
it through the table-level `_execute_query()` path. `AsyncTable` now
preserves namespace context from async namespace connections so async
full reads can make the same pushdown decision. Non-namespace tables and
namespace tables without `QueryTable` pushdown keep their existing
behavior.
### Tests
- `uv run --extra tests --extra dev --no-sync ruff check
python/lancedb/table.py python/lancedb/namespace.py
python/tests/test_namespace.py`
- `uv run --extra tests --extra dev --no-sync ruff format
python/lancedb/table.py python/lancedb/namespace.py
python/tests/test_namespace.py`
- `uv run --extra tests --extra dev --no-sync pytest
python/tests/test_namespace.py::TestPushdownOperations::test_lance_table_to_arrow_uses_query_pushdown
python/tests/test_namespace.py::TestAsyncPushdownOperations::test_async_table_to_arrow_uses_query_pushdown
python/tests/test_namespace.py::test_local_table_to_arrow_and_to_pandas_are_unchanged
-q`
- `uv run --extra tests --extra dev --no-sync pytest
python/tests/test_namespace.py -q`
Updates LanceDB Lance dependencies from v8.0.0-beta.5 to v8.0.0-beta.6
and refreshes Cargo metadata.
No compatibility fixes were required; Java lance-core was bumped to
8.0.0-beta.6 as well.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.6
Updates Lance dependencies from v8.0.0-beta.4 to v8.0.0-beta.5 across
the Rust workspace and Java lance-core version.
No compatibility code changes were required; clippy and rustfmt pass
after installing the missing runner components.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.5
BREAKING CHANGE: direct Rust users lose the `IndexStatistics::loss`
field. Python and Node.js consumers are unaffected in practice for
remote tables (the value was always `None`/absent), but the attribute is
gone for local tables too.
`IndexStatistics::loss` was local-only — LanceDB Cloud never returned
it, so
`RemoteTable::index_stats` always set `loss: None`. It's vestigial; this
removes it.
- Remove `loss` from `IndexStatistics` and the internal `IndexMetadata`
in `rust/lancedb/src/index.rs`, plus the summing logic in
`NativeTable::index_stats`.
- Drop `loss` from the Python and Node.js bindings (and their
tests/docs).
Fixes#3493🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
`AsyncTable.search()` computes the query embedding with
`loop.run_in_executor(None, ...)`, which uses asyncio's **default**
`ThreadPoolExecutor`. That pool is shared with all other
`run_in_executor(None, ...)` work, so a slow embedding call — a heavy
local model or an HTTP request to an embeddings API — ties up those
threads and starves unrelated async I/O under concurrent load.
This moves the (potentially blocking) embedding call onto a **dedicated
executor**, isolating it from the default pool.
Closes#3310.
## Problem
`python/lancedb/table.py`, `AsyncTable.search()`:
```python
return (
await loop.run_in_executor(
None, # asyncio's default executor, shared with other blocking I/O
embedding.function.compute_query_embeddings_with_retry,
query,
)
)[0]
```
Under load, concurrent searches whose embeddings block (or any other
code using the default executor) contend for the same small thread pool.
## Change
- Add a dedicated
`ThreadPoolExecutor(thread_name_prefix="lancedb-embedding")` in
`background_loop.py`, exposed via `embedding_executor()`.
- Use it in `AsyncTable.search()`'s `make_embedding` instead of the
default executor.
- Reset the executor in the existing `_reset_after_fork` hook — its
worker threads don't survive `fork()`, same as the background event
loop. It's recreated lazily, so this is cheap.
## Design notes
The issue asked whether maintainers preferred a configurable executor, a
dedicated internal one, or another approach (no response in the thread).
I went with a **dedicated internal executor**: it fixes the starvation
with no public API change and stays consistent with the existing `LOOP`
singleton. Making the pool size configurable would be an easy follow-up
if preferred.
Scope is limited to `search()`. The broader "embedding functions need
real async support" (including `add()`) is tracked separately in #3268.
## Testing
- Added `test_async_search_runs_embedding_on_dedicated_executor`:
patches the embedding function to record the executing thread during an
async search and asserts it runs on a `lancedb-embedding` thread.
Verified it **fails** against the previous `run_in_executor(None, ...)`
and passes with the fix.
- `ruff format`, `ruff check`, and `pyright` pass on the changed files.
## Summary
Wires `RemoteTable::set_lsm_write_spec` / `unset_lsm_write_spec` to the
sophon REST endpoints added in
[lancedb/sophon#6181](https://github.com/lancedb/sophon/pull/6181),
replacing the previous `NotSupported` stubs.
- `set_lsm_write_spec` maps the `LsmWriteSpec` onto sophon's request DTO
— mode-tagged `sharding` (`unsharded` / `bucket` / `identity`),
`maintained_indexes`, and `writer_config_defaults` — and POSTs to
`/v1/table/{name}/set_lsm_write_spec/`.
- `unset_lsm_write_spec` POSTs to
`/v1/table/{name}/unset_lsm_write_spec/`.
- Both call `check_mutable` first, matching the other remote mutations.
- `maintained_indexes` is sent verbatim (an empty list means "no
maintained indexes", matching native semantics).
## Testing
- Added mocked-endpoint unit tests for unsharded / bucket / identity set
and for unset.
- `cargo check --features remote --tests` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updates Lance dependencies to v8.0.0-beta.2 across the Rust workspace
and Java lance-core metadata.
The update was generated with ci/update_lance_dependency.py and required
no compatibility code changes.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.2
## ⛔ Merge blocker: legal review required
This bump pulls in a new transitive **dev/profiling** dependency chain
`inferno v0.11.21` → `pprof v0.15.0` → `lance-testing`, and `inferno` is
licensed **CDDL-1.0** (copyleft). To get `cargo-deny` green, `CDDL-1.0`
was added to the `deny.toml` allow list.
**Do not merge until legal has reviewed and signed off on allowing
CDDL-1.0.** The dependency is dev/test-only and not distributed, but the
allow-list addition still requires legal approval per our policy.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Add `__reduce__` methods to `LanceDBClientError` and `RetryError` so
that instances can be pickled and unpickled correctly
- `HttpError` inherits the fix from `LanceDBClientError` since it has no
additional `__init__` parameters
- Add tests verifying pickle roundtrip for all three exception classes
Fixes#3447
## Test plan
- [x] Verified pickle roundtrip for `LanceDBClientError` with and
without `status_code`
- [x] Verified pickle roundtrip for `HttpError` (subclass, no extra init
params)
- [x] Verified pickle roundtrip for `RetryError` (subclass with many
extra params)
- [ ] CI tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
## Bug Fix
### What is the bug?
`QueryBuilder.to_pandas(blob_mode="descriptions")` could still fall back
to `self.to_arrow()` for query outputs with blob columns. Custom query
subclasses or wrappers can have `to_arrow()` behavior that is not
compatible with pandas blob-description conversion, which can surface as
low-level Arrow/list-batch conversion failures.
### What issues or incorrect behavior does the bug cause?
Callers need to carry local `to_pandas` or plain-scan adapter special
casing for blob descriptions, and scanner-only kwargs such as row
addresses and fragment selection are not represented in LanceDB query
state.
### How does this PR fix the problem?
This PR routes blob-output query `to_pandas()` through the Lance scanner
path for `lazy`, `bytes`, and `descriptions` modes when the query is a
scanner-backed plain scan. For `blob_mode="descriptions"` with
`flatten`, it collects scanner Arrow/table output, applies LanceDB
`flatten_columns`, and converts to pandas from there. Non-plain blob
query shapes now fail with a clear unsupported error instead of falling
into subclass `to_arrow()` behavior.
It also adds Python query state and builder methods for scanner-only
plain-scan parameters:
- `with_row_address()` for `_rowaddr`
- `with_fragments(...)` for Lance fragment objects
- `fragment_ids([...])` as a convenience wrapper that resolves IDs to
Lance fragments
## Validation
- `cd python && uv run --no-sync ruff format --check
python/lancedb/query.py python/tests/test_query.py`
- `cd python && uv run --no-sync ruff check python/lancedb/query.py
python/tests/test_query.py`
Targeted pytest was intentionally not run locally per maintainer
request.
### Description
This PR exposes native DataFusion expression support in the Rust SDK's
`MergeInsertBuilder` via two new builder methods:
`when_matched_update_all_expr` and
`when_not_matched_by_source_delete_expr`.
For remote LanceDB tables (where operations are serialized over
HTTP/JSON to the SaaS backend), native DataFusion expression trees
cannot be executed directly. The SDK handles this gracefully by
returning a `NotSupported` error.
### Key Changes
- **`MergeFilter` Enum**: Introduced a helper enum to store either a SQL
string or a native `datafusion_expr::Expr`.
- **`MergeInsertBuilder`**: Updated `when_matched_update_all_filt` and
`when_not_matched_by_source_delete_filt` fields to store the new enum,
and added `when_matched_update_all_expr` and
`when_not_matched_by_source_delete_expr` builder methods.
- **Execution & Remote Dispatch**: Dispatched the filter variants during
local execution, and rejected expression filters with a clean
`NotSupported` error in remote table request conversion.
- **Testing**: Added a `test_merge_insert_expr` unit test covering
conditional updates and deletes with programmatically built DataFusion
expressions.
### Verification
- Added integration test `test_merge_insert_expr` which successfully
compiles and passes.
- Formatted and linted the code.
Closes#3416
## Summary
Regression test for [issue
#2654](https://github.com/lancedb/lancedb/issues/2654) — a nullable
struct column whose first batch contains only `None` values crashed in
`_align_field_types` with `AttributeError: 'pyarrow.lib.DataType' object
has no attribute 'fields'`.
The actual fix landed in #3394, but no test was added. This PR adds the
reproducer from the issue as a test.
## Test plan
- `test_add_nullable_struct_with_none`: creates a table with a nullable
struct column, adds a row with a non-null struct value, then a row with
`None` for the struct field. Verifies both rows land correctly.
- Uses Lance file format v2.1 (`new_table_data_storage_version="2.1"`)
because nullable structs aren't supported on v2.0.
## Related
- #3028 (the original fix attempt, now superseded)
Adds a REVIEW.md at the repo root with cross-SDK parity guidance for
automated code review. The Claude Code review feature automatically
loads `REVIEW.md` as review-only context.
This is intentionally a semantic nudge, not a deterministic check, it
relies on the reviewer reading the sibling SDK, so it will catch most
gaps.
## What's broken
Calling `RRFReranker().rerank_multivector([])` crashes with `IndexError:
list index out of range` because the method accesses `vector_results[0]`
for the type-homogeneity check before verifying the list is non-empty.
The `all()` call passes vacuously on an empty iterable so the crash hits
the next lines.
```python
from lancedb.rerankers import RRFReranker
RRFReranker().rerank_multivector([])
# IndexError: list index out of range
```
## Why it happens
The type check uses `vector_results[0]` as the reference type but never
guards against an empty list. `all(...)` short-circuits to `True` when
the iterable is empty, so the bad index access on the lines that follow
is never reached by the existing guard logic.
## Fix
Add an explicit empty-list check before any indexing.
## What's broken
`MRRReranker.rerank_multivector([])` raises `IndexError: list index out
of range`. The crash happens on line 128 (the `all()` type-homogeneity
check passes vacuously on an empty iterable) and on line 134 which
accesses `vector_results[0]` unconditionally, with no prior guard for an
empty list.
## Why it happens
`all()` over an empty iterable returns `True`, so the type check
silently passes and execution falls through to `vector_results[0]` which
crashes.
## Fix
Added a two-line guard at the top of `rerank_multivector` that raises a
clear `ValueError("vector_results must not be empty")` before any
indexing occurs.
## Test
Added `test_mrr_reranker_empty_input` in `test_rerankers.py` which calls
`rerank_multivector([])` and asserts that a `ValueError` with the
message "must not be empty" is raised.
Fixes#3468
Co-authored-by: Aegis Dev <aegis@devteamaegis.com>
"description":"Write, review, debug, and document LanceDB pipelines in Python and TypeScript that work across local LanceDB OSS tables and remote LanceDB Enterprise/Cloud tables.",
description:"Override [project] name in python/pyproject.toml (e.g. 'lancedb-compat'). Default keeps 'lancedb'."
required:false
default:"lancedb"
rustflags:
description:"RUSTFLAGS for the build container, as a single whitespace-free token (e.g. '-Ctarget-cpu=x86-64-v2'). Empty leaves RUSTFLAGS unset, keeping the defaults from .cargo/config.toml."
required:false
default:""
runs:
using:"composite"
steps:
@@ -27,6 +35,18 @@ runs:
ARM_BUILD:${{ inputs.arm-build }}
run:|
echo "ARM BUILD: $ARM_BUILD"
- name:Patch package name for variant build
if:${{ inputs.package-name != 'lancedb' }}
shell:bash
env:
PACKAGE_NAME:${{ inputs.package-name }}
run:|
# Swap the [project] name so this build produces e.g. lancedb-compat
# wheels. The package still installs files under the lancedb/
# namespace -- import lancedb still works after pip install.
sed -i.bak 's/^name = "lancedb"$/name = "'"$PACKAGE_NAME"'"/' python/pyproject.toml
description:"Tag name from Lance. If omitted, the skill will use the latest Lance release that needs an update."
description:"Tag name from Lance (e.g. `v7.2.0-beta.1`). If omitted, the newest release is resolved automatically — stable releases are preferred over pre-releases — and the run is skipped if it is not newer than the version currently pinned in Cargo.toml."
required:false
default:""
type:string
workflow_dispatch:
inputs:
tag:
description:"Tag name from Lance. Leave empty to use the latest Lance release that needs an update."
description:"Tag name from Lance (e.g. `v7.2.0-beta.1`). Leave empty to resolve the newest release automatically — stable releases are preferred over pre-releases — and skip the run if it is not newer than the version currently pinned in Cargo.toml."
echo "Broken documentation links found by [\`$GITHUB_WORKFLOW\`]($run_url)."
echo
echo "This issue is rewritten by every scheduled run and closed automatically once all links resolve."
echo
echo "Entries can be false positives: some sites rate-limit or block automated clients while working fine in a browser. Confirm before editing the docs, and add persistent offenders to \`--exclude\` in \`.github/workflows/docs-link-check.yml\`."
echo
# Timeouts are reported alongside errors: entries land in
# timeout_map with a status text instead of an HTTP code.
jq -r '
"\(.errors) of \(.total) links failed, \(.timeouts) timed out.",
* Use repository-defined Cargo profiles instead of ad hoc LTO overrides.
* Use `release-with-debug` for benchmarks and profiling so optimized builds keep debug symbols without a rebuild.
* Use `release-no-lto` only for local debugging, IO-bound benchmarks, or compile-time-sensitive performance investigation where LTO would not affect the measured bottleneck.
* Format Python: `ruff format .`
* Lint Python: `ruff check .`
* Bootstrap Python dev env: `cd python && uv run --extra tests --extra dev maturin develop --extras tests,dev`
@@ -92,6 +95,8 @@ Python bindings changes:
* Should use `LOOP.run()` to call the corresponding `AsyncTable` method.
6. Add concrete sync method to `RemoteTable` class in `python/python/lancedb/remote/table.py`.
7. Add unit test in `python/tests/test_table.py`.
8. If you added a new public class or module-level function (not just a method on an
existing class), expose it in the API reference. See "Python API reference" below.
TypeScript bindings changes:
@@ -103,6 +108,33 @@ TypeScript bindings changes:
5. Add test in `nodejs/__test__/table.test.ts`.
6. Run `npm run docs` to generate TypeScript documentation.
## Python API reference
`docs/src/python/python.md` is the entire Python API reference. It is maintained by
hand, and anything not listed there is not rendered at all, so new public classes and
module-level functions have to be added explicitly. How depends on the module:
* `lancedb.index`, `lancedb.embeddings`, `lancedb.remote`, and `lancedb.rerankers` are
rendered by a single directive each, driven by the module's `__all__`. Add the new
name to `__all__` and it appears; forget, and it is silently omitted.
* Everything else (`lancedb`, `lancedb.table`, `lancedb.query`, `lancedb.db`, ...) is
listed symbol by symbol. Add a `::: lancedb.<module>.<Name>` line to the matching
section, and remember that the page separates synchronous and asynchronous APIs.
Deliberately undocumented: concrete implementations reached through an abstract base
(`LanceTable`, `LanceDBConnection`, `RemoteDBConnection`), query base classes already
covered by `inherited_members`, and internal helpers.
Cross-references in docstrings use mkdocstrings syntax, `[text][lancedb.table.Table]`.
Plain relative links such as `[Table](Table)` do not resolve. To check your work:
```shell
pip install -r docs/requirements.txt
cd docs && PYTHONPATH=. mkdocs build
```
The docs site only builds on pushes to `main`, so this is not covered by PR CI.
## Review Guidelines
Please consider the following when reviewing code contributions.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Xuanwo
Dan Tasse
Will Jones
Wyatt Alt
LanceDB Robot
XY Zhan
Lance Release
lancedb-gatefixer[bot]
dependabot[bot]
Dan Rammer
Adityaj0
Igor Ganapolsky
Jack Ye
Yang Cen
Sravan Avvaru
ForwardXu
Andrew Chen
David Tolnay
Justin Miller
Drew Gallardo
LuQQiu
Cohen Karnell
Farmer.Chillax
Joaquin Hui
sanskar singh bhardwaj
heart
buduoqiu
kid
Vivek
Heng Ge
Eran Dagan
Prashanth Rao
Expyron
Mateusz Szewczyk
Drew Gallardo
Vitaliy
Kobi Hikri
Tobias
Weston Pace
Mark McDonald
Pranav Achar
Unmilan Mukherjee
ForwardXu
Omkar Kabde
Octopus
LAKSH JAIN
Eric B
Armaan Sandhu
Neo-X7
Raphael Malikian
Ryan Green
whitewooood
JSap0914
Ghxst ☠️
Brendan Clement
Xuanyi Li
nuthalapativarun
Trenton H
devteamaegis
hashwnath
Harikrishna KP