## What changed
- add `block_size` to Python FTS configuration and the deprecated
local/remote helpers
- add `blockSize` to the TypeScript FTS options and propagate it through
the NAPI binding
- serialize the value as `block_size` for remote index creation
- document the existing Rust builder API and generate the TypeScript API
reference
- add local, remote, metadata, search, and invalid-value regression
coverage
## Why
Lance supports configuring the number of documents per compressed FTS
posting block, but LanceDB's Python and TypeScript APIs did not expose
the setting. This made the experimental FTS V3 layout unavailable
through those clients and allowed the value to be dropped before index
creation.
## How it works
The default remains `128`. Supported values are `128` and `256`;
selecting `256` uses the experimental FTS V3 format. Invalid values are
rejected by the Lance builder and surfaced as Python or JavaScript
errors.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo +1.94.0 clippy --quiet --features remote --tests --examples --
-D warnings`
- targeted Rust local and remote index tests
- Rust doctests: 34 passed
- Python Ruff checks, doctest, and targeted local/remote tests: 5 passed
- TypeScript build, Biome lint, generated docs, and targeted Jest tests:
9 passed
- `git diff --check`
## Limitations
The Java client remains unchanged because its external remote REST model
does not currently expose `block_size`.
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
## What
`AnswerdotaiRerankers(return_score="all").rerank_hybrid(...)` (and
`ColbertReranker`, which subclasses it without overriding
`rerank_hybrid`) raises:
```
pyarrow.lib.ArrowInvalid: Invalid sort key column: No match for FieldRef.Name(_relevance_score) in _rowid: int64 ...
```
## Why
```python
combined_results = self.merge_results(vector_results, fts_results)
combined_results = self._rerank(combined_results, query)
if self.score == "relevance":
combined_results = self._keep_relevance_score(combined_results)
elif self.score == "all":
combined_results = self._merge_and_keep_scores(vector_results, fts_results)
```
When `score == "all"`, `combined_results` is unconditionally overwritten
by `_merge_and_keep_scores(vector_results, fts_results)` **after**
`_rerank()` already computed and appended `_relevance_score` —
discarding it. The following `sort_by("_relevance_score", ...)` then has
nothing to sort on.
Every sibling reranker that supports `return_score="all"`
(`cross_encoder`, `openai`, `cohere`, `jinaai`, `voyageai`, `watsonx`)
instead calls `_merge_and_keep_scores()` **before** `_rerank()`. This
file is the one place the ordering got inverted when `"all"` support was
added (#2509) — a copy/paste inconsistency across the six files that PR
touched. Fix mirrors the pattern already used (and tested) by the other
five rerankers.
Also drops the now-stale `"Only 'relevance' is supported for now"`
docstring line on both classes, left over from before `"all"` support
existed.
## Testing
Added `test_answerdotai_reranker_return_all`, mirroring the existing
`test_cross_encoder_reranker_return_all`. Verified locally with the real
built Rust extension: red (reproduces the exact `ArrowInvalid` above) →
green, using the actual `rerank_hybrid`/`_rerank`/`base.py` code path
with the model call mocked out — my local environment's
`rerankers==0.10.0` fails to load the real ColBERT model against the
available `transformers` version (`AttributeError: 'ColBERTModel' object
has no attribute 'all_tied_weights_keys'`), which I confirmed also
breaks the **pre-existing**, unmodified
`test_colbert_reranker`/`test_answerdotai_reranker` baseline tests
identically — an unrelated local dependency-version issue, not a
regression from this change. `ruff check`/`ruff format` clean; full
`test_rerankers.py` run: 9 passed / 8 skipped / 3 failed (the 3 failures
are exactly those two pre-existing tests plus my new one, all failing at
model-loading time for the same unrelated reason before reaching the
changed code).
---
Disclosure: this PR was drafted with AI assistance (Claude); I reviewed,
tested, and take responsibility for the change.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
- Replace legacy model names in `WatsonxEmbeddings` with the current
supported set:
- `ibm/granite-embedding-278m-multilingual` (new default, 768-dim)
- `ibm/slate-125m-english-rtrvr-v2` (768-dim)
- `ibm/slate-30m-english-rtrvr-v2` (384-dim)
- `intfloat/multilingual-e5-large` (1024-dim)
- `sentence-transformers/all-minilm-l6-v2` (384-dim)
- Add `space_id` field — mutually exclusive with `project_id`, mirrors
the
existing pattern in `WatsonxReranker`
- `project_id` / `space_id` resolution now falls back to
`WATSONX_PROJECT_ID` /
`WATSONX_SPACE_ID` env vars; exactly one must be supplied
## Why
The previously hardcoded models (`ibm/slate-125m-english-rtrvr`,
`sentence-transformers/all-minilm-l12-v2`) are legacy and no longer
listed as
supported by the watsonx.ai platform. `space_id` scoping was already
supported
by `WatsonxReranker` but was missing from the embeddings counterpart.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
This PR adds some support for `diff` / `merge` in the remote client as
for local tables we stay `NotSupported` until
https://github.com/lance-format/lance/issues/7263.
This wires the two review-and-land calls against the remote REST API:
- `POST /v1/table/{id}/branches/diff`
- `POST /v1/table/{id}/branches/merge`
Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python
returns the wire JSON, same shape as the REST response.
Merge here means promoting a branch's added columns onto `main`.
### Behavior
- Remote only. Local raises `NotSupported`.
- A rejected merge is not an exception. HTTP 409 still returns `Ok` / a
dict with `status="rejected"` and blockers in `diff.mergeBlockers`.
- Unknown blocker / status codes parse as `Unknown` so a newer server
does not break older clients.
- `MergePreview` tolerates missing fields for the same reason.
- Merge requests are not retried. 409 is final and carries the body you
need.
### Example
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.merge("exp", dry_run=True)
result = table.branches.merge("exp", dry_run=False)
if result["status"] == "merged":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "rejected":
print(result["diff"]["mergeBlockers"])
```
### Testing
cargo test -p lancedb --features remote diff_branch
cargo test -p lancedb --features remote merge_branch
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Problem
On the remote (LanceDB Cloud) write path, each write partition is
uploaded as a **single** `/insert?upload_id=...` request that stays open
until the whole partition has been streamed and the server has written
it to object storage. For large bulk ingests a partition can be many GB,
so a single request can run longer than the client read timeout (default
300s), surfacing as:
```
lancedb.remote.errors.HttpError: operation timed out
```
The server already supports staging **multiple** parts under one
`upload_id` (each `/insert` writes a separate transaction that
`complete` merges atomically), but the client never used that — it sent
one part per partition.
## Change
Split each partition into multiple parts of at most
`max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded
as its own `/insert?upload_id=...&upload_part_id=...` request. This
bounds how long any single request stays open, independent of total data
size or write parallelism.
Key properties:
- **Still streamed, not buffered.** Each part's body is driven through a
bounded channel while the request is in flight (`futures::join!` of a
producer + the send), so peak memory stays at a couple of batches per
partition regardless of the part size. Backpressure from a
slow/throttled server still propagates upstream.
- **Correct part accounting.** An empty partition still sends exactly
one (schema-only) part so `complete` has a transaction to commit; a size
cut landing exactly on the end of input does not emit a trailing empty
part.
- **Multipart only.** The single-request (non-multipart) path is
unchanged.
## Config
New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable
via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable.
**Default 1 GiB** (`Some(0)` disables splitting → one request per
partition). Python users pick up the default/env automatically through
the remote client.
## Tests
- `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each
batch in its own part → N requests, each carrying the shared `upload_id`
and a distinct `upload_part_id`.
- `test_multipart_single_part_when_under_budget`: a large budget keeps
the partition in a single request.
- Verified end-to-end against a live remote table: a forced-chunked
multipart add (many parts) assembles to the correct row count.
Related to ENT-1883.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Fixes#1653.
`infer_vector_column_name` in `util.py` could silently return `None`
when `query is None` and `query_type` is not `"fts"` or `"hybrid"`. This
`None` then propagated into downstream code, causing a cryptic
`TypeError: expected bytes, NoneType found` rather than a clear error
message.
## Changes
- **Removes the no-op `try/except Exception as e: raise e`** around
`inf_vector_column_query` (it was catching and immediately re-raising
without adding any value)
- - **Adds a `None` guard** after the inference block: if
`vector_column_name` is still `None` at this point, raise a clear
`ValueError` pointing the user to pass `vector_column_name` explicitly
## Before / After
**Before:** cryptic `TypeError: expected bytes, NoneType found` deep in
schema lookup code
**After:**
```
ValueError: No vector column found in the schema. Please specify the vector column name explicitly via the `vector_column_name` parameter.
```
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
## Problem
`table.add(dataset)` with a `pyarrow.dataset.Dataset` OOMs the client
during bulk ingestion of wide rows (e.g. embedding columns), even
against a remote table where the upload itself is streaming.
The cause is in `to_scannable`: a `Dataset` is scanned with pyarrow's
default scanner settings (`batch_size=131072` rows,
`batch_readahead=16`, `fragment_readahead=4`). pyarrow's internal
threads prefetch that read-ahead window independently of LanceDB's
backpressure, so for wide rows a large fraction of the dataset is held
in memory. On the remote path this is then multiplied across the
multipart write partitions (one in-flight batch per partition, up to
CPU-core count).
Reproduced on a 10 GB / 1.55M-row dataset with two 768-dim float32
embeddings: peak client RSS ~11.7 GB for the scan alone (6.8 GB after
consuming a *single* batch), ~15.4 GB for the full remote `add()`.
## Fix
`to_scannable` now sizes the scanner from an estimate of bytes-per-row
derived from the schema:
- **Narrow datasets keep pyarrow's defaults** (empty scanner kwargs) —
no throughput regression. The bound only engages above ~410 bytes/row.
- **Wide rows** get a smaller `batch_size` (~16 MiB/batch) and reduced
read-ahead (`batch_readahead=2`, `fragment_readahead=1`) so peak
in-flight memory stays near a ~1 GiB budget. Read-ahead (not just batch
size) has to drop, because pyarrow pins whole row-group buffers.
On the 10 GB dataset this drops peak client RSS to ~1.4 GB, and it stays
flat as the dataset grows. The `Dataset`/`LanceDataset` scannables
remain rescannable (retry-safe).
## Also: expose `write_parallelism` on `add()`
`AddDataBuilder::write_parallelism` already existed in Rust but was not
exposed in Python. This PR forwards it through the async, sync, and
remote `add()` methods, so users can cap the number of parallel write
partitions (each buffers data in flight) to trade throughput for memory
on large uploads.
## Tests
- `test_scannable.py`: bytes-per-row estimation; narrow → defaults; wide
→ bounded; `Dataset` reader streams bounded batches and stays
rescannable.
- `test_table.py`: `write_parallelism` on sync and async `add()`, and
that `write_parallelism=0` is rejected.
Fixes ENT-1883
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tracks #3324. On x86_64 CPUs without AVX2 (Sandy Bridge / Ivy Bridge /
Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD), `import
lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled
function. Per [westonpace's
review](https://github.com/lancedb/lancedb/issues/3324#issuecomment-4328944354),
the default `lancedb` wheel stays fast; pre-Haswell users get a
separately-published `lancedb-compat` wheel.
## Summary
- Adds a `lancedb-compat` matrix entry to `pypi-publish.yml` that builds
with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (Nehalem-class baseline).
Same Python API (`import lancedb` works) — files install to the same
namespace, so the two wheels conflict at install time and users pick
one. Same pattern as `psycopg2` / `psycopg2-binary` and `tensorflow` /
`tensorflow-cpu`.
- Generalizes `build_linux_wheel` and `upload_wheel` composites with
optional `package-name` and `rustflags` inputs (defaults preserve the
existing 4 `lancedb` matrix entries verbatim).
- Documents the choice in `python/README.md`: `pip install
lancedb-compat` for pre-Haswell hosts.
The default `.cargo/config.toml` baseline is unchanged.
## Sequencing
1. ~~lance-format/lance#6630 merges → runtime SIMD dispatch lands in
lance.~~ **Done — merged.**
2. lancedb's lance dep is bumped to a release that includes it (separate
PR / normal cadence).
3. This PR's `lancedb-compat` wheel build path starts producing a wheel
that runs on pre-Haswell hardware. **Maintainer setup**: register
`lancedb-compat` on PyPI and configure trusted publishing.
## Verified end-to-end on Sandy Bridge Xeon E5-2609
Verification was done locally against a fork-pinned lance dep that
includes the runtime dispatch implementation, using the same
`RUSTFLAGS="-C target-cpu=x86-64-v2"` flags this PR uses in CI:
```
$ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
$ pip install ./target/wheels/lancedb-*.whl
$ python verify.py
PASS: import + simd dispatch + table create + vector search all work.
```
Pre-fix on the same CPU (default `pip install lancedb`): `Illegal
instruction (core dumped)`. Full reproducer (deps + clone + build +
verification):
https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df.
Fork-internal verification PR with the dep bump and full logs:
[`tobocop2/lancedb#2`](https://github.com/tobocop2/lancedb/pull/2).
## Benchmarks — no regressions on modern CPUs from the lance-side change
These are the numbers I ran for the lance PR, confirming the runtime
dispatch doesn't slow down the default (`target-cpu=haswell`) wheel that
existing users install. Criterion, one machine, one session, base → PR,
no `RUSTFLAGS` override. Full methodology, null experiments, and logs:
[lance-format/lance#6630 benchmark
comment](https://github.com/lance-format/lance/pull/6630#issuecomment-4933063394)
and the [logs
gist](https://gist.github.com/tobocop2/3c6d0f449cbd736aa2501f89a7fe56a2).
| benchmark | EPYC 7B13 (`avx2`, `fma`, no `avx512f`) | Xeon Cascade
Lake (`avx512f`) |
|---|---|---|
| `Cosine(f32, scalar)` *(control)* | +0.04% | +0.09% |
| `Cosine(f64, scalar)` | −0.34% | −1.94% |
| `Cosine(u8, SIMD)` | +2.30% | +3.63% |
| `Dot(f16, SIMD)` | −0.58% | +0.61% |
| `Dot(f32, SIMD)` | +0.34% | **−6.08%** |
| `Dot(f32, arrow_arity)` | +0.02% | −0.00% |
| `L2(f32, scalar)` | −0.10% | −0.02% |
| `L2(f32, simd)` (dim 1024) | +2.63% | −0.53% |
| **`L2(simd,f32x8)` (dim 8)** | **−45.9%** | **−25.1%** |
| `L2(u8, SIMD)` | +0.42% | −3.11% |
| `NormL2(f32, SIMD)` | −1.02% | −4.17% |
| `NormL2(f64, SIMD)` | +3.51% | −0.58% |
Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector
width — improves 25–46%.
---
To be transparent: this isn't my domain of expertise and the lance-side
implementation is AI-generated. I verified it works end-to-end on the
failing hardware. Happy to roll in feedback.
Routes local sync child-namespace operations through the Rust-backed
connection instead of the Python namespace-client fallback.
Also keeps lazy namespace-client construction for table-to-Lance
conversion and preserves public namespace error mappings.
Validated locally with ruff format/check and targeted namespace pytest.
BREAKING CHANGE: splits generated by the permutation data loader will
not be the same, due to a change in hash function.
Updates the Lance dependencies and Java lance-core to
[v9.0.0-rc.1](https://github.com/lance-format/lance/releases/tag/v9.0.0-rc.1).
Includes the required DataFusion 54 and Lance file-reader compatibility
updates.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds client-side support for analyze_plan distributed metrics modes
across Rust, Python, and TypeScript clients. Defaults to aggregate for
backward compatibility and sends the remote distributed_metrics
parameter only when a non-default mode is requested.
## Summary
Fix `on_bad_vectors="fill"` so it replaces only invalid or missing
vector values instead of replacing the entire vector row.
Fixes#3026.
## Reasoning
The old Python sanitizer detected whether a vector row was bad at row
granularity. For `fill`, it then used that row-level flag to replace the
whole vector with `[fill_value] * dim`. That meant an input like `[1.0,
NaN, 3.0]` became `[0.0, 0.0, 0.0]`, even though the documented and more
useful behavior is to preserve valid values and fill only the bad
element.
I checked whether this should be a Rust-side fix so TypeScript users
would benefit too. Today, Rust core exposes `NaNVectorBehavior::{Error,
Keep}` for rejecting or keeping NaN vectors, while the Python
`on_bad_vectors` API (`error`, `drop`, `fill`, `null`) is implemented in
the Python ingestion sanitizer before data reaches Rust. TypeScript does
not expose the Python `on_bad_vectors="fill"` behavior today. Moving
this exact behavior to Rust would be a broader cross-language API
change, so this PR keeps the fix scoped to the currently affected Python
API.
## What changed
- Added a small helper that fills bad vector rows by preserving valid
elements, replacing NaN elements with `fill_value`, truncating vectors
longer than the expected dimension, and padding short vectors with
`fill_value`.
- Kept the existing fast path unchanged: the helper only runs after bad
vectors are detected and `on_bad_vectors="fill"` is selected.
- Updated sanitizer and table tests to assert element-wise NaN
replacement and short-vector padding for both `create_table` and `add`.
## Validation
- `uv run ruff format .`
- `uv run ruff check .`
- `cd python && uv run --no-sync pytest
python/tests/test_util.py::test_handle_bad_vectors_jagged
python/tests/test_util.py::test_handle_bad_vectors_nan
python/tests/test_table.py::test_create_with_nans
python/tests/test_table.py::test_add_with_nans -vv`
Targeted pytest result: `10 passed`.
## Why this fix is Python-side (and not Rust)
The problematic behavior lives in Python’s `on_bad_vectors` sanitizer,
before data is handed off to Rust. Rust currently only exposes
`NaNVectorBehavior::{Error, Keep}` for add operations, while Python has
the richer `on_bad_vectors={"error","drop","fill","null"}` API.
TypeScript does not currently expose the Python-style fill behavior, so
moving this exact fix into Rust would require designing a broader
cross-language bad-vector handling API.
This PR keeps the change scoped to the existing affected surface:
Python’s `on_bad_vectors="fill"` path. This way, Python users
immediately benefit.
## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage
## Summary
- serialize sync phrase queries consistently for execution and query
plans
- restore the documented no-argument hybrid `phrase_query()` behavior
- keep reranker input as the original user text without mutating the
builder
Fixes#3653.
## Testing
- `python/.venv/bin/python -m pytest <8 focused test nodes> -q` (`8
passed`)
- `python/.venv/bin/python -m ruff format --check
python/python/lancedb/query.py python/python/tests/test_fts.py
python/python/tests/test_hybrid_query.py`
- `python/.venv/bin/python -m ruff check .`
- `git diff --check origin/main...HEAD`
The complete hybrid module and the real native FTS phrase test were not
completed
in the current PyO3 runtime environment: both stalled in the native
`lancedb.connect()` fixture and were interrupted without an assertion
failure.
Carrying over from #2915, this patch introduces:
* Single-API call batching support for Gemini embeddings (up to 100 at a
time, the API limit)
* A versioned user agent header for Gemini API calls
* Support for [variable embedding dimension
size](https://ai.google.dev/gemini-api/docs/embeddings#control-embedding-size)
(Gemini is MRL trained)
## Summary
- preserve explicit `0.0` distance bounds in synchronous hybrid search
- distinguish omitted `None` endpoints from zero-valued endpoints when
configuring the vector child query
- add a public end-to-end regression test for a zero upper bound
## Testing
- `cd python && uv run --extra tests pytest
python/tests/test_hybrid_query.py -q`
- `uv run --project python ruff format --check
python/python/lancedb/query.py python/python/tests/test_hybrid_query.py`
- `uv run --project python ruff check .`
Fixes#3651
Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.
## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.
## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.
## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.
Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.
## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps Lance to v9.0.0-beta.19, which includes lance-format/lance#7687
for side-effect-free DirectoryNamespace read paths.
This fixes root-level read-only table opens that previously could
trigger `__manifest` creation through directory namespace construction,
including Hugging Face bucket reads with read-only tokens. A LanceDB
regression test now covers root listing operations without creating
`__manifest`.
Fixes#3633.
### Summary
`flatten_columns` raises `ValueError` when called with `flatten=False`,
even though `False` should mean "do not flatten". This is reachable from
the public API — `Query.to_pandas(flatten=...)` and
`to_batches(flatten=...)` type their `flatten` param as
`Optional[Union[int, bool]]` and pass it straight to `flatten_columns`.
### Cause
`bool` is a subclass of `int`, so `isinstance(False, int)` is `True`.
`flatten=False` skips the `flatten is True` check, falls into the
integer branch, and `False <= 0` evaluates to `True`, raising:
```
ValueError: Please specify a positive integer for flatten or the boolean value `True`
```
### Reproduction
```python
import lancedb
db = lancedb.connect("/tmp/db")
t = db.create_table("t", data=[{"id": 1, "vector": [0.1, 0.2]}])
t.search([0.1, 0.2]).to_pandas(flatten=False) # -> ValueError
```
### Fix
Guard the integer branch with `not isinstance(flatten, bool)` so that
`flatten=False` (and `None`) mean "do not flatten". Behavior is
otherwise unchanged:
- `flatten=True` → flatten all nested levels
- positive `int` → flatten to that depth
- non-positive `int` (e.g. `0`) → still rejected with `ValueError`
Added a regression test in `tests/test_util.py` covering `None`,
`False`, `True`, a positive depth, and `0`.
This PR fixes a serialization error when using Ollama embeddings in
`create_table`.
The use of `@cached_property` for the Ollama client was causing issues
during serialization/pickling, which is required by certain LanceDB
operations (like when using multiprocessing or certain storage
backends). Switching to a standard `@property` ensures the client is
instantiated when needed without being stored in a way that breaks
serialization.
Verified with the following script:
```python
import lancedb
from lancedb.embeddings import get_registry
import pickle
registry = get_registry().get(\"ollama\")
model = registry(name=\"llama3\")
# This would fail before the fix
pickled = pickle.dumps(model)
unpickled = pickle.loads(pickled)
```
Fixes#2629 (or similar serialization issues reported).
---------
Co-authored-by: Unmilan Mukherjee <Missing-Identity@users.noreply.github.com>
## Summary
Closes#3525
This PR wires up two new optional object-store backends at the LanceDB
layer, exposing capabilities that already exist upstream in `lance` /
`lance-io`:
| Backend | Cargo feature | Default in Rust crate | Default in Python
wheel | Default in Node binding |
| --- | --- | --- | --- | --- |
| **Tencent COS** | `cos` | ❌ off | ✅ on | ❌ off |
| **GooseFS** | `goosefs` | ❌ off | ✅ on | ✅ on |
Both backends are additive and do not affect existing users who don't
opt in.
## Motivation
- **Tencent COS** is the dominant object storage in the China region.
Tencent Cloud users currently need an S3-compatible proxy or a private
fork to use LanceDB against COS buckets.
- **GooseFS** is Tencent Cloud's distributed cache acceleration layer
that sits in front of COS/S3, a common pattern for vector search / AI
training where the same hot dataset is read repeatedly.
- This brings COS / GooseFS to feature parity with the existing
first-class backends (`aws`, `gcs`, `azure`, `oss`, `huggingface`).
See the linked issue #3525 for the full discussion.
## Changes
### `rust/lancedb/Cargo.toml`
Add two new optional features that pull through the corresponding
upstream feature flags:
```toml
cos = ["lance/tencent", "lance-io/tencent"]
goosefs = [
"lance/goosefs",
"lance-io/goosefs",
"lance-namespace-impls/dir-goosefs",
]
```
### `python/Cargo.toml`
Enable both `cos` and `goosefs` by default for the Python wheels, so
`pip install lancedb` works against COS / GooseFS out of the box
(consistent with how `aws` / `gcs` / `azure` / `oss` are bundled today):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/cos", "lancedb/goosefs"]
```
### `nodejs/Cargo.toml`
Enable `goosefs` by default for the Node binding (COS kept opt-in to
limit the default native binary size; can be revisited based on demand):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/goosefs"]
```
### `Cargo.lock`
Regenerated to reflect the transitive dependencies brought in by the new
upstream features. No manual edits.
## Example Usage
### Rust
```toml
# Cargo.toml
lancedb = { version = "0.30", features = ["cos", "goosefs"] }
```
```rust
// Tencent COS
let db = lancedb::connect("cos://my-bucket/my-db").execute().await?;
// GooseFS
let db = lancedb::connect("goosefs://my-namespace/my-db").execute().await?;
```
### Python
```python
import lancedb
db = lancedb.connect(
"cos://my-bucket/my-db",
storage_options={
"secret_id": "...",
"secret_key": "...",
"region": "ap-guangzhou",
},
)
```
## Backwards Compatibility
- All new features are **opt-in** at the Rust crate level (`default =
[]` for `lancedb` itself is unchanged).
- The Python wheel gains both backends by default, increasing wheel size
slightly but matching the existing pattern of bundling all major cloud
backends.
- Node binding only adds `goosefs` to defaults; existing users see no
behavior change.
## Testing
- `cargo check --all-features` ✅
- `cargo check -p lancedb --features cos` ✅
- `cargo check -p lancedb --features goosefs` ✅
- End-to-end COS / GooseFS smoke tests require Tencent Cloud credentials
and are intentionally not added to CI in this PR (same approach used for
`s3-test`). Happy to add a gated test feature in a follow-up if
reviewers prefer.
## Checklist
- [x] Added `cos` and `goosefs` features to `rust/lancedb/Cargo.toml`
- [x] Updated `python/Cargo.toml` default features
- [x] Updated `nodejs/Cargo.toml` default features
- [x] Regenerated `Cargo.lock`
- [x] Verified build with `--all-features`
- [ ] Documentation update (can be done in a follow-up PR once API
stabilizes)
## Related
- Issue: #3525
- Upstream support:
[`lance/tencent`](https://github.com/lance-format/lance),
[`lance/goosefs`](https://github.com/lance-format/lance)
## Summary
Adds `Table::get_lsm_write_spec` returning `Option<LsmWriteSpec>` — the
read counterpart to the existing `set_lsm_write_spec` /
`unset_lsm_write_spec`. Returns `None` when the MemWAL LSM write path is
not enabled; otherwise reconstructs the spec (mode, shard column,
`num_buckets`, `maintained_indexes`, `writer_config_defaults`) exactly
as installed.
## Changes
- **Rust core (`NativeTable`)** — reconstructs the spec from
`mem_wal_index_details()`, resolving the shard column from its Lance
field id via the dataset schema. This is a raw metadata read, so it is
unaffected by `describe_indices` system-index filtering.
- **Remote (`RemoteTable`)** — reads the `__lance_mem_wal` system index
through `index/list` with `include_system: true` (so the curated
`list_indices` surface stays unchanged), then parses the index `details`
JSON. It matches the index by name and ignores `index_type`, so no
client `IndexType` variant is needed. It uses the **server-resolved
`column` name** from the details (Lance field ids do not travel to the
remote client).
- **Python + TypeScript bindings** — sync and async, mirroring
`set`/`unset`, with round-trip tests (bucket / identity / unsharded,
plus `None` when unset).
## Tests
- Rust: native round-trip unit test + remote mock-endpoint tests
(present + absent). All green (`cargo test --features remote -p
lancedb`).
- Python/TS: round-trip tests added; binding-runtime execution runs in
CI.
## Dependencies for the remote path
The remote path is complete on the client side but depends on two
out-of-repo pieces to work end-to-end:
1. **lance** — emit the server-resolved shard **`column`** name in the
MemWAL index `details` JSON (field ids can't reach the client). See
lance-format/lance#7667.
2. **server** — honor `include_system` on `index/list` so the
`__lance_mem_wal` entry is returned for this read.
Against an older server (no `include_system`), the remote getter
degrades gracefully to `Ok(None)` rather than erroring.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Closes#3245.
> **BREAKING CHANGE:** `with_format("torch")` no longer returns a list
of stacked row tensors. It now returns per-row dicts so PyTorch's
default `DataLoader` collate stacks them into `{col: tensor(B,)}`.
Switch to `with_format("torch_row")` to keep the old shape.
### What changed
`"torch"` now returns a list of per-row dicts (`[{col: tensor}, ...]`)
at every indexed access path. The default `DataLoader` collate stacks
them into a column-keyed batched dict, no custom `collate_fn` needed.
The old shape is preserved under a new `"torch_row"` literal.
`"torch_col"` is unchanged.
The unbatching lives inside the transform (`batch_to_tensor_dict`), not
`__getitems__`, so the shape survives pickling and works under
`DataLoader(num_workers>0, multiprocessing_context="spawn")`.
### Format comparison
| Format | `iter(batch_size=N)` | `__getitems__([0,1,2])` | `DataLoader`
default collate |
|---|---|---|---|
| `"torch"` (new) | `list[{col: tensor}]` length N | `list[{col:
tensor}]` length 3 | `{col: tensor(B,)}` |
| `"torch_row"` (old `"torch"` behavior) | `list[tensor(n_cols,)]`
length N | `list[tensor(n_cols,)]` length 3 | `tensor(B, n_cols)` |
| `"torch_col"` (unchanged) | `tensor(n_cols, N)` | `tensor(n_cols, 3)`
| needs `collate_fn=lambda x: x` |
Output matches HuggingFace `Dataset.set_format("torch")` on container
shape, keys, and values at every access path. The only divergence:
HuggingFace downcasts `float64` to `torch.float32` by default, LanceDB
preserves dtype. Verified by `scripts/verify_torch_format.py`.
### Migration
```python
# Old default — column names lost, shape was tensor(B, n_cols)
DataLoader(Permutation.identity(table).with_format("torch"))
# New default — column names preserved
DataLoader(Permutation.identity(table).with_format("torch")) # {col: tensor(B,)}
# Keep old behavior
DataLoader(Permutation.identity(table).with_format("torch_row")) # tensor(B, n_cols)
```
### **Summary**
Closes#3212
Extends the Python `lit()` helper to natively support three additional
types (`date`, `datetime`, and `Decimal`) and implements reflexive
operators for the `Expr` class.
This implementation specifically addresses the blocking feedback
regarding precision loss, CI discovery, and query engine limitations:
* **Logic Refactoring**: Simplified `lit()` by combining `date` and
`datetime` normalization into ISO-8601 strings, ensuring stable SQL
parsing across different engine locales.
* **Precision Preservation**: `decimal.Decimal` objects are now passed
as high-precision strings to the Rust bridge, bypassing intermediate
float conversions and preserving full 128-bit decimal precision for
DataFusion.
* **Averted CI Failures**: Temporarily deferred `bytes` literal support
to a future PR to resolve a known DataFusion `expr_to_sql` limitation
that was crashing the `Doctest` runner.
* **Reflexive Operators**: Added support for "literal-first" arithmetic
and logical operations (e.g., `10 + col('a')` or `True &
col('active')`). Redundant reflexive comparisons (e.g., `__rlt__`) were
pruned as Python's data model handles them automatically.
* **Integration Verification**: Added dedicated integration tests in the
official test directory to ensure the query engine correctly handles the
new types and preserves bit-perfect fidelity.
### **Changes**
####
[python/python/lancedb/expr.py](file:///c:/Users/Laksh/Documents/lancedb/python/python/lancedb/expr.py)
* Updated `lit()` to handle `date`, `datetime`, and `Decimal` natively.
* Implemented reflexive operators (`__radd__`, `__rand__`, `__rmul__`,
etc.) to support literals on the left-hand side.
* Removed the problematic `bytes` doctest example and `lit()` type
support to unblock CI.
####
[python/src/expr.rs](file:///c:/Users/Laksh/Documents/lancedb/python/src/expr.rs)
* Modified the Rust FFI bridge to extract `Decimal` objects as strings.
* Ensured the `expr_lit` handler is ready to receive normalized temporal
strings.
* Consolidated imports and added missing operator documentation.
####
[python/python/lancedb/_lancedb.pyi](file:///c:/Users/Laksh/Documents/lancedb/python/python/lancedb/_lancedb.pyi)
* Updated type stubs for `expr_lit` to include `Any` (allowing for
`Decimal`).
### **Testing**
Added several new advanced test cases in
[python/python/tests/test_expr.py](file:///c:/Users/Laksh/Documents/lancedb/python/python/tests/test_expr.py)
covering:
* **High-precision Decimal preservation**: Verified against 128-bit
boundaries with a "one point off" test case (`1.234567890123456789 <
1.234567890123456790`).
* **Reflexive operator positioning**: Verified successful query
construction with literals on the left.
* **Timezone-aware normalization**: Confirmed stable behavior for
`datetime` objects.
* **Integration Testing**: Confirmed Date32 and Decimal columns return
the correct Python types and values from the engine during `.to_arrow()`
calls.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
# Elastic Streaming Dataloader
## Motivation
Training large models on LanceDB tables today requires loading the
entire dataset
into memory or writing bespoke batching logic. This PR introduces
`StreamingDataset`, a PyTorch `IterableDataset` that streams directly
from a
LanceDB table with two hard guarantees that are difficult to achieve
together:
**elastic determinism** and **resumability**.
## Goals
### Elastic determinism
The dataset partitions the table into a fixed number of *splits*
(controlled by
`num_splits`, `shuffle_seed`, and `epoch`). Samples are yielded by
round-robining
over splits one sample per split per cycle. Because the split structure
is fixed,
the set of samples that makes up each global training step is identical
regardless
of `world_size` or `num_workers`. You can scale your cluster up or down
between
runs and the model sees the same data in the same order — no
re-sharding, no
gradient variance from topology changes.
### Resumability
`state_dict()` / `load_state_dict()` capture how many samples each split
has
consumed. Because all splits are the same size and the round-robin
design keeps
them in lockstep, the state reduces to a single scalar
(`samples_consumed_per_split`)
that is topology-independent. A checkpoint saved with 8 GPUs can resume
correctly
on 4 GPUs or 16 GPUs without any adjustment.
### PyTorch `IterableDataset` / streaming
`StreamingDataset` implements the standard PyTorch `IterableDataset`
interface, so
it drops into any existing `DataLoader` pipeline without modification.
Data is
fetched lazily from Lance in chunks — only the rows needed for the
current batch are
ever in memory.
Compared to the map dataset this takes more work from pytorch and puts
it into the dataset itself (e.g. shuffling, filtering, etc.). We do this
because we cannot achieve things like elastic determinism or
prefiltering otherwise.
### Multi-worker support
DataLoader workers are automatically assigned contiguous sub-blocks of
splits (the
rank's splits are divided evenly across workers). Each worker is
independent:
no shared state, no inter-process coordination. The only constraint is
that
`num_splits` must be divisible by `world_size * num_workers`.
That being said, multi-worker is highly discouraged as it relies on
multiprocessing which is inefficient. Still, we want to support it.
### Filters as prefilters
Filters are applied at *permutation-build time* via
`PermutationBuilder.filter()`,
not re-evaluated on every fetch. The filtered row IDs are stored in the
permutation
table so that subsequent reads see only the matching rows. This allows
us to avoid loading rows that don't match the filter (which is the
default pytorch behavior)
### Prefetching
Two parameters control the I/O pipeline:
- `read_batch_size` (default 64) — number of rows fetched per
`take_offsets` call.
Larger values amortise per-request overhead, which is critical on object
storage
where a single round-trip can cost ~100 ms.
- `prefetch_batches` (default 4) — number of batches prefetched in
parallel per
split via a `ThreadPoolExecutor`. While the model processes the current
batch,
the next several batches are already in flight, hiding storage latency
behind
compute.
If set correctly then you can get good performance even with
num_workers=0 (unless you are bottlenecked on transform).
### Transform parallelism
The underlying `Permutation` API supports a `with_transform()` callback
for
decoding, augmentation, and format conversion. Unfortunately, this is
not parallelized. Pytorch typically parallelizes this with num_workers
which is multiprocessing which is highly inefficient. For simple
transforms we should be able to utilize multithreading and Rust based
UDFs. For complex python UDFs we could have a dedicated multiprocessing
pipeline for just the transform. Or we could just utilize
multithreading. In both cases we would exclude the I/O stage from the
multiprocessing because that ends up being very memory hungry and
inefficient.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- When an embedding function returns an empty list (e.g. `[]`) for an
input row — as can happen when a model produces no output for a blank
string — `_append_vector_columns` crashed with `ArrowInvalid: Length of
item not correct: expected N but got array of size 0` because PyArrow
cannot fit a zero-length value into a fixed-size list element.
- The fix adds a validation step in `gen()`, inside
`_append_vector_columns`, that replaces any vector whose length does not
match the expected `ndims` (including empty lists and `None`) with
`None` before `pa.array()` is called.
- `None` is a valid null in a PyArrow fixed-size list array, so the bad
entry flows into `_handle_bad_vectors` and is handled according to the
caller-supplied `on_bad_vectors` policy (`error` / `drop` / `fill` /
`null`) instead of causing an unconditional crash.
## Test plan
- [ ] Added `test_embedding_with_empty_output_vectors` in
`python/python/tests/test_embeddings.py` that uses an embedding function
returning `[]` for empty-string inputs, calls `table.add(...,
on_bad_vectors="drop")`, and asserts no crash and that bad rows are
correctly dropped.
- [ ] Existing `test_embedding_with_bad_results` continues to pass (NaN
vectors still handled correctly).
- [ ] Verified manually that `pa.array([[1.,2.,3.,4.], []],
type=pa.list_(pa.float32(), 4))` raises `ArrowInvalid` without the fix,
and succeeds with `None` in place of `[]`.
Fixes#1672
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## What
`MRRReranker.rerank_multivector` averages each document's reciprocal
ranks over the wrong denominator. It divides by the number of rankings
the document *happens to appear in*, instead of the total number of
rankings being fused.
```python
# python/python/lancedb/rerankers/mrr.py
for result_id, reciprocal_ranks in mrr_score_map.items():
mean_rr = np.mean(reciprocal_ranks) # divides by len(present systems)
```
`mrr_score_map[doc]` only accumulates a reciprocal rank for the systems
in which the document was returned, so `np.mean` never accounts for the
systems that missed it.
## Why it's wrong
Mean Reciprocal Rank fusion treats a system that didn't return a
document as a reciprocal rank of `0` and averages across **all**
systems. That's the exact mechanism by which it rewards cross-system
consensus. Dividing by the appearance count removes that, so a document
liked by a single ranking can beat one ranked highly by every ranking.
Concretely, fusing 3 vector rankings:
| Doc | Ranks | Current score | Correct score |
|-----|-------|---------------|---------------|
| A | #1 in 1 system only | `mean([1.0]) = 1.000` | `1.0 / 3 = 0.333` |
| B | #1, #1, #2 across all 3 | `mean([1, 1, .5]) = 0.833` | `2.5 / 3 =
0.833` |
The current code ranks **A above B** - a document two of three rankings
ignored outranks one all three ranked at or near the top.
This also makes `rerank_multivector` inconsistent with `rerank_hybrid`
in the same file, which already treats a missing system as `0`
(`vector_rr = 0.0` / `fts_rr = 0.0`), and with the class docstring
("average of reciprocal ranks across different search results").
## Fix
Divide the summed reciprocal ranks by the total number of rankings:
```python
num_systems = len(vector_results)
...
mean_rr = float(np.sum(reciprocal_ranks)) / num_systems
```
## Tests
Adds `test_mrr_multivector_rewards_consensus`, which asserts the exact
MRR scores and that the consensus document ranks first. It fails on
`main` and passes with this change. Existing reranker tests are
unaffected.
BREAKING CHANGE: When passing multiple where clauses to a query, they
now stack instead of replacing the previous filter.
Previously, calling `where`/`only_if` more than once on a query silently
replaced the previous filter, so only the last filter was applied. This
was
surprising and could return rows that an earlier filter should have
excluded.
This implements the alternative suggested in
https://github.com/lancedb/lancedb/pull/3514#issuecomment-4664901580:
instead of
rejecting a second filter, repeated filters are combined with a logical
AND
(`(previous) AND (new)`).
The combination happens in the Rust core (`QueryBase::only_if` and
`only_if_expr`), so it applies to all SDKs at once (Rust, Python async,
and
TypeScript). The Python sync query builder keeps its own filter state,
so it
combines filters in the binding layer as well.
SQL string and expression filters are combined within their own
representation.
When the two representations are mixed, the expression is lowered to SQL
(via
`expr_to_sql_string`) and the filters are combined as SQL strings, so
chaining
`where` works regardless of which form each filter takes.
Fixes#2649
## Tests
- Rust: `cargo test --features remote -p lancedb --lib query`
- Python: `uv run --extra tests pytest python/tests/test_query.py`
- TypeScript: `pnpm test __test__/query.test.ts`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summary:
- Route built-in async namespace-backed connections through the Rust
namespace connector.
- Delegate async namespace/table management methods to the inner
AsyncConnection while keeping the custom implementation Python-client
fallback.
- Add regressions for the native async dir path and lazy
namespace_client() construction.
Validated locally with targeted namespace/db/table pytest, full
test_namespace.py, ruff, cargo fmt/check/clippy, and cargo test -p
lancedb-python.
Summary:
- Route built-in sync namespace connections through the Rust namespace
connector.
- Keep custom namespace clients on the existing Python fallback.
- Preserve namespace-backed to_lance compatibility with lazy Python
client construction and add regressions.
Fixes#2934
## Problem
Passing a `RemoteTable` to `permutation_builder()` raises a cryptic
`AttributeError`:
```
AttributeError: 'RemoteTable' object has no attribute '_inner'
```
This leaves users confused about what went wrong and why.
## Root Cause
`PermutationBuilder.__init__()` calls `async_permutation_builder(table)`
which accesses `table._inner` — the underlying Rust Lance table object.
`RemoteTable` connects to LanceDB Cloud/Enterprise and does not have a
local `_inner` attribute, making permutations fundamentally unsupported
on remote tables.
## Solution
Added an early check in `PermutationBuilder.__init__()` that verifies
the table has `_inner` before calling the Rust function, raising a clear
`TypeError` with an explanation of why permutations don't work on remote
tables.
## Verification
- Syntax validated with `ast.parse()`
- Structural verification: single call site (`permutation_builder()`),
guard placed before Rust FFI call
- Error message tested with mock: `MockRemoteTable()` correctly triggers
`TypeError`
## Changelog
| Date | Change | Author |
|------|--------|--------|
| 2026-06-28 | Added remote table guard in PermutationBuilder.__init__ |
rtmalikian |
### Files Changed
- python/python/lancedb/permutation.py — Added `hasattr(table,
"_inner")` check with clear error
---
**About the Author:** Raphael Malikian — Clinical AI Solutions
Architect. I specialise in building and fixing AI/ML systems for
healthcare, including vector databases, RAG pipelines, and clinical NLP.
If you need help with your project or think I can add value to your
organisation, feel free to reach out — I'd love to connect.
📧rtmalikian@gmail.com🔗 GitHub: https://github.com/rtmalikian🔗 LinkedIn:
http://www.linkedin.com/in/raphael-t-malikian-mbbs-bsc-hons-71075436a
---
**Disclosure:** This code was developed with assistance from
deepseek-v4-pro (DeepSeek) via Hermes Agent (Nous Research). All changes
were reviewed, tested against the actual codebase, and verified for
correctness.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Expose the merged Rust OAuth header provider through the Python async
connection path.
Includes:
- Python OAuthConfig and OAuthFlowType public config objects
- PyO3 conversion into the Rust OAuthConfig
- connect_async(oauth_config=...) plumbing
- repr redaction coverage for client_secret
Local validation: cargo fmt --all; ruff format/check on touched Python
files.
By default the read freshness provider was not included in the namespace
client, preventing the read freshness headers from being included in the
request. This prevents checkout_latest() from working as expected when
using the namespace client.
This fix ensures the provided is built into the client when the
namespace impl and properties are provided.
Fixes#3563
## Summary
- Add `stacklevel=2` to 10 `warnings.warn()` calls across 4 files
- Fix broken message concatenation in `table.py` where the second string
was incorrectly passed as the `category` parameter
## Problem
Multiple `warnings.warn()` calls in the `python/lancedb/` codebase were
missing the `stacklevel` parameter. Without `stacklevel=2`, warnings
point to library internals instead of the caller's code, making it
impossible for users to identify which of their function calls triggered
the warning.
Additionally, two calls in `table.py` (lines 3411 and 3420) had a more
serious bug: the deprecation message was split across two separate
string arguments, causing the second string to be passed as the
`category` parameter instead of being concatenated with the first
string. This would cause `TypeError` when the warning was triggered.
## Changes
| File | Fixes | Description |
|------|-------|-------------|
| `embeddings/colpali.py` | 1 | Add `stacklevel=2` to
`use_token_pooling` deprecation warning |
| `remote/db.py` | 3 | Add `stacklevel=2` to `request_thread_pool`,
`connection_timeout`, `read_timeout` deprecation warnings |
| `remote/table.py` | 3 | Add `stacklevel=2` to `cleanup_old_versions`,
`compact_files`, `optimize` no-op warnings |
| `table.py` | 3 | Fix broken message concatenation for
`data_storage_version` and `enable_v2_manifest_paths` deprecation
warnings + add `stacklevel=2` to `retrain` deprecation warning |
## Verification
```python
# All warnings.warn() calls now have stacklevel
python3 -c "import ast, os; ..."
# Result: All warnings.warn() calls now have stacklevel!
```
## Changelog
| Date | Change | Author |
|------|--------|--------|
| 2026-06-20 | Fix missing stacklevel=2 in 10 warnings.warn() calls +
fix broken message concatenation | rtmalikian |
### Files Changed
- `python/python/lancedb/embeddings/colpali.py` — Add stacklevel=2
- `python/python/lancedb/remote/db.py` — Add stacklevel=2 to 3
deprecation warnings
- `python/python/lancedb/remote/table.py` — Add stacklevel=2 to 3 no-op
warnings
- `python/python/lancedb/table.py` — Fix broken message concatenation +
add stacklevel=2
### Verification
- AST-based audit confirms all `warnings.warn()` calls now include
`stacklevel=2`
- Syntax check passes for all 4 modified files
---
**About the Author:** Raphael Malikian — Clinical AI Solutions
Architect. I specialise in building and fixing AI/ML systems for
healthcare, including vector databases, RAG pipelines, and clinical NLP.
If you need help with your project or think I can add value to your
organisation, feel free to reach out — I'd love to connect.
📧rtmalikian@gmail.com🔗 GitHub: https://github.com/rtmalikian🔗 LinkedIn:
http://www.linkedin.com/in/raphael-t-malikian-mbbs-bsc-hons-71075436a
---
**Disclosure:** This code was developed with assistance from **Hermes
Agent** (Nous Research). All changes were reviewed, tested against the
actual codebase, and verified for correctness.
Signed-off-by: rtmalikian <rtmalikian@gmail.com>
Lance Release
Yang Cen
Andrew Chen
Eran Dagan
Mateusz Szewczyk
LanceDB Robot
Xuanwo
Drew Gallardo
Will Jones
Vitaliy
Tobias
Jack Ye
Prashanth Rao
kid
Mark McDonald
Pranav Achar
Unmilan Mukherjee
ForwardXu
Dan Rammer
Omkar Kabde
LAKSH JAIN
Weston Pace
Eric B
Armaan Sandhu
Neo-X7
Raphael Malikian
Ryan Green