`RemoteDBConnection.open_table` accepts `storage_options` and never uses
it:
```python
def open_table(
self,
name: str,
*,
namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None,
index_cache_size: Optional[int] = None,
...
) -> Table:
...
if index_cache_size is not None:
logging.info("index_cache_size is ignored in LanceDb Cloud ...")
table = LOOP.run(self._conn.open_table(name, namespace_path=namespace_path))
```
The value is never passed down and never mentioned. `index_cache_size`
is ignored on Cloud in the
same way, but it says so.
I checked this at runtime on 0.34.0, not just by reading it: swapping
the inner connection for a
recorder, `open_table("t", storage_options={...})` hands the layer below
`['namespace_path']` and
nothing else, no log record is emitted, and the same probe shows
`index_cache_size` producing its
message as expected.
This adds the matching log line, so the two ignored parameters behave
the same way. `ruff check` and
`ruff format --check` are clean on the file.
A note on severity. This is not a security hole and nothing is exposed.
Someone passing credentials
there gets silence instead of an error, and finds out later.
One thing I am unsure about, and it changes the fix. I have assumed
per-table storage options are
meaningless on Cloud, which is what the `index_cache_size` line next to
it implies about managed
storage. If they are supposed to work, then the right change is to pass
them through to
`self._conn.open_table` instead and this patch is the wrong one. Happy
to redo it that way.
I did not check whether `create_table` or the async connection have the
same gap.
`docs/src/python/python.md` is the whole Python API reference, but it is
maintained by hand and had drifted from the public API. Anything not
listed there simply doesn't get rendered, so a number of public,
documented, tested APIs were invisible to users — most notably branch
management, where `diff` and `merge` live.
I audited every public symbol reachable from `lancedb` and its
subpackages against the `:::` directives on the page. This adds the
missing ones:
- **Branching** — `Branches`, `AsyncBranches` (`list` / `create` /
`checkout` / `delete` / `diff` / `merge`)
- **Tables** — `TableStatistics` (returned by `Table.stats()`; the
fragment-level stats classes were already listed)
- **Full text queries** — `FullTextQuery`, `MatchQuery`, `PhraseQuery`,
`BoostQuery`, `MultiMatchQuery`, `BooleanQuery`, `FullTextOperator`,
`Occur`
- **Querying** — `LanceEmptyQueryBuilder`, `LanceTakeQueryBuilder`,
`AsyncTakeQuery`
- **Indices** — `Fm` (the FM-index for substring search), `IndexConfig`
- **Blobs** — `blob`, `BlobType`, `BlobFile`
- **Namespaces** — `connect_namespace`, `connect_namespace_async`, and
both namespace connection classes
- **Remote config** — `TlsConfig`, `HeaderProvider`, `OAuthConfig`,
`OAuthFlowType`
- **Rerankers** — the `Reranker` base class plus `JinaReranker`,
`RRFReranker`, `MRRReranker`, `AnswerdotaiRerankers`,
`VoyageAIReranker`, `WatsonxReranker` (5 of 12 were listed)
- **Embeddings** — `get_registry`, `register`, and the 14 embedding
functions that were missing (3 of 17 were listed)
- **PyTorch** — `StreamingDataset` and the permutation API it is built
on
- **Misc** — `Session`, `tokenize`, `FtsToken`, `pydantic.Vector`,
`pydantic.MultiVector`, `instrument_lancedb_metrics`, and the two
exception types
It also repairs cross-references in docstrings that no longer resolve:
links into guide pages that have since moved to lancedb.com
(`querying-an-ann-index`, `experimental-full-text-search`),
`lance.dataset` references with no inventory behind them, and the
relative targets `[Table](Table)` and `[PyArrow Table](pyarrow.Table)`.
Deliberately left out: concrete implementation classes reached through
their abstract base (`LanceTable`, `LanceDBConnection`,
`RemoteDBConnection`), query base classes already covered by
`inherited_members: true`, and internal plumbing such as
`FullTextSearchQuery` and `ColumnOrdering`.
## Testing
The docs job only runs on pushes to `main`, so I built the site locally
and compared against a build of `upstream/main`: every added entry
resolves, and no symbol that was rendered before stopped being rendered
when the four packages moved to automodule. `mkdocs build --strict`
exits 0 on this branch, against 61 warnings on `main`.
## Also in this PR
`lancedb.index`, `lancedb.embeddings`, `lancedb.remote` and
`lancedb.rerankers` are now rendered by a single mkdocstrings directive
each, driven by the module's `__all__`, rather than a hand-maintained
list. These four are where most of the drift was, and `__all__` is
harder to forget than a docs page. `lancedb.embeddings` had no
`__all__`; without one mkdocstrings renders no members at all for a
re-export package, so one is added. AGENTS.md gains a section on how the
page is wired up and how to build the docs locally.
Rendering all that code for the first time surfaced ~100 more build
warnings, which would have made #3707 (turning on `mkdocs build
--strict`) harder to land, so the warning backlog is cleared here too.
97 of the 158 warnings were one systematic false positive — griffe
cannot see the generated `__init__` of a pydantic dataclass, so every
documented parameter looks unknown — switched off via
`warn_unknown_params`. The remaining 61 came from 15 docstrings with
real bugs: prose trailing a `Parameters` section (we were rendering
parameters called `The`, `you` and `To`), types dropped because numpydoc
needs spaces around the colon, `num_partitions, default sqrt(num_rows)`
parsing as a list of names and inventing a `default` parameter, and one
parameter indented five spaces. `mkdocs build --strict` now exits 0.
---
#3747 (the coverage test that keeps this from happening again) is
stacked on this branch, so review it after this one.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
In the current LanceDB usage implementation, there is no way to check
whether a table or namespace already exists. This PR introduces the
namespace_exists and table_exists methods to determine the existence of
tables and namespaces.
useage like this:
```
# check table exists
db.table_exists(table_id=['xxx'])
# check namespace exists
db.namespace_exists(namespace_id=['xxx'])
```
fixes: #3419
---------
Signed-off-by: farmer <farmerchillax@outlook.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Remote half of the blob read path. #3578 did local Python. This makes
`RemoteTable` hit the server.
- `fetch_blobs(column, row_ids or hits)` → bytes over `POST
/v1/table/{id}/fetch_blobs/`
- `blob_columns()` from the cached schema (describe already has the
metadata, no extra route)
- search then `fetch_blobs` works. row identity rides inside the blob
descriptor so you do not need a public `_rowid`
- `fetch_blob_files` still `NotSupported` on remote. use `fetch_blobs`
for full bytes for now. Range is a follow up
Accepts Binary / LargeBinary / BinaryView on the way back. Empty
`row_ids` short-circuits. Version + branch go in the request body same
as other read calls.
### Example
```python
db = lancedb.connect(uri="db://my-project", api_key=...)
table = db.open_table("clips")
hits = table.search(query_vec).select(["id", "video"]).limit(10).to_arrow()
# hits is just id + video. row ids are stashed on the descriptor
blobs = table.fetch_blobs("video", hits) # null-aligned, same length as hits
```
Or pass ids yourself:
```python
blobs = table.fetch_blobs("video", [10, 20, 30])
```
### Testing
- `cargo test -p lancedb --features remote --lib`
- `cargo test -p lancedb --features remote --test blob_integration`
- `pytest python/tests/test_remote_db.py -k remote_blob`
- live e2e against a local 0.5.0 remote server (search → fetch, nulls,
nested path, old server gate)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- keep the existing synchronous `connect()` path unchanged
- make `LanceDBConnection.__repr__` and `LanceTable.__repr__`
side-effect-free
- add a regression test that verifies sync reprs do not call the Python
background loop
## Root cause
The freeze is caused by debugger rendering, not by `connect()` itself:
1. debugpy stops at a breakpoint and suspends all Python threads.
2. The debugger renders the new `db_connection` local by calling
`repr()`.
3. `LanceDBConnection.__repr__` reads `read_consistency_interval`.
4. That property calls `LOOP.run(...).result()`.
5. The `LanceDBBackgroundEventLoop` thread is suspended by the debugger,
so `repr()` waits for a thread that cannot run.
This explains why the symptom appears immediately after `connect()`: it
is the first point where a connection object exists in locals and is
automatically rendered. `LanceTable.__repr__` had the same problem
because it also read the connection's consistency interval.
This follows the same principle as #3411: `__repr__` must not trigger
async work or I/O that a debugger assumes is lightweight.
## Evidence
I reproduced the behavior with the real LanceDB classes and debugpy
1.8.21 using a DAP client:
- latest `main` (`ff6ff099`): the debugger reported `allThreadsStopped:
true`, and evaluating `repr(db_connection)` timed out
- this branch (`5755a5ba`): the same evaluation returned
`LanceDBConnection(uri='/tmp/lancedb-debug-repro')` immediately
- setting `PYDEVD_UNBLOCK_THREADS_TIMEOUT=0` also allowed the original
repr path to complete, independently confirming that it was waiting on a
suspended thread
The regression test creates a connection and table, replaces `LOOP.run`
with a function that fails, and verifies that both reprs still work.
## Validation
- `maturin develop --manifest-path python/Cargo.toml`
- `python -m pytest
python/python/tests/test_db.py::test_sync_repr_does_not_use_background_loop
python/python/tests/test_table.py::test_consistency -q` (`4 passed`)
- `ruff check .`
- `ruff format --check python/python/lancedb/db.py
python/python/lancedb/table.py python/python/tests/test_db.py
python/python/tests/test_table.py`
- `git diff --check`
Refs #3611.
## What
Expose custom FTS stop-word lists in the Python and TypeScript public
APIs, including their standalone tokenize helpers and remote index
creation.
This PR supports concrete string lists only. It does not add file or
LanceDB-table stop-word sources.
## Why
Rust already exposes Lance's custom stop-word list option. The Python
and TypeScript APIs did not pass it through, and local index details did
not retain the full tokenizer parameters needed by index-backed
tokenization after reopening a table.
## How
- Add `custom_stop_words` / `customStopWords` to the Python and
TypeScript FTS and tokenize options.
- Preserve `None` / `undefined`, empty lists, and list contents without
normalization.
- Load the persisted FTS segment parameters when returning local index
details.
- Serialize the concrete list in remote create-index requests.
- Keep Python and TypeScript tests thin; behavior, persistence, query
tokenization, and remote JSON coverage live primarily in Rust.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
- Python extension rebuild with `uv` and `maturin`
- Targeted Python tests: 4 passed
- Python `ruff format --check` and `ruff check`
- TypeScript build, typecheck, Biome lint, generated docs, and targeted
tests
---------
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
## Summary
- add a keyword-only `transform_parallelism` option to
`StreamingDataset`
- preserve CPU auto-detection by default and fall back to one worker
when unavailable
- apply the configured limit to both the transform executor and
concurrency semaphore
- document and test explicit, default, fallback, and invalid values
## Testing
- `uv run --extra tests --with torch pytest
python/tests/test_elastic_dataloader.py -q` (`136 passed`)
- `uvx ruff check python/lancedb/streaming.py
python/tests/test_elastic_dataloader.py`
- `uvx ruff format --check python/lancedb/streaming.py
python/tests/test_elastic_dataloader.py`
- `git diff --check origin/main...HEAD`
Closes#3695
Co-authored-by: buduoqiu <yaodong-shen@users.noreply.github.com>
## Summary
Lance can now plan multiple byte ranges for the same blob in one
`read_blob_ranges` operation, but LanceDB users currently cannot expose
a complete set of logical ranges to that planner.
This complements `BlobFile`: file-like consumers such as PyAV can
continue to discover ranges dynamically, while callers that already know
the ranges for a batch can submit them together.
## Motivating example
A training table may store a large video blob together with a small
application-level clip index:
```text
video: blob
clips: [{offset, length}, ...]
```
The caller can select the videos and clips for a batch, obtain their row
IDs from the query, and read all of the selected windows together:
```python
rows = (
table.search()
.select(["clips"])
.with_row_id(True)
.limit(64)
.to_arrow()
.to_pylist()
)
requests = []
for row in rows:
clip = sample_clip(row["clips"])
requests.append(
(row["_rowid"], clip["offset"], clip["length"])
)
chunks = table.fetch_blob_ranges("video", requests)
```
Here, `_rowid` comes from the LanceDB query, while `offset` and `length`
come from the application's clip index and are relative to that row's
video blob. The caller describes only the logical reads; Lance still
handles validation, source grouping, coalescing, scheduling, and byte
backpressure.
Lance v10.0.0-beta.5 returns one logical result per blob selector or
range request and explicitly distinguishes null blobs from valid empty
values. LanceDB consumes that aligned result contract directly and only
adds a cardinality check for unresolved row IDs.
This PR exposes batched blob-range reads on local Rust and Python
tables. Results preserve request identity, duplicates, null slots, and
valid empty ranges while allowing Lance to execute the physical reads
out of order. Scheduler buffer sizing remains an internal Lance concern,
so the LanceDB API does not expose `io_buffer_size`.
Cloud tables continue to report this operation as unsupported until
there is a corresponding remote API.
## What
MemWAL LSM **read** support. When a table has an LSM write spec
(`set_lsm_write_spec`), `merge_insert` upserts live in the MemWAL
active/frozen memtables and flushed SSTables until an external
compaction merges them into the base table, so a normal scan returns
**stale** data. This routes reads through Lance's `LsmScanner` so
queries also surface that in-flight data, deduplicated by primary key
(newest generation wins).
## How
- Adds a **`use_lsm: Option<bool>`** query flag, symmetric with the
`merge_insert` flag:
- **unset** — auto-route through the LSM scanner when the table carries
a write spec
- **`use_lsm(true)`** — force the LSM path; error if there is no spec
- **`use_lsm(false)`** — read the base table only (the escape hatch)
- Plain scan, single-column full-text search, and single-vector ANN all
run through one `LsmScanner` (assembled from on-disk shard manifests
plus the cached writer's in-memory memtables), so a `where` predicate is
honored as a **prefilter** uniformly — including for vector search.
- **Compaction-aware snapshots:** an SSTable generation is dropped only
once it is both compacted into the base table and covered by the arm's
base-index catch-up (`index_catchup`); plain scans use the compaction
watermark alone.
- Query shapes the scanner cannot honor hard-error with guidance to set
`use_lsm(false)`: hybrid, multi/binary vectors, `with_row_id`,
reranking, `order_by`, dynamic/Substrait projection or filters,
`distance_range`, `use_index(false)`, postfilter, take-by-row-id/offset,
reads from a time-traveled version, and an unmaintained or ambiguous
(multiple) FTS/vector index. Namespace-pushdown queries fall back to
local execution when a spec is present; WAL-only writers are handled.
- Exposed across the Rust core and the Python (`use_lsm`) and TypeScript
(`useLsm`) bindings, including `TakeQuery`.
Rebased from Lance `7.2.0-beta.3` to `10.0.0-beta.3`.
Python was versioned and tagged separately from the Rust, Java, and
Node.js SDKs, and had drifted three minor versions ahead (0.36 vs 0.33).
Users had no way to tell which Python version corresponded to which Rust
or Node release, and the gap had no meaning behind it.
This unifies the two tracks so there is one version and one tag for all
four SDKs.
## Version
The shared version is set to `0.37.0-beta.0`. Python continues its own
sequence (highest published: 0.36 → 0.37) while Rust, Java, and Node.js
jump 0.33 → 0.37 to meet it. Picking Python's next minor means Python
users see no discontinuity at all, and only the other SDKs skip forward.
Note that `main` trails the `release/v0.32` branch on both lines (main
is at 0.32.0-beta.3 / 0.35.0-beta.3; the release branch carries
0.33.0-beta.0 / 0.36.0-beta.0), so 0.37 is chosen to clear the highest
tag on either branch. Every index stays monotonic:
| index | publishes | last published | next |
|---|---|---|---|
| PyPI | stable only | 0.34.0 | 0.37.0 |
| Fury | previews | 0.36.0b0 | 0.37.0-beta.1 |
| npm | both | 0.33.0-beta.0 | 0.37.0-beta.1 |
| crates.io | stable only | 0.31.0 | 0.37.0 |
| Maven | both | 0.33.0-beta.0 | 0.37.0-beta.1 |
A one-time jump for three SDKs, versus explaining the offset
indefinitely.
## Mechanism
* `python/.bumpversion.toml` is removed. `python/Cargo.toml` — the
source of the Python package version, since `pyproject.toml` declares
`dynamic = ["version"]` — becomes a tracked file of the root config. Its
`cargo update -p lancedb-python` pre-commit hook is dropped as
redundant: `ci/update_lockfiles.sh` already refreshes every workspace
member version in `Cargo.lock`.
* `pypi-publish.yml` triggers on `v*` instead of `python-v*`, so one tag
releases all four packages. `ci/bump_version.sh` and
`make-release-commit.yml` lose their now-dead tag-prefix and
per-language plumbing, including the `python` / `other` dispatch inputs.
* The two byte-identical GH release jobs in `npm-publish.yml` and
`pypi-publish.yml` are replaced by a single `gh-release.yml`. One
release per tag, named `LanceDB vX.Y.Z`, instead of separate "Python
LanceDB" and "Node/Rust LanceDB" releases for the same commit.
The trade-off: there is no longer a way to ship a Python-only patch
without also releasing crates.io, Maven, and npm. That is the cost of
making drift structurally impossible.
## Beta releases marked "Latest" (#3666)
Both GH release jobs used:
```yaml
prerelease: ${{ contains('beta', github.ref) }}
```
The arguments are reversed. `contains(search, item)` asks whether
*`search`* contains *`item`*, so this evaluated "does the literal string
`'beta'` contain `refs/tags/python-v0.35.0-beta.2`?" — always `false`.
Every beta was published as a full release, and GitHub awards "Latest"
to the newest non-prerelease.
The new workflow derives the flag from the parsed version rather than
the raw ref, and sets `make_latest` explicitly:
```yaml
prerelease: ${{ steps.extract_version.outputs.prerelease }}
make_latest: ${{ steps.extract_version.outputs.prerelease == 'false' }}
```
npm was never affected (`--tag preview` uses correct bash), and PyPI
already excludes pre-releases from resolution.
This only fixes releases published from here on. Already-published betas
need a one-time backfill:
```shell
gh api --paginate /repos/lancedb/lancedb/releases \
--jq '.[] | select(.prerelease == false) | select(.tag_name | test("beta")) | .id' \
| xargs -I{} gh api -X PATCH /repos/lancedb/lancedb/releases/{} -F prerelease=true
```
## Verification
Ran `ci/bump_version.sh` end-to-end against this branch with the release
tooling installed:
* `preview` → tags `v0.37.0-beta.1` (previous tag `v0.33.0-beta.0`
detected, `pre_n` bump)
* `stable` → tags `v0.37.0`
* Both paths update `.bumpversion.toml`, `rust/lancedb/Cargo.toml`,
`nodejs/Cargo.toml`, `python/Cargo.toml`, `nodejs/package.json`, the 7
`nodejs/npm/*/package.json` files, both Java poms, and
`docs/src/java/java.md` together
* `check_breaking_changes.py` resolves the last stable as `v0.31.0`, so
the minor-version gate passes
All five touched workflows parse as valid YAML and the pre-commit hooks
pass.
## Notes for review
* This targets `main` only, so it takes effect at the next
release-branch cut. The in-flight `release/v0.32` branch still carries
`v0.33.0-beta.0` / `python-v0.36.0-beta.0`; if we want the imminent
stable to be 0.37.0, this needs to be applied there too.
* Historical `python-v*` tags are left alone. The changelog builder
scans `^v`, which does not match them, so the first unified release's
notes will compute `fromTag` from the Rust/Node line only — a one-time
gap in the Python-side changelog.
* Pre-existing and not addressed here: `ci/update_lockfiles.sh --amend`
amends the commit that `bump-my-version` has already tagged, so the
lockfile update lands outside the tag on stable releases.
Fixes#3666
## What changed
- add `block_size` to Python FTS configuration and the deprecated
local/remote helpers
- add `blockSize` to the TypeScript FTS options and propagate it through
the NAPI binding
- serialize the value as `block_size` for remote index creation
- document the existing Rust builder API and generate the TypeScript API
reference
- add local, remote, metadata, search, and invalid-value regression
coverage
## Why
Lance supports configuring the number of documents per compressed FTS
posting block, but LanceDB's Python and TypeScript APIs did not expose
the setting. This made the experimental FTS V3 layout unavailable
through those clients and allowed the value to be dropped before index
creation.
## How it works
The default remains `128`. Supported values are `128` and `256`;
selecting `256` uses the experimental FTS V3 format. Invalid values are
rejected by the Lance builder and surfaced as Python or JavaScript
errors.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo +1.94.0 clippy --quiet --features remote --tests --examples --
-D warnings`
- targeted Rust local and remote index tests
- Rust doctests: 34 passed
- Python Ruff checks, doctest, and targeted local/remote tests: 5 passed
- TypeScript build, Biome lint, generated docs, and targeted Jest tests:
9 passed
- `git diff --check`
## Limitations
The Java client remains unchanged because its external remote REST model
does not currently expose `block_size`.
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
## What
`AnswerdotaiRerankers(return_score="all").rerank_hybrid(...)` (and
`ColbertReranker`, which subclasses it without overriding
`rerank_hybrid`) raises:
```
pyarrow.lib.ArrowInvalid: Invalid sort key column: No match for FieldRef.Name(_relevance_score) in _rowid: int64 ...
```
## Why
```python
combined_results = self.merge_results(vector_results, fts_results)
combined_results = self._rerank(combined_results, query)
if self.score == "relevance":
combined_results = self._keep_relevance_score(combined_results)
elif self.score == "all":
combined_results = self._merge_and_keep_scores(vector_results, fts_results)
```
When `score == "all"`, `combined_results` is unconditionally overwritten
by `_merge_and_keep_scores(vector_results, fts_results)` **after**
`_rerank()` already computed and appended `_relevance_score` —
discarding it. The following `sort_by("_relevance_score", ...)` then has
nothing to sort on.
Every sibling reranker that supports `return_score="all"`
(`cross_encoder`, `openai`, `cohere`, `jinaai`, `voyageai`, `watsonx`)
instead calls `_merge_and_keep_scores()` **before** `_rerank()`. This
file is the one place the ordering got inverted when `"all"` support was
added (#2509) — a copy/paste inconsistency across the six files that PR
touched. Fix mirrors the pattern already used (and tested) by the other
five rerankers.
Also drops the now-stale `"Only 'relevance' is supported for now"`
docstring line on both classes, left over from before `"all"` support
existed.
## Testing
Added `test_answerdotai_reranker_return_all`, mirroring the existing
`test_cross_encoder_reranker_return_all`. Verified locally with the real
built Rust extension: red (reproduces the exact `ArrowInvalid` above) →
green, using the actual `rerank_hybrid`/`_rerank`/`base.py` code path
with the model call mocked out — my local environment's
`rerankers==0.10.0` fails to load the real ColBERT model against the
available `transformers` version (`AttributeError: 'ColBERTModel' object
has no attribute 'all_tied_weights_keys'`), which I confirmed also
breaks the **pre-existing**, unmodified
`test_colbert_reranker`/`test_answerdotai_reranker` baseline tests
identically — an unrelated local dependency-version issue, not a
regression from this change. `ruff check`/`ruff format` clean; full
`test_rerankers.py` run: 9 passed / 8 skipped / 3 failed (the 3 failures
are exactly those two pre-existing tests plus my new one, all failing at
model-loading time for the same unrelated reason before reaching the
changed code).
---
Disclosure: this PR was drafted with AI assistance (Claude); I reviewed,
tested, and take responsibility for the change.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
- Replace legacy model names in `WatsonxEmbeddings` with the current
supported set:
- `ibm/granite-embedding-278m-multilingual` (new default, 768-dim)
- `ibm/slate-125m-english-rtrvr-v2` (768-dim)
- `ibm/slate-30m-english-rtrvr-v2` (384-dim)
- `intfloat/multilingual-e5-large` (1024-dim)
- `sentence-transformers/all-minilm-l6-v2` (384-dim)
- Add `space_id` field — mutually exclusive with `project_id`, mirrors
the
existing pattern in `WatsonxReranker`
- `project_id` / `space_id` resolution now falls back to
`WATSONX_PROJECT_ID` /
`WATSONX_SPACE_ID` env vars; exactly one must be supplied
## Why
The previously hardcoded models (`ibm/slate-125m-english-rtrvr`,
`sentence-transformers/all-minilm-l12-v2`) are legacy and no longer
listed as
supported by the watsonx.ai platform. `space_id` scoping was already
supported
by `WatsonxReranker` but was missing from the embeddings counterpart.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
This PR adds some support for `diff` / `merge` in the remote client as
for local tables we stay `NotSupported` until
https://github.com/lance-format/lance/issues/7263.
This wires the two review-and-land calls against the remote REST API:
- `POST /v1/table/{id}/branches/diff`
- `POST /v1/table/{id}/branches/merge`
Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python
returns the wire JSON, same shape as the REST response.
Merge here means promoting a branch's added columns onto `main`.
### Behavior
- Remote only. Local raises `NotSupported`.
- A rejected merge is not an exception. HTTP 409 still returns `Ok` / a
dict with `status="rejected"` and blockers in `diff.mergeBlockers`.
- Unknown blocker / status codes parse as `Unknown` so a newer server
does not break older clients.
- `MergePreview` tolerates missing fields for the same reason.
- Merge requests are not retried. 409 is final and carries the body you
need.
### Example
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.merge("exp", dry_run=True)
result = table.branches.merge("exp", dry_run=False)
if result["status"] == "merged":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "rejected":
print(result["diff"]["mergeBlockers"])
```
### Testing
cargo test -p lancedb --features remote diff_branch
cargo test -p lancedb --features remote merge_branch
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
## Problem
On the remote (LanceDB Cloud) write path, each write partition is
uploaded as a **single** `/insert?upload_id=...` request that stays open
until the whole partition has been streamed and the server has written
it to object storage. For large bulk ingests a partition can be many GB,
so a single request can run longer than the client read timeout (default
300s), surfacing as:
```
lancedb.remote.errors.HttpError: operation timed out
```
The server already supports staging **multiple** parts under one
`upload_id` (each `/insert` writes a separate transaction that
`complete` merges atomically), but the client never used that — it sent
one part per partition.
## Change
Split each partition into multiple parts of at most
`max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded
as its own `/insert?upload_id=...&upload_part_id=...` request. This
bounds how long any single request stays open, independent of total data
size or write parallelism.
Key properties:
- **Still streamed, not buffered.** Each part's body is driven through a
bounded channel while the request is in flight (`futures::join!` of a
producer + the send), so peak memory stays at a couple of batches per
partition regardless of the part size. Backpressure from a
slow/throttled server still propagates upstream.
- **Correct part accounting.** An empty partition still sends exactly
one (schema-only) part so `complete` has a transaction to commit; a size
cut landing exactly on the end of input does not emit a trailing empty
part.
- **Multipart only.** The single-request (non-multipart) path is
unchanged.
## Config
New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable
via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable.
**Default 1 GiB** (`Some(0)` disables splitting → one request per
partition). Python users pick up the default/env automatically through
the remote client.
## Tests
- `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each
batch in its own part → N requests, each carrying the shared `upload_id`
and a distinct `upload_part_id`.
- `test_multipart_single_part_when_under_budget`: a large budget keeps
the partition in a single request.
- Verified end-to-end against a live remote table: a forced-chunked
multipart add (many parts) assembles to the correct row count.
Related to ENT-1883.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Fixes#1653.
`infer_vector_column_name` in `util.py` could silently return `None`
when `query is None` and `query_type` is not `"fts"` or `"hybrid"`. This
`None` then propagated into downstream code, causing a cryptic
`TypeError: expected bytes, NoneType found` rather than a clear error
message.
## Changes
- **Removes the no-op `try/except Exception as e: raise e`** around
`inf_vector_column_query` (it was catching and immediately re-raising
without adding any value)
- - **Adds a `None` guard** after the inference block: if
`vector_column_name` is still `None` at this point, raise a clear
`ValueError` pointing the user to pass `vector_column_name` explicitly
## Before / After
**Before:** cryptic `TypeError: expected bytes, NoneType found` deep in
schema lookup code
**After:**
```
ValueError: No vector column found in the schema. Please specify the vector column name explicitly via the `vector_column_name` parameter.
```
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
## Problem
`table.add(dataset)` with a `pyarrow.dataset.Dataset` OOMs the client
during bulk ingestion of wide rows (e.g. embedding columns), even
against a remote table where the upload itself is streaming.
The cause is in `to_scannable`: a `Dataset` is scanned with pyarrow's
default scanner settings (`batch_size=131072` rows,
`batch_readahead=16`, `fragment_readahead=4`). pyarrow's internal
threads prefetch that read-ahead window independently of LanceDB's
backpressure, so for wide rows a large fraction of the dataset is held
in memory. On the remote path this is then multiplied across the
multipart write partitions (one in-flight batch per partition, up to
CPU-core count).
Reproduced on a 10 GB / 1.55M-row dataset with two 768-dim float32
embeddings: peak client RSS ~11.7 GB for the scan alone (6.8 GB after
consuming a *single* batch), ~15.4 GB for the full remote `add()`.
## Fix
`to_scannable` now sizes the scanner from an estimate of bytes-per-row
derived from the schema:
- **Narrow datasets keep pyarrow's defaults** (empty scanner kwargs) —
no throughput regression. The bound only engages above ~410 bytes/row.
- **Wide rows** get a smaller `batch_size` (~16 MiB/batch) and reduced
read-ahead (`batch_readahead=2`, `fragment_readahead=1`) so peak
in-flight memory stays near a ~1 GiB budget. Read-ahead (not just batch
size) has to drop, because pyarrow pins whole row-group buffers.
On the 10 GB dataset this drops peak client RSS to ~1.4 GB, and it stays
flat as the dataset grows. The `Dataset`/`LanceDataset` scannables
remain rescannable (retry-safe).
## Also: expose `write_parallelism` on `add()`
`AddDataBuilder::write_parallelism` already existed in Rust but was not
exposed in Python. This PR forwards it through the async, sync, and
remote `add()` methods, so users can cap the number of parallel write
partitions (each buffers data in flight) to trade throughput for memory
on large uploads.
## Tests
- `test_scannable.py`: bytes-per-row estimation; narrow → defaults; wide
→ bounded; `Dataset` reader streams bounded batches and stays
rescannable.
- `test_table.py`: `write_parallelism` on sync and async `add()`, and
that `write_parallelism=0` is rejected.
Fixes ENT-1883
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tracks #3324. On x86_64 CPUs without AVX2 (Sandy Bridge / Ivy Bridge /
Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD), `import
lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled
function. Per [westonpace's
review](https://github.com/lancedb/lancedb/issues/3324#issuecomment-4328944354),
the default `lancedb` wheel stays fast; pre-Haswell users get a
separately-published `lancedb-compat` wheel.
## Summary
- Adds a `lancedb-compat` matrix entry to `pypi-publish.yml` that builds
with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (Nehalem-class baseline).
Same Python API (`import lancedb` works) — files install to the same
namespace, so the two wheels conflict at install time and users pick
one. Same pattern as `psycopg2` / `psycopg2-binary` and `tensorflow` /
`tensorflow-cpu`.
- Generalizes `build_linux_wheel` and `upload_wheel` composites with
optional `package-name` and `rustflags` inputs (defaults preserve the
existing 4 `lancedb` matrix entries verbatim).
- Documents the choice in `python/README.md`: `pip install
lancedb-compat` for pre-Haswell hosts.
The default `.cargo/config.toml` baseline is unchanged.
## Sequencing
1. ~~lance-format/lance#6630 merges → runtime SIMD dispatch lands in
lance.~~ **Done — merged.**
2. lancedb's lance dep is bumped to a release that includes it (separate
PR / normal cadence).
3. This PR's `lancedb-compat` wheel build path starts producing a wheel
that runs on pre-Haswell hardware. **Maintainer setup**: register
`lancedb-compat` on PyPI and configure trusted publishing.
## Verified end-to-end on Sandy Bridge Xeon E5-2609
Verification was done locally against a fork-pinned lance dep that
includes the runtime dispatch implementation, using the same
`RUSTFLAGS="-C target-cpu=x86-64-v2"` flags this PR uses in CI:
```
$ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release
$ pip install ./target/wheels/lancedb-*.whl
$ python verify.py
PASS: import + simd dispatch + table create + vector search all work.
```
Pre-fix on the same CPU (default `pip install lancedb`): `Illegal
instruction (core dumped)`. Full reproducer (deps + clone + build +
verification):
https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df.
Fork-internal verification PR with the dep bump and full logs:
[`tobocop2/lancedb#2`](https://github.com/tobocop2/lancedb/pull/2).
## Benchmarks — no regressions on modern CPUs from the lance-side change
These are the numbers I ran for the lance PR, confirming the runtime
dispatch doesn't slow down the default (`target-cpu=haswell`) wheel that
existing users install. Criterion, one machine, one session, base → PR,
no `RUSTFLAGS` override. Full methodology, null experiments, and logs:
[lance-format/lance#6630 benchmark
comment](https://github.com/lance-format/lance/pull/6630#issuecomment-4933063394)
and the [logs
gist](https://gist.github.com/tobocop2/3c6d0f449cbd736aa2501f89a7fe56a2).
| benchmark | EPYC 7B13 (`avx2`, `fma`, no `avx512f`) | Xeon Cascade
Lake (`avx512f`) |
|---|---|---|
| `Cosine(f32, scalar)` *(control)* | +0.04% | +0.09% |
| `Cosine(f64, scalar)` | −0.34% | −1.94% |
| `Cosine(u8, SIMD)` | +2.30% | +3.63% |
| `Dot(f16, SIMD)` | −0.58% | +0.61% |
| `Dot(f32, SIMD)` | +0.34% | **−6.08%** |
| `Dot(f32, arrow_arity)` | +0.02% | −0.00% |
| `L2(f32, scalar)` | −0.10% | −0.02% |
| `L2(f32, simd)` (dim 1024) | +2.63% | −0.53% |
| **`L2(simd,f32x8)` (dim 8)** | **−45.9%** | **−25.1%** |
| `L2(u8, SIMD)` | +0.42% | −3.11% |
| `NormL2(f32, SIMD)` | −1.02% | −4.17% |
| `NormL2(f64, SIMD)` | +3.51% | −0.58% |
Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector
width — improves 25–46%.
---
To be transparent: this isn't my domain of expertise and the lance-side
implementation is AI-generated. I verified it works end-to-end on the
failing hardware. Happy to roll in feedback.
Routes local sync child-namespace operations through the Rust-backed
connection instead of the Python namespace-client fallback.
Also keeps lazy namespace-client construction for table-to-Lance
conversion and preserves public namespace error mappings.
Validated locally with ruff format/check and targeted namespace pytest.
BREAKING CHANGE: splits generated by the permutation data loader will
not be the same, due to a change in hash function.
Updates the Lance dependencies and Java lance-core to
[v9.0.0-rc.1](https://github.com/lance-format/lance/releases/tag/v9.0.0-rc.1).
Includes the required DataFusion 54 and Lance file-reader compatibility
updates.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds client-side support for analyze_plan distributed metrics modes
across Rust, Python, and TypeScript clients. Defaults to aggregate for
backward compatibility and sends the remote distributed_metrics
parameter only when a non-default mode is requested.
## Summary
Fix `on_bad_vectors="fill"` so it replaces only invalid or missing
vector values instead of replacing the entire vector row.
Fixes#3026.
## Reasoning
The old Python sanitizer detected whether a vector row was bad at row
granularity. For `fill`, it then used that row-level flag to replace the
whole vector with `[fill_value] * dim`. That meant an input like `[1.0,
NaN, 3.0]` became `[0.0, 0.0, 0.0]`, even though the documented and more
useful behavior is to preserve valid values and fill only the bad
element.
I checked whether this should be a Rust-side fix so TypeScript users
would benefit too. Today, Rust core exposes `NaNVectorBehavior::{Error,
Keep}` for rejecting or keeping NaN vectors, while the Python
`on_bad_vectors` API (`error`, `drop`, `fill`, `null`) is implemented in
the Python ingestion sanitizer before data reaches Rust. TypeScript does
not expose the Python `on_bad_vectors="fill"` behavior today. Moving
this exact behavior to Rust would be a broader cross-language API
change, so this PR keeps the fix scoped to the currently affected Python
API.
## What changed
- Added a small helper that fills bad vector rows by preserving valid
elements, replacing NaN elements with `fill_value`, truncating vectors
longer than the expected dimension, and padding short vectors with
`fill_value`.
- Kept the existing fast path unchanged: the helper only runs after bad
vectors are detected and `on_bad_vectors="fill"` is selected.
- Updated sanitizer and table tests to assert element-wise NaN
replacement and short-vector padding for both `create_table` and `add`.
## Validation
- `uv run ruff format .`
- `uv run ruff check .`
- `cd python && uv run --no-sync pytest
python/tests/test_util.py::test_handle_bad_vectors_jagged
python/tests/test_util.py::test_handle_bad_vectors_nan
python/tests/test_table.py::test_create_with_nans
python/tests/test_table.py::test_add_with_nans -vv`
Targeted pytest result: `10 passed`.
## Why this fix is Python-side (and not Rust)
The problematic behavior lives in Python’s `on_bad_vectors` sanitizer,
before data is handed off to Rust. Rust currently only exposes
`NaNVectorBehavior::{Error, Keep}` for add operations, while Python has
the richer `on_bad_vectors={"error","drop","fill","null"}` API.
TypeScript does not currently expose the Python-style fill behavior, so
moving this exact fix into Rust would require designing a broader
cross-language bad-vector handling API.
This PR keeps the change scoped to the existing affected surface:
Python’s `on_bad_vectors="fill"` path. This way, Python users
immediately benefit.
## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage
## Summary
- serialize sync phrase queries consistently for execution and query
plans
- restore the documented no-argument hybrid `phrase_query()` behavior
- keep reranker input as the original user text without mutating the
builder
Fixes#3653.
## Testing
- `python/.venv/bin/python -m pytest <8 focused test nodes> -q` (`8
passed`)
- `python/.venv/bin/python -m ruff format --check
python/python/lancedb/query.py python/python/tests/test_fts.py
python/python/tests/test_hybrid_query.py`
- `python/.venv/bin/python -m ruff check .`
- `git diff --check origin/main...HEAD`
The complete hybrid module and the real native FTS phrase test were not
completed
in the current PyO3 runtime environment: both stalled in the native
`lancedb.connect()` fixture and were interrupted without an assertion
failure.
Carrying over from #2915, this patch introduces:
* Single-API call batching support for Gemini embeddings (up to 100 at a
time, the API limit)
* A versioned user agent header for Gemini API calls
* Support for [variable embedding dimension
size](https://ai.google.dev/gemini-api/docs/embeddings#control-embedding-size)
(Gemini is MRL trained)
## Summary
- preserve explicit `0.0` distance bounds in synchronous hybrid search
- distinguish omitted `None` endpoints from zero-valued endpoints when
configuring the vector child query
- add a public end-to-end regression test for a zero upper bound
## Testing
- `cd python && uv run --extra tests pytest
python/tests/test_hybrid_query.py -q`
- `uv run --project python ruff format --check
python/python/lancedb/query.py python/python/tests/test_hybrid_query.py`
- `uv run --project python ruff check .`
Fixes#3651
Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.
## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.
## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.
## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.
Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.
## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps Lance to v9.0.0-beta.19, which includes lance-format/lance#7687
for side-effect-free DirectoryNamespace read paths.
This fixes root-level read-only table opens that previously could
trigger `__manifest` creation through directory namespace construction,
including Hugging Face bucket reads with read-only tokens. A LanceDB
regression test now covers root listing operations without creating
`__manifest`.
Fixes#3633.
### Summary
`flatten_columns` raises `ValueError` when called with `flatten=False`,
even though `False` should mean "do not flatten". This is reachable from
the public API — `Query.to_pandas(flatten=...)` and
`to_batches(flatten=...)` type their `flatten` param as
`Optional[Union[int, bool]]` and pass it straight to `flatten_columns`.
### Cause
`bool` is a subclass of `int`, so `isinstance(False, int)` is `True`.
`flatten=False` skips the `flatten is True` check, falls into the
integer branch, and `False <= 0` evaluates to `True`, raising:
```
ValueError: Please specify a positive integer for flatten or the boolean value `True`
```
### Reproduction
```python
import lancedb
db = lancedb.connect("/tmp/db")
t = db.create_table("t", data=[{"id": 1, "vector": [0.1, 0.2]}])
t.search([0.1, 0.2]).to_pandas(flatten=False) # -> ValueError
```
### Fix
Guard the integer branch with `not isinstance(flatten, bool)` so that
`flatten=False` (and `None`) mean "do not flatten". Behavior is
otherwise unchanged:
- `flatten=True` → flatten all nested levels
- positive `int` → flatten to that depth
- non-positive `int` (e.g. `0`) → still rejected with `ValueError`
Added a regression test in `tests/test_util.py` covering `None`,
`False`, `True`, a positive depth, and `0`.
This PR fixes a serialization error when using Ollama embeddings in
`create_table`.
The use of `@cached_property` for the Ollama client was causing issues
during serialization/pickling, which is required by certain LanceDB
operations (like when using multiprocessing or certain storage
backends). Switching to a standard `@property` ensures the client is
instantiated when needed without being stored in a way that breaks
serialization.
Verified with the following script:
```python
import lancedb
from lancedb.embeddings import get_registry
import pickle
registry = get_registry().get(\"ollama\")
model = registry(name=\"llama3\")
# This would fail before the fix
pickled = pickle.dumps(model)
unpickled = pickle.loads(pickled)
```
Fixes#2629 (or similar serialization issues reported).
---------
Co-authored-by: Unmilan Mukherjee <Missing-Identity@users.noreply.github.com>
## Summary
Closes#3525
This PR wires up two new optional object-store backends at the LanceDB
layer, exposing capabilities that already exist upstream in `lance` /
`lance-io`:
| Backend | Cargo feature | Default in Rust crate | Default in Python
wheel | Default in Node binding |
| --- | --- | --- | --- | --- |
| **Tencent COS** | `cos` | ❌ off | ✅ on | ❌ off |
| **GooseFS** | `goosefs` | ❌ off | ✅ on | ✅ on |
Both backends are additive and do not affect existing users who don't
opt in.
## Motivation
- **Tencent COS** is the dominant object storage in the China region.
Tencent Cloud users currently need an S3-compatible proxy or a private
fork to use LanceDB against COS buckets.
- **GooseFS** is Tencent Cloud's distributed cache acceleration layer
that sits in front of COS/S3, a common pattern for vector search / AI
training where the same hot dataset is read repeatedly.
- This brings COS / GooseFS to feature parity with the existing
first-class backends (`aws`, `gcs`, `azure`, `oss`, `huggingface`).
See the linked issue #3525 for the full discussion.
## Changes
### `rust/lancedb/Cargo.toml`
Add two new optional features that pull through the corresponding
upstream feature flags:
```toml
cos = ["lance/tencent", "lance-io/tencent"]
goosefs = [
"lance/goosefs",
"lance-io/goosefs",
"lance-namespace-impls/dir-goosefs",
]
```
### `python/Cargo.toml`
Enable both `cos` and `goosefs` by default for the Python wheels, so
`pip install lancedb` works against COS / GooseFS out of the box
(consistent with how `aws` / `gcs` / `azure` / `oss` are bundled today):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/cos", "lancedb/goosefs"]
```
### `nodejs/Cargo.toml`
Enable `goosefs` by default for the Node binding (COS kept opt-in to
limit the default native binary size; can be revisited based on demand):
```diff
-default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface"]
+default = ["remote", "lancedb/aws", "lancedb/gcs", "lancedb/azure", "lancedb/dynamodb", "lancedb/oss", "lancedb/huggingface", "lancedb/goosefs"]
```
### `Cargo.lock`
Regenerated to reflect the transitive dependencies brought in by the new
upstream features. No manual edits.
## Example Usage
### Rust
```toml
# Cargo.toml
lancedb = { version = "0.30", features = ["cos", "goosefs"] }
```
```rust
// Tencent COS
let db = lancedb::connect("cos://my-bucket/my-db").execute().await?;
// GooseFS
let db = lancedb::connect("goosefs://my-namespace/my-db").execute().await?;
```
### Python
```python
import lancedb
db = lancedb.connect(
"cos://my-bucket/my-db",
storage_options={
"secret_id": "...",
"secret_key": "...",
"region": "ap-guangzhou",
},
)
```
## Backwards Compatibility
- All new features are **opt-in** at the Rust crate level (`default =
[]` for `lancedb` itself is unchanged).
- The Python wheel gains both backends by default, increasing wheel size
slightly but matching the existing pattern of bundling all major cloud
backends.
- Node binding only adds `goosefs` to defaults; existing users see no
behavior change.
## Testing
- `cargo check --all-features` ✅
- `cargo check -p lancedb --features cos` ✅
- `cargo check -p lancedb --features goosefs` ✅
- End-to-end COS / GooseFS smoke tests require Tencent Cloud credentials
and are intentionally not added to CI in this PR (same approach used for
`s3-test`). Happy to add a gated test feature in a follow-up if
reviewers prefer.
## Checklist
- [x] Added `cos` and `goosefs` features to `rust/lancedb/Cargo.toml`
- [x] Updated `python/Cargo.toml` default features
- [x] Updated `nodejs/Cargo.toml` default features
- [x] Regenerated `Cargo.lock`
- [x] Verified build with `--all-features`
- [ ] Documentation update (can be done in a follow-up PR once API
stabilizes)
## Related
- Issue: #3525
- Upstream support:
[`lance/tencent`](https://github.com/lance-format/lance),
[`lance/goosefs`](https://github.com/lance-format/lance)
## Summary
Adds `Table::get_lsm_write_spec` returning `Option<LsmWriteSpec>` — the
read counterpart to the existing `set_lsm_write_spec` /
`unset_lsm_write_spec`. Returns `None` when the MemWAL LSM write path is
not enabled; otherwise reconstructs the spec (mode, shard column,
`num_buckets`, `maintained_indexes`, `writer_config_defaults`) exactly
as installed.
## Changes
- **Rust core (`NativeTable`)** — reconstructs the spec from
`mem_wal_index_details()`, resolving the shard column from its Lance
field id via the dataset schema. This is a raw metadata read, so it is
unaffected by `describe_indices` system-index filtering.
- **Remote (`RemoteTable`)** — reads the `__lance_mem_wal` system index
through `index/list` with `include_system: true` (so the curated
`list_indices` surface stays unchanged), then parses the index `details`
JSON. It matches the index by name and ignores `index_type`, so no
client `IndexType` variant is needed. It uses the **server-resolved
`column` name** from the details (Lance field ids do not travel to the
remote client).
- **Python + TypeScript bindings** — sync and async, mirroring
`set`/`unset`, with round-trip tests (bucket / identity / unsharded,
plus `None` when unset).
## Tests
- Rust: native round-trip unit test + remote mock-endpoint tests
(present + absent). All green (`cargo test --features remote -p
lancedb`).
- Python/TS: round-trip tests added; binding-runtime execution runs in
CI.
## Dependencies for the remote path
The remote path is complete on the client side but depends on two
out-of-repo pieces to work end-to-end:
1. **lance** — emit the server-resolved shard **`column`** name in the
MemWAL index `details` JSON (field ids can't reach the client). See
lance-format/lance#7667.
2. **server** — honor `include_system` on `index/list` so the
`__lance_mem_wal` entry is returned for this read.
Against an older server (no `include_system`), the remote getter
degrades gracefully to `Ok(None)` rather than erroring.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Closes#3245.
> **BREAKING CHANGE:** `with_format("torch")` no longer returns a list
of stacked row tensors. It now returns per-row dicts so PyTorch's
default `DataLoader` collate stacks them into `{col: tensor(B,)}`.
Switch to `with_format("torch_row")` to keep the old shape.
### What changed
`"torch"` now returns a list of per-row dicts (`[{col: tensor}, ...]`)
at every indexed access path. The default `DataLoader` collate stacks
them into a column-keyed batched dict, no custom `collate_fn` needed.
The old shape is preserved under a new `"torch_row"` literal.
`"torch_col"` is unchanged.
The unbatching lives inside the transform (`batch_to_tensor_dict`), not
`__getitems__`, so the shape survives pickling and works under
`DataLoader(num_workers>0, multiprocessing_context="spawn")`.
### Format comparison
| Format | `iter(batch_size=N)` | `__getitems__([0,1,2])` | `DataLoader`
default collate |
|---|---|---|---|
| `"torch"` (new) | `list[{col: tensor}]` length N | `list[{col:
tensor}]` length 3 | `{col: tensor(B,)}` |
| `"torch_row"` (old `"torch"` behavior) | `list[tensor(n_cols,)]`
length N | `list[tensor(n_cols,)]` length 3 | `tensor(B, n_cols)` |
| `"torch_col"` (unchanged) | `tensor(n_cols, N)` | `tensor(n_cols, 3)`
| needs `collate_fn=lambda x: x` |
Output matches HuggingFace `Dataset.set_format("torch")` on container
shape, keys, and values at every access path. The only divergence:
HuggingFace downcasts `float64` to `torch.float32` by default, LanceDB
preserves dtype. Verified by `scripts/verify_torch_format.py`.
### Migration
```python
# Old default — column names lost, shape was tensor(B, n_cols)
DataLoader(Permutation.identity(table).with_format("torch"))
# New default — column names preserved
DataLoader(Permutation.identity(table).with_format("torch")) # {col: tensor(B,)}
# Keep old behavior
DataLoader(Permutation.identity(table).with_format("torch_row")) # tensor(B, n_cols)
```
### **Summary**
Closes#3212
Extends the Python `lit()` helper to natively support three additional
types (`date`, `datetime`, and `Decimal`) and implements reflexive
operators for the `Expr` class.
This implementation specifically addresses the blocking feedback
regarding precision loss, CI discovery, and query engine limitations:
* **Logic Refactoring**: Simplified `lit()` by combining `date` and
`datetime` normalization into ISO-8601 strings, ensuring stable SQL
parsing across different engine locales.
* **Precision Preservation**: `decimal.Decimal` objects are now passed
as high-precision strings to the Rust bridge, bypassing intermediate
float conversions and preserving full 128-bit decimal precision for
DataFusion.
* **Averted CI Failures**: Temporarily deferred `bytes` literal support
to a future PR to resolve a known DataFusion `expr_to_sql` limitation
that was crashing the `Doctest` runner.
* **Reflexive Operators**: Added support for "literal-first" arithmetic
and logical operations (e.g., `10 + col('a')` or `True &
col('active')`). Redundant reflexive comparisons (e.g., `__rlt__`) were
pruned as Python's data model handles them automatically.
* **Integration Verification**: Added dedicated integration tests in the
official test directory to ensure the query engine correctly handles the
new types and preserves bit-perfect fidelity.
### **Changes**
####
[python/python/lancedb/expr.py](file:///c:/Users/Laksh/Documents/lancedb/python/python/lancedb/expr.py)
* Updated `lit()` to handle `date`, `datetime`, and `Decimal` natively.
* Implemented reflexive operators (`__radd__`, `__rand__`, `__rmul__`,
etc.) to support literals on the left-hand side.
* Removed the problematic `bytes` doctest example and `lit()` type
support to unblock CI.
####
[python/src/expr.rs](file:///c:/Users/Laksh/Documents/lancedb/python/src/expr.rs)
* Modified the Rust FFI bridge to extract `Decimal` objects as strings.
* Ensured the `expr_lit` handler is ready to receive normalized temporal
strings.
* Consolidated imports and added missing operator documentation.
####
[python/python/lancedb/_lancedb.pyi](file:///c:/Users/Laksh/Documents/lancedb/python/python/lancedb/_lancedb.pyi)
* Updated type stubs for `expr_lit` to include `Any` (allowing for
`Decimal`).
### **Testing**
Added several new advanced test cases in
[python/python/tests/test_expr.py](file:///c:/Users/Laksh/Documents/lancedb/python/python/tests/test_expr.py)
covering:
* **High-precision Decimal preservation**: Verified against 128-bit
boundaries with a "one point off" test case (`1.234567890123456789 <
1.234567890123456790`).
* **Reflexive operator positioning**: Verified successful query
construction with literals on the left.
* **Timezone-aware normalization**: Confirmed stable behavior for
`datetime` objects.
* **Integration Testing**: Confirmed Date32 and Decimal columns return
the correct Python types and values from the engine during `.to_arrow()`
calls.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
# Elastic Streaming Dataloader
## Motivation
Training large models on LanceDB tables today requires loading the
entire dataset
into memory or writing bespoke batching logic. This PR introduces
`StreamingDataset`, a PyTorch `IterableDataset` that streams directly
from a
LanceDB table with two hard guarantees that are difficult to achieve
together:
**elastic determinism** and **resumability**.
## Goals
### Elastic determinism
The dataset partitions the table into a fixed number of *splits*
(controlled by
`num_splits`, `shuffle_seed`, and `epoch`). Samples are yielded by
round-robining
over splits one sample per split per cycle. Because the split structure
is fixed,
the set of samples that makes up each global training step is identical
regardless
of `world_size` or `num_workers`. You can scale your cluster up or down
between
runs and the model sees the same data in the same order — no
re-sharding, no
gradient variance from topology changes.
### Resumability
`state_dict()` / `load_state_dict()` capture how many samples each split
has
consumed. Because all splits are the same size and the round-robin
design keeps
them in lockstep, the state reduces to a single scalar
(`samples_consumed_per_split`)
that is topology-independent. A checkpoint saved with 8 GPUs can resume
correctly
on 4 GPUs or 16 GPUs without any adjustment.
### PyTorch `IterableDataset` / streaming
`StreamingDataset` implements the standard PyTorch `IterableDataset`
interface, so
it drops into any existing `DataLoader` pipeline without modification.
Data is
fetched lazily from Lance in chunks — only the rows needed for the
current batch are
ever in memory.
Compared to the map dataset this takes more work from pytorch and puts
it into the dataset itself (e.g. shuffling, filtering, etc.). We do this
because we cannot achieve things like elastic determinism or
prefiltering otherwise.
### Multi-worker support
DataLoader workers are automatically assigned contiguous sub-blocks of
splits (the
rank's splits are divided evenly across workers). Each worker is
independent:
no shared state, no inter-process coordination. The only constraint is
that
`num_splits` must be divisible by `world_size * num_workers`.
That being said, multi-worker is highly discouraged as it relies on
multiprocessing which is inefficient. Still, we want to support it.
### Filters as prefilters
Filters are applied at *permutation-build time* via
`PermutationBuilder.filter()`,
not re-evaluated on every fetch. The filtered row IDs are stored in the
permutation
table so that subsequent reads see only the matching rows. This allows
us to avoid loading rows that don't match the filter (which is the
default pytorch behavior)
### Prefetching
Two parameters control the I/O pipeline:
- `read_batch_size` (default 64) — number of rows fetched per
`take_offsets` call.
Larger values amortise per-request overhead, which is critical on object
storage
where a single round-trip can cost ~100 ms.
- `prefetch_batches` (default 4) — number of batches prefetched in
parallel per
split via a `ThreadPoolExecutor`. While the model processes the current
batch,
the next several batches are already in flight, hiding storage latency
behind
compute.
If set correctly then you can get good performance even with
num_workers=0 (unless you are bottlenecked on transform).
### Transform parallelism
The underlying `Permutation` API supports a `with_transform()` callback
for
decoding, augmentation, and format conversion. Unfortunately, this is
not parallelized. Pytorch typically parallelizes this with num_workers
which is multiprocessing which is highly inefficient. For simple
transforms we should be able to utilize multithreading and Rust based
UDFs. For complex python UDFs we could have a dedicated multiprocessing
pipeline for just the transform. Or we could just utilize
multithreading. In both cases we would exclude the I/O stage from the
multiprocessing because that ends up being very memory hungry and
inefficient.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- When an embedding function returns an empty list (e.g. `[]`) for an
input row — as can happen when a model produces no output for a blank
string — `_append_vector_columns` crashed with `ArrowInvalid: Length of
item not correct: expected N but got array of size 0` because PyArrow
cannot fit a zero-length value into a fixed-size list element.
- The fix adds a validation step in `gen()`, inside
`_append_vector_columns`, that replaces any vector whose length does not
match the expected `ndims` (including empty lists and `None`) with
`None` before `pa.array()` is called.
- `None` is a valid null in a PyArrow fixed-size list array, so the bad
entry flows into `_handle_bad_vectors` and is handled according to the
caller-supplied `on_bad_vectors` policy (`error` / `drop` / `fill` /
`null`) instead of causing an unconditional crash.
## Test plan
- [ ] Added `test_embedding_with_empty_output_vectors` in
`python/python/tests/test_embeddings.py` that uses an embedding function
returning `[]` for empty-string inputs, calls `table.add(...,
on_bad_vectors="drop")`, and asserts no crash and that bad rows are
correctly dropped.
- [ ] Existing `test_embedding_with_bad_results` continues to pass (NaN
vectors still handled correctly).
- [ ] Verified manually that `pa.array([[1.,2.,3.,4.], []],
type=pa.list_(pa.float32(), 4))` raises `ArrowInvalid` without the fix,
and succeeds with `None` in place of `[]`.
Fixes#1672
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Cohen Karnell
Will Jones
Farmer.Chillax
Drew Gallardo
heart
LanceDB Robot
Yang Cen
Lance Release
buduoqiu
kid
Xuanwo
Heng Ge
Andrew Chen
Eran Dagan
Mateusz Szewczyk
Drew Gallardo
Vitaliy
Tobias
Jack Ye
Prashanth Rao
Mark McDonald
Pranav Achar
Unmilan Mukherjee
ForwardXu
Dan Rammer
Omkar Kabde
LAKSH JAIN
Weston Pace
Eric B