## Summary
`AsyncTable.search()` computes the query embedding with
`loop.run_in_executor(None, ...)`, which uses asyncio's **default**
`ThreadPoolExecutor`. That pool is shared with all other
`run_in_executor(None, ...)` work, so a slow embedding call — a heavy
local model or an HTTP request to an embeddings API — ties up those
threads and starves unrelated async I/O under concurrent load.
This moves the (potentially blocking) embedding call onto a **dedicated
executor**, isolating it from the default pool.
Closes#3310.
## Problem
`python/lancedb/table.py`, `AsyncTable.search()`:
```python
return (
await loop.run_in_executor(
None, # asyncio's default executor, shared with other blocking I/O
embedding.function.compute_query_embeddings_with_retry,
query,
)
)[0]
```
Under load, concurrent searches whose embeddings block (or any other
code using the default executor) contend for the same small thread pool.
## Change
- Add a dedicated
`ThreadPoolExecutor(thread_name_prefix="lancedb-embedding")` in
`background_loop.py`, exposed via `embedding_executor()`.
- Use it in `AsyncTable.search()`'s `make_embedding` instead of the
default executor.
- Reset the executor in the existing `_reset_after_fork` hook — its
worker threads don't survive `fork()`, same as the background event
loop. It's recreated lazily, so this is cheap.
## Design notes
The issue asked whether maintainers preferred a configurable executor, a
dedicated internal one, or another approach (no response in the thread).
I went with a **dedicated internal executor**: it fixes the starvation
with no public API change and stays consistent with the existing `LOOP`
singleton. Making the pool size configurable would be an easy follow-up
if preferred.
Scope is limited to `search()`. The broader "embedding functions need
real async support" (including `add()`) is tracked separately in #3268.
## Testing
- Added `test_async_search_runs_embedding_on_dedicated_executor`:
patches the embedding function to record the executing thread during an
async search and asserts it runs on a `lancedb-embedding` thread.
Verified it **fails** against the previous `run_in_executor(None, ...)`
and passes with the fix.
- `ruff format`, `ruff check`, and `pyright` pass on the changed files.
## Summary
Wires `RemoteTable::set_lsm_write_spec` / `unset_lsm_write_spec` to the
sophon REST endpoints added in
[lancedb/sophon#6181](https://github.com/lancedb/sophon/pull/6181),
replacing the previous `NotSupported` stubs.
- `set_lsm_write_spec` maps the `LsmWriteSpec` onto sophon's request DTO
— mode-tagged `sharding` (`unsharded` / `bucket` / `identity`),
`maintained_indexes`, and `writer_config_defaults` — and POSTs to
`/v1/table/{name}/set_lsm_write_spec/`.
- `unset_lsm_write_spec` POSTs to
`/v1/table/{name}/unset_lsm_write_spec/`.
- Both call `check_mutable` first, matching the other remote mutations.
- `maintained_indexes` is sent verbatim (an empty list means "no
maintained indexes", matching native semantics).
## Testing
- Added mocked-endpoint unit tests for unsharded / bucket / identity set
and for unset.
- `cargo check --features remote --tests` passes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updates Lance dependencies to v8.0.0-beta.2 across the Rust workspace
and Java lance-core metadata.
The update was generated with ci/update_lance_dependency.py and required
no compatibility code changes.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.2
## ⛔ Merge blocker: legal review required
This bump pulls in a new transitive **dev/profiling** dependency chain
`inferno v0.11.21` → `pprof v0.15.0` → `lance-testing`, and `inferno` is
licensed **CDDL-1.0** (copyleft). To get `cargo-deny` green, `CDDL-1.0`
was added to the `deny.toml` allow list.
**Do not merge until legal has reviewed and signed off on allowing
CDDL-1.0.** The dependency is dev/test-only and not distributed, but the
allow-list addition still requires legal approval per our policy.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Add `__reduce__` methods to `LanceDBClientError` and `RetryError` so
that instances can be pickled and unpickled correctly
- `HttpError` inherits the fix from `LanceDBClientError` since it has no
additional `__init__` parameters
- Add tests verifying pickle roundtrip for all three exception classes
Fixes#3447
## Test plan
- [x] Verified pickle roundtrip for `LanceDBClientError` with and
without `status_code`
- [x] Verified pickle roundtrip for `HttpError` (subclass, no extra init
params)
- [x] Verified pickle roundtrip for `RetryError` (subclass with many
extra params)
- [ ] CI tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
## Bug Fix
### What is the bug?
`QueryBuilder.to_pandas(blob_mode="descriptions")` could still fall back
to `self.to_arrow()` for query outputs with blob columns. Custom query
subclasses or wrappers can have `to_arrow()` behavior that is not
compatible with pandas blob-description conversion, which can surface as
low-level Arrow/list-batch conversion failures.
### What issues or incorrect behavior does the bug cause?
Callers need to carry local `to_pandas` or plain-scan adapter special
casing for blob descriptions, and scanner-only kwargs such as row
addresses and fragment selection are not represented in LanceDB query
state.
### How does this PR fix the problem?
This PR routes blob-output query `to_pandas()` through the Lance scanner
path for `lazy`, `bytes`, and `descriptions` modes when the query is a
scanner-backed plain scan. For `blob_mode="descriptions"` with
`flatten`, it collects scanner Arrow/table output, applies LanceDB
`flatten_columns`, and converts to pandas from there. Non-plain blob
query shapes now fail with a clear unsupported error instead of falling
into subclass `to_arrow()` behavior.
It also adds Python query state and builder methods for scanner-only
plain-scan parameters:
- `with_row_address()` for `_rowaddr`
- `with_fragments(...)` for Lance fragment objects
- `fragment_ids([...])` as a convenience wrapper that resolves IDs to
Lance fragments
## Validation
- `cd python && uv run --no-sync ruff format --check
python/lancedb/query.py python/tests/test_query.py`
- `cd python && uv run --no-sync ruff check python/lancedb/query.py
python/tests/test_query.py`
Targeted pytest was intentionally not run locally per maintainer
request.
### Description
This PR exposes native DataFusion expression support in the Rust SDK's
`MergeInsertBuilder` via two new builder methods:
`when_matched_update_all_expr` and
`when_not_matched_by_source_delete_expr`.
For remote LanceDB tables (where operations are serialized over
HTTP/JSON to the SaaS backend), native DataFusion expression trees
cannot be executed directly. The SDK handles this gracefully by
returning a `NotSupported` error.
### Key Changes
- **`MergeFilter` Enum**: Introduced a helper enum to store either a SQL
string or a native `datafusion_expr::Expr`.
- **`MergeInsertBuilder`**: Updated `when_matched_update_all_filt` and
`when_not_matched_by_source_delete_filt` fields to store the new enum,
and added `when_matched_update_all_expr` and
`when_not_matched_by_source_delete_expr` builder methods.
- **Execution & Remote Dispatch**: Dispatched the filter variants during
local execution, and rejected expression filters with a clean
`NotSupported` error in remote table request conversion.
- **Testing**: Added a `test_merge_insert_expr` unit test covering
conditional updates and deletes with programmatically built DataFusion
expressions.
### Verification
- Added integration test `test_merge_insert_expr` which successfully
compiles and passes.
- Formatted and linted the code.
Closes#3416
## Summary
Regression test for [issue
#2654](https://github.com/lancedb/lancedb/issues/2654) — a nullable
struct column whose first batch contains only `None` values crashed in
`_align_field_types` with `AttributeError: 'pyarrow.lib.DataType' object
has no attribute 'fields'`.
The actual fix landed in #3394, but no test was added. This PR adds the
reproducer from the issue as a test.
## Test plan
- `test_add_nullable_struct_with_none`: creates a table with a nullable
struct column, adds a row with a non-null struct value, then a row with
`None` for the struct field. Verifies both rows land correctly.
- Uses Lance file format v2.1 (`new_table_data_storage_version="2.1"`)
because nullable structs aren't supported on v2.0.
## Related
- #3028 (the original fix attempt, now superseded)
Adds a REVIEW.md at the repo root with cross-SDK parity guidance for
automated code review. The Claude Code review feature automatically
loads `REVIEW.md` as review-only context.
This is intentionally a semantic nudge, not a deterministic check, it
relies on the reviewer reading the sibling SDK, so it will catch most
gaps.
## What's broken
Calling `RRFReranker().rerank_multivector([])` crashes with `IndexError:
list index out of range` because the method accesses `vector_results[0]`
for the type-homogeneity check before verifying the list is non-empty.
The `all()` call passes vacuously on an empty iterable so the crash hits
the next lines.
```python
from lancedb.rerankers import RRFReranker
RRFReranker().rerank_multivector([])
# IndexError: list index out of range
```
## Why it happens
The type check uses `vector_results[0]` as the reference type but never
guards against an empty list. `all(...)` short-circuits to `True` when
the iterable is empty, so the bad index access on the lines that follow
is never reached by the existing guard logic.
## Fix
Add an explicit empty-list check before any indexing.
## What's broken
`MRRReranker.rerank_multivector([])` raises `IndexError: list index out
of range`. The crash happens on line 128 (the `all()` type-homogeneity
check passes vacuously on an empty iterable) and on line 134 which
accesses `vector_results[0]` unconditionally, with no prior guard for an
empty list.
## Why it happens
`all()` over an empty iterable returns `True`, so the type check
silently passes and execution falls through to `vector_results[0]` which
crashes.
## Fix
Added a two-line guard at the top of `rerank_multivector` that raises a
clear `ValueError("vector_results must not be empty")` before any
indexing occurs.
## Test
Added `test_mrr_reranker_empty_input` in `test_rerankers.py` which calls
`rerank_multivector([])` and asserts that a `ValueError` with the
message "must not be empty" is raised.
Fixes#3468
Co-authored-by: Aegis Dev <aegis@devteamaegis.com>
## Feature
- What is the new feature?
- Adds `blob_mode` support to sync and async Python query `to_pandas()`
APIs.
- Enables plain scan queries to return blob columns as lazy `BlobFile`
objects, raw bytes, or blob descriptions.
- Lets namespace-backed local tables use Lance native blob-aware pandas
conversion for lazy blobs.
- Why do we need this feature?
- Table and Lance dataset/scanner APIs already support blob-aware pandas
conversion, but LanceDB query builders did not expose that capability.
- Geneva and other callers should be able to use query-level
`to_pandas(blob_mode=...)` without manually constructing Lance scanners.
- How does it work?
- Plain scan queries route through Lance scanner native
`to_pandas(blob_mode=...)`, preserving filter, projection, limit,
offset, row id, and alias/expression projection behavior.
- Non-native query shapes keep existing Arrow fallback semantics and
raise a clear error when they return blob columns with
`blob_mode="lazy"` or `blob_mode="bytes"`.
- Focused tests cover table/query blob modes,
filter/select/limit/offset/alias query cases, async query behavior,
vector-query error boundaries, and namespace-backed lazy blobs.
## Validation
- `cd python && .venv/bin/maturin develop --uv --extras tests,dev
--profile dev`
- `cd python && uv run --frozen --no-sync pytest
python/tests/test_table.py::test_table_to_pandas_blob_modes
python/tests/test_table.py::test_async_table_to_pandas_blob_bytes
python/tests/test_query.py::test_plain_scan_query_to_pandas_blob_modes
python/tests/test_query.py::test_plain_scan_query_to_pandas_blob_projection
python/tests/test_query.py::test_async_plain_scan_query_to_pandas_blob_projection
python/tests/test_query.py::test_vector_query_to_pandas_blob_mode_requires_native_path
python/tests/test_namespace.py::TestNamespaceConnection::test_table_to_pandas_blob_lazy_through_namespace
-q`
- `cd python && uv run --frozen --no-sync ruff format --check .`
- `cd python && uv run --frozen --no-sync ruff check .`
- `git diff --check`
### Summary
Deprecates the Python replace_field_metadata (on Table and AsyncTable)
in favor of update_field_metadata. Mirrors Lance, which already
deprecated Dataset.replace_field_metadata for update_field_metadata.
Stacked on top of #3482 as this was a follow-up task after adding
update_field_metadata
### Summary
Adds update_field_metadata to the client SDK (Rust core, Python, and
TypeScript) so clients can edit per-field (column) Arrow metadata
(schema.fields[].metadata)
### Testing
- added unit tests
- ran E2E against a local server on both local and remote tables (set →
merge → delete), across Python sync/async and TypeScript
### Next steps
- deprecate replace_field_metadata in the python lancedb favor of this
(typescript didn't have replace_field_metadata method). This matches
Lance's API direction (Lance already deprecated replace_field_metadata
for update_field_metadata)
Moves the Lance dependency bump process into an in-repository skill so
local agents and GitHub Actions share the same workflow definition.
The update workflow is now an explicit, optional-tag entrypoint;
latest-release resolution, duplicate PR handling, Java/Rust dependency
updates, and Sophon follow-up are documented in the skill and backed by
a small deterministic helper.
This PR makes remote LanceDB tables usable from PyTorch multiprocessing
workers. Remote tables now carry enough safe JSON connection state to
reopen themselves after pickle/spawn or fork, and permutations lazily
rebuild their reader from restored tables instead of trying to reuse
process-local handles.
This addresses the remote-table gap in the PyTorch dataset path while
preserving the explicit connection factory escape hatch for custom
worker-side credential loading or non-serializable header providers.
Validated with targeted remote table, permutation, and PyTorch
DataLoader tests.
This updates the workspace Lance dependencies from `v7.1.0-beta.4` to
`v7.2.0-beta.3` and refreshes `Cargo.lock`.
The lockfile now points at Lance commit
`7c070f760fa8e24c8015cb2afbd22c5e6b7898e8` and includes the transitive
dependency updates required by the new beta.
Update the Rust workspace Lance git dependencies and Java lance-core
dependency to v7.2.0-beta.1.
This keeps LanceDB aligned with the latest Lance beta release and
refreshes the Cargo lockfile for the new Lance dependency graph.
## Summary
When an `LsmWriteSpec` is installed on a table (#3396), `merge_insert`
upsert
calls are dispatched through Lance's MemWAL `ShardWriter` (LSM-style
append)
instead of the standard merge path.
- **`use_lsm_write`** — a `merge_insert` builder option, default `true`;
set it
`false` to use the standard path for a call even when a spec is set.
- **`assume_pre_sharded`** — a `merge_insert` builder option, default
`false`;
skips the per-row shard check and routes by the first row only.
- **`close_lsm_writers`** — drains and closes the table's cached MemWAL
shard
writers.
- The `merge_insert` **`on`** columns default to, and are validated
against,
the table's unenforced primary key.
- Shard writers are cached alongside the dataset (in
`DatasetConsistencyWrapper`) and reused for the session.
- `MergeResult` gains **`num_rows`** — on the LSM path the insert/update
breakdown is unknown until compaction, so only the total is reported.
Routing covers all three sharding strategies — bucket (murmur3,
Iceberg-compatible), identity, and unsharded. Each `merge_insert` call
targets
a single shard; the whole input is collected and validated before a
single
atomic `ShardWriter::put`, so a validation failure leaves the MemWAL
untouched.
Bindings: Python (`merge_insert(...).use_lsm_write(...)` /
`.assume_pre_sharded(...)`, `Table.close_lsm_writers`) and TypeScript
(`mergeInsert(...).useLsmWrite(...)` / `.assumePreSharded(...)`,
`Table.closeLsmWriters`).
## Context
Reconstructed from the original #3354 branch onto current `main`: the
branch
predated the #3394 (unenforced primary key) / #3396 (`LsmWriteSpec`)
split and
has been rebuilt on that merged foundation. Depends on Lance
`v7.0.0-beta.13`.
The MemWAL read path (reading un-flushed shard data back into queries)
and
remote (LanceDB Cloud) LSM support are follow-ups.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
Bumps the pinned toolchain in `rust-toolchain.toml` from 1.94.0 to
1.95.0.
Fixes new lints surfaced by clippy on 1.95.0:
- `manual_checked_ops` — fragment size mean in `table.rs` uses
`checked_div`
- `explicit_counter_loop` — shuffle test loop in `shuffle.rs`
No rustc warnings were introduced.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The latest main Python workflow fails across multiple matrix jobs
because `test_remote_create_index_new_api` opens a remote table whose
mocked schema only exposes `id`, while the new `create_index(...,
config=...)` path validates the requested indexed columns.
This updates the remote-table fixture to include the indexed columns
used by the smoke test and checks the emitted column payloads, keeping
the test aligned with the schema-aware API path.
## Summary
- Transitions `LanceTable` and `RemoteTable` to use the unified
`create_index()` API matching `AsyncTable`
- Deprecates `create_scalar_index()` and `create_fts_index()` with
deprecation warnings
- Adds detection logic to distinguish legacy vs new API calls
- Adds `@overload` decorators for type checker compatibility
- Adds `accelerator` parameter to IVF config classes for GPU support
**New API:**
```python
table.create_index("vec", config=IvfPq(distance_type="l2"))
table.create_index("col", config=BTree())
table.create_index("text_col", config=FTS(with_position=True))
```
**Legacy API (deprecated):**
```python
table.create_index("l2", vector_column_name="vec") # emits DeprecationWarning
table.create_scalar_index("col", index_type="BTREE") # deprecated
table.create_fts_index("text_col") # deprecated
```
Fixes#2879🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
manylinux2_17 reached EOL in 2024 and pyarrow stopped publishing 2_17
wheels long ago. We already build manylinux2_28 wheels, so drop the 2_17
matrix entries.
Fixes#3452
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The python-v0.32.0 publish run failed on every build matrix entry. Three
independent issues:
1. **Mac and Windows**: `pypa/gh-action-pypi-publish` only runs on
Linux, but was being called inline from each build job.
2. **Linux (all arches)**: `pypa/gh-action-pypi-publish` derives its
docker image name from `github.action_repository`, which is empty when
the action is invoked from inside a composite action
(actions/runner#2473 — pypa's own `action.yml` references this bug). It
falls back to `github.repository`, generating
`docker://ghcr.io/lancedb/lancedb:<tag>`, which doesn't exist →
`denied`. Only the ARM matrix entry surfaced this because it failed
first and cancel-cascaded the rest.
3. **Windows**: `upload-artifact` in `build_windows_wheel` pointed at
`python\target\wheels`, but maturin writes to the workspace-root
`target/wheels`. The artifact was always empty. Also, `pypi-publish.yml`
passed a `vcpkg_token` input that the composite doesn't declare.
## Changes
- Build jobs (linux/mac/windows) now upload their wheels as
`actions/upload-artifact` artifacts.
- New Linux `publish` job downloads all wheel artifacts and runs the
Fury or PyPA publish step directly (not via a composite), so
`github.action_repository` resolves correctly.
- Delete the unused `upload_wheel` composite action.
- Drop the broken upload-artifact step inside `build_windows_wheel`.
- Remove the bogus `vcpkg_token` input.
- Fury upload now loops over all wheels instead of just the first.
- Bump `actions/checkout`, `actions/upload-artifact`,
`actions/download-artifact` to current major versions (Node 24) to clear
deprecation warnings.
- Bump Windows job timeout 60 → 90 minutes; previous run was
cancel-timing-out on a 60m cap.
- Use `rust-lld` as the Windows MSVC linker via
`CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER`. `link.exe` is
single-threaded and the long pole on Windows builds.
Fixes#3445
## Test plan
- [x] Open this PR — `paths` filter triggers a dry-run build on all
three platforms.
- [x] Verify all three builds produce wheels.
- [x] Confirm the `pypa/gh-action-pypi-publish` container actually
starts (the actions/runner#2473 bug) via the `publish-dry-run` job
pointed at TestPyPI.
- [x] **REMOVE BEFORE MERGE**: drop the `publish-dry-run` job and the
now-redundant `actions/upload-artifact` runs on PRs (currently always-on
so the dry-run has wheels to publish).
- [ ] After merge, cherry-pick onto `python-v0.32.0` and force-push the
tag to re-trigger the publish.
When a table is created with `pa.json_()` (PyArrow's JSON extension
type),
it is stored internally as `lance.json` (LargeBinary with `lance.json`
extension metadata). Calling `table.add()` with `pa.json_()` data failed
with:
```
RuntimeError: lance error: Append with different schema:
`data` should have type json but type was large_binary
```
`build_field_exprs` in `rust/lancedb/src/table/datafusion/cast.rs` saw
that
the input field (`Utf8` with `arrow.json` metadata) differed from the
table
field (`LargeBinary` with `lance.json` metadata). Since
`can_cast_types(Utf8, LargeBinary)` is true, it inserted a DataFusion
`Utf8 → LargeBinary` cast. That cast preserved the input field's
`arrow.json`
extension metadata instead of adopting the table's `lance.json`
metadata, so
lance-core detected a schema mismatch and rejected the append.
This adds a special case in `build_field_exprs`: when the input is
`arrow.json` and the table field is `lance.json`, the expression is
passed
through unchanged. Lance-core's write path already handles the
`arrow.json → lance.json` conversion (including JSONB encoding), so no
DataFusion cast is needed.
Fixes#3144
Continues #3291 from a fork (the original author's branch could not be
pushed to). The original commits are preserved; an additional commit
fixes
the CI failures on that PR — formatting, a missing trait import, and
read-back assertions that assumed binary storage when a lance.json
column
is read back as `Utf8`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: yunju.lly <yunju.lly@antgroup.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Bump lance dependency from `v7.0.0-beta.13` to `v7.0.0-rc.1`
- Remove PK constraint from `LsmWriteSpec::Bucket` docs and
`Table::set_lsm_write_spec` docs
- Remove test assertions that expected rejection when no PK is set or
when bucket column != PK
Closes https://github.com/lance-format/lance/issues/6917
## Problem
`LinearCombinationReranker.merge_results` has two related bugs that make
it return **inverted relevance rankings** — the least relevant document
ranks first (closes#3154).
### Bug 1 — `_combine_score` subtracts from 1, inverting the final
ranking
```python
def _combine_score(self, vector_score, fts_score):
return 1 - (self.weight * vector_score + (1 - self.weight) * fts_score)
```
Both `vector_score` (already converted via `_invert_score`) and
`fts_score` (BM25 relevance) are in **higher-is-better** space. Wrapping
the weighted average in `1 - (...)` flips the direction: a perfectly
matching document (`vector_score=1, fts_score=1`) gets `_relevance_score
= 0.0`, while a non-matching document gets a high score.
### Bug 2 — Documents missing an FTS score are rewarded, not penalised
```python
fts_score = result.get("_score", fill) # fill=1.0 by default
```
When a document has no FTS match, `fts_score = fill = 1.0`. In
`_combine_score` (with the bug-1 formula), this large value becomes a
**negative penalty** via `1 - (... + 0.3 * 1.0)`, counterintuitively
*boosting* the document's score. By contrast, missing vector results
correctly receive `_invert_score(fill) = 0.0` (penalised).
## Fix
**Bug 1** — remove the `1 -` inversion from `_combine_score`:
```python
def _combine_score(self, vector_score, fts_score):
return self.weight * vector_score + (1 - self.weight) * fts_score
```
**Bug 2** — use `1 - fill` for missing FTS scores so both penalties are
symmetric (mirror of what `_invert_score(fill)` already does for missing
vector scores):
```python
fts_score = result.get("_score", 1 - fill) # was: fill
```
With `fill=1.0` (default): `1 - 1.0 = 0.0` — missing-FTS entries
contribute `0` to the FTS term, identical to how missing-vector entries
contribute `0` to the vector term.
## Verification
Concrete example from the issue. With `weight=0.7`, `fill=1.0`:
| Document | `_distance` | `_score` | Old `_relevance_score` | New
`_relevance_score` |
|----------|-------------|----------|------------------------|------------------------|
| `apple orange` | 0.0 (best) | 2.41 (only FTS) | 0.30 (**wrong: ranked
2nd**) | 1.42 (**correct: ranked 1st**) |
| `banana grape` | 0.9999 (worst) | — | 0.70 (**wrong: ranked 1st**) |
0.00 (**correct: ranked last**) |
## Tests
Two regression tests added to `python/python/tests/test_rerankers.py`:
- `test_linear_combination_best_match_ranks_first` — the document with
the smallest distance **and** an FTS match must have the highest
`_relevance_score`.
- `test_linear_combination_missing_fts_is_penalised` — a document with
any FTS score must beat an otherwise-equal document with no FTS match.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Closes client side work of #3370
### Summary
- Plumbs `read_consistency_interval` from `ConnectBuilder` through
`RestfulLanceDbClient` so remote reads attach an
`x-lancedb-min-timestamp` freshness header. None = no header (default),
zero = "now", positive = `now - interval`.
- Adds per-table `FreshnessState` on `RemoteTable`: write responses
(`update`, `delete`, `merge_insert`, `add_columns`, `alter_columns`,
`drop_columns`) track the committed version, and the next read sends
`x-lancedb-min-version` so the server's cache honors read-your-write.
- `checkout(v)` / `checkout_tag(t)` / `checkout_latest()` / `restore()`
reset the freshness state appropriately; the validating `/describe/` and
tag-resolve requests are sent without freshness headers so they don't
carry stale state.
- Updates Rust, Python, and Node docstrings and calls out that stronger
consistency raises per-read latency and cost.
### Testing
- Unit tests cover default behavior, interval=0, positive interval,
checkout_latest baseline, min_version-after-write, checkout clears
state, and the two no-stale-header invariants on `checkout(v)` and
`checkout_tag(t)`.
- Ran smoke tests against local remote table to verify functionality
Modified the parameter of delete to a Predicate that could be
constructed from either datafusion Expr, from str (to support SQL
predicate), or from String to support python and javascript bindings.
When a datafusion Expr is used, it avoids the overhead of serializing to
SQL and re-parsing when callers already have an Expr (e.g. from query
planning).
The native implementation uses lance's `DeleteBuilder::from_expr`. The
remote implementation converts the Expr to SQL via `expr_to_sql_string`
before sending to the server, consistent with the existing query and
count_rows paths.
Closes#3204
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Previously `check_lance_release.py` used `git/refs/tags` with
`--paginate --jq`, which drops the last page in some `gh` versions. The
7.x Lance tags all landed on the final (partial) page, causing the
script to report `v6.0.1` as the latest and never triggering an update.
Switch to the releases API with `per_page=20`, which returns the 20 most
recent releases sorted newest-first — one API call, no pagination
needed.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Problem
Issue #2247 notes that the Python docs do not show how to use LanceDB's
in-memory backend via `connect("memory://")`.
## Solution
Add `memory://` examples to the sync and async `connect` docstrings, and
call out that in-memory databases are intended for tests/temporary data
and are not persisted.
## Validation
- `python3 -m py_compile python/python/lancedb/__init__.py`
- `git diff --check`
## Confidence
82/100 — docs-only update, directly tied to the documented missing
`memory://` usage. It changes API documentation only and was syntax/diff
validated.
Closes#2247.
The `test_resolve_user_id_*` tests in `remote/client.rs` mutate the
process-global `LANCEDB_USER_ID` and `LANCEDB_USER_ID_ENV_KEY`
environment variables. cargo runs tests in a binary across multiple
threads, so one test's `remove_var` can race another's `set_var` between
when it's set and when `resolve_user_id()` reads it.
This surfaced as an intermittent failure of
`test_resolve_user_id_from_env_key` on Windows CI:
```
assertion `left == right` failed
left: None
right: Some("custom-env-user-id")
```
Annotates the five env-mutating tests with `serial_test`'s
`#[serial(user_id_env)]` so they run serially with respect to each
other.
Should be backported to `release/v0.28` (CI for #3421 hit this same
flake).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes#3407.
Remote tables now resolve create-index field paths against the table
schema before sending requests, so nested, escaped, and case-insensitive
inputs use the same canonical path contract as local tables. Remote
`list_indices()` also canonicalizes returned columns against the current
schema, and the remote query tests lock explicit nested vector and FTS
request payloads.
lancedb automation
Armaan Sandhu
Dan RammerandClaude Opus 4.8
dependabot[bot]
Lance Release
Yang Cen
Harikrishna KP
Brendan Clement
devteamaegis
Xuanwo
Heng GeandJack Ye
Yuval LifshitzandClaude Code
Zhaocun Sun