Fixes#4399.
Remote connections currently discard local-only settings, including
table write options that determine storage format. The synchronous
Python client also binds the fourth positional `create_table` argument
to `on_bad_vectors`, so an intended overwrite is sent as a create.
This change aligns the remote Python signature with
`DBConnection.create_table`, preserves the legacy remote timeout
keywords, and raises `ValueError` for other unknown connection keywords.
The shared Rust client rejects unsupported connection, create, and open
options before sending HTTP requests; open validation also runs before a
table-cache hit. Python reports these as `NotImplementedError`,
including explicit `enable_v2_manifest_paths=False`. Supported
Enterprise connection-level storage options remain available.
This implements option 2 from the issue: explicit rejection. It
deliberately does not add unverified server write parameters. Calls that
previously supplied ignored options must omit them to use server-managed
storage and caching. Database URI prefixes are already preserved on
current main and are unchanged here.
Regression tests cover Python sync/async, TypeScript create/open calls,
and the Rust remote client, including empty-table creation, overwrite
mode, and cached table handles. Python API documentation describes the
behavior and corrected positional order.
Reconciled with main at `0ea932da`, preserving upstream blob and
error-handling tests and updating the Python test that previously
expected open options to be ignored.
Validation on the merged source:
- Rebuilt the Python native extension; 304 tests passed across remote
options, database, remote database, and session suites.
- Rust remote database suite: 135 passed. Rebuilt the Node native
extension; TypeScript remote suite: 62 passed. TypeScript compilation
and documentation generation passed.
- Full Python SDK lint and formatting pass (159 files); full TypeScript
lint and formatting pass (66 files). Rust formatting and `git diff
--check` pass.
- Repository-wide Python lint/format reports existing issues in
unchanged `docs/web-tests` files (49 lint findings and 11 formatting
files).
Environment: Ubuntu under WSL2, Linux x86-64, CPython 3.12.13, Rust
1.97.0, and Node.js 22.16.0. Locked dependencies and the repository `ci`
build profile. Remote tests use HTTP fixtures; this does not claim
validation against the original ARM64 single-node deployment.
Documentation review follow-up: added a draft Enterprise migration note
in `docs/web/basics/sdk.mdx`. It explains that `db://` connections
reject local session/cache and per-table storage/write options, while
supported connection-level Enterprise storage options remain available.
Following the repository's release conventions, the note stays in an MDX
comment below the migration list until the behavior is released. Vale
3.24.0 with the pinned LanceDB style package reports no findings; `git
diff --check` passes. This follow-up changes documentation only.
BREAKING CHANGE: Remote connections now reject unsupported local
session/cache and per-table storage/write options instead of silently
ignoring them. Callers must omit these options and use server-managed
storage and caching; supported Enterprise connection-level storage
options remain available.
Integration with current main preserves its synchronous API signatures,
including the remote `create_table(mode=None)` default, and the table
compatibility defaults. The positional-order repair is already on main
via #4608; this PR retains the distinct unsupported-option rejection
behavior.
CI follow-up: the inherited IVF search-effort test occasionally measured
recall 0.375 against its unchanged 0.5 threshold after random index
training. A matching local reproducer observed this once in 30 builds.
The fixture now places the eight exact neighbors together, preserving
the 16-partition setup, two writes, all probe/precedence assertions, and
the recall threshold. The adjusted fixture returned recall 1.0 in 100
builds; the full query and remote suites passed (102 tests). This
stabilizes test data and does not change search behavior or promise
production recall.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.
The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.
Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.
BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.
Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.
Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):
- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.
All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.
Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
Updates the Rust workspace Lance dependencies and Java lance-core from
14.0.0-beta.1 to
[v14.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.6),
and refreshes Cargo.lock. No compatibility fixes were required.
Validation passed: `cargo clippy --quiet --workspace --tests
--all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff
--check`.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Empty local TypeScript queries returned a zero-row Arrow table with no
columns because `toArrow()` built the table solely from record batches.
Local streams can yield no batches, while remote streams often supply an
empty batch carrying the schema.
Expose the executed stream's schema through the native iterator and
decode it into an empty Arrow table when no batches arrive. This
preserves projected and search columns, types, nullability, and metadata
without executing another query. Document the empty-result behavior.
Validation:
- Eleven regression cases failed with the original `toArrow()`
implementation; all twelve now pass, including remote schema-only and
empty-batch responses.
- All 1,075 tests in the query, remote, Arrow, and table suites pass.
- Native build, TypeScript compilation, Rust formatting and Clippy (`-D
warnings`), TypeScript lint/formatting, and documentation generation
pass.
Fixes#4386
<!-- lance-gatekeeper-fix:v1 agent=7d589da1f7db5c7383f9600490e111a9
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add Node regression coverage for vector-search offset pagination
- add equivalent coverage for full-text search
- compare later pages with the corresponding complete-result slice and
assert page sizes
## Root cause
The historical query path requested only the user limit from
nearest-neighbor or full-text search before applying the offset, so a
page became empty when its offset reached that limit. The production
query path on current main already incorporates the later fix from
#2592; this change adds the missing Node binding coverage for the
still-open report and protects both affected APIs from regression.
## Validation
- corepack pnpm build
- corepack pnpm test -- query.test.ts --runInBand
--testNamePattern="Search pagination"
- corepack pnpm lint-ci
- corepack pnpm tsc
- corepack pnpm run docs
Fixes#2229
<!-- lance-gatekeeper-fix:v1 agent=8ba8b18a18260a68a3e605d1bbfa518e
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
BREAKING CHANGE: When passing multiple where clauses to a query, they
now stack instead of replacing the previous filter.
Previously, calling `where`/`only_if` more than once on a query silently
replaced the previous filter, so only the last filter was applied. This
was
surprising and could return rows that an earlier filter should have
excluded.
This implements the alternative suggested in
https://github.com/lancedb/lancedb/pull/3514#issuecomment-4664901580:
instead of
rejecting a second filter, repeated filters are combined with a logical
AND
(`(previous) AND (new)`).
The combination happens in the Rust core (`QueryBase::only_if` and
`only_if_expr`), so it applies to all SDKs at once (Rust, Python async,
and
TypeScript). The Python sync query builder keeps its own filter state,
so it
combines filters in the binding layer as well.
SQL string and expression filters are combined within their own
representation.
When the two representations are mixed, the expression is lowered to SQL
(via
`expr_to_sql_string`) and the filters are combined as SQL strings, so
chaining
`where` works regardless of which form each filter takes.
Fixes#2649
## Tests
- Rust: `cargo test --features remote -p lancedb --lib query`
- Python: `uv run --extra tests pytest python/tests/test_query.py`
- TypeScript: `pnpm test __test__/query.test.ts`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
KarthikSandXuanwo
Yang Cen
lancedb-gatefixer[bot]
Will JonesandClaude Opus 4.8
Xin Sun
Weston Pace