Retain the upstream nested-blob missing-field checks and regression tests while preserving non-scan blob pandas conversion and remote snapshot safeguards.
## Why
lance-format/lance#9803 makes the MemWAL's memtable index kinds
pluggable: a `MemIndexRegistry` on the writer config decides which
base-table index kinds are maintained in memory, so a kind Lance does
not build in can be added by registering a plugin. LanceDB opens the
MemWAL writer itself for `merge_insert`, and checks a named maintained
set itself, both against Lance's built-ins only. So a kind registered
outside Lance could not be named in
`LsmWriteSpec::with_maintained_indexes`, and under "maintain every
index" it was skipped.
## What changes
- `NativeTable::set_mem_index_registry(registry)` sets the memtable
index kinds a table maintains. The default is Lance's built-ins, so
nothing changes unless a caller sets one. The registry checks a named
maintained set (`validate_maintained_indexes_with`), is recorded on
`initialize_mem_wal`, and goes on the writer config `merge_insert`
opens. `lancedb::table` re-exports `MemIndexRegistry` and Lance's plugin
traits (`mem_index`), so a caller needs no direct `lance` dependency.
- The LSM vector read no longer checks resident memtables at all, and
the `carries_vector_index` helper goes with it. Lance already compares
every row of a memtable whose index can't answer the search, so no rows
are missed. It still requires the base vector index to be maintained.
## Limits
- The registry belongs to the table handle. Clones of a handle and its
branch handles share it; a handle opened separately starts from the
built-ins. Set it before the handle's first LSM write: a writer already
open keeps the kinds it opened with.
- The maintained set is stored with the table but the registry is not.
Once a spec names an index only a registered plugin maintains, an LSM
write from a handle without that registry is refused with Lance's "no
registered plugin maintains" error rather than skipping the index. That
includes the Python and Node bindings, which have no way to set a
registry yet.
- Rust only. A registry holds Rust plugin objects, which the Python,
TypeScript and Java bindings cannot construct, so they get no setter;
their documented behaviour (built-in kinds only) is unchanged.
- Per-index plugin settings (`ShardWriterConfig::index_overrides`, from
lance-format/lance#9805) are not exposed; writers use each plugin's
defaults.
## Open decision
Whether the registry should also be settable on the connection, so every
table a connection opens carries it. This PR keeps it per table; a
connection-level option would sit on top of `set_mem_index_registry`
without changing it.
## Depends on
Lance v14.0.0-beta.13, which `main` already uses. It includes
lance-format/lance#9803 and #9805 (pluggable memtable index kinds) and
#9857 (flushed generations keep the table's field ids).
## Testing
- `lsm_maintains_an_index_kind_from_the_tables_registry`: a stand-in
plugin maintains the base table's bitmap index as a B-tree. Naming the
index is refused without the registry and accepted with it; a separately
opened handle without the registry is refused an LSM write. Rows written
through `merge_insert` are found by id before and after the flush. Each
generation the shard manifests publish carries a bitmap index on
`region`, and a filter on it is answered through that index.
- `lsm_default_set_maintains_a_registry_kind_created_while_writing`:
under "maintain every index", a bitmap index created after the writer
opened is picked up by that writer and flushed. A clone of the table
handle shares the registry.
- `lsm_vector_search_compares_every_row_when_no_resident_index_answers`:
with a vector plugin that answers no search, LSM vector search still
returns the right rows, including a second page.
- `cargo test -p lancedb --features remote --lib` passes; `cargo clippy
-p lancedb --features remote --tests -- -D warnings` and `cargo fmt
--all -- --check` are clean.
---------
Co-authored-by: XYZhan <xuanyu.zhan@lancedb.com>
Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.
The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.
Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.
BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.
Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.
Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):
- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.
All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.
Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core to
[v14.0.0-beta.11](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.11).
Adapts the LSM resident-index guards to Lance's query-based index API
and adds regression coverage for index presence and type checks.
Validation: workspace Clippy with all features and warnings denied, Rust
formatting, and targeted LSM query unit tests.
## Summary
The Java SDK pinned `jackson-databind` 2.17.1, and `jackson-core` and
the other Jackson modules came in transitively via `arrow-vector` and
`lance-namespace-apache-client` at 2.16.x–2.17.x, all affected by known
CVEs.
This imports `jackson-bom` 2.18.11 in the parent POM, and declares
`jackson-core`, `jackson-annotations`, `jackson-datatype-jsr310`, and
`jackson-jaxrs-json-provider` as direct (BOM-managed) dependencies of
`lancedb-core`. The direct declarations are needed because the parent's
dependency management doesn't apply to a consumer's transitive
resolution; without them, Maven's nearest-wins mediation still hands
downstream apps the older versions from `arrow-vector`.
## Evidence
`mvn dependency:list` in a separate consumer project depending only on
`lancedb-core`:
| Artifact | Before | After |
|---|---|---|
| jackson-core | 2.16.0 | 2.18.11 |
| jackson-annotations | 2.16.0 | 2.18.11 |
| jackson-databind | 2.17.1 | 2.18.11 |
| jackson-datatype-jsr310 | 2.16.0 | 2.18.11 |
| jackson-jaxrs-base / json-provider | 2.17.1 | 2.18.11 |
| jackson-module-jaxb-annotations | 2.17.1 | 2.18.11 |
## Merge Risk
**Door:** Two-way. Dependency versions only, and they stay within
Jackson 2.x.
**Blast Radius:** `java` dependencies. Downstream users of
`lancedb-core` now get Jackson 2.18.11 transitively.
Fixes OSS-2360, OSS-2337
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Fixes the Dependabot alerts for Rust crates that have a patched release.
- `Cargo.lock`: aws-smithy-json 0.62.5 → 0.62.7, quinn-proto 0.11.14 →
0.11.19, xxhash-rust 0.8.15 → 0.8.19.
- `docs/web-tests/rs/Cargo.lock`: rustls 0.23.35 → 0.23.45,
rustls-webpki 0.103.13 → 0.103.15, serde_with 3.16.1 → 3.22.0, rand
0.8.5 → 0.8.8 and 0.9.2 → 0.9.5, xxhash-rust 0.8.15 → 0.8.19.
- `rust/lancedb/Cargo.toml`: the AWS SDK crates' default features
include the legacy `rustls` feature, which pulls in rustls 0.21 and the
vulnerable rustls-webpki 0.101.7 via hyper 0.14.
`aws-sdk-bedrockruntime` (behind the optional `bedrock` feature) and the
`aws-sdk-s3`/`dynamodb`/`kms` dev-dependencies now use `default-features
= false` with `default-https-client` + `rt-tokio`, the same rustls 0.23
client `aws-config` already uses. Minimum versions are raised to the
first releases that have `default-https-client`. This removes rustls
0.21, rustls-webpki 0.101, hyper 0.14 and h2 0.3 from the lockfile.
Not fixed: fast-float 0.2.0 has no patched release. It comes from
polars-arrow via the optional `polars` feature, pinned to `<0.40`;
removing it needs a polars upgrade.
## Merge Risk
**Door:** two-way; lockfile and dependency-feature changes only.
**Blast Radius:** `bedrock` embedding function's HTTP client, and
S3/DynamoDB/KMS integration tests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
LanceDB's remote HTTP client (reqwest) and gRPC client (tonic) used
rustls with the `ring` crypto provider, while the AWS SDK and
object_store use aws-lc-rs. This moves both clients to aws-lc-rs:
- `tonic`: `tls-ring` → `tls-aws-lc`.
- `reqwest`: 0.12 → 0.13. reqwest 0.12 has no aws-lc-rs feature; 0.13's
`rustls` feature uses it. 0.13 was already in the lockfile through
object_store. No code changes were needed.
- `macos-system-configuration` is replaced by its 0.13 name,
`system-proxy`, which also reads the system proxy on Windows.
- `query` is now an opt-in feature in 0.13; it's used for request query
parameters.
- Certificate verification moves from `rustls-native-certs` to
`rustls-platform-verifier`, which uses the OS verifier on macOS and
Windows.
`ring` is still compiled in, because `lance-namespace-impls` and
`lance-namespace-reqwest-client` enable it through reqwest 0.12
(lance-format/lance#9855, lance-format/lance-namespace#369). Both
clients here pick aws-lc-rs explicitly, so they don't depend on which
providers are compiled in.
## Evidence
A temporary example that connects the remote client to real HTTPS hosts:
```
https://example.com: Http error: ... 404 Not Found: <!doctype html>... # TLS handshake OK
https://expired.badssl.com: Hit retry limit ... connect_failures=3/3 # bad cert rejected
```
## Merge Risk
**Door:** two-way; dependency feature changes only.
**Blast Radius:** `remote` TLS and certificate verification (Python,
Node, and Rust LanceDB Cloud clients).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Python `Decimal` values and Arrow decimal scalars raise
`NotImplementedError` in `table.update(values=...)`. Register a Decimal
converter that emits quoted fixed-point text so the native cast to the
target column preserves every digit, including decimal128 and decimal256
values. Non-finite Decimal values raise `ValueError` with a static
message.
Closes#4360.
Validation with the native extension built from the current source at
7f593359, on macOS arm64 and Python 3.12.14:
- All 33 new regression cases failed before the fix and passed
afterward. They cover Python and Arrow scalar updates, signs, zero,
exponents, full 38/76-digit precision, and a Decimal context precision
of 6.
- 16 existing converter, update, and Decimal expression tests passed.
- Repository Ruff formatting and lint passed; formatting left all 156
files unchanged.
- A native round-trip check confirmed that an unquoted SQL number rounds
a high-precision value while the quoted Decimal update preserves it
exactly.
The full suite, other Python versions, and remote API were not
exercised.
AI assistance: Codex helped implement and validate the Decimal
conversion and its regression tests.
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.
The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
Centralize the entity name validation for tables, views, functions,
namespaces, secrets, and databases. This helps in several ways. One is
that percent-encoding is no longer needed in most cases (the exception
is database names, which can contain slashes). Another good thing is
that LLMs looking at this repository can now understand what the naming
rules are, since they will be the same here as on the server side.
Finally, this will help with the upcoming authZ code, since it also
needs to serialize these names, and needs to know that they don't
contain (among other things) colons.
Moves Function timestamps on the wire from RFC 3339 strings to epoch
milliseconds.
## What
- `FunctionVersion.created_at` (an RFC 3339 string) becomes
`created_at_millis` (milliseconds since the epoch, as an integer).
That's the shape Secrets, jobs and `FunctionErrorRecord` already use.
- Rust: `FunctionVersion::created_at() -> &str` becomes
`created_at_millis() -> i64`.
- Python: `FunctionVersion.created_at: str` becomes `created_at_millis:
int`.
- Fixtures and the mock server now send `1787270400000`, the old
`2026-08-21T00:00:00Z`.
Node.js doesn't expose `FunctionVersion`, so it doesn't change.
## Compatibility
Breaking, and it must ship together with the matching server change: a
server that still sends `created_at` fails to deserialize on this
client. Functions aren't deployed yet, so nothing is migrated.
## Tests
- `cargo test --features remote --test first_class_function_slice1`: 10
passed
- `cargo test --features remote --lib function`: 51 passed
- `pytest python/tests/test_first_class_function*.py`: 100 passed
- clippy, `cargo fmt --check`, ruff check and ruff format are clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BY5xcEGEAa1E11VG55Kd1w
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Covers lance-format/lance#9750's `TopKLateMaterialization` rule through
lancedb's DataFusion `TableProvider`, now that main is on lance
v14.0.0-beta.7: SQL `ORDER BY ... LIMIT` over `BaseTableAdapter` with
the rule registered returns the same rows as without it, the plan
contains a `Take`, and the output schema metadata stays erased. No
certification of `MetadataEraserExec` is needed because DataFusion
pushes the top-k sort below it.
ENT-2777
`FTS()` rejects JSON columns because validation only considers their
`LargeBinary` storage type. Validate the complete Arrow field in
embedded and remote creation so `lance.json` columns reach Lance's
existing JSON-aware inverted index, while ordinary binary fields remain
rejected.
JSON searches use path-qualified triples such as `meta.tag,str,nature`;
the API documentation describes this syntax and its field-scoping
behavior. Remote creation preserves the supplied FTS configuration.
Two integration limits remain: malformed triples can still panic in the
pinned Lance dependency (tracked by lance-format/lance#8812 and the
pending fix in lance-format/lance#8813), and the opt-in server test has
not run against a real deployment because no remote database is
configured. That test covers index completion, statistics, search
results, and index use; HTTP mock coverage only verifies the client
request.
Refs #3891.
TypeScript rejected `Index.fts({ baseTokenizer: "code" })` because the
public `BaseTokenizer` union omitted `"code"`, although the native
tokenizer already supports it.
Add `"code"` to the shared union, document its identifier behavior, and
regenerate the API reference. The regression test creates and searches a
code-tokenized FTS index with a typed configuration across Arrow
versions 15–18.
Validation:
- The regression failed before the fix with TS2322: `"code"` was not
assignable to `BaseTokenizer`.
- After the fix, all four cases pass: `pnpm test __test__/table.test.ts
--runInBand -t 'full text search code tokenizer' --verbose=false`.
- `pnpm build`, `pnpm lint`, `pnpm run docs`, and formatting of the
changed TypeScript files pass.
Fixes#4366
<!-- lance-gatekeeper-fix:v1 agent=8c37ac529cec8862bd5dbc218c5aaa8d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`Index.fts({ language: "english" })` and other unsupported language
values abort the Node process because the native factory unwraps the
error returned by `FtsIndexBuilder::language`.
Convert that error into a catchable JavaScript `Error`, using the same
message as `tokenize` and Python. Add regression coverage for
`"Klingon"` and lowercase `"english"`, plus a valid English stemming and
search case across Arrow 15–18.
Validation:
- Native and TypeScript builds: `pnpm build:debug --profile ci` and
`pnpm tsc`.
- Full-text and tokenization tests: `pnpm test __test__/table.test.ts
--runInBand --testNamePattern 'full text|FTS|tokeniz'` — 69 passed.
- Rust formatting and lint: `cargo fmt --all`; `cargo clippy --profile
ci --target x86_64-unknown-linux-gnu -p lancedb-nodejs -- -D warnings`.
- TypeScript formatting and lint: `pnpm exec biome format --write
__test__/table.test.ts` and `pnpm lint`; `pnpm run docs` passed without
generated changes.
Fixes#4367
<!-- lance-gatekeeper-fix:v1 agent=271bb96150580678f4e03b8e36beca03
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4379
Remote `table_names` discarded the continuation token after the first
root-listing request, silently returning the server's default page of 10
names. Remote `list_tables` also inherited that server default, while
local listings had no SDK page limit. Python's synchronous and namespace
wrappers still imposed a default limit of 10 names. The namespace
adapter also discarded continuation tokens, so an unlimited REST
namespace listing could stop at the server's page cap.
Follow server-provided root listing tokens until the requested name
limit or the end of the listing, preserving `start_after`, cache
population, and guards against empty or repeated tokens. The namespace
adapter collects every server page, then sorts names and applies the
name cursor and total limit: namespace tokens are opaque, and page order
need not be name order. Empty tokens end the listing; token cycles
return an error. Python sync and namespace `table_names` wrappers now
default to `None`, matching the asynchronous and TypeScript APIs.
Namespace overrides accept the base connection's optional limit,
avoiding import-time signature errors on Python 3.10 and 3.11. Set the
`list_tables` default to 100 at the shared connection boundary,
including namespaces, and return empty results for a zero limit without
sending an invalid server request. Update the SDK documentation to
describe these defaults.
`list_tables()` is now a paginated call in both modes: callers with more
than 100 tables must follow the returned token or supply an explicit
larger limit. `table_names()` without a limit returns every name.
Namespace table-name calls enumerate the namespace before applying name
cursors and limits; use `list_tables()` for paginated access.
Validation:
- Regression cases reproduce the original root-listing truncation and
cover 130 tables, opaque tokens, short pages, explicit limits,
`start_after`, and zero limits.
- Relevant Rust remote database, connection, and local listing test
groups: 175 passed. Namespace adapter tests: 23 passed, including opaque
tokens, unordered and empty pages, name cursors, limits, zero requests,
and token cycles.
- Rebuilt the Python extension; namespace and local/mocked remote
listing tests using `uv.lock`: 46 passed on Python 3.10 and 46 passed on
Python 3.13.
- REST namespace regressions reproduced the truncation before the fix
and now return all 12 names from a server capped at 10 per response, for
sync/async connections at root and child paths. They also cover
unordered pages, name cursors, explicit limits, and zero requests. The
exact Python 3.10 import error also reproduced before the
optional-signature fix.
- TypeScript connection test file: 38 passed; `pnpm build`, `pnpm lint`,
and `pnpm run docs` passed.
- `cargo check --quiet --features remote --tests --examples` and `cargo
clippy --quiet --features remote --tests --examples` passed.
- Rust formatting, Python Ruff formatting/lint, and diff whitespace
checks passed.
<!-- lance-gatekeeper-fix:v1 agent=43f92f1f05ac9ca74229af590945cb88
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Cause
Async remote pandas conversion could call `AsyncTable.to_lance()`, which
opened the server's table storage URI in the client process. This
bypassed server query and authorization APIs. The first repair routed
blob reads through the server, but a query and subsequent blob fetch
could resolve different table snapshots after replacement or compaction.
## Fix
- Reject `AsyncTable.to_lance()` for remote tables before requesting a
table URI.
- Convert remote nonblob and blob description results through the server
Arrow path.
- Reject automatic remote byte and lazy blob materialization until the
server can bind the query and fetches to one verified table identity and
version. The default lazy mode now raises `NotImplementedError` when a
blob column is projected; callers can request
`blob_mode="descriptions"`.
## Validation
- Mocked remote tests cover direct storage rejection, description
results, byte and lazy rejection before any blob fetch, and nonblob
projections.
- Focused local and remote blob tests passed (19 tests). `ruff check .`,
`ruff format .`, and pyright passed.
Fixes#4272
<!-- lance-gatekeeper-fix:v1 agent=35937e6e1d968f62eea59adcc56c84e9
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Python's FTS bindings always forwarded `stem=True` and
`remove_stop_words=True`, overriding Lance's code tokenizer defaults. As
a result, code indexes stemmed identifiers and discarded stop words, and
`lancedb.tokenize` reproduced the same behavior.
Use `None` for omitted filter options and only apply explicit booleans
in the Rust bindings. This preserves tokenizer defaults across local and
remote index creation, the deprecated `create_fts_index` helpers, and
standalone tokenization. Update the signatures and documentation to
describe the inherited defaults.
Regression tests cover stored local settings, token output, stop-word
searches, remote request payloads, explicit overrides, and the existing
simple tokenizer behavior. The regression suite produced 16 failures
before the fix and passes all 28 cases after it.
Validation:
- Rebuilt the Python extension using the uv-managed development
environment.
- Full Python FTS and index test files plus relevant remote index tests:
105 passed.
- `cargo clippy --quiet --features remote --tests --examples -p
lancedb-python`
- `cargo fmt --all`, Ruff format/lint with CI's pinned version 0.9.9,
and Pyright (zero errors).
Fixes#4365
<!-- lance-gatekeeper-fix:v1 agent=9c1db2d3fc25c49c5b340f8debf43a0f
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Python sync remote searches default `fast_search` to `False` and omitted
full-text search columns to `None`, while local searches leave fast
search unset and use `[]`. This makes otherwise identical
`to_query_object()` results depend on the backend.
Align these omitted defaults in `RemoteTable.search` while preserving
explicit fast-search settings. Keep the shared vector-column inference
and automatic FTS fallback provided by the base branch.
Regression tests compare complete local and remote query objects for
both reported vector dimensions, NumPy and batch vectors, explicit
columns, scans, and full-text queries. They also cover matching
ambiguity errors, explicit fast-search values, automatic FTS without
embeddings, and automatic vector search with embedding metadata, using
the shared remote fixture.
Validation after merging `main` at `bbc4aebe`:
- `cd python && uv run --no-sync --extra tests pytest
python/tests/test_remote_db.py -q`: 128 passed.
- `cd python && uv run --no-sync --extra tests pytest
python/tests/test_query.py python/tests/test_table.py -q -k
'query_serialization or async_search_without_query or none_query or
nprobes_zero or empty_vector'`: 27 passed.
- `ruff format .` and `ruff check .` from the repository root (Ruff
0.9.9): passed.
- `git diff --check`: passed.
Fixes#4391
<!-- lance-gatekeeper-fix:v1 agent=7cad560371d7aa64fb2511e510dd1bfd
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote SDKs report Cloud-specific errors against self-hosted servers,
and some Python wrappers raise different exception types for the same
unsupported operation. The sync index wrapper also warns that `replace`
is ignored even though Rust forwards it to the server.
Use backend-neutral table/server diagnostics and the existing
`NotSupported` error conversions for unsupported manifest operations,
blob ranges, primary keys, optimization, and HNSW-PQ indexes. Sync
Python delegates to the async paths, so manifest operations and
`optimize()` consistently raise `NotImplementedError`. Legacy and
config-based HNSW-PQ creation share the same error and alternative-index
advice.
Remove the false replacement warning and simplify the ignored
GPU-accelerator warning. Keep the deprecated cleanup/compaction no-ops,
with backend-neutral warnings that do not assume automatic server
maintenance.
Embedding configuration forwarding and stale warning removal are now
supplied by #4408 in `main`. Retain the regression that verifies full
embedding metadata and warning-free remote creation alongside the
diagnostics tests.
Validation against the merged `main` revision `bbc4aebe` and Lance
14.0.0-beta.1:
- Rebuilt the Python extension with `uv run --extra tests --extra dev
maturin develop --extras tests,dev`; all 122 remote Python tests passed.
Root Ruff checks and formatting passed using the uv-managed environment.
- `pnpm build`, `pnpm lint`, formatting, and `pnpm run docs` passed; all
47 remote Node tests and the focused local vector-index regression
passed.
- All 583 Rust remote tests and 54 query unit tests passed, along with
`cargo check --quiet --features remote --tests --examples`, `cargo
clippy --quiet --features remote --tests --examples`, and `cargo fmt
--all`.
Fixes#4393
<!-- lance-gatekeeper-fix:v1 agent=10bafd337999f4910855c3a9b71e7dcf
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote `merge_insert` rejected `MergeFilter::Expr` for conditional
updates and source-absent deletes before sending a request. Convert both
filters with `expr_to_sql_string`, matching the existing remote query
and delete paths.
Add a mocked-request regression covering all four SQL/Expr combinations,
including qualified `target`/`source` references and returned merge
statistics. Before the fix, the three expression cases failed with the
reported `NotSupported` error and the SQL-only case passed.
This fixes the reported Python delete-condition reproducer and both
existing Rust expression setters. In the current checkout, Python's
`when_matched_update_all(where=...)` parameter is string-only; this
change does not extend that binding.
Validation passed:
- All four regression cases.
- All 273 remote-table tests: `cargo test --quiet --profile ci
--features remote -p lancedb --lib remote::table::tests`.
- `cargo fmt --all` and `cargo fmt --all --check`.
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples -- -D warnings`.
Fixes#4438
<!-- lance-gatekeeper-fix:v1 agent=bbd935c2160bd3868f4cb0f1d3d7cfe5
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`MatchQuery(..., fuzziness=None)` raised a Pydantic validation error
before a search could run because the field accepted only integers.
Allow `Optional[int]`, retain the exact-match default of `0`, and make
the documented parameter type explicit. The existing Rust binding
already accepts and serializes the optional value.
Regression coverage checks construction and JSON serialization for
`None`, `0`, `1`, and `2`, plus native indexed searches with the
default, explicit exact matching, automatic mode, and one- and two-edit
controls. The two `None` regressions reproduced the original constructor
error before the fix.
Validation against the source-built extension in the locked uv
environment:
- `uv run --extra tests --extra dev maturin develop --extras tests,dev`
- `uv run --no-sync --extra tests --extra dev pytest
python/tests/test_fts.py -k 'match_query_fuzziness or fts_query_to_json'
-q` — 10 passed.
- `uv run --no-sync --extra tests --extra dev pytest
python/tests/test_fts.py -q --disable-warnings` — 61 passed.
- Locked-environment `ruff check .`, `ruff format .`, and `git diff
--check` passed.
The reported engine limitation remains observable with pinned Lance
`14.0.0-beta.1`: its dataset planner compatibility gate rewrites `None`
to zero, so a five-character AUTO query still omits one-edit matches.
Automatic expansion for longer terms requires upstream Lance activation.
The native acceptance regression uses a two-character term, whose
documented automatic distance is zero, while the JSON regression
verifies that Python forwards `None` as `null`. This keeps the repair
within the issue's requested Python constructor scope.
Fixes#4466
<!-- lance-gatekeeper-fix:v1 agent=f5f78166a4f0c7433e7890792b729c3d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`execute_query` / `execute_query_async` can now bind values to `$1` /
`$name` placeholders instead of splicing them into the SQL text. The
values travel as Arrow, so a float32 keeps its exact bits and type and a
vector stays a compact `FixedSizeList` (3 KB for 768 dims rather than ~9
KB of decimal literals parsed back as Float64).
```rust
db.execute_query_async("SELECT id FROM docs ORDER BY distance(vector, $vector) LIMIT $k")
.parameters(batch) // one-row RecordBatch
.execute()
.await?;
```
```python
db.execute_query(sql, parameters={"vector": np.float32_vector, "k": 10})
```
- A statement with parameters is sent on one `DoExchange`: the first
message carries the `CommandStatementQuery` descriptor and the parameter
schema, the second the one-row batch, and the result rows stream back on
the same call. Statements without parameters keep `PollFlightInfo` +
`DoGet` unchanged.
- Column `i` binds `$<i+1>`; a column whose name is not a number also
binds `$<name>`. Python accepts a list (positional), a dict (named), or
a one-row `RecordBatch` / `Table`; 1-D numpy / pyarrow arrays become a
`FixedSizeList` of their element type.
- A parameterized query runs on its call: `cancel()`, dropping the
handle before reading, or dropping the reader ends the call and stops
the server-side statement. It is still registered for `describe_query`.
- `Database::execute_query_async` now takes an `ExecuteQueryRequest`
options struct.
Server side: lancedb/sophon#8256.
Updates the Rust workspace Lance dependencies and Java lance-core from
14.0.0-beta.1 to
[v14.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.6),
and refreshes Cargo.lock. No compatibility fixes were required.
Validation passed: `cargo clippy --quiet --workspace --tests
--all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff
--check`.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Lance beta and RC artifacts are moving to `https://maven.lance.org`. Add
that release repository to the Java parent POM so LanceDB can resolve
explicit preview versions from it, while keeping Maven Central available
for historical releases and other dependencies.
Validated resolution of `org.lance:lance-core:14.0.0-beta.3` from a
fresh Maven cache using the actual `lancedb-core` POM. Maven recorded
the new `lance` repository as the source of both the POM and JAR. The
pinned Lance version and LanceDB's own publishing workflow are
unchanged.
Remove scaffold that accidentally caused the post-merge Rust
license-header check to fail. Include `docs/web-tests/**` in the PR
workflow paths so future docs test changes run the license checks before
merge.
Validation: `cargo fmt --all`; `git diff --check`;
license-header-checker v1.5.0 using all four workflow commands on a
clean checkout; Cargo metadata confirms all six Rust example targets
remain unchanged.
Refuse user writes (add, insert, update, delete, merge_insert, column
changes) to a table carrying `mv.definition`, so a materialized view is
written only by refresh. Index, optimize, and computed-column operations
stay allowed. Sophon's REST write paths are handled separately in
ENT-2878.
Fixes ENT-2876.
Move the public website pages into `docs/web` beside the SDK, with
shared navigation and executable example sources in `docs/web-tests`.
The combined website is assembled by
[lancedb/docs#355](https://github.com/lancedb/docs/pull/355), with six
Enterprise replacements from
[lancedb/sophon#7681](https://github.com/lancedb/sophon/pull/7681).
This includes Function guidance, runnable Quickstarts, regenerated
snippets, production contact/demo fixes, the legacy SQL route, and the
LeRobot redirect. LeRobot remains attributed to Geneva; all 34 Geneva
pages remain in the docs repository until Function officially launches.
Materialized-view guidance follows the full rebuild refresh introduced
by #4373: local refresh writes the complete result, has no `full`
argument, supports `source_version`, and no longer requires stable row
IDs. Deployment SQL refresh options and stable-ID setup are qualified by
version/configuration. Function input guidance distinguishes ordinary
stored columns from computed inputs supported by newer deployments.
**Validation:** independently accepted local checks include 95 Python
tests (real Enterprise excluded), eight TypeScript tests, Rust
Quickstart, snippet parity, typos, deterministic assembly, Mint
validation and links/anchors/redirects. The combined baseline retains
all 181 production routes and 17 legacy redirects. Examples ran locally
against source-built Python SDK 0.40.0b13; existing GitHub workflows do
not run `docs/web-tests`. New main at `069b2f7d` is 0.41.0b0 and adds
computed Function inputs; that SDK was not exercised by the accepted
tests. Latest prose corrections receive focused site validation.
**Merge/rollout:** merge this and #7681 before #355's combined CI, which
reads producer `main`. This is migration infrastructure: section
overlays, source-owned skills and automatic versioned paired releases
remain follow-ups. The current manual publisher is transitional. Hosted
validation, a supported SDK/Enterprise release pair and a real
Enterprise walkthrough remain pending.
Production Mintlify watches `lancedb/docs:deploy-freeze`, directory
`docs`; preserve that source through review. The separate SDK GitHub
Pages reference workflow rebuilds on main merges.
**Updated-head CI (9a50f8db):**
[Typos](https://github.com/lancedb/lancedb/actions/runs/37654185443/job/112904990558),
[PR
title](https://github.com/lancedb/lancedb/actions/runs/37654179507/job/112904974595)
and
[Aikido](https://app.aikido.dev/featurebranch/scan/215386145?groupId=74229)
pass. Critical findings in the example lockfiles were patched: all Sharp
copies use 0.35.5, protobufjs 7.6.6, tar 7.5.22, quinn-proto 0.11.19,
zerovec 0.11.8 and zerovec-derive 0.11.6. The SDK's product dependencies
are unchanged. The revised lockfiles pass eight TypeScript examples,
HuggingFace decode/resize/crop/rescale/normalize plus AVIF roundtrip,
and Rust Quickstart. Aikido passing does not mean every advisory is
cleared; npm still reports 29 high and five moderate advisories in the
example dependency graph. No scanner bypass or suppression was added.
The Function prose correction passes combined/standalone Mint,
deterministic assembly, routes and assets. The revised 325-file site
checksum is
`175bec06864abb400dd292e667ba93448c0479b8359fd7bf6146afabf9040647`. This
PR remains open for independent review; further merges are on hold.
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
add to table stats the number of rows that has
been marked for deletion, but not yet deleted.
note that when a table fragment is removed due to
deletion of all of its rows, these rows are not counted. since the main
use of this feature is to determine when table compaction is needed, the
fact that these rows are not reported should not impact that decision
(deleted table fragments don't need compaction).
Fixes: https://github.com/lancedb/lancedb/issues/3761
Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com>
Lets a registered-Function column read another computed column (SQL or
Function). Resolves ENT-2780.
Nothing records which rows of a computed column have been filled, so a
refresh treats a null computed input as "not yet computed". It skips the
row and leaves the outputs null until a later refresh, after the input
is filled. Because the Function never receives a null from a computed
input, that input binds a non-nullable parameter.
The trade-off is that a computed input which is null by definition (a
SQL expression that yields null, or a Function row skipped on error)
leaves the downstream row unfilled.
## Changes
- `plan_function_application` and `ensure_binding_matches_schema` now
accept computed inputs and record them as non-nullable. They still
refuse inputs that are:
- unrecognized computed kinds,
- incomplete declarations,
- internal Function assignment columns.
- A new public `function_input_is_computed` reports whether an input
reads a computed column. A new public `computed_function_inputs` lists
the inputs a server refresh must find non-null before it computes a row.
- `ensure_supported_function_metadata` now refuses a schema whose
computed columns read each other in a cycle. A declaration can only read
columns that already exist, so only a schema supplied whole can contain
one.
- Dropping an upstream output while a downstream binding still reads it
is refused by the existing binding guard. This is now covered by a test.
The server-side refresh change (skipping rows that are not ready,
clearing them on recompute) lives in sophon and follows this PR.
## Testing
- `cargo test -p lancedb --features remote --lib -- computed refresh
freshness schema_evolution function`: 255 passed.
- `cargo clippy -p lancedb --features remote --tests -- -D warnings`:
clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sync remote string searches ignored the table's embedding function when
`vector_column_name` was omitted. `auto` silently fell back to full-text
search, while `vector` and `hybrid` failed with `No embedding function
for None`.
Resolve vector columns from the schema before creating the query
builder, using the same inference helper as local search. Automatic
string queries without embedding functions select full-text search
first, so text-only tables and tables with multiple unconfigured vector
columns remain searchable. Vector and hybrid searches still reject
ambiguous columns, infer numeric queries by dimension, and honor
explicit column selection. Structured full-text queries and empty
queries retain their existing behavior.
Regression coverage uses embedding metadata returned by a mocked remote
server rather than a mocked embedding-function lookup. It also checks
automatic and explicit full-text queries on schemas with zero, one, or
multiple vector columns and no embedding functions.
Validation after merging `main` at `87ecfeac`:
- Confirmed the original embedding and ambiguity failures before the
fix, and reproduced the automatic FTS inference failures before
restoring the fallback.
- Remote, query, hybrid, and embedding test files through the uv-managed
environment: 240 passed, 4 skipped, 9 deselected (`not slow and not
s3_test`).
- Python extension built with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`.
- `ruff check .` and `ruff format .` from the repository root, using
CI's Ruff version 0.9.9.
- `cargo fmt --all`.
Fixes#4377
<!-- lance-gatekeeper-fix:v1 agent=c167c6c89d857342e5d7415b08142589
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote vector serialization unwrapped `serde_json::Number::from_f64`,
which returns `None` for NaN and infinity, causing a Tokio worker panic.
Validate vectors after conversion to Float32 in the shared `nearest_to`
/ `add_query_vector` builder so local and remote queries consistently
return `Error::InvalidInput` (mapped to `ValueError` by the Python
bindings). This also rejects finite Float64 inputs that overflow
Float32. Make remote serialization fallible to protect requests
constructed directly, including single-vector queries and both remote
batch-query paths.
Validation:
- All 116 query-related Rust tests passed with the `remote` feature,
including the three new regression cases.
- The three regression cases fail with the original production code and
pass with the fix. Coverage includes NaN, positive and negative
infinity, Float16/32/64 Arrow inputs, Float64 overflow, and rejection
before any HTTP request.
- `cargo fmt --all -- --check`, `cargo check --profile ci --locked -p
lancedb --features remote --tests --examples`, and `cargo clippy
--profile ci --locked -p lancedb --features remote --tests --examples --
-D warnings` passed.
Fixes#4375
<!-- lance-gatekeeper-fix:v1 agent=97d6925d4af7cd09767da8b552c9bcf3
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Sync remote `create_table(..., embedding_functions=...)` discarded the
supplied configuration and only logged an unsupported-feature warning.
Schema-only creation therefore lost the metadata needed by `add()` to
embed text, while text-only initial data produced no vector column.
Forward the configuration to the existing async creation path, remove
the stale warning, and document the parameter. The shared path stores
the embedding metadata and generates vectors for initial data and
subsequent additions.
Add a regression test against the mocked remote HTTP API for both
schema-only and text-only creation. It checks the outgoing Arrow schema
metadata, the metadata returned by the remote table, inferred vector
types, and generated vectors during creation and `add()`.
Validation:
- Both regression cases failed before the fix because the outgoing
embedding metadata was missing, then passed after the fix.
- `uv run --extra tests --extra dev pytest
python/tests/test_remote_db.py python/tests/test_embeddings.py -q -m
'not slow and not s3_test'`: 82 passed, 4 skipped, 8 deselected.
- Repository-wide `ruff check .` and `ruff format .` passed using the
uv-managed environment; `git diff --check` passed.
Fixes#4376
<!-- lance-gatekeeper-fix:v1 agent=a6816b1f3335438bd8ad192f36395213
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote `create_table` with `exist_ok` opened an existing table without
validating the supplied schema, so Python and TypeScript silently
accepted mismatches.
Capture the request schema before consuming its data stream, compare it
with the opened table's schema, and return the same `Error::Schema` and
message as the local backend when they differ. The existing open-table
callback continues to run before validation.
Regression coverage exercises matching schemas and differences in field
name, type, and nullability, for both empty and data-backed creation
with cached and uncached table opens. All 12 mismatch cases failed
against the original implementation; all 16 cases now pass.
Validation:
- `cargo test --profile ci --features remote -p lancedb --lib
remote::db::tests`: 113 passed.
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`: passed.
- `cargo fmt --all` and `git diff --check`: passed.
Fixes#4378
<!-- lance-gatekeeper-fix:v1 agent=5781a39ba124cf9094d36715730342b4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote handles at the latest version have no checkout version, so
`restore()` sent `{"version":null}` and surfaced a server
deserialization error. Validate the checkout version before constructing
the request and return `Error::InvalidInput` with the same message as
local tables. This shared Rust fix covers Python sync/async and
TypeScript callers.
Add a regression test proving that restore sends no request on a fresh
handle, after `checkout_latest()`, or after a successful restore clears
checkout state. It also verifies that a valid restore sends the
checked-out version. Update the schema-cache and branch restore tests to
check out a version before restoring and require success.
Validation:
- The new regression failed against the original implementation when the
mock received a null version.
- `cargo test --locked --profile ci --features remote -p lancedb --lib
restore` — 6 tests passed.
- `cargo fmt --all` and `git diff --check` passed.
- `cargo clippy --quiet --locked --profile ci --features remote -p
lancedb --tests --examples` passed.
Fixes#4381
<!-- lance-gatekeeper-fix:v1 agent=cc67544dfcbb46d631762752d55b39e3
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote update responses report their count as `updated_rows`, but the
client only recognized `rows_updated` and silently defaulted the count
to zero. Python sync/async and TypeScript both use this Rust result.
Accept `updated_rows` as a Serde alias while retaining `rows_updated`
and compatibility with older responses. Extend the remote mock test to
cover both keys, zero and missing counts, returned versions, and empty
responses, with and without a predicate.
Validation:
- Both nonzero `updated_rows` regression cases failed before the fix (0
instead of 5); all 11 focused remote tests pass after it.
- All 3 native update tests pass.
- `cargo fmt --all`
- `cargo clippy --quiet --profile ci --features remote -p lancedb
--tests --examples`
Fixes#4443
<!-- lance-gatekeeper-fix:v1 agent=423249073212c9840d96ccf6d4452905
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`maintained_indexes` on a MemWAL table has three values: a named list,
`[]` for none, and `null` for every index the table has, including ones
created later. This PR makes every client read all three back as they
were set.
- **Rust remote client** (also used by the Python and Node SDKs) reads
`null` as `None`.
- **Java client** reads `null` as `null`, not as an empty list.
- **Docs** for the setter, the field and the getter describe the three
values the same way in Rust, Python, Node and Java. The generated JS
docs are regenerated to match.
## Testing
- `rust/lancedb`: 73 LSM tests pass, including one case per value. With
the decoder typed as a list, the `null` case fails.
- Java: two cases for `null` and `[]` on the read path, run by the Java
CI job.
- `cargo fmt`, `cargo clippy -D warnings` and `ruff format` are clean.
BaseTableAdapter wraps every scan in MetadataEraserExec, which reported
an unknown cardinality effect. Lance's CountPushdown only looks through
nodes that keep every row, so a SQL COUNT(*) over the adapter always
scanned, even with the rule registered. The eraser now reports Equal.
with_new_children kept the old schema and panicked when the new child's
schema differed; it now rebuilds from the child.
Gatefixer
XY ZhanandXYZhan
Lance Release
Yang Cen
LanceDB Robot
Will JonesandClaude Opus 5.5
Kevin Dong
Jack Ye
Colin Patrick McCabe
Jonathan HsiehandClaude Opus 5.5
Wyatt Alt
Xuanwo
Brendan Clement
Yuval LifshitzandClaude Opus 5.0