Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.
The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.
Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.
BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.
Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.
Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):
- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.
All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.
Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
Python `Decimal` values and Arrow decimal scalars raise
`NotImplementedError` in `table.update(values=...)`. Register a Decimal
converter that emits quoted fixed-point text so the native cast to the
target column preserves every digit, including decimal128 and decimal256
values. Non-finite Decimal values raise `ValueError` with a static
message.
Closes#4360.
Validation with the native extension built from the current source at
7f593359, on macOS arm64 and Python 3.12.14:
- All 33 new regression cases failed before the fix and passed
afterward. They cover Python and Arrow scalar updates, signs, zero,
exponents, full 38/76-digit precision, and a Decimal context precision
of 6.
- 16 existing converter, update, and Decimal expression tests passed.
- Repository Ruff formatting and lint passed; formatting left all 156
files unchanged.
- A native round-trip check confirmed that an unquoted SQL number rounds
a high-precision value while the quoted Decimal update preserves it
exactly.
The full suite, other Python versions, and remote API were not
exercised.
AI assistance: Codex helped implement and validate the Decimal
conversion and its regression tests.
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.
The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
Moves Function timestamps on the wire from RFC 3339 strings to epoch
milliseconds.
## What
- `FunctionVersion.created_at` (an RFC 3339 string) becomes
`created_at_millis` (milliseconds since the epoch, as an integer).
That's the shape Secrets, jobs and `FunctionErrorRecord` already use.
- Rust: `FunctionVersion::created_at() -> &str` becomes
`created_at_millis() -> i64`.
- Python: `FunctionVersion.created_at: str` becomes `created_at_millis:
int`.
- Fixtures and the mock server now send `1787270400000`, the old
`2026-08-21T00:00:00Z`.
Node.js doesn't expose `FunctionVersion`, so it doesn't change.
## Compatibility
Breaking, and it must ship together with the matching server change: a
server that still sends `created_at` fails to deserialize on this
client. Functions aren't deployed yet, so nothing is migrated.
## Tests
- `cargo test --features remote --test first_class_function_slice1`: 10
passed
- `cargo test --features remote --lib function`: 51 passed
- `pytest python/tests/test_first_class_function*.py`: 100 passed
- clippy, `cargo fmt --check`, ruff check and ruff format are clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BY5xcEGEAa1E11VG55Kd1w
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`FTS()` rejects JSON columns because validation only considers their
`LargeBinary` storage type. Validate the complete Arrow field in
embedded and remote creation so `lance.json` columns reach Lance's
existing JSON-aware inverted index, while ordinary binary fields remain
rejected.
JSON searches use path-qualified triples such as `meta.tag,str,nature`;
the API documentation describes this syntax and its field-scoping
behavior. Remote creation preserves the supplied FTS configuration.
Two integration limits remain: malformed triples can still panic in the
pinned Lance dependency (tracked by lance-format/lance#8812 and the
pending fix in lance-format/lance#8813), and the opt-in server test has
not run against a real deployment because no remote database is
configured. That test covers index completion, statistics, search
results, and index use; HTTP mock coverage only verifies the client
request.
Refs #3891.
Fixes#4379
Remote `table_names` discarded the continuation token after the first
root-listing request, silently returning the server's default page of 10
names. Remote `list_tables` also inherited that server default, while
local listings had no SDK page limit. Python's synchronous and namespace
wrappers still imposed a default limit of 10 names. The namespace
adapter also discarded continuation tokens, so an unlimited REST
namespace listing could stop at the server's page cap.
Follow server-provided root listing tokens until the requested name
limit or the end of the listing, preserving `start_after`, cache
population, and guards against empty or repeated tokens. The namespace
adapter collects every server page, then sorts names and applies the
name cursor and total limit: namespace tokens are opaque, and page order
need not be name order. Empty tokens end the listing; token cycles
return an error. Python sync and namespace `table_names` wrappers now
default to `None`, matching the asynchronous and TypeScript APIs.
Namespace overrides accept the base connection's optional limit,
avoiding import-time signature errors on Python 3.10 and 3.11. Set the
`list_tables` default to 100 at the shared connection boundary,
including namespaces, and return empty results for a zero limit without
sending an invalid server request. Update the SDK documentation to
describe these defaults.
`list_tables()` is now a paginated call in both modes: callers with more
than 100 tables must follow the returned token or supply an explicit
larger limit. `table_names()` without a limit returns every name.
Namespace table-name calls enumerate the namespace before applying name
cursors and limits; use `list_tables()` for paginated access.
Validation:
- Regression cases reproduce the original root-listing truncation and
cover 130 tables, opaque tokens, short pages, explicit limits,
`start_after`, and zero limits.
- Relevant Rust remote database, connection, and local listing test
groups: 175 passed. Namespace adapter tests: 23 passed, including opaque
tokens, unordered and empty pages, name cursors, limits, zero requests,
and token cycles.
- Rebuilt the Python extension; namespace and local/mocked remote
listing tests using `uv.lock`: 46 passed on Python 3.10 and 46 passed on
Python 3.13.
- REST namespace regressions reproduced the truncation before the fix
and now return all 12 names from a server capped at 10 per response, for
sync/async connections at root and child paths. They also cover
unordered pages, name cursors, explicit limits, and zero requests. The
exact Python 3.10 import error also reproduced before the
optional-signature fix.
- TypeScript connection test file: 38 passed; `pnpm build`, `pnpm lint`,
and `pnpm run docs` passed.
- `cargo check --quiet --features remote --tests --examples` and `cargo
clippy --quiet --features remote --tests --examples` passed.
- Rust formatting, Python Ruff formatting/lint, and diff whitespace
checks passed.
<!-- lance-gatekeeper-fix:v1 agent=43f92f1f05ac9ca74229af590945cb88
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Cause
Async remote pandas conversion could call `AsyncTable.to_lance()`, which
opened the server's table storage URI in the client process. This
bypassed server query and authorization APIs. The first repair routed
blob reads through the server, but a query and subsequent blob fetch
could resolve different table snapshots after replacement or compaction.
## Fix
- Reject `AsyncTable.to_lance()` for remote tables before requesting a
table URI.
- Convert remote nonblob and blob description results through the server
Arrow path.
- Reject automatic remote byte and lazy blob materialization until the
server can bind the query and fetches to one verified table identity and
version. The default lazy mode now raises `NotImplementedError` when a
blob column is projected; callers can request
`blob_mode="descriptions"`.
## Validation
- Mocked remote tests cover direct storage rejection, description
results, byte and lazy rejection before any blob fetch, and nonblob
projections.
- Focused local and remote blob tests passed (19 tests). `ruff check .`,
`ruff format .`, and pyright passed.
Fixes#4272
<!-- lance-gatekeeper-fix:v1 agent=35937e6e1d968f62eea59adcc56c84e9
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Python's FTS bindings always forwarded `stem=True` and
`remove_stop_words=True`, overriding Lance's code tokenizer defaults. As
a result, code indexes stemmed identifiers and discarded stop words, and
`lancedb.tokenize` reproduced the same behavior.
Use `None` for omitted filter options and only apply explicit booleans
in the Rust bindings. This preserves tokenizer defaults across local and
remote index creation, the deprecated `create_fts_index` helpers, and
standalone tokenization. Update the signatures and documentation to
describe the inherited defaults.
Regression tests cover stored local settings, token output, stop-word
searches, remote request payloads, explicit overrides, and the existing
simple tokenizer behavior. The regression suite produced 16 failures
before the fix and passes all 28 cases after it.
Validation:
- Rebuilt the Python extension using the uv-managed development
environment.
- Full Python FTS and index test files plus relevant remote index tests:
105 passed.
- `cargo clippy --quiet --features remote --tests --examples -p
lancedb-python`
- `cargo fmt --all`, Ruff format/lint with CI's pinned version 0.9.9,
and Pyright (zero errors).
Fixes#4365
<!-- lance-gatekeeper-fix:v1 agent=9c1db2d3fc25c49c5b340f8debf43a0f
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Python sync remote searches default `fast_search` to `False` and omitted
full-text search columns to `None`, while local searches leave fast
search unset and use `[]`. This makes otherwise identical
`to_query_object()` results depend on the backend.
Align these omitted defaults in `RemoteTable.search` while preserving
explicit fast-search settings. Keep the shared vector-column inference
and automatic FTS fallback provided by the base branch.
Regression tests compare complete local and remote query objects for
both reported vector dimensions, NumPy and batch vectors, explicit
columns, scans, and full-text queries. They also cover matching
ambiguity errors, explicit fast-search values, automatic FTS without
embeddings, and automatic vector search with embedding metadata, using
the shared remote fixture.
Validation after merging `main` at `bbc4aebe`:
- `cd python && uv run --no-sync --extra tests pytest
python/tests/test_remote_db.py -q`: 128 passed.
- `cd python && uv run --no-sync --extra tests pytest
python/tests/test_query.py python/tests/test_table.py -q -k
'query_serialization or async_search_without_query or none_query or
nprobes_zero or empty_vector'`: 27 passed.
- `ruff format .` and `ruff check .` from the repository root (Ruff
0.9.9): passed.
- `git diff --check`: passed.
Fixes#4391
<!-- lance-gatekeeper-fix:v1 agent=7cad560371d7aa64fb2511e510dd1bfd
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote SDKs report Cloud-specific errors against self-hosted servers,
and some Python wrappers raise different exception types for the same
unsupported operation. The sync index wrapper also warns that `replace`
is ignored even though Rust forwards it to the server.
Use backend-neutral table/server diagnostics and the existing
`NotSupported` error conversions for unsupported manifest operations,
blob ranges, primary keys, optimization, and HNSW-PQ indexes. Sync
Python delegates to the async paths, so manifest operations and
`optimize()` consistently raise `NotImplementedError`. Legacy and
config-based HNSW-PQ creation share the same error and alternative-index
advice.
Remove the false replacement warning and simplify the ignored
GPU-accelerator warning. Keep the deprecated cleanup/compaction no-ops,
with backend-neutral warnings that do not assume automatic server
maintenance.
Embedding configuration forwarding and stale warning removal are now
supplied by #4408 in `main`. Retain the regression that verifies full
embedding metadata and warning-free remote creation alongside the
diagnostics tests.
Validation against the merged `main` revision `bbc4aebe` and Lance
14.0.0-beta.1:
- Rebuilt the Python extension with `uv run --extra tests --extra dev
maturin develop --extras tests,dev`; all 122 remote Python tests passed.
Root Ruff checks and formatting passed using the uv-managed environment.
- `pnpm build`, `pnpm lint`, formatting, and `pnpm run docs` passed; all
47 remote Node tests and the focused local vector-index regression
passed.
- All 583 Rust remote tests and 54 query unit tests passed, along with
`cargo check --quiet --features remote --tests --examples`, `cargo
clippy --quiet --features remote --tests --examples`, and `cargo fmt
--all`.
Fixes#4393
<!-- lance-gatekeeper-fix:v1 agent=10bafd337999f4910855c3a9b71e7dcf
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`MatchQuery(..., fuzziness=None)` raised a Pydantic validation error
before a search could run because the field accepted only integers.
Allow `Optional[int]`, retain the exact-match default of `0`, and make
the documented parameter type explicit. The existing Rust binding
already accepts and serializes the optional value.
Regression coverage checks construction and JSON serialization for
`None`, `0`, `1`, and `2`, plus native indexed searches with the
default, explicit exact matching, automatic mode, and one- and two-edit
controls. The two `None` regressions reproduced the original constructor
error before the fix.
Validation against the source-built extension in the locked uv
environment:
- `uv run --extra tests --extra dev maturin develop --extras tests,dev`
- `uv run --no-sync --extra tests --extra dev pytest
python/tests/test_fts.py -k 'match_query_fuzziness or fts_query_to_json'
-q` — 10 passed.
- `uv run --no-sync --extra tests --extra dev pytest
python/tests/test_fts.py -q --disable-warnings` — 61 passed.
- Locked-environment `ruff check .`, `ruff format .`, and `git diff
--check` passed.
The reported engine limitation remains observable with pinned Lance
`14.0.0-beta.1`: its dataset planner compatibility gate rewrites `None`
to zero, so a five-character AUTO query still omits one-edit matches.
Automatic expansion for longer terms requires upstream Lance activation.
The native acceptance regression uses a two-character term, whose
documented automatic distance is zero, while the JSON regression
verifies that Python forwards `None` as `null`. This keeps the repair
within the issue's requested Python constructor scope.
Fixes#4466
<!-- lance-gatekeeper-fix:v1 agent=f5f78166a4f0c7433e7890792b729c3d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`execute_query` / `execute_query_async` can now bind values to `$1` /
`$name` placeholders instead of splicing them into the SQL text. The
values travel as Arrow, so a float32 keeps its exact bits and type and a
vector stays a compact `FixedSizeList` (3 KB for 768 dims rather than ~9
KB of decimal literals parsed back as Float64).
```rust
db.execute_query_async("SELECT id FROM docs ORDER BY distance(vector, $vector) LIMIT $k")
.parameters(batch) // one-row RecordBatch
.execute()
.await?;
```
```python
db.execute_query(sql, parameters={"vector": np.float32_vector, "k": 10})
```
- A statement with parameters is sent on one `DoExchange`: the first
message carries the `CommandStatementQuery` descriptor and the parameter
schema, the second the one-row batch, and the result rows stream back on
the same call. Statements without parameters keep `PollFlightInfo` +
`DoGet` unchanged.
- Column `i` binds `$<i+1>`; a column whose name is not a number also
binds `$<name>`. Python accepts a list (positional), a dict (named), or
a one-row `RecordBatch` / `Table`; 1-D numpy / pyarrow arrays become a
`FixedSizeList` of their element type.
- A parameterized query runs on its call: `cancel()`, dropping the
handle before reading, or dropping the reader ends the call and stops
the server-side statement. It is still registered for `describe_query`.
- `Database::execute_query_async` now takes an `ExecuteQueryRequest`
options struct.
Server side: lancedb/sophon#8256.
Updates the Rust workspace Lance dependencies and Java lance-core from
14.0.0-beta.1 to
[v14.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.6),
and refreshes Cargo.lock. No compatibility fixes were required.
Validation passed: `cargo clippy --quiet --workspace --tests
--all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff
--check`.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Refuse user writes (add, insert, update, delete, merge_insert, column
changes) to a table carrying `mv.definition`, so a materialized view is
written only by refresh. Index, optimize, and computed-column operations
stay allowed. Sophon's REST write paths are handled separately in
ENT-2878.
Fixes ENT-2876.
add to table stats the number of rows that has
been marked for deletion, but not yet deleted.
note that when a table fragment is removed due to
deletion of all of its rows, these rows are not counted. since the main
use of this feature is to determine when table compaction is needed, the
fact that these rows are not reported should not impact that decision
(deleted table fragments don't need compaction).
Fixes: https://github.com/lancedb/lancedb/issues/3761
Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com>
Sync remote string searches ignored the table's embedding function when
`vector_column_name` was omitted. `auto` silently fell back to full-text
search, while `vector` and `hybrid` failed with `No embedding function
for None`.
Resolve vector columns from the schema before creating the query
builder, using the same inference helper as local search. Automatic
string queries without embedding functions select full-text search
first, so text-only tables and tables with multiple unconfigured vector
columns remain searchable. Vector and hybrid searches still reject
ambiguous columns, infer numeric queries by dimension, and honor
explicit column selection. Structured full-text queries and empty
queries retain their existing behavior.
Regression coverage uses embedding metadata returned by a mocked remote
server rather than a mocked embedding-function lookup. It also checks
automatic and explicit full-text queries on schemas with zero, one, or
multiple vector columns and no embedding functions.
Validation after merging `main` at `87ecfeac`:
- Confirmed the original embedding and ambiguity failures before the
fix, and reproduced the automatic FTS inference failures before
restoring the fallback.
- Remote, query, hybrid, and embedding test files through the uv-managed
environment: 240 passed, 4 skipped, 9 deselected (`not slow and not
s3_test`).
- Python extension built with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`.
- `ruff check .` and `ruff format .` from the repository root, using
CI's Ruff version 0.9.9.
- `cargo fmt --all`.
Fixes#4377
<!-- lance-gatekeeper-fix:v1 agent=c167c6c89d857342e5d7415b08142589
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Sync remote `create_table(..., embedding_functions=...)` discarded the
supplied configuration and only logged an unsupported-feature warning.
Schema-only creation therefore lost the metadata needed by `add()` to
embed text, while text-only initial data produced no vector column.
Forward the configuration to the existing async creation path, remove
the stale warning, and document the parameter. The shared path stores
the embedding metadata and generates vectors for initial data and
subsequent additions.
Add a regression test against the mocked remote HTTP API for both
schema-only and text-only creation. It checks the outgoing Arrow schema
metadata, the metadata returned by the remote table, inferred vector
types, and generated vectors during creation and `add()`.
Validation:
- Both regression cases failed before the fix because the outgoing
embedding metadata was missing, then passed after the fix.
- `uv run --extra tests --extra dev pytest
python/tests/test_remote_db.py python/tests/test_embeddings.py -q -m
'not slow and not s3_test'`: 82 passed, 4 skipped, 8 deselected.
- Repository-wide `ruff check .` and `ruff format .` passed using the
uv-managed environment; `git diff --check` passed.
Fixes#4376
<!-- lance-gatekeeper-fix:v1 agent=a6816b1f3335438bd8ad192f36395213
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`maintained_indexes` on a MemWAL table has three values: a named list,
`[]` for none, and `null` for every index the table has, including ones
created later. This PR makes every client read all three back as they
were set.
- **Rust remote client** (also used by the Python and Node SDKs) reads
`null` as `None`.
- **Java client** reads `null` as `null`, not as an empty list.
- **Docs** for the setter, the field and the getter describe the three
values the same way in Rust, Python, Node and Java. The generated JS
docs are regenerated to match.
## Testing
- `rust/lancedb`: 73 LSM tests pass, including one case per value. With
the decoder typed as a list, the `null` case fails.
- Java: two cases for `null` and `[]` on the read path, run by the Java
CI job.
- `cargo fmt`, `cargo clippy -D warnings` and `ruff format` are clean.
Simplify experimental materialized views to one output-only storage
layout and one full-rebuild refresh implementation.
## Current layout
Local materialized views currently maintain incremental-refresh state
alongside the materialized values:
- an internal `__source_row_id` column for source provenance;
- `mv.definition` plus `mv.incarnation`;
- source version/timestamp, view version, and last-refresh watermarks;
- optional `mv.staging` state for table-function sources; and
- stable-row-ID requirements on the source table.
Refresh classifies source changes, appends or evicts rows when possible,
and falls back to a rebuild for unsupported deltas. Remote
create/refresh also use separate REST mutation paths.
## New layout
An MV is an ordinary Lance backing table containing only:
- the defining query's output columns;
- its current data fragments; and
- `mv.definition`, encoded as a versioned SQL envelope such as
`{"kind":"query","format":1,"query":"SELECT ..."}`.
There is no provenance column, incarnation, source/view watermark,
staging metadata, or stable-row-ID requirement. Local refresh may still
pin a source version for reproducibility, but it always recomputes the
complete result.
Full refresh streams the new result into uncommitted fragments and then
publishes a single transaction that removes every old fragment and
installs the new set. A shared conflict token prevents concurrent
replacements from both committing. This fragment-replacement primitive
is also used by Sophon's SQL worker.
## New remote create workflow
```text
Connection::create_materialized_view(...)
-> builder renders the defining SELECT
-> remote Database receives a CreateMaterializedViewRequest
-> submit CREATE MATERIALIZED VIEW ... AS ... through Flight SQL
-> retain the SQL Query handle while the Job is running
-> consume the durable SQL-job result
-> open the created table and read mv.definition
-> return MaterializedView
```
Remote refresh similarly submits `REFRESH MATERIALIZED VIEW` through
Flight SQL. Cancelling either wrapper forwards cancellation to the
retained SQL query before aborting the local task, so the server
operation is not silently left running.
## API changes
- remove incremental/no-op modes and the `full` switch;
- remove distributed grouped/partitioned refresh APIs and native
vector-source MV APIs;
- keep full-refresh results across Rust, Python, and TypeScript;
- preserve list/describe/drop as object-management operations while
routing create/refresh mutations through SQL.
The storage contract is experimental and has no compatibility shim;
views using the removed layout must be recreated.
CatalogConnection.list_databases should return an iterator so that users
don't have to manually re-implement pagination. For those who do want
more control over pagination, add num_page_results and page_token to the
Iterator object, and also allow the iterator object to be created at a
specific page token.
Refs #4404.
AsyncTable.list_versions split the nanosecond timestamp with float
division/modulo (`ts // 1e9`, `ts % 1e9`).
At ~1.8e18 a float can't hold every integer, so the microseconds drifted
(.274000 came back as .273999).
Use `divmod(ts, 1_000_000_000)` and integer // 1000 instead.
Only the precision fix is included here. Timezone-aware UTC (option 1 in
#4404) is on hold for now,
and I can add it here or in a follow-up depending on what they prefer.
Added test_list_versions_timestamp_precision using the value from the
issue (1790989429274000000), expecting microsecond == 274000.
Empty list and NumPy vector queries were validated too late: local
search inferred a vector column first, producing misleading schema
errors, while remote search indexed `vector[0]` and raised `IndexError`.
Add a shared empty-vector check before column inference and query
creation, and validate direct vector-builder inputs. Local and remote
sync search now raise `ValueError: Query vector must not be empty` for
these inputs.
Validation:
- All 26 new regression cases failed on the original code and pass with
the fix, covering list/NumPy inputs, automatic/explicit vector search,
inferred/explicit column names, one/two local vector columns, and remote
search before query submission.
- After merging current `main`, all 32 focused cases passed, covering
these regressions and upstream async searches without a query.
- The query, table, remote, utility, hybrid-query, and database suites
passed in the rebuilt uv environment: 541 passed, 1 skipped, 1 slow test
deselected.
- `ruff check .`, `ruff format .`, and `cargo fmt --all` passed from the
repository root.
Fixes#4389
<!-- lance-gatekeeper-fix:v1 agent=f1f4875090ef391bb1aa912dd54026dc
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`nprobes(0)` bypassed the validation used by the minimum and maximum
probe setters, allowing zero into query requests and producing
inconsistent local and remote results. Sync Python could additionally
interpret the zero maximum as an unbounded search.
Reject zero in the Rust vector-query builder and propagate its input
error through the Python vector/hybrid and Node bindings. Validate
nonpositive counts in the sync Python vector and hybrid builders so
every SDK and backend reports `nprobes must be greater than 0` before
execution. Update the Rust examples and document the positive-count
requirement.
Regression tests cover local and remote vector/hybrid builders,
rejection before remote query requests, promised TypeScript query
vectors, and positive counts replacing both probe bounds.
Validation:
- Rust probe regression tests: 2 passed.
- Focused Python query, remote, hybrid, and serialization tests: 22
passed. The four sync regression cases fail with the original setters.
- Focused TypeScript indexed local and mocked remote tests: 2 passed.
- Workspace `cargo check --features remote --tests --examples` and
Clippy passed.
- Rust formatting, Python formatting/lint, and TypeScript
build/lint/docs passed. Repository-wide `ruff check .` reports 29
existing errors in unchanged CI scripts; `ruff check python` passes.
Fixes#4388
BREAKING CHANGE: Rust `VectorQuery::nprobes` now returns `Result<Self>`
instead of `Self`; callers must handle the result, as with the existing
minimum/maximum probe setters.
<!-- lance-gatekeeper-fix:v1 agent=e92fda5a212c7f9e80fa55e7b4e4634d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Synchronous local Python `create_table` callers using `mode="exist_ok"`
received `ValueError` instead of creating or opening a table.
Related to [#2900](https://github.com/lancedb/lancedb/issues/2900). The
local wrapper rejected this mode before the shared Rust parser could
validate it. Removing that guard delegates mode validation to the
parser, and the sync and async docs now describe `exist_ok`. Remote
option gaps and the TypeScript table type gap remain outside this
change.
The added tests retain both calls from [@wjones127's
comment](https://github.com/lancedb/lancedb/issues/2900#issuecomment-5964167206).
This builds on the code @wjones127 posted there. Three regression tests
fail before the change, pass after it, fail with the original `db.py`
restored, then pass again on HEAD.
Tested on Linux x86_64 with Python 3.14.7. From the Python directory,
`uv run --no-sync --extra tests --extra dev python -m pytest
python/tests/test_db.py -q` reports 53 passed. `uv run --no-sync --extra
tests --extra dev python -m pytest python/tests -q -rs -m "not slow and
not s3_test"` reports 1428 passed, 34 skipped, 68 deselected, and one
failure: the existing `test_get_lsm_write_spec`, also failing on clean
base (1425 passed, one failed). Ruff 0.9.9 `ruff format --check .` and
`ruff check .` pass. Tests used worktree Python code with the released
`lancedb==0.40.0b12` native wheel. No native source build was run.
Co-authored-by: Will Jones <willjones127@gmail.com>
Fixes#4390
Lance accepts column alterations with every change field unset, so a
path-only Python call could commit a new local version without changing
the schema. The Python binding also silently discarded unknown keys such
as `nulable`.
Validate every alteration in the shared Rust `Table` API before
dispatching to either backend, and reject unknown dictionary keys in the
Python binding while preserving the legacy `name` alias. Invalid batches
leave local schema and version unchanged and send no remote request.
Validation:
- Reproduced both reported calls on the original extension: versions
advanced from 1 to 2 to 3 with an unchanged schema; all 13 new
invalid-input Python cases failed before the fix.
- Rebuilt the extension with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`; all 18 focused Python cases pass across
sync/async and local/mocked remote tables.
- `cargo test --quiet --features remote -p lancedb --lib alter_column`:
all eight tests pass, including both new regressions.
- `cargo clippy --quiet --features remote --tests --examples` passes
across the workspace.
- `cargo fmt --all`, `ruff check .`, and `ruff format .` pass.
- `typos` 1.26.0 passes across the repository; intentional misspellings
in regression fixtures use the existing line-specific spelling
exemptions.
<!-- lance-gatekeeper-fix:v1 agent=36a547cd11108898893456042e38628b
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`AsyncTable.search()` and `search(None)` reached vector search without
assigning `vector_query`, raising `UnboundLocalError` on both local and
remote tables. Return a plain `AsyncQuery` before inferring the search
type, equivalent to `table.query()`, and update the overloads and
documentation to describe that return value.
Add six regression cases for omitted and explicit `None` queries on
local tables with and without vectors and on a mocked remote endpoint.
They verify scans, limits, filters, column projection, and the remote
request body.
Validation: all six regressions reproduced the original
`UnboundLocalError`; all 11 focused query tests pass with the fix,
including existing vector, full-text, hybrid, and sync scan coverage.
The uv development bootstrap, locked Ruff formatting/lint checks, and
focused Pyright return-type assertions also pass.
Fixes#4385
<!-- lance-gatekeeper-fix:v1 agent=bb9a7509f6dadde030d8113b172abc9a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Local synchronous `create_table(...,
enable_v2_manifest_paths=True/False)` failed with `TypeError` because
the deprecated argument was forwarded as a boolean in `storage_options`,
while the Rust binding expects string values.
Convert the argument to lowercase `"true"` or `"false"` before
forwarding it, preserving the deprecation warning. Add regression
coverage for both values with populated and schema-only tables, checking
stored data and the manifest mode after reopening.
Validation:
- All four regression cases reproduced the original `TypeError` before
the fix and pass afterward.
- `uv run --no-sync --extra tests --extra dev pytest
python/tests/test_db.py -q --tb=short -m 'not slow and not s3_test'`: 54
passed.
- Repository-wide `ruff format .` and `ruff check .` completed
successfully.
Fixes#4384
<!-- lance-gatekeeper-fix:v1 agent=9c14955b90d0c12d6a46605ec5ef32c0
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4383
`LanceTable.migrate_v2_manifest_paths()` raised `AttributeError` because
it called a nonexistent `AsyncTable.migrate_v2_manifest_paths()` method.
Delegate to the existing async method, `migrate_manifest_paths_v2()`,
while preserving the public sync API.
Add a local regression test for tables starting with either v1 or v2
manifest paths. It checks repeated migration, v2 manifest filenames on
disk, and preservation of data and version history after reopening the
table.
Validation:
- Both new regression cases failed with the reported `AttributeError`
before the fix.
- `cd python && uv run --extra tests pytest
python/tests/test_table.py::test_migrate_v2_manifest_paths
python/tests/test_db.py::test_create_table_v2_manifest_paths_async -q` —
3 passed.
- Repository-wide `ruff check .` and `ruff format .` passed using Ruff
0.15.20 from `python/uv.lock`.
- `git diff --check` passed.
<!-- lance-gatekeeper-fix:v1 agent=bb500f0d1ddefeba3c7d7d8390a2fb74
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Calling `close()` on a synchronous remote connection returned an
unawaited coroutine and left the connection usable. Local synchronous
connections had no close or context-manager API.
Add synchronous `close()` and `is_open()` to the connection interface,
delegate them to the existing connection handle for local, remote, and
namespace connections, and provide shared context-manager cleanup.
Closing is idempotent, and subsequent database operations use the
existing `Connection is closed` error. Document the lifecycle with
examples.
Validation:
- Reproduced the reported coroutine warning and missing local lifecycle
API on the original code; all 15 new regression cases failed before the
fix.
- All 168 tests in `test_db.py`, `test_remote_db.py`, and
`test_namespace.py` passed through the uv-managed environment, including
local, in-memory, namespace, native-derived, and mocked remote lifecycle
coverage.
- All 6 synchronous connection doctests passed.
- Repository-wide Ruff 0.9.9 lint and formatting passed; `git diff
--check` passed.
Fixes#4382
<!-- lance-gatekeeper-fix:v1 agent=a743139af98198f1d4c8934c9cc1d0c9
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
**The maintained set is an intent, not a snapshot.** An unnamed set is
stored as
the intent to maintain every index, which lance re-resolves whenever it
builds a
MemTable, so an index created after the spec installs is maintained too.
A named
set is checked here, where the caller can be told which names exist; an
index of
a kind the MemWAL cannot mirror fails a named set and is skipped by an
unnamed
one.
**A newly created index reaches a writer that is already open.** lancedb
caches a
`ShardWriter` per shard and reuses it, so `create_index` installs the
current set
on the cached writer. A failure there is reported rather than
propagated: the
index is committed durably by then, and the writer this process holds
may be
fenced or poisoned, which is no reason to fail creating an index.
**A table carries one spec.** `set_lsm_write_spec` fails while one is
installed
and names `unset_lsm_write_spec` in the error, because the generations
already
written were homed under that spec. Accepting a repeat that changed only
the
maintained set would hand a caller who also passed a different sharding
a success
that changed nothing they asked for.
**The read path asks what is actually carried.** An unnamed set leaves
the stored
list empty, so the list alone cannot decide whether an index is
maintained. The
read asks the spec *and* every resident MemTable whether it carries the
index, so
a MemTable built before the index joined the set refuses the read rather
than
answering without its rows. Both spec routes report an unnamed set as
`null`
rather than `[]`: maintain whatever the table gains, and maintain
nothing, are
different settings.
## Known gap
The guard above covers resident MemTables. Once a writer flushes and
closes none
is resident, so a retained generation written before the index existed
still gets
an indexed read that omits its rows.
This window is **wider than on main**: a set resolved at install time
excluded an
index created later, so the read was refused; an intent includes it, and
the
refusal only holds while a MemTable is resident. Closing it needs the
generation's
own coverage, which this layer cannot see — its snapshot carries a
generation and
a path — so it belongs in lance's scanner or flush path.
## Testing
`cargo nextest run -p lancedb --lib --features remote` — **1479 passed,
3
skipped**. `cargo fmt --all --check`, and `cargo clippy --features
remote --lib
--tests --examples -- -D warnings`, both clean.
Covered: the round trip of an unnamed set, a set over an installed spec
refused,
a spec replaced through unset, a read refused while a resident MemTable
predates
the index, and an index created mid-write reaching the open writer.
Two Python binding tests move with the semantics: an unnamed set now
reads back
as `None` rather than the list it resolves to, and the refusal is
matched on
`already set`. Neither was run locally — no Python build here — so CI is
the
check on them.
Updates the Rust workspace and Java Lance dependencies to
[v13.1.0-beta.1](https://github.com/lance-format/lance/releases/tag/v13.1.0-beta.1)
and refreshes Cargo.lock. Removes a redundant `pub(crate)` declaration
in the private Node.js blob module to satisfy the required Clippy check.
Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, `pnpm build`, and `pnpm test
__test__/blob.test.ts --runInBand` (26 tests passed) in `nodejs`.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adds namespace addressing to Functions, matching what Secrets already
have (#4150).
## API
`create_function` / `create_function_async`, `get_function`,
`list_functions`, `drop_function` / `drop_function_async` take
`namespace_path` — keyword-only in Python (sync, async and remote
connections) and a trailing `&[String]` in Rust — defaulting to the
root. The namespace is addressed in the route identifier
(`/v1/function/analytics$features$embed/describe`,
`/v1/namespace/analytics$features/function/list`); no body carries it.
`FunctionVersion` and `FunctionVersionRef` name their Function **by
parts** — a short `name` plus `namespace_path` — the same shape as
`SecretReference`, so a returned version reopens from its own fields:
```python
fv = db.create_function(embed, namespace_path=["analytics", "features"])
fv.name, fv.namespace_path # "embed", ("analytics", "features")
db.get_function(fv.name, version=fv.version, namespace_path=list(fv.namespace_path))
```
An empty path is omitted from the wire, so root documents keep their
shape. `namespace_path` is passed straight to the binding, so a bare
string (`namespace_path="prod"`) is a `TypeError` rather than four
one-character segments; segments are checked for addressability before a
request is built.
The binding and application allow-lists now share one field list,
`FUNCTION_VERSION_REF_FIELDS`, which admits `namespace_path` — without
it a namespaced binding or application read as coming from a newer
contract.
## Service side
The Sophon half serves and persists FunctionVersion / FunctionVersionRef
documents in this split form, keeping the joined id as its internal
catalog key. It lands in sophon with the submodule bump.
## Tests
- Rust: every Function verb addresses a namespace in the path; an
unaddressable segment never reaches the transport; `FunctionVersion`
round-trips a namespace and `reference()` carries it; a namespaced
binding and application are known shapes.
- Python: sync and async verbs address the namespace; a created Function
reopens from its own `name`/`namespace_path`; a bare-string path is
refused before any request.
Draft: compiles, `cargo fmt --check` and clippy `-D warnings` clean;
lancedb lib Function tests (116) and `first_class_function_slice2` pass
on this base.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
- infer fixed-size float32 Arrow vectors from `VectorField` metadata
when a model uses the standard `list[float]` annotation
- document the mypy-compatible embedding-field declaration while
preserving the existing `Vector(dim)` runtime API
- add regression coverage for schema conversion and model construction
## Root cause
`Vector(dim)` is a runtime constrained-type factory, but Python's static
typing rules do not permit call expressions in annotation positions.
Mypy's suggested `Vector[...]` spelling also cannot represent a
dimension computed at runtime and fails because `Vector` is a function.
`VectorField` already carries the embedding function and its dimension,
but schema conversion previously ignored that metadata for a statically
valid `list[float]` annotation.
The runtime factory remains available for dimension-constrained Pydantic
validation. Embedding-backed models that require mypy compatibility can
now use `list[float] = embedding.VectorField()`; LanceDB derives the
same nullable fixed-size float32 Arrow type from the field metadata.
## Validation
- `cd python && uv run --extra tests pytest
python/tests/test_pydantic.py -q` (23 passed)
- focused mypy check reproducing the original error and accepting the
new `LanceModel` declaration
- isolated schema checks with Pydantic 1.10.24 and 2.11.7
- `ruff format .`
- `ruff check .`
Fixes#1982
<!-- lance-gatekeeper-fix:v1 agent=877facbc659cdc9424bd367e12426366
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Updates the Rust workspace dependencies, Cargo lockfile, and Java
`lance-core` to [Lance
v13.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.18).
It includes lance-format/lance#9574, which bounds the memory an IVF
partition split uses when it re-reads raw vectors during
`optimize_indices`, and lance-format/lance#9582, which keeps the mTLS
REST namespace client on rustls.
Lance's new combined-fields FTS query variant (introduced by
v13.0.0-beta.17) is handled the same way as in #4336: the remote
capability check and Rust/Node bindings accept it, with a regression
test for the remote request payload, and the Python conversion reports
that this query type is not yet supported. This PR supersedes #4336.
## Cause
Python list and pandas rows were converted to Arrow before the target
blob v2 schema was applied. In a mixed bytes and URI batch, Arrow
inferred a binary column and encoded the URI as inline bytes; dict and
`lance.Blob` values could fail conversion.
## Fix
Build blob v2 columns from the original Python values using the target
field's storage type before converting the rest of each row. This
preserves inline bytes, external URIs, descriptor dictionaries, blob
objects, and nulls for `add`, `merge_insert`, and `create_table`.
## Validation
- `python/tests/test_blob.py`: 76 passed, 1 skipped.
- Python project Ruff check passed.
- Repository-wide Ruff check still reports 29 existing findings in `ci/`
files outside this change.
Fixes#4271
<!-- lance-gatekeeper-fix:v1 agent=54c95f06f11c88187b07a430aac6be2f
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Cause
On latest-tracking remote tables, `fetch_blob_files` probed a concrete
table version but retained `None` on each handle. Later range reads
resolved the latest version again, so an update or delete could make an
opened handle fail.
## Fix
For nonempty blobs, capture `x-lancedb-version` from the 206 size probe
and include it on every later range request. Reject a nonempty probe
without a valid version. Accept an empty-blob 416 probe with or without
that header, because an empty handle makes no later data range request.
The Python remote mock now covers both response shapes.
## Limitation
A table dropped and recreated with the same name can reuse version
numbers and row addresses. Version pinning does not identify the table
itself, so an old handle can still return data from the recreated table.
That identity case needs a separate fix. Callers that replace a table
should discard its old blob handles.
## Validation
- `cargo test --quiet --features remote -p lancedb --lib
remote::table::blobs::tests` (24 passed)
- Rebuilt the Python extension; `uv run --extra tests pytest
python/tests/test_remote_db.py -q -k remote_blob` (6 passed)
- `cargo clippy --quiet --features remote -p lancedb --lib --tests
--examples`
- `cargo fmt --all -- --check`, `ruff check .`, and `ruff format .`
Fixes#4276
<!-- lance-gatekeeper-fix:v1 agent=e309bdf75b1a87f6492a6dc3ee1bfd81
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Cause
Remote `fetch_blobs` sent every row ID in one Cloud request, exceeding
the server's 1024-row or 64 MiB byte cap. A single blob over the byte
cap could not be read through this method.
## Fix
- Send at most 1024 row IDs per request and split batches further when
the server reports the blob-byte cap, preserving input order and nulls.
- Resolve one exact dataset version before multi-request reads. If an
initial single request hits the byte cap, resolve the version before any
successful split or Range read; use that version for every resulting
request.
- Read an individual oversized blob through the existing Range route.
- Document the per-request caps and transparent handling in the Rust,
Python, and TypeScript APIs and generated JS docs.
## Validation
- Focused Rust `fetch_blobs` tests: 26 passed, including row-count,
byte-cap, Range fallback, and exact-version assertions.
- Rust clippy and formatting passed.
- Focused Python blob tests: 13 passed; Ruff lint and formatting passed.
- Node build, lint, docs generation, and focused `fetchBlobs` test
passed.
Fixes#4275
<!-- lance-gatekeeper-fix:v1 agent=6ae94d49bf1d4cae057bba72584e7b90
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## What's wrong
`RRFReranker.rerank_multivector` concatenates every vector result list
and passes the concatenation to `rerank_hybrid` as a single list.
`rerank_hybrid` then scores each row by its position in that
concatenation, so a row's rank depends on how many rows came before it
in *earlier* lists.
With two searches of 3 rows each (`K=60`):
| row | rank in its own list | score today | RRF score |
|---|---|---|---|
| 1 (top of list 1) | 1 | 1/61 | 1/61 |
| 3 (third of list 1) | 3 | 1/63 | 1/63 |
| 4 (top of list 2) | 1 | **1/64** | 1/61 |
| 5 (second of list 2) | 2 | **1/65** | 1/62 |
So the best hit of the second search ranks below the worst hit of the
first, and the order of the lists passed in decides the fused ranking.
`MRRReranker.rerank_multivector` already scores each list on its own
("MRR semantics require treating each vector result as a separate
ranking system"). RRF is defined the same way.
## Fix
Score each result list with its own ranks (`1 / (rank + K)`, summed
across lists for rows found by more than one), then dedupe the
concatenation by `_rowid`, attach `_relevance_score`, and sort. The
`return_score` handling is unchanged.
## Tests
`test_rrf_multivector_ranks_each_result_list` in
`python/python/tests/test_rerankers.py`:
- On `main`, it fails with `assert 0.015625 == 0.01639344262295082` (the
top hit of list 2 got `1/64` instead of `1/61`).
- With the fix, it passes, and so do the other tests in
`test_rerankers.py` (16 passed, 12 skipped for missing API keys or
models).
- `ruff format --check` and `ruff check` are clean.
The fix was found and written by an AI agent (Breken) and checked
against the tests above.
Co-authored-by: breken-ai <312387581+breken-ai@users.noreply.github.com>
Lance Release
Yang Cen
Kevin Dong
Jack Ye
Jonathan HsiehandClaude Opus 5.5
Xuanwo
lancedb-gatefixer[bot]
Wyatt Alt
Colin Patrick McCabe
Yuval LifshitzandClaude Opus 5.0
XY Zhan
Aviral Saxena
Chi-Wei LeeandWill Jones
brekenandbreken-ai