Sync remote table and connection methods inherited bare base-class
stubs, so unsupported conversion and bulk-drop operations raised
`NotImplementedError` with an empty message and differed from the async
API.
Delegate `to_lance()`, `drop_all_tables()`, and `drop_database()` to the
async implementation. Sync and async remote Lance conversion share
#4306's native-table guard and reject before requesting a URI or opening
client storage. Conversion documentation directs remote reads through
server queries.
Add an explicit Polars error, suggest query conversions in
Arrow/Pandas/Polars errors, and give the remaining base-class errors an
operation name. Preserve upstream `blob_mode` validation for remote
Arrow and Pandas conversion. The shared remote bulk-drop error suggests
dropping tables individually.
Regression tests cover sync/async rejection parity, accessible and
missing advertised locations, pinned versions, and bulk-drop rejection
without requests. Sync blob query tests verify that descriptions come
from the server and unsupported modes cannot open client storage. Keep
the remote closed-blob-handle assertion consistent with the upstream
`ValueError` contract.
Validation:
- All 30 focused regression cases passed.
- 222 Python tests passed with rebuilt bindings in the locked uv
environment: the full remote and error suites, plus selected local
conversion, blob, and deletion tests.
- Python lint and formatting, Rust formatting, `cargo check --quiet
--features remote --tests --examples`, and `cargo clippy --quiet
--features remote --tests --examples` passed.
- Root Ruff checking reports 49 existing errors in upstream
`docs/web-tests`; those files are unchanged relative to `main`.
Fixes#4392
<!-- lance-gatekeeper-fix:v1 agent=95db5dd7002043a2c48924ded18b51b2
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
`Table.take_offsets` and `Table.take_row_ids` declare a keyword-only
`with_row_id: bool = False`, but none of the three implementations
accept it — `LanceTable`, `AsyncTable` and `RemoteTable` all raise
`TypeError: ... got an unexpected keyword argument 'with_row_id'` on the
documented call. The builders they return already support it
(`LanceTakeQueryBuilder.with_row_id()` /
`AsyncTakeQuery.with_row_id()`), so this is a signature that never
reached the implementations. The base docstrings also don't mention the
parameter and still say `take_offsets` returns a `pa.RecordBatch` (it
returns a `LanceTakeQueryBuilder`), so those are corrected too.
Each implementation now accepts the kwarg and applies `.with_row_id()`
to the builder it returns. Tests: sync and async assert `_rowid` appears
in the result columns; the remote test uses the suite's
request-capturing fixture and asserts `with_row_id: true` reaches the
wire (and is absent by default). All three fail on `main` with the
`TypeError`. ruff format/check clean.
A related mismatch I left alone: the deprecated
`LanceTable.create_scalar_index`/`create_fts_index` also drop a
`wait_timeout` the base declares — deprecated since 0.25, so not worth
touching here.
Co-authored-by: Xuanwo <github@xuanwo.io>
Addresses option 2 of #4440.
tokenize() builds the FTS analyzer on the client. For jieba/* and
lindera/* that needs model files under LANCE_LANGUAGE_MODEL_HOME, and
when they’re missing you only got lance’s raw Invalid directory path:
... error, even though server-side search works.
Now, if a jieba/* or lindera/* tokenizer can’t be built, the error keeps
the original cause and adds that tokenize() runs on the client, which
directory the files need to be in, and that for remote tables they must
match the server’s. I also updated the tokenize docs (Rust, Python, TS,
generated JS docs) to say a mismatch can’t be detected, so tokens may
differ from the server’s index.
Behavior when the files exist is unchanged. Detecting a mismatch needs a
server-side tokenize endpoint (option 1), so I left it out. The hint
isn’t limited to remote tables, since Table::as_native is slated for
removal and the wording fits local tables too.
Tests: Rust test with mocked jieba/default index_details and an empty
model dir, one checking other tokenizer errors don’t get the hint, and
Python sync/async tests.
## What
Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.
### 1. MemWAL APIs drop "LSM"
The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.
| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |
TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.
Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.
### 2. Stats use SSTable and table shard
Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.
**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.
**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.
| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |
Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.
`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.
## Compatibility
Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:
- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies
## Validation
- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
`RemoteTable::set_unenforced_primary_key` returned `NotSupported`, so
the call failed against LanceDB Cloud and enterprise from every SDK —
Python and TypeScript both forward to it. That also blocked sharded LSM
writes, since the server rejects bucket/identity sharding on a table
that declares no unenforced primary key, and the SDK had no way to
declare one.
No server-side support was missing. The unenforced primary key is Lance
schema field metadata, and the existing `update_field_metadata` endpoint
writes exactly that, so the remote table now installs the key through
it. The commit layer behind that endpoint installs the position,
enforces immutability and runs `verify_primary_key()` — the same code a
native table reaches, so both paths agree on semantics and not just on
error messages.
## Changes
- `rust/lancedb/src/table/primary_key.rs`: the request validation
(single column, column exists, supported dtype, key not already set) and
the metadata edit (set the 1-indexed position, clear the legacy boolean
key) move into `validate()` and `install_edit()`, shared by both table
types so they cannot drift. Native behaviour is unchanged.
- `rust/lancedb/src/remote/table.rs`: real implementation —
`check_mutable()`, validate against the described schema, then one
`update_field_metadata` request.
- `python/python/lancedb/remote/table.py`: drop the "Not supported on
LanceDB Cloud." docstring.
No binding changes were needed; Python and TypeScript already forward to
the Rust `BaseTable` method.
## Tests
Three mocked-server tests in `rust/lancedb/src/remote/table.rs`:
- the happy path, asserting the request body carries
`lance-schema:unenforced-primary-key:position: "1"` and a null legacy
key on the right field path;
- the invalid requests a native table also rejects (empty, compound,
unknown column, unsupported dtype), each failing before any write
reaches the server;
- the already-set case, rejected from the schema the server already
reports.
`cargo test --features remote -p lancedb --lib`: 1186 passed, 0 failed.
`cargo clippy --features remote -p lancedb --lib --tests` is clean.
## Note for enterprise
Sophon pins the lancedb submodule at `v0.39.0-beta.6`, so this reaches
enterprise on the next bump.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011xV8EebEZDmDDV8hVBm6MD
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.
The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
`FTS()` rejects JSON columns because validation only considers their
`LargeBinary` storage type. Validate the complete Arrow field in
embedded and remote creation so `lance.json` columns reach Lance's
existing JSON-aware inverted index, while ordinary binary fields remain
rejected.
JSON searches use path-qualified triples such as `meta.tag,str,nature`;
the API documentation describes this syntax and its field-scoping
behavior. Remote creation preserves the supplied FTS configuration.
Two integration limits remain: malformed triples can still panic in the
pinned Lance dependency (tracked by lance-format/lance#8812 and the
pending fix in lance-format/lance#8813), and the opt-in server test has
not run against a real deployment because no remote database is
configured. That test covers index completion, statistics, search
results, and index use; HTTP mock coverage only verifies the client
request.
Refs #3891.
Fixes#4379
Remote `table_names` discarded the continuation token after the first
root-listing request, silently returning the server's default page of 10
names. Remote `list_tables` also inherited that server default, while
local listings had no SDK page limit. Python's synchronous and namespace
wrappers still imposed a default limit of 10 names. The namespace
adapter also discarded continuation tokens, so an unlimited REST
namespace listing could stop at the server's page cap.
Follow server-provided root listing tokens until the requested name
limit or the end of the listing, preserving `start_after`, cache
population, and guards against empty or repeated tokens. The namespace
adapter collects every server page, then sorts names and applies the
name cursor and total limit: namespace tokens are opaque, and page order
need not be name order. Empty tokens end the listing; token cycles
return an error. Python sync and namespace `table_names` wrappers now
default to `None`, matching the asynchronous and TypeScript APIs.
Namespace overrides accept the base connection's optional limit,
avoiding import-time signature errors on Python 3.10 and 3.11. Set the
`list_tables` default to 100 at the shared connection boundary,
including namespaces, and return empty results for a zero limit without
sending an invalid server request. Update the SDK documentation to
describe these defaults.
`list_tables()` is now a paginated call in both modes: callers with more
than 100 tables must follow the returned token or supply an explicit
larger limit. `table_names()` without a limit returns every name.
Namespace table-name calls enumerate the namespace before applying name
cursors and limits; use `list_tables()` for paginated access.
Validation:
- Regression cases reproduce the original root-listing truncation and
cover 130 tables, opaque tokens, short pages, explicit limits,
`start_after`, and zero limits.
- Relevant Rust remote database, connection, and local listing test
groups: 175 passed. Namespace adapter tests: 23 passed, including opaque
tokens, unordered and empty pages, name cursors, limits, zero requests,
and token cycles.
- Rebuilt the Python extension; namespace and local/mocked remote
listing tests using `uv.lock`: 46 passed on Python 3.10 and 46 passed on
Python 3.13.
- REST namespace regressions reproduced the truncation before the fix
and now return all 12 names from a server capped at 10 per response, for
sync/async connections at root and child paths. They also cover
unordered pages, name cursors, explicit limits, and zero requests. The
exact Python 3.10 import error also reproduced before the
optional-signature fix.
- TypeScript connection test file: 38 passed; `pnpm build`, `pnpm lint`,
and `pnpm run docs` passed.
- `cargo check --quiet --features remote --tests --examples` and `cargo
clippy --quiet --features remote --tests --examples` passed.
- Rust formatting, Python Ruff formatting/lint, and diff whitespace
checks passed.
<!-- lance-gatekeeper-fix:v1 agent=43f92f1f05ac9ca74229af590945cb88
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Cause
Async remote pandas conversion could call `AsyncTable.to_lance()`, which
opened the server's table storage URI in the client process. This
bypassed server query and authorization APIs. The first repair routed
blob reads through the server, but a query and subsequent blob fetch
could resolve different table snapshots after replacement or compaction.
## Fix
- Reject `AsyncTable.to_lance()` for remote tables before requesting a
table URI.
- Convert remote nonblob and blob description results through the server
Arrow path.
- Reject automatic remote byte and lazy blob materialization until the
server can bind the query and fetches to one verified table identity and
version. The default lazy mode now raises `NotImplementedError` when a
blob column is projected; callers can request
`blob_mode="descriptions"`.
## Validation
- Mocked remote tests cover direct storage rejection, description
results, byte and lazy rejection before any blob fetch, and nonblob
projections.
- Focused local and remote blob tests passed (19 tests). `ruff check .`,
`ruff format .`, and pyright passed.
Fixes#4272
<!-- lance-gatekeeper-fix:v1 agent=35937e6e1d968f62eea59adcc56c84e9
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Python's FTS bindings always forwarded `stem=True` and
`remove_stop_words=True`, overriding Lance's code tokenizer defaults. As
a result, code indexes stemmed identifiers and discarded stop words, and
`lancedb.tokenize` reproduced the same behavior.
Use `None` for omitted filter options and only apply explicit booleans
in the Rust bindings. This preserves tokenizer defaults across local and
remote index creation, the deprecated `create_fts_index` helpers, and
standalone tokenization. Update the signatures and documentation to
describe the inherited defaults.
Regression tests cover stored local settings, token output, stop-word
searches, remote request payloads, explicit overrides, and the existing
simple tokenizer behavior. The regression suite produced 16 failures
before the fix and passes all 28 cases after it.
Validation:
- Rebuilt the Python extension using the uv-managed development
environment.
- Full Python FTS and index test files plus relevant remote index tests:
105 passed.
- `cargo clippy --quiet --features remote --tests --examples -p
lancedb-python`
- `cargo fmt --all`, Ruff format/lint with CI's pinned version 0.9.9,
and Pyright (zero errors).
Fixes#4365
<!-- lance-gatekeeper-fix:v1 agent=9c1db2d3fc25c49c5b340f8debf43a0f
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Python sync remote searches default `fast_search` to `False` and omitted
full-text search columns to `None`, while local searches leave fast
search unset and use `[]`. This makes otherwise identical
`to_query_object()` results depend on the backend.
Align these omitted defaults in `RemoteTable.search` while preserving
explicit fast-search settings. Keep the shared vector-column inference
and automatic FTS fallback provided by the base branch.
Regression tests compare complete local and remote query objects for
both reported vector dimensions, NumPy and batch vectors, explicit
columns, scans, and full-text queries. They also cover matching
ambiguity errors, explicit fast-search values, automatic FTS without
embeddings, and automatic vector search with embedding metadata, using
the shared remote fixture.
Validation after merging `main` at `bbc4aebe`:
- `cd python && uv run --no-sync --extra tests pytest
python/tests/test_remote_db.py -q`: 128 passed.
- `cd python && uv run --no-sync --extra tests pytest
python/tests/test_query.py python/tests/test_table.py -q -k
'query_serialization or async_search_without_query or none_query or
nprobes_zero or empty_vector'`: 27 passed.
- `ruff format .` and `ruff check .` from the repository root (Ruff
0.9.9): passed.
- `git diff --check`: passed.
Fixes#4391
<!-- lance-gatekeeper-fix:v1 agent=7cad560371d7aa64fb2511e510dd1bfd
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Remote SDKs report Cloud-specific errors against self-hosted servers,
and some Python wrappers raise different exception types for the same
unsupported operation. The sync index wrapper also warns that `replace`
is ignored even though Rust forwards it to the server.
Use backend-neutral table/server diagnostics and the existing
`NotSupported` error conversions for unsupported manifest operations,
blob ranges, primary keys, optimization, and HNSW-PQ indexes. Sync
Python delegates to the async paths, so manifest operations and
`optimize()` consistently raise `NotImplementedError`. Legacy and
config-based HNSW-PQ creation share the same error and alternative-index
advice.
Remove the false replacement warning and simplify the ignored
GPU-accelerator warning. Keep the deprecated cleanup/compaction no-ops,
with backend-neutral warnings that do not assume automatic server
maintenance.
Embedding configuration forwarding and stale warning removal are now
supplied by #4408 in `main`. Retain the regression that verifies full
embedding metadata and warning-free remote creation alongside the
diagnostics tests.
Validation against the merged `main` revision `bbc4aebe` and Lance
14.0.0-beta.1:
- Rebuilt the Python extension with `uv run --extra tests --extra dev
maturin develop --extras tests,dev`; all 122 remote Python tests passed.
Root Ruff checks and formatting passed using the uv-managed environment.
- `pnpm build`, `pnpm lint`, formatting, and `pnpm run docs` passed; all
47 remote Node tests and the focused local vector-index regression
passed.
- All 583 Rust remote tests and 54 query unit tests passed, along with
`cargo check --quiet --features remote --tests --examples`, `cargo
clippy --quiet --features remote --tests --examples`, and `cargo fmt
--all`.
Fixes#4393
<!-- lance-gatekeeper-fix:v1 agent=10bafd337999f4910855c3a9b71e7dcf
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
add to table stats the number of rows that has
been marked for deletion, but not yet deleted.
note that when a table fragment is removed due to
deletion of all of its rows, these rows are not counted. since the main
use of this feature is to determine when table compaction is needed, the
fact that these rows are not reported should not impact that decision
(deleted table fragments don't need compaction).
Fixes: https://github.com/lancedb/lancedb/issues/3761
Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com>
Sync remote string searches ignored the table's embedding function when
`vector_column_name` was omitted. `auto` silently fell back to full-text
search, while `vector` and `hybrid` failed with `No embedding function
for None`.
Resolve vector columns from the schema before creating the query
builder, using the same inference helper as local search. Automatic
string queries without embedding functions select full-text search
first, so text-only tables and tables with multiple unconfigured vector
columns remain searchable. Vector and hybrid searches still reject
ambiguous columns, infer numeric queries by dimension, and honor
explicit column selection. Structured full-text queries and empty
queries retain their existing behavior.
Regression coverage uses embedding metadata returned by a mocked remote
server rather than a mocked embedding-function lookup. It also checks
automatic and explicit full-text queries on schemas with zero, one, or
multiple vector columns and no embedding functions.
Validation after merging `main` at `87ecfeac`:
- Confirmed the original embedding and ambiguity failures before the
fix, and reproduced the automatic FTS inference failures before
restoring the fallback.
- Remote, query, hybrid, and embedding test files through the uv-managed
environment: 240 passed, 4 skipped, 9 deselected (`not slow and not
s3_test`).
- Python extension built with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`.
- `ruff check .` and `ruff format .` from the repository root, using
CI's Ruff version 0.9.9.
- `cargo fmt --all`.
Fixes#4377
<!-- lance-gatekeeper-fix:v1 agent=c167c6c89d857342e5d7415b08142589
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Sync remote `create_table(..., embedding_functions=...)` discarded the
supplied configuration and only logged an unsupported-feature warning.
Schema-only creation therefore lost the metadata needed by `add()` to
embed text, while text-only initial data produced no vector column.
Forward the configuration to the existing async creation path, remove
the stale warning, and document the parameter. The shared path stores
the embedding metadata and generates vectors for initial data and
subsequent additions.
Add a regression test against the mocked remote HTTP API for both
schema-only and text-only creation. It checks the outgoing Arrow schema
metadata, the metadata returned by the remote table, inferred vector
types, and generated vectors during creation and `add()`.
Validation:
- Both regression cases failed before the fix because the outgoing
embedding metadata was missing, then passed after the fix.
- `uv run --extra tests --extra dev pytest
python/tests/test_remote_db.py python/tests/test_embeddings.py -q -m
'not slow and not s3_test'`: 82 passed, 4 skipped, 8 deselected.
- Repository-wide `ruff check .` and `ruff format .` passed using the
uv-managed environment; `git diff --check` passed.
Fixes#4376
<!-- lance-gatekeeper-fix:v1 agent=a6816b1f3335438bd8ad192f36395213
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Empty list and NumPy vector queries were validated too late: local
search inferred a vector column first, producing misleading schema
errors, while remote search indexed `vector[0]` and raised `IndexError`.
Add a shared empty-vector check before column inference and query
creation, and validate direct vector-builder inputs. Local and remote
sync search now raise `ValueError: Query vector must not be empty` for
these inputs.
Validation:
- All 26 new regression cases failed on the original code and pass with
the fix, covering list/NumPy inputs, automatic/explicit vector search,
inferred/explicit column names, one/two local vector columns, and remote
search before query submission.
- After merging current `main`, all 32 focused cases passed, covering
these regressions and upstream async searches without a query.
- The query, table, remote, utility, hybrid-query, and database suites
passed in the rebuilt uv environment: 541 passed, 1 skipped, 1 slow test
deselected.
- `ruff check .`, `ruff format .`, and `cargo fmt --all` passed from the
repository root.
Fixes#4389
<!-- lance-gatekeeper-fix:v1 agent=f1f4875090ef391bb1aa912dd54026dc
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`nprobes(0)` bypassed the validation used by the minimum and maximum
probe setters, allowing zero into query requests and producing
inconsistent local and remote results. Sync Python could additionally
interpret the zero maximum as an unbounded search.
Reject zero in the Rust vector-query builder and propagate its input
error through the Python vector/hybrid and Node bindings. Validate
nonpositive counts in the sync Python vector and hybrid builders so
every SDK and backend reports `nprobes must be greater than 0` before
execution. Update the Rust examples and document the positive-count
requirement.
Regression tests cover local and remote vector/hybrid builders,
rejection before remote query requests, promised TypeScript query
vectors, and positive counts replacing both probe bounds.
Validation:
- Rust probe regression tests: 2 passed.
- Focused Python query, remote, hybrid, and serialization tests: 22
passed. The four sync regression cases fail with the original setters.
- Focused TypeScript indexed local and mocked remote tests: 2 passed.
- Workspace `cargo check --features remote --tests --examples` and
Clippy passed.
- Rust formatting, Python formatting/lint, and TypeScript
build/lint/docs passed. Repository-wide `ruff check .` reports 29
existing errors in unchanged CI scripts; `ruff check python` passes.
Fixes#4388
BREAKING CHANGE: Rust `VectorQuery::nprobes` now returns `Result<Self>`
instead of `Self`; callers must handle the result, as with the existing
minimum/maximum probe setters.
<!-- lance-gatekeeper-fix:v1 agent=e92fda5a212c7f9e80fa55e7b4e4634d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4390
Lance accepts column alterations with every change field unset, so a
path-only Python call could commit a new local version without changing
the schema. The Python binding also silently discarded unknown keys such
as `nulable`.
Validate every alteration in the shared Rust `Table` API before
dispatching to either backend, and reject unknown dictionary keys in the
Python binding while preserving the legacy `name` alias. Invalid batches
leave local schema and version unchanged and send no remote request.
Validation:
- Reproduced both reported calls on the original extension: versions
advanced from 1 to 2 to 3 with an unchanged schema; all 13 new
invalid-input Python cases failed before the fix.
- Rebuilt the extension with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`; all 18 focused Python cases pass across
sync/async and local/mocked remote tables.
- `cargo test --quiet --features remote -p lancedb --lib alter_column`:
all eight tests pass, including both new regressions.
- `cargo clippy --quiet --features remote --tests --examples` passes
across the workspace.
- `cargo fmt --all`, `ruff check .`, and `ruff format .` pass.
- `typos` 1.26.0 passes across the repository; intentional misspellings
in regression fixtures use the existing line-specific spelling
exemptions.
<!-- lance-gatekeeper-fix:v1 agent=36a547cd11108898893456042e38628b
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`AsyncTable.search()` and `search(None)` reached vector search without
assigning `vector_query`, raising `UnboundLocalError` on both local and
remote tables. Return a plain `AsyncQuery` before inferring the search
type, equivalent to `table.query()`, and update the overloads and
documentation to describe that return value.
Add six regression cases for omitted and explicit `None` queries on
local tables with and without vectors and on a mocked remote endpoint.
They verify scans, limits, filters, column projection, and the remote
request body.
Validation: all six regressions reproduced the original
`UnboundLocalError`; all 11 focused query tests pass with the fix,
including existing vector, full-text, hybrid, and sync scan coverage.
The uv development bootstrap, locked Ruff formatting/lint checks, and
focused Pyright return-type assertions also pass.
Fixes#4385
<!-- lance-gatekeeper-fix:v1 agent=bb9a7509f6dadde030d8113b172abc9a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Calling `close()` on a synchronous remote connection returned an
unawaited coroutine and left the connection usable. Local synchronous
connections had no close or context-manager API.
Add synchronous `close()` and `is_open()` to the connection interface,
delegate them to the existing connection handle for local, remote, and
namespace connections, and provide shared context-manager cleanup.
Closing is idempotent, and subsequent database operations use the
existing `Connection is closed` error. Document the lifecycle with
examples.
Validation:
- Reproduced the reported coroutine warning and missing local lifecycle
API on the original code; all 15 new regression cases failed before the
fix.
- All 168 tests in `test_db.py`, `test_remote_db.py`, and
`test_namespace.py` passed through the uv-managed environment, including
local, in-memory, namespace, native-derived, and mocked remote lifecycle
coverage.
- All 6 synchronous connection doctests passed.
- Repository-wide Ruff 0.9.9 lint and formatting passed; `git diff
--check` passed.
Fixes#4382
<!-- lance-gatekeeper-fix:v1 agent=a743139af98198f1d4c8934c9cc1d0c9
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Cause
On latest-tracking remote tables, `fetch_blob_files` probed a concrete
table version but retained `None` on each handle. Later range reads
resolved the latest version again, so an update or delete could make an
opened handle fail.
## Fix
For nonempty blobs, capture `x-lancedb-version` from the 206 size probe
and include it on every later range request. Reject a nonempty probe
without a valid version. Accept an empty-blob 416 probe with or without
that header, because an empty handle makes no later data range request.
The Python remote mock now covers both response shapes.
## Limitation
A table dropped and recreated with the same name can reuse version
numbers and row addresses. Version pinning does not identify the table
itself, so an old handle can still return data from the recreated table.
That identity case needs a separate fix. Callers that replace a table
should discard its old blob handles.
## Validation
- `cargo test --quiet --features remote -p lancedb --lib
remote::table::blobs::tests` (24 passed)
- Rebuilt the Python extension; `uv run --extra tests pytest
python/tests/test_remote_db.py -q -k remote_blob` (6 passed)
- `cargo clippy --quiet --features remote -p lancedb --lib --tests
--examples`
- `cargo fmt --all -- --check`, `ruff check .`, and `ruff format .`
Fixes#4276
<!-- lance-gatekeeper-fix:v1 agent=e309bdf75b1a87f6492a6dc3ee1bfd81
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Cancellation is the only job control the SDK exposes, and it is
terminal, so a long-running server-side job cannot be parked and picked
up again. This adds `pause_job` and `resume_job` to the Rust core
connection and the Python bindings (sync and async), posting to the
server's `/v1/jobs/pause` and `/v1/jobs/resume` endpoints.
A pause parks the job until it is resumed: its workers drain and stop.
The outcome strings mirror the server's answers — a job finalizing its
results reports `"committing"` and cannot be parked, and a resume before
the drain is confirmed reports `"still_pausing"`; both are retried
rather than failed. Resuming re-queues the job and its workers pick
their work back up from checkpoints. Local connections report the
operations as unsupported, like the rest of the jobs API.
## Cause
`io.RawIOBase.__del__` checks `closed` and calls `close()` during GC.
Both PyO3 methods entered `block_on`, which panics when GC runs on a
LanceDB Tokio worker.
## Fix
Track Python blob handle closure atomically so `closed` is synchronous.
`close()` marks the handle closed immediately and, when called on a
Tokio thread, schedules the underlying async cleanup without blocking
that thread. Reads and seeks after close fail immediately; ordinary
callers still wait for cleanup.
## Validation
- Rebuilt the Python extension with `uv run --extra tests --extra dev
maturin develop --extras tests,dev`.
- Ran the blob test file and remote blob handle test: 69 passed, 1
skipped.
- Ran `cargo fmt --all`, `ruff format .`, and `ruff check .`.
Fixes#4283
<!-- lance-gatekeeper-fix:v1 agent=15a7bd545e63dea4b69faa0429f0a684
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
A view is a named query a database stores and plans on every read. It
holds no rows, which is the whole difference from a materialized view.
## API
| Verb | Route |
| --- | --- |
| `create_view(name, query, namespace_path)` | `POST
/v1/view/{id}/create` |
| `describe_view(name, namespace_path)` | `POST /v1/view/{id}/describe`
|
| `drop_view(name, namespace_path)` | `POST /v1/view/{id}/drop` |
| `list_views(namespace_path)` | `GET /v1/namespace/{id}/view/list` |
On `Connection` and the `Database` trait, with the remote client, Python
(sync and async) and Node bindings. Local databases return
`NotSupported`: the server side is Sophon's, where a view is an object
of the database manifest.
`ViewDescription` carries the defining query, the database *and
namespace path* unqualified names in it resolve against, and the schema
the query resolved to. `create_view` returns one, so a caller has the
schema without a second call.
Both defaults travel with the view because it outlives the session that
declared it: the server re-plans the stored query on every read, so a
reader resolving an unqualified name against its own defaults would read
a different table. `default_namespace_path` crosses the wire as
`default_namespace`, a path like `namespace`, absent for the root.
There is no replace: a name already taken is an error, and changing a
view is a drop followed by a create, each authorized against what it
actually touches.
Querying a view stays SQL's job. There are no rows behind a view, so
there is no `open_view` returning a `Table`.
A refresh running under a skip policy records each row it skipped, with
the failing input and the error, but the client could not read that
store: the server exposes it over SQL and, since recently, a REST route.
A user who hit per-row failures still had to open a SQL session.
`Table::function_errors` calls the route. The listing is table-addressed
with optional job and column filters, the same addressing the SQL
surface uses, so the two cannot disagree about what a table's errors
are. The two non-record signals come back as their own fields rather
than as rows: capped-fragment summaries, and whether the listing stopped
at its limit. Local tables refuse rather than answer with an empty list.
Python and Node expose the same call, with the same optional filters.
A 1M-row column refresh over 200 fragments produced no visible result,
and the client could only ever say `"running"`. Everything needed to
diagnose it already existed server-side — the job registry records a
`claim`/`claim_complete` pair per fragment carrying `rows_processed` —
but none of it was reachable.
## Before
Four ways to ask about a job, none of which told you much.
```python
job = table.refresh_column_async("embedding")
job.status() # "running". That was the entire debug surface.
db.get_job(job_id) # state, and a spec. No result, no progress.
db.job_history(job_id) # raw record batches, no limit, no filter
db.job(job_id) # a handle that knew nothing
```
## After
Open a job the way you open a table; the handle answers everything.
```python
job = db.open_job(job_id) # raises JobNotFoundError if there is no such job
```
```python
>>> print(job)
Job(
id='job-1',
state='failed',
job_type='refresh_column',
creation_ms=1757000000000,
spec={
"column": "embedding",
"num_workers": 4
},
failure=JobFailureInfo(phase='execute', message='worker died', retryable=True),
)
```
Individual fields are there too — `job.state`, `job.job_type`,
`job.creation_ms`, `job.spec`, `job.result`, `job.failure` — and
`job.result` carries `rows_assigned` / `rows_failed` as soon as the job
succeeds, with no `wait()` required.
Per-fragment progress *while it is still running*:
```python
done = job.events(filter="state = 'claim_complete'", limit=10_000)
done.column("rows_processed").to_pylist() # [5000, 5000, ...]
```
The handle an async action returns is the same object, one `refresh()`
away:
```python
job = table.refresh_column_async("embedding")
job.refresh()
job.state, job.result
```
TypeScript is the same experience, down to `console.log`:
```ts
const job = await db.openJob(jobId); // rejects if there is no such job
console.log(job); // same multi-line layout
job.state; job.jobType; job.spec; job.result; job.failure;
const done = await job.events({ filter: "state = 'claim_complete'", limit: 10_000 });
```
## Why each piece matters
- **A result without waiting.** `rows_assigned` / `rows_failed` used to
live only on the terminal result, so a job that never terminated
reported nothing at all.
- **`limit`.** The server caps event rows at 1000 and truncates without
saying so, which silently hid most of a 200-fragment job's history.
- **`filter`.** `claim_complete` rows carry per-claim `rows_processed` —
the only progress signal that exists mid-flight.
- **Events outlive the worker.** They live in the job registry, not in
pod logs that vanish with the pod.
- **One place to ask.** `open_job` replaces `describe_job`,
`query_job_events` and `job`, so a question about a job has one answer
instead of one per calling location.
- **A missing job is an error, not a `None`.** The common case is a job
id copied out of a log, where absence is the surprise worth raising —
and it matches `open_table`.
- **Printing is the debug surface.** Every field on its own line, JSON
payloads keeping their structure. An unrefreshed handle stays on one
line, because there is nothing to lay out.
- **In-process jobs say so.** A local refresh reports `state` and leaves
the rest null rather than inventing fields it has no record for.
`list_jobs` and `cancel_job` stay as they were: one lists, the other is
a one-shot action that should not need a describe first.
## Breaking
All shipped in 0.38.0. No deprecated aliases.
| Was | Now |
| --- | --- |
| `Connection.get_job` → `describe_job` | `Connection.open_job` returns
a populated `Job`, or raises |
| `Connection.job_history` → `query_job_events` | `job.events(...)` |
| `Connection.job` | `Connection.open_job` |
| Python events → `List[pa.RecordBatch]` | `pa.Table` |
| `JobDescription.spec_json` / `.result_json` | internal; use `job.spec`
/ `job.result` |
Node's `Job` is now a TypeScript class wrapping the native handle, so it
returns an Arrow table and parsed values like Python does. New
`Error::JobNotFound` / `JobNotFoundError`; the three job exceptions are
now in the Python API reference.
Remote create-index requests already expose `replace` on the builder,
but the remote client did not consistently forward an explicit
`replace=false` over REST. That meant create-only intent could be lost
before it reached a remote server, even though local builders and Python
APIs can express it. This PR forwards `replace=false` on the existing
`create_index` endpoint and keeps the current default behavior unchanged
for compatibility.
This was accomplished with the following changes:
- Serialize `replace: false` into the existing remote create-index
request body when the builder is configured with `.replace(false)`.
- Forward `replace` through the synchronous Python remote `create_index`
wrapper so `RemoteTable.create_index(..., replace=False)` reaches the
repaired path.
- Continue omitting `replace` for the default path so existing remote
create-index requests keep their current semantics.
- Document `name` and `replace` on the existing OpenAPI create-index
request schema.
- Add coverage that verifies the remote client uses the existing
`/create_index/` route and forwards `replace=false`, including the
synchronous Python unified API.
### Testing
- `cargo fmt --all --check`
- `cargo test -p lancedb --features remote
test_create_index_forwards_replace_false_on_existing_route --locked`
- `uv tool run maturin develop --extras tests,dev,embeddings`
- `uv run --frozen pytest
python/tests/test_remote_db.py::test_remote_create_index_new_api`
- `uv run ruff format --check python/lancedb/remote/table.py
python/tests/test_remote_db.py`
- `cargo build -p lancedb --features remote --locked`
- `cargo clippy -p lancedb --features remote --all-targets --locked --
-D warnings`
## Summary
- preserve repeated table offsets without adding a public ordering
guarantee
- retain exact requested ordering in identity and persisted permutations
- cover local, projected, multi-batch, and mocked-remote query paths
## Root cause
Take queries lowered offsets to a set-like IN predicate and discarded
repeated occurrences. Persisted permutation loading also compared the
distinct base-table result count with the requested occurrence count,
rejecting repeated row IDs before its existing reordering step could
expand them.
## Fix
The shared take-query path now deduplicates the predicate for efficient
lookup, requests row-offset metadata internally, and expands each
matching row to the requested multiplicity in backend result order. An
internal opt-in keeps exact requested order for identity
PermutationReader reads, while persisted permutations continue using
their existing ordering map.
## Validation
- cargo test --quiet --features remote --tests
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples
- targeted Python local and mocked-remote regression tests
- exact issue reproduction
Fixes#2820
<!-- lance-gatekeeper-fix:v1 agent=75acf840afa6f4be4bff98b567b504bd
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
LanceDB could not request Lance's list-element FTS document granularity
through Python or Remote APIs, and generic nested-field resolution
exposed Arrow's internal `item` segment instead of the public field
path.
This exposes typed `row | list_element` configuration for Python FTS
index creation and match/phrase queries, preserves `_doc_index`, and
keeps nested FTS paths public (for example, `docs.content`). Remote
list-element requests require server API version 0.6.0 so older servers
cannot silently execute them with row semantics; explicit row requests
remain compatible.
## Compatibility
Omitted index and query parameters retain row granularity. Remote row
index creation omits the new wire field.
## Tracking
[ENT-2342](https://linear.app/lancedb/issue/ENT-2342/expose-list-element-fts-document-granularity-end-to-end)
## Problem
`refresh_column_async` returned a unit-result job even though durable
refresh jobs carry a canonical terminal result. Python callers could not
obtain row counts or source and published versions through the public
`Job` API, and local and remote refresh jobs exposed different result
semantics.
## Behavior
`refresh_column_async` now returns `Job[RefreshColumnResult]` for local
and remote tables. The general typed-job bridge binds each endpoint to
its public result model while preserving unit-result jobs and existing
status, wait, cancel, and timeout behavior. A local no-op refresh
reports no published version.
The Node.js API continues to resolve `wait()` as `void`; its binding
erases the Rust result type internally to preserve the existing public
contract.
## Ownership and integration boundary
LanceDB owns the language-neutral `Job<T>` contract and language-binding
decode. Sophon owns production and durable persistence of terminal
payloads. Sophon #7348 and #7378 now publish the canonical refresh
result for Function-backed and expression-backed refresh jobs,
respectively. The remote client fixture matches the merged server
schema; live deployment and end-to-end demo acceptance remain separate
rollout checks.
This PR is a **breaking** rename of #3686.
merge reads like git merge w/ three-way, replay history, combine two
lines of work. That is not this API.
This call takes one additive change on a branch and lands it on main.
New column, including a blob column. Main's existing columns are not
rewritten. If it cannot land, you get `status="failed"` and
`diff.errors`, not a merge conflict to resolve.
Cherry-pick is terminology that aligns more with that.
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.cherry_pick("exp", dry_run=True)
result = table.branches.cherry_pick("exp")
if result["status"] == "cherryPicked":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "failed":
print(result["diff"]["errors"])
```
### Behavior
- Remote / Enterprise only. Local still NotSupported.
- HTTP 409 is not an exception. It is Ok with status="failed" and
diff.errors (CherryPickError).
- Unknown error / status codes still parse as Unknown.
- Requests are not retried. 409 is final and carries the body.
- Endpoint is POST /v1/table/{id}/branches/cherry_pick/.
- merge_insert and Table.merge are unchanged.
### Testing
- `cargo test -p lancedb --features remote diff_branch`
- `cargo test -p lancedb --features remote cherry_pick`
- `pytest python/python/tests/test_remote_db.py -k cherry_pick`
- node `remote.test.ts` diffs / cherry-picks path
## Summary
The sync `RemoteTable` carried `set_lsm_write_spec`,
`unset_lsm_write_spec`, `get_lsm_write_spec`, and `close_lsm_writers`,
but not `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, or
`get_lsm_stats`.
That left the four LSM control methods reachable from `AsyncTable` only.
They are also the four that *only* work against a remote table —
`NativeTable` does not override the `BaseTable` defaults, so on a local
table they return `NotSupported` (`rust/lancedb/src/table.rs:679-701`).
The net effect for sync users:
| | `checkpoint_lsm` / `get_lsm_stats` |
|---|---|
| `LanceTable` (sync, local) | present, but always `NotSupported` |
| `RemoteTable` (sync, remote) | `AttributeError` — method absent |
| `AsyncTable` (remote) | works |
So there was no working sync path at all, despite the Rust `RemoteTable`
implementing every one of these against real endpoints.
## Changes
* Add `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, and `get_lsm_stats`
to `lancedb.remote.table.RemoteTable`, mirroring the delegation style of
their neighbours.
* Correct the docstrings on `set_lsm_write_spec` /
`unset_lsm_write_spec`, which read `"""Not supported on LanceDB
Cloud."""` although `rust/lancedb/src/remote/table.rs:2549-2601`
implements both against `/v1/table/{}/set_lsm_write_spec/` and
`/unset_lsm_write_spec/`. They appear to have been copy-pasted from
`set_unenforced_primary_key` directly above.
No Rust or PyO3 changes — the bindings and the `AsyncTable` methods
already existed. The `Table` ABC is left alone, matching how the
existing `*_lsm_write_spec` methods are declared on the concrete classes
only.
## Tests
Four new tests in `python/python/tests/test_remote_db.py`, against the
existing mock HTTP server:
* `test_get_lsm_stats_sync` — the server payload round-trips into the
dict, and `include_generation_rows` defaults to `False` and is forwarded
when set.
* `test_get_lsm_stats_sync_returns_none_when_lsm_disabled` — a
`{"lsm_stats": null}` envelope yields `None` rather than an error.
* `test_flush_and_compact_lsm_sync` — both are one-shot POSTs answered
`202` with no body.
* `test_checkpoint_lsm_sync` — pins the binding to the endpoints it
drives (`flush_lsm` then `get_lsm_stats`); the convergence loop itself
is already covered in Rust.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- validate the generated LanceDB Cloud hostname during connection setup
- return a clear invalid-input error for empty, overlong, or oversized
DNS names before network resolution
- add Rust and Python regression coverage for malformed `db://`
authorities
## Root cause
The `db://` authority and region were interpolated into the Cloud API
hostname without DNS length validation. Empty or overlong labels
therefore reached the resolver and surfaced as an opaque IDNA
`UnicodeError` instead of a useful connection error.
## Validation
- `cargo test --quiet --features remote -p lancedb
test_rejects_invalid_cloud_dns_hostname --lib`
- `cargo check --quiet --features remote --tests --examples`
- `uv run --no-sync --extra tests pytest
python/tests/test_remote_db.py::test_async_remote_db
python/tests/test_remote_db.py::test_connect_rejects_invalid_cloud_dns_hostname
-q`
- `cargo fmt --all -- --check`
- `ruff check .`
- `ruff format --check python/python/tests/test_remote_db.py`
Fixes#799
<!-- lance-gatekeeper-fix:v1 agent=4d1597b3d244b58f0603ed40a8a59cf9
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- Implements Cloud `fetch_blob_files`: returns real seekable `BlobFile`
handles over HTTP Range instead of `NotSupported`.
- Completes the second Cloud blob read verb after #3684 (`fetch_blobs` =
eager whole bytes; this = lazy / partial / sequential reads).
- Same public handle API as local (`read_range`, `read_up_to`, `seek`,
`tell`, `close`), so one code path works for local and Cloud.
Large blobs (video, audio, PDFs) should not require downloading the
whole object to inspect a header or stream a slice. After search,
callers open a handle and read only what they need:
```python
hits = table.search(vec).select(["id", "video"]).limit(5).to_arrow()
with table.fetch_blob_files("video", hits)[0] as f:
header = f.read_range(0, 256)
f.seek(keyframe_offset)
chunk = f.read_up_to(1 << 20)
```
### Behavior
- Handle creation probes size with `bytes=0-0` (bounded concurrency,
input order preserved).
- `204` → null (`None`); `416` with `bytes */0` → valid empty blob;
other `416` → error.
- `read_range` validates `Content-Range` and body length; OOB ranges
fail with `invalid_input` before the request (aligned with Lance).
- `read_up_to` reuses one open-ended Range response across sequential
reads; `seek` drops it.
- Servers older than 0.5.0 get a clear `NotSupported` (does not suggest
`fetch_blobs`, which they also lack).
## Testing
- `cargo test --features remote -p lancedb remote_blob`
- `cargo test --features remote -p lancedb test_blob`
- `cargo clippy --features remote --tests --examples` (no new warnings
from this change)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Adds job operations to the connection surface, building on the Job
handle from #3742: job(id), list_jobs, get_job, cancel_job, and
job_history, plus a non-blocking Job.status(). Implemented on the
Database trait (defaulting to NotSupported), the remote backend
(/v1/jobs), and the Python and Node bindings; job_history returns Arrow
batches.
errors() and progress() are not included.
Tested with mocked endpoints in all three languages.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
IndexBuilder::execute now returns a Job with wait and cancel methods.
Local tables build the index synchronously and return an already-done
job. Remote tables read the job id the server returns from create_index
and track it through the /v1/jobs API: wait polls describe until the job
reaches a terminal state and cancel posts a cancellation. Servers that
return no job id yield a done job, so behavior against older servers is
unchanged. The job id is not exposed on the handle.
The Python and TypeScript bindings keep their current signatures and
discard the handle; exposing Job there is left to follow-ups.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Remote half of the blob read path. #3578 did local Python. This makes
`RemoteTable` hit the server.
- `fetch_blobs(column, row_ids or hits)` → bytes over `POST
/v1/table/{id}/fetch_blobs/`
- `blob_columns()` from the cached schema (describe already has the
metadata, no extra route)
- search then `fetch_blobs` works. row identity rides inside the blob
descriptor so you do not need a public `_rowid`
- `fetch_blob_files` still `NotSupported` on remote. use `fetch_blobs`
for full bytes for now. Range is a follow up
Accepts Binary / LargeBinary / BinaryView on the way back. Empty
`row_ids` short-circuits. Version + branch go in the request body same
as other read calls.
### Example
```python
db = lancedb.connect(uri="db://my-project", api_key=...)
table = db.open_table("clips")
hits = table.search(query_vec).select(["id", "video"]).limit(10).to_arrow()
# hits is just id + video. row ids are stashed on the descriptor
blobs = table.fetch_blobs("video", hits) # null-aligned, same length as hits
```
Or pass ids yourself:
```python
blobs = table.fetch_blobs("video", [10, 20, 30])
```
### Testing
- `cargo test -p lancedb --features remote --lib`
- `cargo test -p lancedb --features remote --test blob_integration`
- `pytest python/tests/test_remote_db.py -k remote_blob`
- live e2e against a local 0.5.0 remote server (search → fetch, nulls,
nested path, old server gate)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What
Expose custom FTS stop-word lists in the Python and TypeScript public
APIs, including their standalone tokenize helpers and remote index
creation.
This PR supports concrete string lists only. It does not add file or
LanceDB-table stop-word sources.
## Why
Rust already exposes Lance's custom stop-word list option. The Python
and TypeScript APIs did not pass it through, and local index details did
not retain the full tokenizer parameters needed by index-backed
tokenization after reopening a table.
## How
- Add `custom_stop_words` / `customStopWords` to the Python and
TypeScript FTS and tokenize options.
- Preserve `None` / `undefined`, empty lists, and list contents without
normalization.
- Load the persisted FTS segment parameters when returning local index
details.
- Serialize the concrete list in remote create-index requests.
- Keep Python and TypeScript tests thin; behavior, persistence, query
tokenization, and remote JSON coverage live primarily in Rust.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
- Python extension rebuild with `uv` and `maturin`
- Targeted Python tests: 4 passed
- Python `ruff format --check` and `ruff check`
- TypeScript build, typecheck, Biome lint, generated docs, and targeted
tests
---------
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
## What changed
- add `block_size` to Python FTS configuration and the deprecated
local/remote helpers
- add `blockSize` to the TypeScript FTS options and propagate it through
the NAPI binding
- serialize the value as `block_size` for remote index creation
- document the existing Rust builder API and generate the TypeScript API
reference
- add local, remote, metadata, search, and invalid-value regression
coverage
## Why
Lance supports configuring the number of documents per compressed FTS
posting block, but LanceDB's Python and TypeScript APIs did not expose
the setting. This made the experimental FTS V3 layout unavailable
through those clients and allowed the value to be dropped before index
creation.
## How it works
The default remains `128`. Supported values are `128` and `256`;
selecting `256` uses the experimental FTS V3 format. Invalid values are
rejected by the Lance builder and surfaced as Python or JavaScript
errors.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo +1.94.0 clippy --quiet --features remote --tests --examples --
-D warnings`
- targeted Rust local and remote index tests
- Rust doctests: 34 passed
- Python Ruff checks, doctest, and targeted local/remote tests: 5 passed
- TypeScript build, Biome lint, generated docs, and targeted Jest tests:
9 passed
- `git diff --check`
## Limitations
The Java client remains unchanged because its external remote REST model
does not currently expose `block_size`.
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
This PR adds some support for `diff` / `merge` in the remote client as
for local tables we stay `NotSupported` until
https://github.com/lance-format/lance/issues/7263.
This wires the two review-and-land calls against the remote REST API:
- `POST /v1/table/{id}/branches/diff`
- `POST /v1/table/{id}/branches/merge`
Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python
returns the wire JSON, same shape as the REST response.
Merge here means promoting a branch's added columns onto `main`.
### Behavior
- Remote only. Local raises `NotSupported`.
- A rejected merge is not an exception. HTTP 409 still returns `Ok` / a
dict with `status="rejected"` and blockers in `diff.mergeBlockers`.
- Unknown blocker / status codes parse as `Unknown` so a newer server
does not break older clients.
- `MergePreview` tolerates missing fields for the same reason.
- Merge requests are not retried. 409 is final and carries the body you
need.
### Example
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.merge("exp", dry_run=True)
result = table.branches.merge("exp", dry_run=False)
if result["status"] == "merged":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "rejected":
print(result["diff"]["mergeBlockers"])
```
### Testing
cargo test -p lancedb --features remote diff_branch
cargo test -p lancedb --features remote merge_branch
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
# Elastic Streaming Dataloader
## Motivation
Training large models on LanceDB tables today requires loading the
entire dataset
into memory or writing bespoke batching logic. This PR introduces
`StreamingDataset`, a PyTorch `IterableDataset` that streams directly
from a
LanceDB table with two hard guarantees that are difficult to achieve
together:
**elastic determinism** and **resumability**.
## Goals
### Elastic determinism
The dataset partitions the table into a fixed number of *splits*
(controlled by
`num_splits`, `shuffle_seed`, and `epoch`). Samples are yielded by
round-robining
over splits one sample per split per cycle. Because the split structure
is fixed,
the set of samples that makes up each global training step is identical
regardless
of `world_size` or `num_workers`. You can scale your cluster up or down
between
runs and the model sees the same data in the same order — no
re-sharding, no
gradient variance from topology changes.
### Resumability
`state_dict()` / `load_state_dict()` capture how many samples each split
has
consumed. Because all splits are the same size and the round-robin
design keeps
them in lockstep, the state reduces to a single scalar
(`samples_consumed_per_split`)
that is topology-independent. A checkpoint saved with 8 GPUs can resume
correctly
on 4 GPUs or 16 GPUs without any adjustment.
### PyTorch `IterableDataset` / streaming
`StreamingDataset` implements the standard PyTorch `IterableDataset`
interface, so
it drops into any existing `DataLoader` pipeline without modification.
Data is
fetched lazily from Lance in chunks — only the rows needed for the
current batch are
ever in memory.
Compared to the map dataset this takes more work from pytorch and puts
it into the dataset itself (e.g. shuffling, filtering, etc.). We do this
because we cannot achieve things like elastic determinism or
prefiltering otherwise.
### Multi-worker support
DataLoader workers are automatically assigned contiguous sub-blocks of
splits (the
rank's splits are divided evenly across workers). Each worker is
independent:
no shared state, no inter-process coordination. The only constraint is
that
`num_splits` must be divisible by `world_size * num_workers`.
That being said, multi-worker is highly discouraged as it relies on
multiprocessing which is inefficient. Still, we want to support it.
### Filters as prefilters
Filters are applied at *permutation-build time* via
`PermutationBuilder.filter()`,
not re-evaluated on every fetch. The filtered row IDs are stored in the
permutation
table so that subsequent reads see only the matching rows. This allows
us to avoid loading rows that don't match the filter (which is the
default pytorch behavior)
### Prefetching
Two parameters control the I/O pipeline:
- `read_batch_size` (default 64) — number of rows fetched per
`take_offsets` call.
Larger values amortise per-request overhead, which is critical on object
storage
where a single round-trip can cost ~100 ms.
- `prefetch_batches` (default 4) — number of batches prefetched in
parallel per
split via a `ThreadPoolExecutor`. While the model processes the current
batch,
the next several batches are already in flight, hiding storage latency
behind
compute.
If set correctly then you can get good performance even with
num_workers=0 (unless you are bottlenecked on transform).
### Transform parallelism
The underlying `Permutation` API supports a `with_transform()` callback
for
decoding, augmentation, and format conversion. Unfortunately, this is
not parallelized. Pytorch typically parallelizes this with num_workers
which is multiprocessing which is highly inefficient. For simple
transforms we should be able to utilize multithreading and Rust based
UDFs. For complex python UDFs we could have a dedicated multiprocessing
pipeline for just the transform. Or we could just utilize
multithreading. In both cases we would exclude the I/O stage from the
multiprocessing because that ends up being very memory hungry and
inefficient.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
lancedb-gatefixer[bot]andXuanwo
Joaquin Hui GomezandXuanwo
Aviral Saxena
Dan RammerandClaude Opus 5.5
Drew
Jack Ye
Colin Patrick McCabe
Yuval LifshitzandClaude Opus 5.0
Will Jones
Wyatt Alt
Bruno Ramirez
Yang CenandYang Cen
Drew GallardoandCursor
Weston PaceandClaude Sonnet 4.6