SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.
### Python experience
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.
The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.
For an async table:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
### Execution
The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.
Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.
Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.
Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
Remote materialized-view creation currently expects a REST response
containing a registry job ID, while refresh already submits SQL. Route
both through the SQL query backend and preserve Python's synchronous and
asynchronous return types.
### Query to Job mapping
`SqlClient::submit_as_job()` submits an `ExecuteQueryRequest` through
the same backend as `execute_query_async()`. Once submission succeeds,
it retains the returned `Query` and starts one client-side task that
consumes its result reader to completion. `Job::from_query_task()` wraps
that task, its cached outcome, a single ID and a cancellation callback.
Python exposes this as `Job[T]` or `AsyncJob[T]`.
The existing `job_type` property maps SQL-backed handles to the new
`"query"` kind. The shared `id` contains that query's UUID string, so
Python callers can inspect it with `db.describe_query(job.id)` on the
submitting connection. No separate `query_id`, backend enum or `type()`
API is added.
| Handle | `job_type` | `id` | Available lookup |
| --- | --- | --- | --- |
| SQL-backed | `"query"`, known at submission | Connection-scoped query
UUID string | `describe_query(job.id)` |
| Registry-backed | Existing server-reported kind, after `status()` or
`refresh()` | Registry job ID string | Registry job APIs, descriptions
and events |
| Local | `None` | `None` | The current in-process handle |
| Job concept | SQL-backed behavior |
| --- | --- |
| Submission | Await initial SQL submission before returning the handle.
Submission errors propagate immediately. |
| `status()` | Tracks the adapter task: running during result
consumption/finalization, finished after both succeed, failed if either
fails, or cancelled after cancellation. |
| `wait()` | Returns the operation's typed terminal result. One task
consumes the query reader once; repeated waits reuse its cached outcome
without resubmitting SQL or reopening the reader. |
| `cancel()` | Delegates to `Query.cancel()`. Rejected or unsupported
cancellation leaves the task running; an accepted request stops the
task. Cancelling a finished operation is a no-op. |
| Registry details | Query-backed handles expose no registry
description, event history, pause/resume or reattachment after a client
restart. `refresh()` updates cached lifecycle state. |
The Job tracks command completion rather than exposing SQL batches. A
wait timeout does not cancel the query. Query IDs remain scoped to the
submitting connection and cannot be used with registry APIs or as
durable resume tokens. Python `describe_query()` accepts both UUID
objects and the string `id` exposed by query-backed jobs. Rust, Python
and TypeScript expose the same classification through their existing
job-type accessors.
### Materialized-view operations
- **CREATE:** Render `CREATE MATERIALIZED VIEW` with the view namespace
and optional auto-refresh settings. After command completion, resolve
the job to `None`. Synchronous creation waits on the job, then opens and
returns the materialized view.
- **REFRESH:** Submit `REFRESH MATERIALIZED VIEW` through the same
adapter. After command completion, read the refreshed table's version
and row count to construct the existing typed refresh result.
Synchronous refresh waits on the same path.
Remote creation requires the SQL service. Execution placement remains a
server decision. REST endpoints are unchanged. The job guide documents
the ID/job-type contract and query-backed limitations.
Regression coverage exercises SQL submission, namespaces, refresh
options, job-type/ID preservation through typed result mapping, repeated
waits, failures and cancellation rejection. No local builds or tests
were run.
SQL execution now starts with one Flight call. The server can stream
rows immediately or return a polling continuation, which the SDK follows
through the existing query handle and reader APIs. The same behavior
applies with and without parameters.
Both initial transports share direct-stream lifecycle handling. Handoffs
are explicitly negotiated; existing servers can continue returning
ordinary streams. Add coverage for streaming, deferred results,
cancellation, malformed decisions, and legacy responses.
```mermaid
flowchart TD
A[execute_query or execute_query_async] --> B{Parameters supplied?}
B -->|No| C[DoGet: SQL command ticket]
B -->|Yes| D[DoExchange: SQL and Arrow parameter row]
C --> E[Server chooses execution path]
D --> E
E -->|Stream| F[Initial response: streaming decision and Arrow batches]
E -->|Defer| G[Initial response: PollInfo continuation]
G --> H[SDK follows PollFlightInfo]
H -->|Still running| H
H -->|Result endpoints available| I[SDK fetches tickets with DoGet]
F --> J[Query reader yields Arrow batches]
I --> J
```
`execute_query` opens the reader; `execute_query_async` returns its
query handle. The caller does not select the transport or execution
path.
Infer Blob V2 input access from the Python Function signature. `video:
BlobReader` requests a lazy, seekable reader; `video: bytes` requests
the complete payload. Users do not need a separate `blob_inputs`
registration option.
## Python experience
For a video table joined to a table of frame timestamps, the Function
can pass its reader directly to a media decoder:
```python
import io
import av
from lancedb import BlobReader, udf
@udf(pip=["av", "Pillow"])
def extract_frame(video: BlobReader, timestamp_s: float) -> bytes:
with av.open(video) as container:
stream = container.streams.video[0]
container.seek(int(timestamp_s / stream.time_base), stream=stream)
for frame in container.decode(stream):
if frame.time is not None and frame.time >= timestamp_s:
output = io.BytesIO()
frame.to_image().save(output, format="JPEG")
return output.getvalue()
raise ValueError("No frame at or after the requested timestamp")
```
After registering `extract_frame`, the query keeps the usual Function
call:
```sql
SELECT f.video_id, f.timestamp_s,
extract_frame(v.video, f.timestamp_s) AS image
FROM videos AS v
JOIN frames AS f ON v.id = f.video_id
```
The returned JPEG can feed a subsequent image-model step. The reader
provides `read`, `readinto`, `seek`, `tell`, `size()`, and
`read_range(offset, length)`; range reads leave the sequential cursor
unchanged. Each row invocation owns its reader, which closes after the
call. Decoder seeks can request the parts of the MP4 needed to locate
and decode the requested frame; the amount read depends on the file
layout and codec.
Functions that need the whole value can keep a bytes signature:
```python
@udf
def video_length(video: bytes) -> int:
return len(video)
```
## Execution contract
- Signature inference maps `BlobReader` to a Blob V2 field and records
`blob_access="reader"`. `bytes` keeps its binary Arrow type and records
`blob_access="bytes"`; it can also bind to a Blob V2 source. Explicit
schemas support the same choice, including a Blob V2 field annotated as
`bytes`.
- Reader execution carries a compact blob reference into the Function
worker. Payload reads happen when Python calls the reader, with requests
split into at most 1 MiB ranges. The full video does not need to enter
the Arrow input batch just to invoke the Function.
- Bytes execution materializes the full source before the Function runs
and remains subject to eager-input memory admission. Calling
`reader.read()` without a size also materializes the remaining contents
in Python; lazy access depends on how the Function consumes it.
- Input access metadata survives Python/Rust Function serialization and
registration. Packaged Functions import the worker-provided reader
implementation. This PR defines the SDK contract; execution requires
corresponding server support.
- This changes blob inputs. The example still returns ordinary image
bytes; it does not add streamed outputs or run an image model.
Adds coverage for signature inference, explicit schemas, nullable
readers, signature round trips, seeking, bounded range reads, buffering,
and invalid responses.
## What
Move typed job decode error construction behind a feature-aware helper
so builds without the `remote` feature do not reference `Error::Http`.
Local decode failures continue to surface as `Error::Runtime`, while
remote builds retain HTTP errors with request IDs.
Add a regression test for local typed-result decode failures.
Closes#4096.
## Testing
- `cargo fmt --all -- --check`
- `cargo check -p lancedb --no-default-features`
- `cargo test -p lancedb --no-default-features
job::tests::local_typed_result_decode_errors_are_runtime_errors`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
Fixes#4401.
- Validate cached remote table entries with the existing `describe`
endpoint before returning a handle.
- Preserve cached server-version behavior after a successful validation.
- Add a regression test for a table dropped by another connection after
its entry has been cached.
A cached entry can otherwise outlive an external drop, allowing
`open_table()` to return a handle that fails only on its first later
operation.
## Verification
[Linux
validation](https://github.com/Ganesh1729-ui/lancedb/actions/runs/39420560974)
passed against the exact runtime and test sources at PR head
`d39092b01f6b49921b5d66ccedf29709a7ddc23d`, using the repository's Rust
toolchain and `ci` profile:
- `cargo fmt --all -- --check`
- `cargo test --profile ci --locked -p lancedb --features remote --lib
remote::db::tests::test_open_table -- --nocapture`: **8 passed**,
including cached-table deletion, independent handle checkout,
branch/version handling, schema seeding and describe-response handling.
- Removing only the cached-hit existence guard makes
`test_open_table_checks_that_a_cached_table_still_exists` fail at its
`TableNotFound` assertion; restoring the fix makes the same test pass
again.
- `git diff --check`
The validation workflow is on a separate fork branch and is excluded
from this PR. These are focused, mock-backed remote unit tests; the full
upstream matrix and live Cloud integration tests still need upstream CI
approval. Local Windows compilation was unavailable because the MSVC
environment has no `link.exe`; the linked Linux run completed
successfully.
---------
Co-authored-by: Ganesh172919 <ganeshreddy17291122@gmail.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#2211.
Local queries converted `limit`/`offset` with `as i64`, so
`limit(usize::MAX)` wrapped to `-1` and Lance failed with `Limit must be
non-negative`. The remote client already worked around this (#2354);
this fixes the local path.
- Saturate scanner offsets to `i64::MAX` and cap limits at `i64::MAX -
offset`, keeping both conversion and full-text pagination addition safe.
- Use `saturating_add` for vector `top_k = limit + offset`, then cap
candidates at the dataset row count to avoid indexed-search capacity and
refinement overflows. Keep a positive candidate bound for empty tables
because Lance rejects `k = 0`.
- Leave public APIs and remote query handling unchanged.
### Testing
- Regression coverage includes plain scans, indexed IVF search,
refinement, index bypass, empty vector tables, and indexed full-text
pagination with finite and extreme limits/offsets.
- Reproduced the indexed-search and FTS overflows before applying the
follow-up fix.
- `cargo test --quiet --features remote -p lancedb --lib query::` — 81
passed.
- `cargo clippy --quiet --features remote -p lancedb --tests --examples`
— clean.
- `cargo fmt --all` and `git diff --check` — clean.
Vector planning now reads the dataset row count (metadata fast path) to
bound candidate allocation; this was not performance-benchmarked.
---------
Co-authored-by: Romin Parekh <roparekh@adobe.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- allow `db://` to select the remote storage root when a host override
supplies the endpoint
- preserve the empty `x-lancedb-database` header used by the legacy
clients for root storage
- reject root-storage connections without a host override before
constructing an invalid Cloud URL
- document the supported URI and cover both accepted and rejected
configurations
## Root cause
The shared Rust remote URL parser required every `db://` URI to contain
a host. That validation was introduced when the Node and Python clients
moved to the Rust implementation, but the legacy Node client
intentionally treated the empty host in `db://` as the storage-root
selector.
## Validation
- `cargo fmt --all`
- `cargo test --quiet --features remote -p lancedb
test_connect_remote_storage_root`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
Fixes#1944
<!-- lance-gatekeeper-fix:v1 agent=c71ca188791ad36fe5644c12960f758c
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Sync remote table and connection methods inherited bare base-class
stubs, so unsupported conversion and bulk-drop operations raised
`NotImplementedError` with an empty message and differed from the async
API.
Delegate `to_lance()`, `drop_all_tables()`, and `drop_database()` to the
async implementation. Sync and async remote Lance conversion share
#4306's native-table guard and reject before requesting a URI or opening
client storage. Conversion documentation directs remote reads through
server queries.
Add an explicit Polars error, suggest query conversions in
Arrow/Pandas/Polars errors, and give the remaining base-class errors an
operation name. Preserve upstream `blob_mode` validation for remote
Arrow and Pandas conversion. The shared remote bulk-drop error suggests
dropping tables individually.
Regression tests cover sync/async rejection parity, accessible and
missing advertised locations, pinned versions, and bulk-drop rejection
without requests. Sync blob query tests verify that descriptions come
from the server and unsupported modes cannot open client storage. Keep
the remote closed-blob-handle assertion consistent with the upstream
`ValueError` contract.
Validation:
- All 30 focused regression cases passed.
- 222 Python tests passed with rebuilt bindings in the locked uv
environment: the full remote and error suites, plus selected local
conversion, blob, and deletion tests.
- Python lint and formatting, Rust formatting, `cargo check --quiet
--features remote --tests --examples`, and `cargo clippy --quiet
--features remote --tests --examples` passed.
- Root Ruff checking reports 49 existing errors in upstream
`docs/web-tests`; those files are unchanged relative to `main`.
Fixes#4392
<!-- lance-gatekeeper-fix:v1 agent=95db5dd7002043a2c48924ded18b51b2
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
job.wait() polled on a backoff that doubled up to 5 s, so a job's
completion was seen 2.5 s late on average, as long as a small refresh
itself takes. Cap the interval at 1 s; the first polls still back off
from 200 ms, so a quick job is seen quickly and a long one costs one
request a second.
Linear: OSS-2704 https://linear.app/lancedb/issue/OSS-2704
Make table and namespace listings consistent with other resource
listings across Rust, Python, and TypeScript. Callers can iterate over
individual names across all pages, with configurable page sizes and
continuation tokens.
In order to avoid breaking existing Python users, this change adds some
compatibility methods to the returned Python iterators. Code such as:
```
for t in db.list_tables().tables:
print(t)
```
will continue to work as expected because the iterator has a `tables`
field and a `page_token` field, which function as expected to reveal the
first page of results. In other words, it is never necessary to use the
iterator __as an iterator__. It may be used as if it were an old-style
response object. (As the iterator progresses, the `tables` list may
switch to the second, third, etc. page of results.) Similar
compatibility code allows the async python API to keep working as well.
`optimize()` fails once a tagged version is older than the cleanup
cutoff, although the docs say tagged versions
are exempt from cleanup, and the failed prune also skips the
index-optimize step (repro and numbers in the issue).
Change: `OptimizeAction::Prune` already exposes
`error_if_tagged_old_versions: Option<bool>`, so no new API.
`cleanup_old_versions` and `cleanup_old_versions_before` in
`rust/lancedb/src/table/optimize.rs` now use
`unwrap_or(false)`, which covers `OptimizeAction::Prune`,
`OptimizeAction::All` and `optimize_prune_before`. The
Python and Node bindings pass `None`, which now means "retain tagged
versions"; I left `python/src/table.rs` and
`nodejs/src/table.rs` untouched (this also avoids a conflict with #3870,
which edits that region). `Some(true)`
still opts in to the error. Doc comments updated; the sentence in
`docs/web/management/table.mdx` is accurate once
this lands.
Tests: Rust `test_optimize_prune_retains_tagged_versions` (`Some(true)`
still errors; the default retains the tag;
`optimize_prune_before`; `checkout_tag` plus a data read), a Python sync
test with a scalar index (index step runs,
a second optimize works, untagged versions are pruned,
`checkout("release-1")` works), a Python async test and a
Node test.
**What I could and could not verify (please read):**
- I could NOT compile Rust here (not enough disk), so `cargo check`,
`cargo test`, the Python/Node binding builds
and therefore the fixed-path assertions of the new Rust, Python and Node
tests have not been run. This is why the
PR is a draft; I will mark it ready once CI is green and fix whatever it
finds.
- Ran: both new Python tests fail on the released 0.40.0 wheel with the
exact error above (so they do catch the
bug); pylance 13.0.0 confirms the semantics
(`cleanup_old_versions(error_if_tagged_old_versions=False)` removes
the untagged versions and keeps the tagged one, still readable with
`lance.dataset(version="keep")`); in Lance
v14.0.0-beta.7 `cleanup.rs` the policy field is `bool`, the builder
default is `true`, and Lance's own
`auto_cleanup_hook` uses `Some(false)`; `rustfmt --edition 2024 --check`
passes on `optimize.rs`.
- The Rust and Node tests rely on timestamp ordering
(`nextMillisecond()`, `days(0)`/`now()`), so they could be
flaky: tell me if you prefer another approach.
Behaviour change: anyone who relied on the error to learn that a tag
blocks cleanup now gets a silent retain.
Not changed: the deprecated sync `LanceTable.cleanup_old_versions` calls
pylance directly and still errors on
tagged versions (the keyword may not exist in the oldest supported
pylance); `freshness.rs:1377` and
`refresh.rs:1077` still pass `None` (I did not check what those paths do
with tagged versions).
Fixes#4543
Addresses option 2 of #4440.
tokenize() builds the FTS analyzer on the client. For jieba/* and
lindera/* that needs model files under LANCE_LANGUAGE_MODEL_HOME, and
when they’re missing you only got lance’s raw Invalid directory path:
... error, even though server-side search works.
Now, if a jieba/* or lindera/* tokenizer can’t be built, the error keeps
the original cause and adds that tokenize() runs on the client, which
directory the files need to be in, and that for remote tables they must
match the server’s. I also updated the tokenize docs (Rust, Python, TS,
generated JS docs) to say a mismatch can’t be detected, so tokens may
differ from the server’s index.
Behavior when the files exist is unchanged. Detecting a mismatch needs a
server-side tokenize endpoint (option 1), so I left it out. The hint
isn’t limited to remote tables, since Table::as_native is slated for
removal and the wording fits local tables too.
Tests: Rust test with mocked jieba/default index_details and an empty
model dir, one checking other tokenizer errors don’t get the hint, and
Python sync/async tests.
A positive query timeout emits one error and then ends, but
`Duration::ZERO` leaves `TimeoutStream` in `NotStarted`, returning the
same error on every subsequent poll.
Transition the zero-duration path to `Completed` before returning its
error. The initial error is unchanged. Extend both timeout tests to
check that subsequent polls return `None`; the normal-completion test is
unchanged. This affects callers that continue polling after an error,
not `try_collect`, which already stops at the first error.
Validation:
- `cargo fmt --all` and `cargo fmt --all --check`: pass.
- Mechanically extracted the unchanged implementation and three upstream
timeout tests into a native Rust test binary using real cached Arrow
58.4.0, DataFusion 54.1.0, futures 0.3.32 and Tokio 1.53.1 libraries
(matching the current lock). With the new assertions, baseline: 2 pass /
1 fail; fix: 3 pass.
- Full-crate `cargo test --locked --offline --features remote -p lancedb
--lib test_timeout_stream` could not start because this environment
lacks the newly locked Lance `v14.0.0-beta.10` checkout. The component
comparison is not a claim that the full Cargo suite passed; CI still
needs to validate integration.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Keep the request ID when a multipart create/complete response has a
missing or incorrectly typed `upload_id`/`version`. These errors
currently use an empty ID, unlike JSON parse errors, so callers lose the
ID needed to correlate the failed request.
Add six cases through `Table.add(...).write_parallelism(2).execute()`,
capturing the actual outgoing `x-request-id`: four missing/wrong-type
cases fail on unchanged main, while two invalid-JSON controls already
pass. After the fix all six pass, and all 15 multipart tests pass.
Native Linux validation (locked dependencies, `ci` profile):
- `cargo test --offline --profile ci --locked -p lancedb
--no-default-features --features remote --lib
remote::table::tests::test_multipart_write`: 15 passed.
- `cargo fmt --all -- --check`: passed.
- `cargo clippy --offline --profile ci --locked -p lancedb
--no-default-features --features remote --lib --tests -- -D warnings`:
passed.
The committed source bytes match the tested source. HTTP endpoints are
mocked; this does not claim a live Cloud or full-repository test run.
RemoteTable::alter_columns currently unwraps the JsonDataType
conversion. A valid Arrow input that the remote wire format cannot
represent (for example Utf8View, BinaryView, or a struct containing
Utf8View) therefore panics instead of returning an error.
Propagate conversion/serialization failures as Error::InvalidInput,
including the column path. Unsupported inputs are rejected before any
HTTP request; supported alteration payloads are unchanged. This does not
add server support for view types or change server HTTP-error
classification.
Three new native Rust cases reproduce the panic on unchanged production
and return InvalidInput with the fix. The existing supported-type cases
remain passing.
Validation at head 88184ea, Linux/Rust 1.97.0, locked offline
dependencies:
- cargo test --offline --profile ci --locked -p lancedb
--no-default-features --features remote --lib remote::table::tests::
(269 passed)
- cargo clippy --offline --profile ci --locked -p lancedb
--no-default-features --features remote --lib --tests -- -D warnings
- cargo fmt --all -- --check
Current main merges cleanly; the intervening remote-table change is
unrelated nprobes validation. Live Cloud, rebuilt Python/Node bindings
and full-workspace integration were not tested.
AI-assisted; native regression and before/after results independently
checked.
Addresses the client-side waiting portion of #4402.
When create_index returns a job ID, wait_timeout now waits for that
submission rather than an already-ready index with the same name. It
propagates JobFailed / JobCancelled and bounds job polling by the
requested timeout.
Older servers without job IDs keep name-based waiting, calls without
wait_timeout remain fire-and-forget, and the existing two-hour maximum
is preserved. The job wrapper is shared with create_index_async; no
synchronous server-side config validation is added.
Rebased on main 81d52a98, preserving the new materialized-view SQL
constructor. Validation on Linux / Rust 1.97, exact head 61122f92:
- All five new native HTTP-mock tests pass (DONE/FAILED/CANCELLED,
pending deadline, legacy response, no wait, maximum timeout). On
unchanged current-main production the job-tracking and deadline
regressions fail by returning early against the old index.
- All 274 remote-table tests pass, no failures/skips.
- cargo fmt --all -- --check and Clippy --lib --tests -- -D warnings
pass.
Tests and Clippy used --offline --profile ci --locked -p lancedb
--no-default-features --features remote. The affected package was
rebuilt from each selected source root; dependency caches were reused.
This is Rust mocked-HTTP coverage, not live-cloud, Python/Node binding,
full-workspace or performance qualification.
Preserve the compaction option tests and upstream CUDA tests by retaining both Any and MagicMock imports.
Validation: rebuilt the Python extension against Lance v15; 29 optimize/CUDA tests and 7 storage documentation tests passed. Rust formatting and Python source checks passed. Full repository Ruff still reports 49 inherited documentation example errors.
## Root cause
The pinned Lance Arrow crate rebuilt a sliced struct child with its
original Arrow offset but supplied a validity bitmap containing only the
visible rows. Arrow requires the bitmap to cover the offset as well, so
filtered vector searches involving nullable nested fields could panic
with `null_bit_buffer size too small`.
## Fix
- Carry an offset-aligned validity bitmap when propagating parent nulls
to a sliced child.
- Vendor the pinned `lance-arrow` crate until a Lance release includes
the fix; include it in Maturin source distributions.
- Add unit regressions for sliced children and parent bitmaps, plus a
record-batch merge regression.
## Validation
- The issue's 8,600-row Python reproducer completes its filtered flat
dot search and returns 8,448 rows without a panic.
- The targeted Rust merge regression and two focused Python query tests
pass.
- Rust formatting, Ruff formatting and lint, the vendored crate tests,
and the workspace Cargo checks passed. The Maturin source archive
includes the patched crate.
Fixes#4260
<!-- lance-gatekeeper-fix:v1 agent=e6045f701d9b708ee26528ba36353aba
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
Fixes#4142: `lancedb` failed to compile without the `remote` feature.
`TerminalResult::decode` in `rust/lancedb/src/job.rs` constructed the
`Error::Http` variant unconditionally, but that variant is declared
behind
`#[cfg(feature = "remote")]` in `rust/lancedb/src/error.rs`. Since the
crate's default feature set is empty, a plain `lancedb = "0.38"`
dependency
failed with:
```
error[E0599]: no variant named `Http` found for enum `error::Error`
--> rust/lancedb/src/job.rs:102:40
```
The module itself is not remote-only (it carries local job support), so
instead of gating `pub mod job`, both `Error::Http` constructions now go
through a small `remote_job_error` helper that resolves to `Error::Http`
with the `remote` feature and falls back to `Error::Runtime` without it.
A `request_id` is only ever `Some` for remote jobs, so the fallback is
unreachable in practice without the feature.
## Testing
- Reproduced the reported failure on current `main`:
`cargo check -p lancedb --no-default-features` failed with the exact
E0599 errors from the issue before the change, and passes after it.
- `cargo check -p lancedb --features remote` passes with no errors and
no
new warnings.
- Added two regression tests in `rust/lancedb/src/job.rs`
(`decode_missing_result_uses_feature_appropriate_error`,
`decode_unparsable_result_uses_feature_appropriate_error`) asserting the
error variant matches the feature configuration. The test code compiles
under the fixed configuration (`cargo check --tests` passes), and the
branching logic was executed standalone against both feature cfgs
(3/3 green with and without `remote`). The full `cargo test` binary link
could not complete on the small build VM available here (the linker
stalled), so CI coverage of the new tests is appreciated.
- `cargo fmt --all -- --check` is clean.
## Disclosure
AI assistance was used while preparing this change (code search,
editing,
and verification commands were agent-driven); the diagnosis, fix design,
and all test results above were verified by actually running the builds.
`table_names()` enumerates with `read_dir`, which returns files as well
as directories, filtered only on the `.lance` extension — so a loose
`foo.lance` *file* at the database prefix is reported as a table.
`list_tables()` already applies the correct rule ("Only child
directories can be tables"), and
`test_listing_ignores_non_table_children` documents it, so the two
listing APIs currently disagree.
The practical damage is in `drop_all_tables()`, which is built on
`table_names`: with a loose `.lance` file present it deletes some real
tables, fails on the file with `os error 20 (NotADirectory)`, and then
fails the same way on every retry — the database can never be emptied
through the API, and the partial drop is not recoverable.
`table_names` now lists via `list_with_delimiter` and filters
`common_prefixes` through the same `table_name` helper `list_tables`
uses; the sort/`start_after`/`limit` handling is unchanged.
`drop_tables` additionally attempts every name before returning, still
reporting the first error unchanged (kept rather than an aggregate error
so the mapped exception types in the bindings stay exactly as they are).
Extended the non-table-children test to cover `table_names`, plus tests
for the drop behavior — all three fail without the fix. Full `cargo test
-p lancedb --lib --features remote` 1320 passed on the repo-pinned 1.97
toolchain; clippy/fmt clean.
## What
Preserve unbounded plain-query semantics in the Rust Namespace
`QueryTable` adapter.
Plain queries without an explicit limit are now encoded with `i32::MAX`.
Explicit limits remain unchanged, while vector and FTS queries continue
to use the default top-k of 10.
## Root cause
The Rust Namespace adapter converted a missing limit using:
```rust
q.limit.unwrap_or(10)
```
This turned an unbounded plain query into an explicit `k=10`, silently
truncating the results returned by Namespace pushdown.
## Tests
Added regression coverage for unbounded and explicitly limited plain
queries, while verifying that vector and FTS queries retain their
default limit.
```shell
cargo fmt --all -- --check
cargo test --quiet --features remote -p lancedb \
test_convert_to_namespace_query --lib
```
close#3926.
Native `list_indices()` drops registered scalar plugin types absent from
LanceDB's fixed enum, and `index_stats()` reports them as `Unknown`.
Preserve their registered display names through
`IndexType::Custom(String)`.
Resolve these names through Lance's shared `IndexDetails` registry entry
point. Registered types such as MinHashLSH become visible; system and
unregistered types remain excluded. Existing builtin aliases retain
their canonical names. Rust consumers with exhaustive `IndexType`
matches need to handle the added variant.
Manual validation with [Lance
#10224](https://github.com/lance-format/lance/pull/10224) at
`fcc070f88a`: a downstream statistics-only plugin appears as
`Custom("Sketch")` in native listing and statistics. Its `load_index`
deliberately returns an error, and repeated calls do not construct
additional plugin instances, verifying metadata-only lookup through the
shared registry.
Expose optional `auto_refresh_enabled` and
`auto_refresh_interval_seconds` when creating materialized views from
Python and Rust. Omitted intervals remain unset: the server determines
when to refresh.
```python
view = db.create_materialized_view(
"recent_orders", "orders",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
## Architecture and workflow
- Python synchronous, asynchronous, job-returning and namespace wrappers
forward both optional values through PyO3 to the Rust creation builder.
Both default to `None`; no client interval is selected.
- Remote creation calls `POST /v1/materialized_view/{id}/create` with
the defining query and only the supplied refresh properties. A `202
Accepted` response must contain a job ID. Existing job tracking waits
for the server's ordinary SQL CREATE job before synchronous creation
opens the view.
- The server validates and authorizes the CREATE, then uses the existing
SQL worker to populate the view and persist its refresh settings. The
Lance agent subsequently uses those settings to submit REFRESH SQL jobs.
- Explicit positive finite fractional seconds are preserved. An interval
alone does not enable refresh. Embedded connections retain manual
refresh, accept explicit disabled state, and reject requests for
automatic scheduling rather than ignore them.
Includes API documentation and coverage for omission, false,
fractional/invalid intervals, namespace forwarding, server job IDs and
invalid acceptance responses.
Requires the server REST-create endpoint to be deployed before SDK
release. Existing remote `with_no_data=True` remains unsupported.
## Context
`StreamingDataset`'s shipped split strategies (`split_random`,
`split_sequential`, `split_hash`, `split_calculated`) all share one
trait: every row in the base table is assigned to exactly one split,
exactly once. They only differ in *how* that one-to-one assignment is
chosen -- none of them can express "give me a resampled stream of N
rows, allowing repeats."
That's a distinct, common need in ML training: **the bootstrap** -- the
statistical technique (Efron, 1979) of estimating a dataset's properties
by resampling it *with replacement*. It's the same sense scikit-learn's
bagging classifiers mean by `bootstrap=True`. `SplitStrategy::Bootstrap`
adds exactly this to the data loader: sample rows uniformly with
replacement from the (optionally filtered) base table, independently per
split.
## Main use cases
- **Bagging / ensemble training** (e.g. random forests): each split
trains on an independent bootstrap resample of the training set.
- **Oversampling a rare class**: filter to the minority class, then
sample it up to whatever count balances it against the majority, instead
of exhausting its small set once per epoch.
- **Fixed-size training steps independent of dataset size**: want
exactly N batches per epoch regardless of whether the table has 50 rows
or 50 million? Set `num_samples` directly instead of it being implicitly
`len(dataset)`.
- **Weighted sampling without a weight column**: filter to "eligible"
rows (e.g. a windowed-sampling workload filtering to valid window-start
positions), then bootstrap the filtered set -- an item with more
eligible rows is naturally drawn proportionally more often, with no
explicit per-row weight needed.
## Example (an actual run, not a hypothetical one)
A 5-row base table (`row_id` 0..4), sampled via `Bootstrap {
num_samples: 8, sizes: Fixed(2), seed: Some(42) }`:
```
row_id split_id
0 0
2 0
4 0
4 0 <- row 4 repeated within split 0
4 1 <- row 4 also drawn in split 1 (independent draw)
4 1
1 1
2 1
```
8 output rows from a 5-row table -- not 5. Row 4 drawn 4 times total,
row 3 never drawn. That's with-replacement sampling actually behaving
like with-replacement sampling, not an even round-robin dressed up with
a new name.
## What changed
Adds `SplitStrategy::Bootstrap` (uniform sampling with replacement,
optionally filtered, independently seeded per split -- a given split's
content doesn't depend on how many other splits there are, the same
elastic-determinism property documented for the loader's other
strategies). No new `PermutationBuilder` Rust method was needed:
`with_split_strategy` already accepts any `SplitStrategy` value
generically. Python binding:
`PermutationBuilder.bootstrap_sample(num_samples, *, ratios=None,
counts=None, fixed=None, seed=None, split_names=None)`, mirroring
`split_random`'s existing shape.
Also adds a generic `split_strategy` parameter to `StreamingDataset`'s
constructor (a callable taking/returning a `PermutationBuilder`), since
it previously could only reach `split_random`/`split_sequential` even
though `split_hash`/`split_calculated` already shipped. This unblocks
`Bootstrap` and any future strategy without further constructor changes.
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
## Problem
`permutation_builder(table).execute()` and `StreamingDataset(table,
...)` (which builds a permutation at construction) panic with
`RustPanic` on any table whose Arrow schema has schema-level metadata:
```python
schema = pa.schema([pa.field("x", pa.int64())], metadata={b"huggingface": b"{}"})
t = lancedb.connect(tmp).create_table("t", pa.table({"x": list(range(100))}, schema=schema))
permutation_builder(t).split_random(fixed=2, seed=1).execute() # RustPanic
```
Every Hugging Face Lance dataset triggers this, e.g.
`hf://datasets/lance-format/lerobot-xvla-soft-fold/data` table `frames`
carries `source-column-name-map`.
## Root cause
`PermutationBuilder::add_config_metadata` replaced the stream schema's
metadata with the permutation config and then called
`batch.with_schema(schema).unwrap()`. The base-table scan's batches
carry the base schema metadata, and `RecordBatch::with_schema` requires
the target to be a superset of the current schema, metadata included. It
returned `SchemaError("target schema is not superset of current schema
...")`, and the `unwrap` turned that into a panic on a tokio worker.
## Fix
- The permutation table deliberately does **not** inherit base-table
schema metadata. That metadata describes columns the permutation table
(`row_id`, `split_id`) does not have, and its keys can collide with
config keys such as `split_names`. A merge would let a base table's
`split_names` leak into, or override, the permutation's split names.
Batches are rebuilt with `RecordBatch::try_new(schema, columns)`.
- Errors are propagated (`map_ok` → `and_then`) instead of `unwrap`.
- The same panic-on-worker pattern in the splitter's streaming closures
(`split.rs`: `try_new`/`filter(...).unwrap()`) now propagates errors
too. Nothing else in `dataloader/` used `with_schema(...).unwrap()`.
## Tests
- Rust: `test_permutation_over_table_with_schema_metadata` in
`builder.rs` builds a random split + shuffled permutation over a table
with `huggingface` and a colliding `split_names` key. It panics with the
original `SchemaError` on the old code and passes now.
- Python: `test_permutation_over_table_with_schema_metadata` in
`test_permutation.py` covers the repro and reads both splits through
`Permutations`/`Permutation.from_tables`.
`test_streaming_dataset_over_table_with_schema_metadata` in
`test_elastic_dataloader.py` covers `StreamingDataset` construction and
iteration.
- `cargo fmt`, `cargo clippy --features remote --tests --examples`, the
dataloader Rust tests (42), `test_permutation.py` (72), and
`test_elastic_dataloader.py` all pass.
`test_torch.py::test_remote_permutation_dataloader_multiprocessing`
fails locally with `take query result did not include ordering column
'__lancedb_take_row_offset'`. It uses `Permutation.identity`, which does
not go through `PermutationBuilder`, so this change does not cause it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
udf() rejected a large_binary input or output, and large_binary nested in a list or struct, outside a Blob v2 payload. It is now a grammar primitive like large_utf8, and the shared type golden lists it.
Creating a view now follows its source through each view's stored
coordinate, up to 32 levels, and refuses when the walk reaches the name
being created. One table open per level; no catalog scan.
Part of ENT-2922.
## Why
Every MemWAL write and read resolves rows by the table's unenforced
primary key, but `enable_memwal` accepted a table without one. The spec
installed, and every MemWAL `merge_insert` then failed with "table has a
MemWAL index but no unenforced primary key" until MemWAL was disabled.
Lance's MemWAL initialization doesn't catch it: its key checks pass when
there is no key.
## What changes
`enable_memwal` refuses a spec on a table with no unenforced primary
key, for every sharding mode, before anything is committed. The check
runs against the dataset the spec is committed to; a key cannot be
removed once set, and an overwrite or restore of the table conflicts
with that commit. Servers that install specs through this method get the
check too.
The Rust doc on `Table::enable_memwal` now states the requirement; the
Python and TypeScript docs already did.
## Tests
- New `test_enable_memwal_requires_primary_key` (Rust and Python):
unsharded, bucket and identity specs are all refused without a key, a
refused spec leaves nothing installed, and the same table accepts a spec
once a key is set. The Rust test fails with the check disabled.
- Tests that installed a spec on a table without a key now declare one
first (6 Rust, 3 Python).
- `cargo nextest run -p lancedb --features remote --lib`: 1529 passed.
`cargo clippy -p lancedb --features remote --tests --examples -- -D
warnings` and `cargo fmt --check` are clean; no integration test or
example installs a spec.
- Python: `test_memwal_spec.py` and `test_merge_insert_lsm.py`, 43
passed; `ruff format` and `ruff check` are clean.
## What
Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.
### 1. MemWAL APIs drop "LSM"
The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.
| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |
TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.
Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.
### 2. Stats use SSTable and table shard
Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.
**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.
**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.
| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |
Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.
`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.
## Compatibility
Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:
- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies
## Validation
- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
`RemoteTable::set_unenforced_primary_key` returned `NotSupported`, so
the call failed against LanceDB Cloud and enterprise from every SDK —
Python and TypeScript both forward to it. That also blocked sharded LSM
writes, since the server rejects bucket/identity sharding on a table
that declares no unenforced primary key, and the SDK had no way to
declare one.
No server-side support was missing. The unenforced primary key is Lance
schema field metadata, and the existing `update_field_metadata` endpoint
writes exactly that, so the remote table now installs the key through
it. The commit layer behind that endpoint installs the position,
enforces immutability and runs `verify_primary_key()` — the same code a
native table reaches, so both paths agree on semantics and not just on
error messages.
## Changes
- `rust/lancedb/src/table/primary_key.rs`: the request validation
(single column, column exists, supported dtype, key not already set) and
the metadata edit (set the 1-indexed position, clear the legacy boolean
key) move into `validate()` and `install_edit()`, shared by both table
types so they cannot drift. Native behaviour is unchanged.
- `rust/lancedb/src/remote/table.rs`: real implementation —
`check_mutable()`, validate against the described schema, then one
`update_field_metadata` request.
- `python/python/lancedb/remote/table.py`: drop the "Not supported on
LanceDB Cloud." docstring.
No binding changes were needed; Python and TypeScript already forward to
the Rust `BaseTable` method.
## Tests
Three mocked-server tests in `rust/lancedb/src/remote/table.rs`:
- the happy path, asserting the request body carries
`lance-schema:unenforced-primary-key:position: "1"` and a null legacy
key on the right field path;
- the invalid requests a native table also rejects (empty, compound,
unknown column, unsupported dtype), each failing before any write
reaches the server;
- the already-set case, rejected from the schema the server already
reports.
`cargo test --features remote -p lancedb --lib`: 1186 passed, 0 failed.
`cargo clippy --features remote -p lancedb --lib --tests` is clean.
## Note for enterprise
Sophon pins the lancedb submodule at `v0.39.0-beta.6`, so this reaches
enterprise on the next bump.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011xV8EebEZDmDDV8hVBm6MD
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Updates the Rust workspace dependencies and Java lance-core to [Lance
v14.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.18).
Mirrors Lance's pinned object_store patch to support the multipart write
mode API required by this release.
Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed.
---------
Co-authored-by: Lu Qiu <luqiujob@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Yang Cen <bubble-cal@outlook.com>
Gatefixer
Jack Ye
Shiduo LiandXuanwo
Wyatt Alt
Colin Patrick McCabe
Avanish Garg
Aviral Saxena
adenzhou1350
SIDDARTHA REDDY
Joaquin Hui Gomez
Li Zinuo
Lance Release
Dan RammerandClaude Opus 5.5
Lei XuandClaude Opus 5.5
XY Zhan
