LanceDBConnection.session read AsyncConnection.session, but
AsyncConnection has no session attribute, so access raised
AttributeError. Store the session passed to the constructor and return
that object.
Assisted by Cursor (Grok).
Co-authored-by: Preston Zhao <195942578+canyang25@users.noreply.github.com>
TypeScript CreateTableOptions.mode omitted exist_ok even though
createTable already forwards that mode to the native client.
closes: #2900
---------
Co-authored-by: Preston Zhao <195942578+canyang25@users.noreply.github.com>
RemoteTable::alter_columns currently unwraps the JsonDataType
conversion. A valid Arrow input that the remote wire format cannot
represent (for example Utf8View, BinaryView, or a struct containing
Utf8View) therefore panics instead of returning an error.
Propagate conversion/serialization failures as Error::InvalidInput,
including the column path. Unsupported inputs are rejected before any
HTTP request; supported alteration payloads are unchanged. This does not
add server support for view types or change server HTTP-error
classification.
Three new native Rust cases reproduce the panic on unchanged production
and return InvalidInput with the fix. The existing supported-type cases
remain passing.
Validation at head 88184ea, Linux/Rust 1.97.0, locked offline
dependencies:
- cargo test --offline --profile ci --locked -p lancedb
--no-default-features --features remote --lib remote::table::tests::
(269 passed)
- cargo clippy --offline --profile ci --locked -p lancedb
--no-default-features --features remote --lib --tests -- -D warnings
- cargo fmt --all -- --check
Current main merges cleanly; the intervening remote-table change is
unrelated nprobes validation. Live Cloud, rebuilt Python/Node bindings
and full-workspace integration were not tested.
AI-assisted; native regression and before/after results independently
checked.
Addresses the client-side waiting portion of #4402.
When create_index returns a job ID, wait_timeout now waits for that
submission rather than an already-ready index with the same name. It
propagates JobFailed / JobCancelled and bounds job polling by the
requested timeout.
Older servers without job IDs keep name-based waiting, calls without
wait_timeout remain fire-and-forget, and the existing two-hour maximum
is preserved. The job wrapper is shared with create_index_async; no
synchronous server-side config validation is added.
Rebased on main 81d52a98, preserving the new materialized-view SQL
constructor. Validation on Linux / Rust 1.97, exact head 61122f92:
- All five new native HTTP-mock tests pass (DONE/FAILED/CANCELLED,
pending deadline, legacy response, no wait, maximum timeout). On
unchanged current-main production the job-tracking and deadline
regressions fail by returning early against the old index.
- All 274 remote-table tests pass, no failures/skips.
- cargo fmt --all -- --check and Clippy --lib --tests -- -D warnings
pass.
Tests and Clippy used --offline --profile ci --locked -p lancedb
--no-default-features --features remote. The affected package was
rebuilt from each selected source root; dependency caches were reused.
This is Rust mocked-HTTP coverage, not live-cloud, Python/Node binding,
full-workspace or performance qualification.
Since #3232, `pydantic_to_schema` maps an `Enum` field to the Arrow type
of its values, including enums without a `str`/`int` mixin. Writing such
models still failed: `model_to_dict()` returns `model.model_dump()`,
which keeps the Enum members, and Arrow can only convert members that
are also `str`/`int` instances.
```python
class Color(Enum):
RED = "red"
class Item(LanceModel):
color: Color
tbl = db.create_table("items", schema=Item)
tbl.add([Item(color=Color.RED)])
# pyarrow.lib.ArrowTypeError: Expected bytes, got a 'Color' object
```
`create_table(data=[...])` fails the same way, and int-valued enums
raise `ArrowInvalid`.
`model_to_dict()` now replaces Enum members with their values, including
in nested models and lists. Lists of plain scalars (vectors in
particular) are returned without being walked, so the per-row cost stays
about the same as `model_dump()` alone (21.3µs vs 20.3µs for a 1536-dim
model). Reading back is unchanged: Pydantic already validates the stored
value into the Enum member.
Added `test_plain_enum_values_are_written`, which covers str and int
valued enums, `Optional` and `List` fields, and an Enum inside a nested
model, and checks that `to_pydantic` round-trips the rows. It fails with
the `ArrowTypeError` above before the change. `test_pydantic.py`,
`test_embeddings.py`, `test_table.py` and `test_db.py` pass, and `ruff
format --check` / `ruff check` are clean.
`EmbeddingFunctionRegistry.register()` documents that the class name is
used when no alias is given, and it does register the class under that
name. But it stored the alias itself (`None`) as
`__embedding_function_registry_alias__`, which is what
`function_to_metadata()` writes into the table schema. A table using
such a function gets `"name": null` in its `embedding_functions`
metadata, and any later read of the functions fails:
```python
@get_registry().register()
class MyEmbed(TextEmbeddingFunction): ...
f = MyEmbed.create()
class Schema(LanceModel):
text: str = f.SourceField()
vector: Vector(4) = f.VectorField()
tbl = db.create_table("t", schema=Schema)
tbl.add([{"text": "hi"}]) # KeyError: None
tbl.search("hi") # KeyError: None
```
This stores the key the class is registered under instead, so the class
name round-trips. For aliased classes the key is the alias, so their
metadata is unchanged. The TypeScript registry already falls back to
`ctor.name` in the same way.
Added `test_embedding_function_registered_without_alias`, which creates
a table with such a function, reopens it, and adds a row. It fails with
`KeyError` before the change. `test_embeddings.py`, `test_pydantic.py`,
`test_table.py` and `test_db.py` pass, and `ruff format --check` / `ruff
check` are clean.
## Summary
With `rerank(..., normalize="rank")`, a hybrid query returns ranks (1.0,
2.0, 3.0, ...) in the `_distance` and `_score` columns instead of the
real vector distances and BM25 scores.
`_combine_hybrid_results` converted both result sets to ranks first, and
only then captured `original_distances` / `original_scores`. So the step
that restores the original values after reranking put the ranks back.
With `normalize="score"`, the real values are returned as expected.
Repro (RRFReranker(return_score="all"), vector distances 0 / 25 / 100):
| normalize | `_distance` returned | `_score` returned |
| --- | --- | --- |
| `"score"` | 0.0, 25.0, 100.0 | 0.47, 0.67, None |
| `"rank"` (before) | 1.0, 2.0, 3.0 | 1.0, 2.0, None |
| `"rank"` (after) | 0.0, 25.0, 100.0 | 0.47, 0.67, None |
## Changes
- Capture the original `_distance` / `_score` columns before converting
them to ranks, then normalize the ranked column. The reranker input is
unchanged; only the values restored afterwards are fixed.
`AsyncHybridQuery` uses the same function, so both sync and async are
covered.
## Test plan
- [x] New `test_hybrid_return_all_restores_original_scores` in
`test_rerankers.py`, parametrized over `score` / `rank`: the `rank` case
fails before the fix and passes after
- [x] `pytest python/tests/test_rerankers.py
python/tests/test_hybrid_query.py`
- [x] `ruff format --check` / `ruff check`
I have no local Rust toolchain, so I ran the tests against the released
0.39.0 native module with the changed Python files overlaid. The changed
functions are identical between 0.39.0 and main.
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
#3971 added `test_zero_lifetime_token_is_refreshed`, but its expected
headers omit `x-lancedb-credential-type: oidc`, which
`OAuthProvider.get_headers()` has emitted since #2631/#4070. The test
fails against current main and breaks the PR CI matrix for every open
PR.
## Changes
- Align the two zap assertions in
`test_zero_lifetime_token_is_refreshed` with the provider contract (see
`test_token_refresh` and `test_sync_method`).
## Tests
- `pytest python/tests/test_header_provider.py -q`
## Summary
Fixes#4583.
`Connection::list_namespaces` narrowed `Option<u32>` to `i32` with an
unchecked `as` cast, so a JavaScript `limit` between 2^31 and 2^32 - 1
(e.g. `3000000000`) wrapped negative before reaching
`ListNamespacesRequest`. The server received a negative page size
instead of a clear client-side error.
Replaced the cast with a `checked_limit` helper using `i32::try_from`,
returning `napi::Error::from_reason` on overflow, matching the existing
`try_from` validation used for `limit` elsewhere in the same file.
## Tests
Unit tests cover the wrap case (`u32` values above `i32::MAX` reject),
the boundary (`i32::MAX` passes), and `None`/zero passthrough. `cargo
test -p lancedb-nodejs` passes 8/8 and `pnpm build` compiles the binding
and TypeScript definitions cleanly.
Fixes#4331
With `return_score="all"`, model rerankers reconstruct merged rows
without their Arrow schema. This changes fixed-size float32 vectors into
variable-length float64 lists, loses metadata, and can overflow valid
uint64 row IDs.
Unify the original input schemas before inserting placeholder scores,
then use the unified schema when reconstructing the merged table. Row
merging and reranking behavior stay the same. Capturing the schema first
also preserves float64 scores without conflicting with the float32 null
placeholders.
Validation:
- New regression cases: 5 failures and 5 passing controls before the
fix; all 10 pass afterward.
- Reranker, hybrid-query and TypeSafe reranker tests: 150 passed, 17
skipped. Optional live model/API tests were not run.
- The fake-client tests that use random metadata passed again with seeds
7 and 41.
- Repository-wide Ruff lint/format checks, configured Pyright check and
applicable pre-commit hooks pass.
- Tests used the locally built Rust/Python extension and the
repository's locked uv environment. No Rust code changed.
AI disclosure: I used Codex for investigation, implementation, tests,
local checks, and drafting this description.
## Root cause
The pinned Lance Arrow crate rebuilt a sliced struct child with its
original Arrow offset but supplied a validity bitmap containing only the
visible rows. Arrow requires the bitmap to cover the offset as well, so
filtered vector searches involving nullable nested fields could panic
with `null_bit_buffer size too small`.
## Fix
- Carry an offset-aligned validity bitmap when propagating parent nulls
to a sliced child.
- Vendor the pinned `lance-arrow` crate until a Lance release includes
the fix; include it in Maturin source distributions.
- Add unit regressions for sliced children and parent bitmaps, plus a
record-batch merge regression.
## Validation
- The issue's 8,600-row Python reproducer completes its filtered flat
dot search and returns 8,448 rows without a panic.
- The targeted Rust merge regression and two focused Python query tests
pass.
- Rust formatting, Ruff formatting and lint, the vendored crate tests,
and the workspace Cargo checks passed. The Maturin source archive
includes the patched crate.
Fixes#4260
<!-- lance-gatekeeper-fix:v1 agent=e6045f701d9b708ee26528ba36353aba
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- preflight CUDA availability and initialization before accelerated
index training
- direct unavailable or misconfigured CUDA environments to the explicit
CPU path
- cover both legacy and unified Python index APIs
## Root cause
LanceDB delegated accelerator requests directly to PyLance. When PyTorch
lacked CUDA support or CUDA initialization failed, users received a
low-level assertion or runtime error with no indication that omitting
`accelerator` or setting `accelerator=None` creates the index on CPU.
The repair validates CUDA before training and preserves explicit
accelerator semantics without silently changing the requested backend.
## Scope
This PR addresses only the separately reproduced CUDA availability and
initialization error. It does not close#1766; the reported query
failure after successful CUDA index creation remains open pending
reproducible evidence.
## Validation
- `uv run --extra tests pytest python/tests/test_table.py -k
create_index -q` (10 passed)
- `ruff format .`
- `ruff check .`
- CPU-only integration check confirms the actionable fallback message
Related to #1766
<!-- lance-gatekeeper-fix:v1 agent=d05c37237b04c2c8f88395a82e00bc67
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
Fixes#4142: `lancedb` failed to compile without the `remote` feature.
`TerminalResult::decode` in `rust/lancedb/src/job.rs` constructed the
`Error::Http` variant unconditionally, but that variant is declared
behind
`#[cfg(feature = "remote")]` in `rust/lancedb/src/error.rs`. Since the
crate's default feature set is empty, a plain `lancedb = "0.38"`
dependency
failed with:
```
error[E0599]: no variant named `Http` found for enum `error::Error`
--> rust/lancedb/src/job.rs:102:40
```
The module itself is not remote-only (it carries local job support), so
instead of gating `pub mod job`, both `Error::Http` constructions now go
through a small `remote_job_error` helper that resolves to `Error::Http`
with the `remote` feature and falls back to `Error::Runtime` without it.
A `request_id` is only ever `Some` for remote jobs, so the fallback is
unreachable in practice without the feature.
## Testing
- Reproduced the reported failure on current `main`:
`cargo check -p lancedb --no-default-features` failed with the exact
E0599 errors from the issue before the change, and passes after it.
- `cargo check -p lancedb --features remote` passes with no errors and
no
new warnings.
- Added two regression tests in `rust/lancedb/src/job.rs`
(`decode_missing_result_uses_feature_appropriate_error`,
`decode_unparsable_result_uses_feature_appropriate_error`) asserting the
error variant matches the feature configuration. The test code compiles
under the fixed configuration (`cargo check --tests` passes), and the
branching logic was executed standalone against both feature cfgs
(3/3 green with and without `remote`). The full `cargo test` binary link
could not complete on the small build VM available here (the linker
stalled), so CI coverage of the new tests is appreciated.
- `cargo fmt --all -- --check` is clean.
## Disclosure
AI assistance was used while preparing this change (code search,
editing,
and verification commands were agent-driven); the diagnosis, fix design,
and all test results above were verified by actually running the builds.
`table_names()` enumerates with `read_dir`, which returns files as well
as directories, filtered only on the `.lance` extension — so a loose
`foo.lance` *file* at the database prefix is reported as a table.
`list_tables()` already applies the correct rule ("Only child
directories can be tables"), and
`test_listing_ignores_non_table_children` documents it, so the two
listing APIs currently disagree.
The practical damage is in `drop_all_tables()`, which is built on
`table_names`: with a loose `.lance` file present it deletes some real
tables, fails on the file with `os error 20 (NotADirectory)`, and then
fails the same way on every retry — the database can never be emptied
through the API, and the partial drop is not recoverable.
`table_names` now lists via `list_with_delimiter` and filters
`common_prefixes` through the same `table_name` helper `list_tables`
uses; the sort/`start_after`/`limit` handling is unchanged.
`drop_tables` additionally attempts every name before returning, still
reporting the first error unchanged (kept rather than an aggregate error
so the mapped exception types in the bindings stay exactly as they are).
Extended the non-table-children test to cover `table_names`, plus tests
for the drop behavior — all three fail without the fix. Full `cargo test
-p lancedb --lib --features remote` 1320 passed on the repo-pinned 1.97
toolchain; clippy/fmt clean.
`LanceHybridQueryBuilder._validate_query`'s ValueError concatenated two
adjacent string literals without a separator, rendering "...explicitly
for hybrid search.**But** not both.". Added the trailing space.
String-only change.
`VoyageAIReranker`'s `api_key` docstring said it falls back to
`OPENAI_API_KEY`, but the class actually reads `VOYAGE_API_KEY` (and
raises a "VOYAGE_API_KEY not set" error without it) — the docstring was
copy-pasted from the OpenAI reranker. Docstring-only fix.
This PR converts the uri parameter to string before calling urlparse in
get_uri_scheme and get_uri_location, allowing pathlib.Path objects to be
passed seamlessly without raising a TypeError.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#4033
`create_table(data=...)` without an explicit schema raised `IndexError:
index out of bounds` when a `vector`/`embedding`-named list column was
all-null. `_modal_list_size` infers the dimension with
`pc.mode(pc.list_value_length(arr))[0]`, and pyarrow's `mode` skips
nulls, so an all-null column produces a zero-length result and the `[0]`
subscript blows up.
The sibling `_infer_vector_dim` already guards the identical
`pc.mode(...)` with a `len == 0` check and returns `None`;
`_modal_list_size` just didn't. This makes `_modal_list_size` return
`Optional[int]` and return `None` when every value is null, and the call
site then leaves the column as a variable-length list instead of
inventing a fixed size — the dimension is genuinely unknowable when
there are no non-null rows.
Added a regression test that creates a table from an all-null
`list<float32>` vector column and asserts it succeeds, the column stays
a `list<float32>`, and the rows are queryable. It fails on `main` with
the `IndexError` and passes with this change.
Retries individual inputs one at a time when the OpenAI API rejects a
batch with BadRequestError, instead of discarding the whole batch.
Valid inputs now still get embeddings; only the actually-bad input
resolves to None, and the log names the exact failing input.
Fixes#1677
## Summary
Sync `Table.search()` declares tuple, `pyarrow.Array`, and
`pyarrow.ChunkedArray` query vectors as supported, and the async query
path already accepts them. The sync query builder only classified lists
and NumPy arrays as vectors, however, so these inputs were routed
through embedding/FTS resolution and then rejected. Arrow and tuple
inputs also did not contribute a query dimension when selecting among
multiple vector columns.
This change:
- recognizes the supported tuple and Arrow containers as sync vector
queries for both `auto` and explicit `vector` query types;
- converts them to the existing internal `float32` NumPy representation
at the sync builder boundary;
- infers dimensions from tuple, primitive Arrow, and list-valued Arrow
queries so schemas with differently sized vector columns select the
matching column.
The regression test uses two vector columns with dimensions 2 and 3. It
verifies tuple, Arrow array, and chunked Arrow queries through both sync
query modes and confirms the dimension-2 column is selected.
## Test plan
- `pytest
python/tests/test_table.py::test_sync_search_accepts_supported_vector_containers
-q` (6 passed)
- `pytest python/tests/test_util.py python/tests/test_query.py
python/tests/test_table.py -q -m "not slow and not s3_test"` (308
passed, 1 skipped, 1 deselected)
- `pytest python/tests -q -m "not slow and not s3_test"
--ignore=python/tests/docs` (981 passed, 25 skipped, 65 deselected)
- `ruff format --check python`
- `ruff check python`
- `cargo fmt --all --check`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- treat an OAuth `expires_in` value of zero as an explicit immediate
expiration
- refresh zero-lifetime tokens instead of caching them indefinitely
- add a regression test covering two consecutive header requests
## Tests
- `pytest python/tests/test_header_provider.py -q` (16 passed)
- `ruff format --check python/lancedb/remote/header.py
python/tests/test_header_provider.py`
- `ruff check python/lancedb/remote/header.py
python/tests/test_header_provider.py`
## What
Preserve unbounded plain-query semantics in the Rust Namespace
`QueryTable` adapter.
Plain queries without an explicit limit are now encoded with `i32::MAX`.
Explicit limits remain unchanged, while vector and FTS queries continue
to use the default top-k of 10.
## Root cause
The Rust Namespace adapter converted a missing limit using:
```rust
q.limit.unwrap_or(10)
```
This turned an unbounded plain query into an explicit `k=10`, silently
truncating the results returned by Namespace pushdown.
## Tests
Added regression coverage for unbounded and explicitly limited plain
queries, while verifying that vector and FTS queries retain their
default limit.
```shell
cargo fmt --all -- --check
cargo test --quiet --features remote -p lancedb \
test_convert_to_namespace_query --lib
```
close#3926.
## Summary
- move synchronous iterator preparation off LanceDB's singleton
background event loop
- keep the loop available when an input iterator executes a synchronous
table query
- cover query-backed iterators for both `Table.add` and `create_table`
## Root cause
The async implementations prepared input data on the singleton
background loop and peeked at the first item of iterators there. If that
iterator called a synchronous LanceDB query, `LOOP.run()` scheduled the
query on the same loop and blocked waiting for it, so neither operation
could progress.
## Validation
- `ruff format` on the changed Python files
- `ruff check .`
- `cd python && uv run --frozen --extra tests pytest
python/tests/test_table.py -q` (138 passed)
- the broader non-slow suite was attempted but hit a native segmentation
fault in the external DataFusion `register_table_provider` path; an
isolated control using the pre-change inline preparation behavior
reproduced the same unrelated crash
Fixes#2107
<!-- lance-gatekeeper-fix:v1 agent=8b4aaf556014ac0e457c9d63495697e4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a Python regression for filtered scans over deeply nested
list/struct values
- verify the public query path returns every nested sibling field
without an Arrow schema mismatch
## Root cause
Older Lance scanner code split filter projections into partial nested
arrays and did not recursively merge List<Struct> children, so a sibling
field could be omitted from the returned array while remaining in the
declared schema. That merge was fixed upstream in
lance-format/lance#4838, and the Lance revision currently pinned by
LanceDB contains that fix. This test makes the repair durable at the
LanceDB Python API boundary.
## Validation
- cd python && uv run --extra tests pytest
python/tests/test_query.py::test_filter_preserves_deeply_nested_struct_fields
-q
- cd python && uv run --extra tests pytest python/tests/test_query.py -q
- uv run --project python --extra tests --extra dev ruff check .
- uv run --project python --extra tests --extra dev ruff format --check
python/python/tests/test_query.py
Fixes#2217
<!-- lance-gatekeeper-fix:v1 agent=9ddc0efd1f13888b2a7a896881c6e68a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add regression coverage for an IVF_RQ index created through the Lance
Python API and consumed through LanceDB
- verify a fresh LanceDB handle can list and query the
externally-created index
## Root cause
Lance 4.0.0-beta.5 could serialize `rotate_mat_position` as null, while
the Lance 2.0.0 reader used by LanceDB 0.29.2 required a `u32`. Upstream
Lance fixed the incompatible IVF_RQ metadata handling and index
versioning, and the current LanceDB dependency includes that repair.
LanceDB was missing coverage for the reported mixed-API workflow,
allowing the compatibility regression to go undetected.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_index.py
-q` (24 passed)
- `uv run --project python --extra dev ruff format .`
- `uv run --project python --extra dev ruff check .`
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_index.py`
- `uv run --project python --extra dev ruff check
python/python/tests/test_index.py`
Fixes#3092
<!-- lance-gatekeeper-fix:v1 agent=7d8c849bd8421be65d32355c42a60b8a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- inspect the assembled x64 musl addon for the unresolved AVX-512
distance-table symbol before npm publication
- fail the publish lifecycle with an actionable error if the broken
symbol is present
- add regression coverage using the dynamic-symbol output from the
affected 0.27.x artifact
## Root cause
LanceDB 0.27.x could enable the Rust AVX-512 distance-table callsite
when the separate C kernel build did not emit its object, leaving
`sum_4bit_dist_table_32bytes_batch_avx512` undefined in the musl addon.
The current Lance dependency includes the upstream split-kernel cfg fix,
and the published 0.33.0 musl artifact no longer has the undefined
symbol, but the LanceDB publish path did not verify that property before
releasing an addon.
This adds a release guard around the corrected implementation so the
same invalid binary cannot be published again.
## Validation
- `pnpm lint`
- `pnpm run build`
- `pnpm test --runInBand` (680 passed, 5 skipped)
- `pnpm run docs`
- checked published artifacts: 0.27.1 is rejected for the unresolved
symbol; 0.33.0 is accepted
Fixes#3200
<!-- lance-gatekeeper-fix:v1 agent=4094b9b6de4b43665a9ff50fa35322fb
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Native `list_indices()` drops registered scalar plugin types absent from
LanceDB's fixed enum, and `index_stats()` reports them as `Unknown`.
Preserve their registered display names through
`IndexType::Custom(String)`.
Resolve these names through Lance's shared `IndexDetails` registry entry
point. Registered types such as MinHashLSH become visible; system and
unregistered types remain excluded. Existing builtin aliases retain
their canonical names. Rust consumers with exhaustive `IndexType`
matches need to handle the added variant.
Manual validation with [Lance
#10224](https://github.com/lance-format/lance/pull/10224) at
`fcc070f88a`: a downstream statistics-only plugin appears as
`Custom("Sketch")` in native listing and statistics. Its `load_index`
deliberately returns an error, and repeated calls do not construct
additional plugin instances, verifying metadata-only lookup through the
shared registry.
Expose optional `auto_refresh_enabled` and
`auto_refresh_interval_seconds` when creating materialized views from
Python and Rust. Omitted intervals remain unset: the server determines
when to refresh.
```python
view = db.create_materialized_view(
"recent_orders", "orders",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
## Architecture and workflow
- Python synchronous, asynchronous, job-returning and namespace wrappers
forward both optional values through PyO3 to the Rust creation builder.
Both default to `None`; no client interval is selected.
- Remote creation calls `POST /v1/materialized_view/{id}/create` with
the defining query and only the supplied refresh properties. A `202
Accepted` response must contain a job ID. Existing job tracking waits
for the server's ordinary SQL CREATE job before synchronous creation
opens the view.
- The server validates and authorizes the CREATE, then uses the existing
SQL worker to populate the view and persist its refresh settings. The
Lance agent subsequently uses those settings to submit REFRESH SQL jobs.
- Explicit positive finite fractional seconds are preserved. An interval
alone does not enable refresh. Embedded connections retain manual
refresh, accept explicit disabled state, and reject requests for
automatic scheduling rather than ignore them.
Includes API documentation and coverage for omission, false,
fractional/invalid intervals, namespace forwarding, server job IDs and
invalid acceptance responses.
Requires the server REST-create endpoint to be deployed before SDK
release. Existing remote `with_no_data=True` remains unsupported.
## Context
`StreamingDataset`'s shipped split strategies (`split_random`,
`split_sequential`, `split_hash`, `split_calculated`) all share one
trait: every row in the base table is assigned to exactly one split,
exactly once. They only differ in *how* that one-to-one assignment is
chosen -- none of them can express "give me a resampled stream of N
rows, allowing repeats."
That's a distinct, common need in ML training: **the bootstrap** -- the
statistical technique (Efron, 1979) of estimating a dataset's properties
by resampling it *with replacement*. It's the same sense scikit-learn's
bagging classifiers mean by `bootstrap=True`. `SplitStrategy::Bootstrap`
adds exactly this to the data loader: sample rows uniformly with
replacement from the (optionally filtered) base table, independently per
split.
## Main use cases
- **Bagging / ensemble training** (e.g. random forests): each split
trains on an independent bootstrap resample of the training set.
- **Oversampling a rare class**: filter to the minority class, then
sample it up to whatever count balances it against the majority, instead
of exhausting its small set once per epoch.
- **Fixed-size training steps independent of dataset size**: want
exactly N batches per epoch regardless of whether the table has 50 rows
or 50 million? Set `num_samples` directly instead of it being implicitly
`len(dataset)`.
- **Weighted sampling without a weight column**: filter to "eligible"
rows (e.g. a windowed-sampling workload filtering to valid window-start
positions), then bootstrap the filtered set -- an item with more
eligible rows is naturally drawn proportionally more often, with no
explicit per-row weight needed.
## Example (an actual run, not a hypothetical one)
A 5-row base table (`row_id` 0..4), sampled via `Bootstrap {
num_samples: 8, sizes: Fixed(2), seed: Some(42) }`:
```
row_id split_id
0 0
2 0
4 0
4 0 <- row 4 repeated within split 0
4 1 <- row 4 also drawn in split 1 (independent draw)
4 1
1 1
2 1
```
8 output rows from a 5-row table -- not 5. Row 4 drawn 4 times total,
row 3 never drawn. That's with-replacement sampling actually behaving
like with-replacement sampling, not an even round-robin dressed up with
a new name.
## What changed
Adds `SplitStrategy::Bootstrap` (uniform sampling with replacement,
optionally filtered, independently seeded per split -- a given split's
content doesn't depend on how many other splits there are, the same
elastic-determinism property documented for the loader's other
strategies). No new `PermutationBuilder` Rust method was needed:
`with_split_strategy` already accepts any `SplitStrategy` value
generically. Python binding:
`PermutationBuilder.bootstrap_sample(num_samples, *, ratios=None,
counts=None, fixed=None, seed=None, split_names=None)`, mirroring
`split_random`'s existing shape.
Also adds a generic `split_strategy` parameter to `StreamingDataset`'s
constructor (a callable taking/returning a `PermutationBuilder`), since
it previously could only reach `split_random`/`split_sequential` even
though `split_hash`/`split_calculated` already shipped. This unblocks
`Bootstrap` and any future strategy without further constructor changes.
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
Cargo.lock: webpki-root-certs 1.0.10; webpki-roots 1.0.10 and
rustls-webpki 0.103.16 are already current.
docs/web-tests/rs/Cargo.lock: rustls-webpki 0.103.16. python/uv.lock:
certifi 2026.10.19, the only entry that changes.
The root crate drops the root CA whose intermediates were compromised
(Mozilla bug 2083449), and rustls-webpki 0.103.16 rejects chains that
carry the compromised eMudhra intermediate keys. certifi 2026.10.19 is
the release the certifi advisory names as fixed: it drops the same root
and keeps the issuer's other roots, so HTTPS connections that chain to
them keep working. Relative to the previous 2026.6.17 pin it also adds
Telia EC and RSA TLS Root CA v3 and SECOM TLS ECC and RSA Root CA 2024
and drops Atos TrustedRoot 2011.
The webpki-root-certs entry is written by hand with the crates.io
checksum.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Esteban Gutierrez <esteban@lancedb.com>
Co-authored-by: Claude Opus 5.1 <noreply@anthropic.com>
Pin the Rust workspace and Java lance-core dependency to [Lance
v14.0.0-beta.22](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.22),
including managed data file parts and descriptor reuse from
lance-format/lance#10176. This enables the next LanceDB preview release
to provide the tagged dependency needed by lancedb/sophon#8898. No
compatibility code changes are required.
Update rustls-webpki to 0.103.16 and webpki-roots to 1.0.10 in the
lockfile to resolve the two TLS advisories blocking dependency CI
(RUSTSEC-2026-0382 and RUSTSEC-2026-0381), without adding advisory
exceptions.
The Java dependency availability blocker is resolved: both the beta.22
POM and JAR return HTTP 200 from maven.lance.org. On `ea043bf93`, the
review reproducer (`cd java && ./mvnw -U -q test -pl lancedb-core
-Dspotless.skip=true`) passes all 37 tests, and [Java CI attempt
2](https://github.com/lancedb/lancedb/actions/runs/39556336205/attempts/2)
passes.
Lines 105-106 of AGENTS.md tell agents to add TypeScript methods in
`nodejs/src/table.ts` and `nodejs/src/native_table.ts`. Neither file
exists; the abstract `Table` class and the concrete `LocalTable` class
are both in `nodejs/lancedb/table.ts`. An agent following these steps
searches for files that are not there.
This updates both paths. Docs-only, no code change.
Suggested by LanceDB support (Daisuke) after I reported it by email.
## Summary
- Rework `docs/web/training/torch.mdx` for people who already know the
PyTorch `DataLoader`, built around `lancedb.streaming.StreamingDataset`.
- The quickstart trains a state-to-action policy on the real
`lance-format/lerobot-xvla-soft-fold` dataset loaded via `hf://`, using
a zero-copy Arrow-to-tensor transform.
- Add a short table mapping PyTorch idioms (`IterableDataset`,
`DistributedSampler`, `collate_fn`, `random_split`,
`StatefulDataLoader`) to their LanceDB equivalents.
- Add an Optimizations section: Transform (including an `ExtractFrames`
example using blob v2 and torchcodec), Distributed Training, and the
existing Output Formats content moved underneath.
## Notes
- The quickstart depends on #4602 (permutation building over tables with
schema metadata), which is merged.
- `ExtractFrames` takes a URI and opens the table lazily because a local
`LanceTable` cannot yet be pickled into DataLoader workers.
- The Permutation-based "Selecting columns" and "Using multiple
DataLoader workers" sections further down are unchanged from main.
## Test plan
- [x] Ran the quickstart, transform, `ExtractFrames` (main process and
forkserver workers) and distributed snippets locally.
- [x] Rendered the page with `mint dev` and checked for console errors.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds a user-facing realtime engine (RTE) page at
`docs/web/tables/realtime-engine.mdx`, listed under **Management** after
Branches. It is adapted from the internal WAL field guide
(lancedb/sophon#6213), with the deployment and operations material left
out.
The page covers:
- what RTE is and how it works (memtable → SSTables → compaction; reads
merge the base table with the fresh tier)
- getting started: `set_unenforced_primary_key`, indexes, `enable_rte` /
`ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial
updates, deletes, and `checkpoint_rte` / `disable_rte`
- performance: write throughput and shard scaling, and read latency with
and without RTE
- a note on operations not yet supported (time travel, branching,
`count_rows`, row IDs, `TABLESAMPLE`)
Also aligns related docs with current RTE naming and behavior:
- `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` /
`UNSET RTE`
- `maintenance/background-jobs.mdx`: job names are `memwal_compaction`
and `memwal_catchup_<type>`
- `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine
- Python `AsyncTable.enable_rte` /
`RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes
created after enabling RTE are maintained too
The examples are written directly in the page rather than generated from
`docs/web-tests`, because RTE needs an Enterprise server to run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: XY Zhan <zhaner08@hotmail.com>
## Problem
`permutation_builder(table).execute()` and `StreamingDataset(table,
...)` (which builds a permutation at construction) panic with
`RustPanic` on any table whose Arrow schema has schema-level metadata:
```python
schema = pa.schema([pa.field("x", pa.int64())], metadata={b"huggingface": b"{}"})
t = lancedb.connect(tmp).create_table("t", pa.table({"x": list(range(100))}, schema=schema))
permutation_builder(t).split_random(fixed=2, seed=1).execute() # RustPanic
```
Every Hugging Face Lance dataset triggers this, e.g.
`hf://datasets/lance-format/lerobot-xvla-soft-fold/data` table `frames`
carries `source-column-name-map`.
## Root cause
`PermutationBuilder::add_config_metadata` replaced the stream schema's
metadata with the permutation config and then called
`batch.with_schema(schema).unwrap()`. The base-table scan's batches
carry the base schema metadata, and `RecordBatch::with_schema` requires
the target to be a superset of the current schema, metadata included. It
returned `SchemaError("target schema is not superset of current schema
...")`, and the `unwrap` turned that into a panic on a tokio worker.
## Fix
- The permutation table deliberately does **not** inherit base-table
schema metadata. That metadata describes columns the permutation table
(`row_id`, `split_id`) does not have, and its keys can collide with
config keys such as `split_names`. A merge would let a base table's
`split_names` leak into, or override, the permutation's split names.
Batches are rebuilt with `RecordBatch::try_new(schema, columns)`.
- Errors are propagated (`map_ok` → `and_then`) instead of `unwrap`.
- The same panic-on-worker pattern in the splitter's streaming closures
(`split.rs`: `try_new`/`filter(...).unwrap()`) now propagates errors
too. Nothing else in `dataloader/` used `with_schema(...).unwrap()`.
## Tests
- Rust: `test_permutation_over_table_with_schema_metadata` in
`builder.rs` builds a random split + shuffled permutation over a table
with `huggingface` and a colliding `split_names` key. It panics with the
original `SchemaError` on the old code and passes now.
- Python: `test_permutation_over_table_with_schema_metadata` in
`test_permutation.py` covers the repro and reads both splits through
`Permutations`/`Permutation.from_tables`.
`test_streaming_dataset_over_table_with_schema_metadata` in
`test_elastic_dataloader.py` covers `StreamingDataset` construction and
iteration.
- `cargo fmt`, `cargo clippy --features remote --tests --examples`, the
dataloader Rust tests (42), `test_permutation.py` (72), and
`test_elastic_dataloader.py` all pass.
`test_torch.py::test_remote_permutation_dataloader_multiprocessing`
fails locally with `take query result did not include ordering column
'__lancedb_take_row_offset'`. It uses `Permutation.identity`, which does
not go through `PermutationBuilder`, so this change does not cause it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Document explicit Managed Blob v2 opt-in, descriptor queries and payload
fetching, Function inputs and outputs, and whole-object cleanup. Explain
null versus empty values, snapshot ownership, packed-object retention,
and the distinction from the legacy large-binary Blob API.
The local ingestion and Enterprise Function examples have been executed
end to end. The Function lifecycle description depends on
lance-format/lance#10176 and the corresponding Enterprise migration. The
existing Functions page remains outside the published navigation until
launch.
Validation: the 347-file assembled site has no broken links. Mintlify
4.2.888 build validation reports four existing
`RsExampleBody`/`RsExampleTail` import warnings in the reranking and
vector-search pages; the same diagnostics reproduce with the unchanged
base versions of all three pages edited here.
The existing docs have 254 findings from our shared Vale rules.
Contributors would otherwise inherit these when editing the same lines.
Clean up the prose across 80 pages: correct product names and a repeated
word, simplify a few phrases, and replace dashes with punctuation that
fits each sentence. The pinned Vale scan now reports zero findings, with
no suppressions.
Closes#4246.
lancedb publishes no source distribution, so a platform without a
prebuilt wheel (#4104, #4239) has nothing to install. This adds an sdist
to the Python release.
## The sdist job
The release workflow builds an sdist with the maturin version of the
wheel jobs. On a tag, the publish job uploads it with the wheels: to
PyPI, and to Fury for beta tags. The job also runs on the pull request
dry run, which now starts on a change to `python/pyproject.toml`.
## Checks
- The job builds a wheel from the sdist with `--locked` and smoke-tests
it in a fresh venv.
- `ci/check_sdist.py` fails if a tracked file of the Python package is
missing from the tarball.
- `twine check --strict` checks that the metadata renders. `pydistcheck`
enforces PyPI's 100 MiB size limit.
- The unit tests of both check scripts run in the CI scripts workflow.
## Cargo.lock
maturin ships the repository lockfile, which pins crates that only the
dropped `nodejs` member uses, so `cargo --locked` fails
(PyO3/maturin#2609). The job first runs `cargo metadata --locked` on the
repository workspace, so a lockfile that has drifted from `Cargo.toml`
fails there. It then removes the nodejs-only pins and repacks the sdist.
`ci/check_cargo_lock.py` fails if the prune changes any package it
keeps, or any dependency of one. The published file is the repacked one.
## Behavior change
After the first release with an sdist, pip on a platform with no current
wheel builds the newest version from source, which needs a Rust
toolchain and `protoc`. Before, pip installed the last version with a
wheel for that platform.
Preston ZhaoandPreston Zhao
adenzhou1350
Mohammad Hijjawi
Prince Jain
Yingzi Zhang
lancedb-gatefixer[bot]
SIDDARTHA REDDY
Joaquin Hui Gomez
陈志谦
Mohd Quamar TyagiandXuanwo
Madan Kumar
Vinayak Vathare
Zhewen TanandXuanwo
primorLee
Li Zinuo
Lance Release
Jack Ye
Colin Patrick McCabe
Yang Cen
IOCRYPOSTO
Lei XuandClaude Sonnet 5.5
LanceDB RobotandLuQQiu
Brendan Clement
Tobias