This PR converts the uri parameter to string before calling urlparse in
get_uri_scheme and get_uri_location, allowing pathlib.Path objects to be
passed seamlessly without raising a TypeError.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#4033
`create_table(data=...)` without an explicit schema raised `IndexError:
index out of bounds` when a `vector`/`embedding`-named list column was
all-null. `_modal_list_size` infers the dimension with
`pc.mode(pc.list_value_length(arr))[0]`, and pyarrow's `mode` skips
nulls, so an all-null column produces a zero-length result and the `[0]`
subscript blows up.
The sibling `_infer_vector_dim` already guards the identical
`pc.mode(...)` with a `len == 0` check and returns `None`;
`_modal_list_size` just didn't. This makes `_modal_list_size` return
`Optional[int]` and return `None` when every value is null, and the call
site then leaves the column as a variable-length list instead of
inventing a fixed size — the dimension is genuinely unknowable when
there are no non-null rows.
Added a regression test that creates a table from an all-null
`list<float32>` vector column and asserts it succeeds, the column stays
a `list<float32>`, and the rows are queryable. It fails on `main` with
the `IndexError` and passes with this change.
Retries individual inputs one at a time when the OpenAI API rejects a
batch with BadRequestError, instead of discarding the whole batch.
Valid inputs now still get embeddings; only the actually-bad input
resolves to None, and the log names the exact failing input.
Fixes#1677
## Summary
Sync `Table.search()` declares tuple, `pyarrow.Array`, and
`pyarrow.ChunkedArray` query vectors as supported, and the async query
path already accepts them. The sync query builder only classified lists
and NumPy arrays as vectors, however, so these inputs were routed
through embedding/FTS resolution and then rejected. Arrow and tuple
inputs also did not contribute a query dimension when selecting among
multiple vector columns.
This change:
- recognizes the supported tuple and Arrow containers as sync vector
queries for both `auto` and explicit `vector` query types;
- converts them to the existing internal `float32` NumPy representation
at the sync builder boundary;
- infers dimensions from tuple, primitive Arrow, and list-valued Arrow
queries so schemas with differently sized vector columns select the
matching column.
The regression test uses two vector columns with dimensions 2 and 3. It
verifies tuple, Arrow array, and chunked Arrow queries through both sync
query modes and confirms the dimension-2 column is selected.
## Test plan
- `pytest
python/tests/test_table.py::test_sync_search_accepts_supported_vector_containers
-q` (6 passed)
- `pytest python/tests/test_util.py python/tests/test_query.py
python/tests/test_table.py -q -m "not slow and not s3_test"` (308
passed, 1 skipped, 1 deselected)
- `pytest python/tests -q -m "not slow and not s3_test"
--ignore=python/tests/docs` (981 passed, 25 skipped, 65 deselected)
- `ruff format --check python`
- `ruff check python`
- `cargo fmt --all --check`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- treat an OAuth `expires_in` value of zero as an explicit immediate
expiration
- refresh zero-lifetime tokens instead of caching them indefinitely
- add a regression test covering two consecutive header requests
## Tests
- `pytest python/tests/test_header_provider.py -q` (16 passed)
- `ruff format --check python/lancedb/remote/header.py
python/tests/test_header_provider.py`
- `ruff check python/lancedb/remote/header.py
python/tests/test_header_provider.py`
## Summary
- move synchronous iterator preparation off LanceDB's singleton
background event loop
- keep the loop available when an input iterator executes a synchronous
table query
- cover query-backed iterators for both `Table.add` and `create_table`
## Root cause
The async implementations prepared input data on the singleton
background loop and peeked at the first item of iterators there. If that
iterator called a synchronous LanceDB query, `LOOP.run()` scheduled the
query on the same loop and blocked waiting for it, so neither operation
could progress.
## Validation
- `ruff format` on the changed Python files
- `ruff check .`
- `cd python && uv run --frozen --extra tests pytest
python/tests/test_table.py -q` (138 passed)
- the broader non-slow suite was attempted but hit a native segmentation
fault in the external DataFusion `register_table_provider` path; an
isolated control using the pre-change inline preparation behavior
reproduced the same unrelated crash
Fixes#2107
<!-- lance-gatekeeper-fix:v1 agent=8b4aaf556014ac0e457c9d63495697e4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a Python regression for filtered scans over deeply nested
list/struct values
- verify the public query path returns every nested sibling field
without an Arrow schema mismatch
## Root cause
Older Lance scanner code split filter projections into partial nested
arrays and did not recursively merge List<Struct> children, so a sibling
field could be omitted from the returned array while remaining in the
declared schema. That merge was fixed upstream in
lance-format/lance#4838, and the Lance revision currently pinned by
LanceDB contains that fix. This test makes the repair durable at the
LanceDB Python API boundary.
## Validation
- cd python && uv run --extra tests pytest
python/tests/test_query.py::test_filter_preserves_deeply_nested_struct_fields
-q
- cd python && uv run --extra tests pytest python/tests/test_query.py -q
- uv run --project python --extra tests --extra dev ruff check .
- uv run --project python --extra tests --extra dev ruff format --check
python/python/tests/test_query.py
Fixes#2217
<!-- lance-gatekeeper-fix:v1 agent=9ddc0efd1f13888b2a7a896881c6e68a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add regression coverage for an IVF_RQ index created through the Lance
Python API and consumed through LanceDB
- verify a fresh LanceDB handle can list and query the
externally-created index
## Root cause
Lance 4.0.0-beta.5 could serialize `rotate_mat_position` as null, while
the Lance 2.0.0 reader used by LanceDB 0.29.2 required a `u32`. Upstream
Lance fixed the incompatible IVF_RQ metadata handling and index
versioning, and the current LanceDB dependency includes that repair.
LanceDB was missing coverage for the reported mixed-API workflow,
allowing the compatibility regression to go undetected.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_index.py
-q` (24 passed)
- `uv run --project python --extra dev ruff format .`
- `uv run --project python --extra dev ruff check .`
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_index.py`
- `uv run --project python --extra dev ruff check
python/python/tests/test_index.py`
Fixes#3092
<!-- lance-gatekeeper-fix:v1 agent=7d8c849bd8421be65d32355c42a60b8a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Expose optional `auto_refresh_enabled` and
`auto_refresh_interval_seconds` when creating materialized views from
Python and Rust. Omitted intervals remain unset: the server determines
when to refresh.
```python
view = db.create_materialized_view(
"recent_orders", "orders",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
## Architecture and workflow
- Python synchronous, asynchronous, job-returning and namespace wrappers
forward both optional values through PyO3 to the Rust creation builder.
Both default to `None`; no client interval is selected.
- Remote creation calls `POST /v1/materialized_view/{id}/create` with
the defining query and only the supplied refresh properties. A `202
Accepted` response must contain a job ID. Existing job tracking waits
for the server's ordinary SQL CREATE job before synchronous creation
opens the view.
- The server validates and authorizes the CREATE, then uses the existing
SQL worker to populate the view and persist its refresh settings. The
Lance agent subsequently uses those settings to submit REFRESH SQL jobs.
- Explicit positive finite fractional seconds are preserved. An interval
alone does not enable refresh. Embedded connections retain manual
refresh, accept explicit disabled state, and reject requests for
automatic scheduling rather than ignore them.
Includes API documentation and coverage for omission, false,
fractional/invalid intervals, namespace forwarding, server job IDs and
invalid acceptance responses.
Requires the server REST-create endpoint to be deployed before SDK
release. Existing remote `with_no_data=True` remains unsupported.
## Context
`StreamingDataset`'s shipped split strategies (`split_random`,
`split_sequential`, `split_hash`, `split_calculated`) all share one
trait: every row in the base table is assigned to exactly one split,
exactly once. They only differ in *how* that one-to-one assignment is
chosen -- none of them can express "give me a resampled stream of N
rows, allowing repeats."
That's a distinct, common need in ML training: **the bootstrap** -- the
statistical technique (Efron, 1979) of estimating a dataset's properties
by resampling it *with replacement*. It's the same sense scikit-learn's
bagging classifiers mean by `bootstrap=True`. `SplitStrategy::Bootstrap`
adds exactly this to the data loader: sample rows uniformly with
replacement from the (optionally filtered) base table, independently per
split.
## Main use cases
- **Bagging / ensemble training** (e.g. random forests): each split
trains on an independent bootstrap resample of the training set.
- **Oversampling a rare class**: filter to the minority class, then
sample it up to whatever count balances it against the majority, instead
of exhausting its small set once per epoch.
- **Fixed-size training steps independent of dataset size**: want
exactly N batches per epoch regardless of whether the table has 50 rows
or 50 million? Set `num_samples` directly instead of it being implicitly
`len(dataset)`.
- **Weighted sampling without a weight column**: filter to "eligible"
rows (e.g. a windowed-sampling workload filtering to valid window-start
positions), then bootstrap the filtered set -- an item with more
eligible rows is naturally drawn proportionally more often, with no
explicit per-row weight needed.
## Example (an actual run, not a hypothetical one)
A 5-row base table (`row_id` 0..4), sampled via `Bootstrap {
num_samples: 8, sizes: Fixed(2), seed: Some(42) }`:
```
row_id split_id
0 0
2 0
4 0
4 0 <- row 4 repeated within split 0
4 1 <- row 4 also drawn in split 1 (independent draw)
4 1
1 1
2 1
```
8 output rows from a 5-row table -- not 5. Row 4 drawn 4 times total,
row 3 never drawn. That's with-replacement sampling actually behaving
like with-replacement sampling, not an even round-robin dressed up with
a new name.
## What changed
Adds `SplitStrategy::Bootstrap` (uniform sampling with replacement,
optionally filtered, independently seeded per split -- a given split's
content doesn't depend on how many other splits there are, the same
elastic-determinism property documented for the loader's other
strategies). No new `PermutationBuilder` Rust method was needed:
`with_split_strategy` already accepts any `SplitStrategy` value
generically. Python binding:
`PermutationBuilder.bootstrap_sample(num_samples, *, ratios=None,
counts=None, fixed=None, seed=None, split_names=None)`, mirroring
`split_random`'s existing shape.
Also adds a generic `split_strategy` parameter to `StreamingDataset`'s
constructor (a callable taking/returning a `PermutationBuilder`), since
it previously could only reach `split_random`/`split_sequential` even
though `split_hash`/`split_calculated` already shipped. This unblocks
`Bootstrap` and any future strategy without further constructor changes.
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
Cargo.lock: webpki-root-certs 1.0.10; webpki-roots 1.0.10 and
rustls-webpki 0.103.16 are already current.
docs/web-tests/rs/Cargo.lock: rustls-webpki 0.103.16. python/uv.lock:
certifi 2026.10.19, the only entry that changes.
The root crate drops the root CA whose intermediates were compromised
(Mozilla bug 2083449), and rustls-webpki 0.103.16 rejects chains that
carry the compromised eMudhra intermediate keys. certifi 2026.10.19 is
the release the certifi advisory names as fixed: it drops the same root
and keeps the issuer's other roots, so HTTPS connections that chain to
them keep working. Relative to the previous 2026.6.17 pin it also adds
Telia EC and RSA TLS Root CA v3 and SECOM TLS ECC and RSA Root CA 2024
and drops Atos TrustedRoot 2011.
The webpki-root-certs entry is written by hand with the crates.io
checksum.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Esteban Gutierrez <esteban@lancedb.com>
Co-authored-by: Claude Opus 5.1 <noreply@anthropic.com>
Adds a user-facing realtime engine (RTE) page at
`docs/web/tables/realtime-engine.mdx`, listed under **Management** after
Branches. It is adapted from the internal WAL field guide
(lancedb/sophon#6213), with the deployment and operations material left
out.
The page covers:
- what RTE is and how it works (memtable → SSTables → compaction; reads
merge the base table with the fresh tier)
- getting started: `set_unenforced_primary_key`, indexes, `enable_rte` /
`ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial
updates, deletes, and `checkpoint_rte` / `disable_rte`
- performance: write throughput and shard scaling, and read latency with
and without RTE
- a note on operations not yet supported (time travel, branching,
`count_rows`, row IDs, `TABLESAMPLE`)
Also aligns related docs with current RTE naming and behavior:
- `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` /
`UNSET RTE`
- `maintenance/background-jobs.mdx`: job names are `memwal_compaction`
and `memwal_catchup_<type>`
- `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine
- Python `AsyncTable.enable_rte` /
`RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes
created after enabling RTE are maintained too
The examples are written directly in the page rather than generated from
`docs/web-tests`, because RTE needs an Enterprise server to run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: XY Zhan <zhaner08@hotmail.com>
## Problem
`permutation_builder(table).execute()` and `StreamingDataset(table,
...)` (which builds a permutation at construction) panic with
`RustPanic` on any table whose Arrow schema has schema-level metadata:
```python
schema = pa.schema([pa.field("x", pa.int64())], metadata={b"huggingface": b"{}"})
t = lancedb.connect(tmp).create_table("t", pa.table({"x": list(range(100))}, schema=schema))
permutation_builder(t).split_random(fixed=2, seed=1).execute() # RustPanic
```
Every Hugging Face Lance dataset triggers this, e.g.
`hf://datasets/lance-format/lerobot-xvla-soft-fold/data` table `frames`
carries `source-column-name-map`.
## Root cause
`PermutationBuilder::add_config_metadata` replaced the stream schema's
metadata with the permutation config and then called
`batch.with_schema(schema).unwrap()`. The base-table scan's batches
carry the base schema metadata, and `RecordBatch::with_schema` requires
the target to be a superset of the current schema, metadata included. It
returned `SchemaError("target schema is not superset of current schema
...")`, and the `unwrap` turned that into a panic on a tokio worker.
## Fix
- The permutation table deliberately does **not** inherit base-table
schema metadata. That metadata describes columns the permutation table
(`row_id`, `split_id`) does not have, and its keys can collide with
config keys such as `split_names`. A merge would let a base table's
`split_names` leak into, or override, the permutation's split names.
Batches are rebuilt with `RecordBatch::try_new(schema, columns)`.
- Errors are propagated (`map_ok` → `and_then`) instead of `unwrap`.
- The same panic-on-worker pattern in the splitter's streaming closures
(`split.rs`: `try_new`/`filter(...).unwrap()`) now propagates errors
too. Nothing else in `dataloader/` used `with_schema(...).unwrap()`.
## Tests
- Rust: `test_permutation_over_table_with_schema_metadata` in
`builder.rs` builds a random split + shuffled permutation over a table
with `huggingface` and a colliding `split_names` key. It panics with the
original `SchemaError` on the old code and passes now.
- Python: `test_permutation_over_table_with_schema_metadata` in
`test_permutation.py` covers the repro and reads both splits through
`Permutations`/`Permutation.from_tables`.
`test_streaming_dataset_over_table_with_schema_metadata` in
`test_elastic_dataloader.py` covers `StreamingDataset` construction and
iteration.
- `cargo fmt`, `cargo clippy --features remote --tests --examples`, the
dataloader Rust tests (42), `test_permutation.py` (72), and
`test_elastic_dataloader.py` all pass.
`test_torch.py::test_remote_permutation_dataloader_multiprocessing`
fails locally with `take query result did not include ordering column
'__lancedb_take_row_offset'`. It uses `Permutation.identity`, which does
not go through `PermutationBuilder`, so this change does not cause it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Closes#4246.
lancedb publishes no source distribution, so a platform without a
prebuilt wheel (#4104, #4239) has nothing to install. This adds an sdist
to the Python release.
## The sdist job
The release workflow builds an sdist with the maturin version of the
wheel jobs. On a tag, the publish job uploads it with the wheels: to
PyPI, and to Fury for beta tags. The job also runs on the pull request
dry run, which now starts on a change to `python/pyproject.toml`.
## Checks
- The job builds a wheel from the sdist with `--locked` and smoke-tests
it in a fresh venv.
- `ci/check_sdist.py` fails if a tracked file of the Python package is
missing from the tarball.
- `twine check --strict` checks that the metadata renders. `pydistcheck`
enforces PyPI's 100 MiB size limit.
- The unit tests of both check scripts run in the CI scripts workflow.
## Cargo.lock
maturin ships the repository lockfile, which pins crates that only the
dropped `nodejs` member uses, so `cargo --locked` fails
(PyO3/maturin#2609). The job first runs `cargo metadata --locked` on the
repository workspace, so a lockfile that has drifted from `Cargo.toml`
fails there. It then removes the nodejs-only pins and repacks the sdist.
`ci/check_cargo_lock.py` fails if the prune changes any package it
keeps, or any dependency of one. The published file is the repacked one.
## Behavior change
After the first release with an sdist, pip on a platform with no current
wheel builds the newest version from source, which needs a Rust
toolchain and `protoc`. Before, pip installed the last version with a
wheel for that platform.
BREAKING CHANGE: `SentenceTransformerEmbeddings.trust_remote_code` now
defaults to `False` (was `True`). When an embedding function is restored
from table metadata, `trust_remote_code=True` and custom endpoints
(OpenAI `base_url`, Ollama `host`/proxy, watsonx `url`) are only honored
if the process approved them; otherwise computing embeddings raises
`UntrustedEmbeddingConfigError`. Functions created in the same process
are approved automatically. To reopen such a table in a new process,
call `get_registry().allow_remote_code(model, revision=...)` or
`get_registry().allow_endpoint(url)` first.
## Summary
Embedding functions are rebuilt from table metadata, which anyone who
can write the table controls. Since sentence-transformers defaulted
`trust_remote_code` to `True`, opening and querying a crafted table ran
arbitrary code from a model repository (GHSA-x8v2-vj73-gfwm). Metadata
could likewise point `base_url` at another server, sending it queries
and the `OPENAI_API_KEY` from the environment.
The table can no longer grant itself trust:
```python
registry = get_registry()
registry.allow_remote_code("nomic-ai/nomic-embed-text-v1.5", revision="<commit>")
registry.allow_endpoint("http://my-ollama-host:11434")
```
Trust is keyed by model name (optionally pinned to a revision), so
approving one model doesn't let a table swap in another. Restored
functions that aren't approved are replaced by a placeholder that raises
on use without constructing the real function (`huggingface` loads its
model in `__init__`), so non-embedding operations on the table keep
working. `sentence-transformers` and `huggingface` gain a `revision`
parameter. Custom embedding functions can opt in by overriding the new
`remote_code_model()` / `endpoints()` hooks.
Not covered: TypeScript (its built-ins have neither remote code nor
configurable endpoints), Bedrock `assumed_role`, and third-party
functions that don't implement the hooks.
## Evidence
`test_embeddings_trust.py` covers restore with and without approval,
revision pinning, swapped model names, endpoint gating, and that the
table stays usable. Opening a table with crafted metadata and searching
it now gives:
```
UntrustedEmbeddingConfigError: Embedding function 'sentence-transformers' was restored from table metadata with settings this process has not approved:
- runs code from model 'attacker/model' at revision None. If you trust it, call get_registry().allow_remote_code('attacker/model', revision=None)
```
## Merge Risk
**Door:** One-way: changes a default and makes previously working
reopen-and-search flows require explicit approval.
**Blast Radius:** public-api (Python embeddings); users of remote-code
models or custom endpoints who reopen tables in a new process.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
- `StreamingDataset`'s Stage 1 (I/O) queue (`raw_batches`) had no cap:
`_fill_io` only limited *in-flight* fetch futures (`io_pending`) to
`io_queue_depth`, but every completed fetch was unconditionally drained
into `raw_batches` and the freed slot immediately refilled, so I/O could
race arbitrarily far ahead of a slow transform stage with no bound on
buffered memory.
- `transform_queue_depth`'s admission check (`_try_submit_tx`) couldn't
compensate either, since it only counts `cooked` rows and in-flight
transform futures — it never looks at `raw_batches`.
- Fix: `_fill_io` now counts both `io_pending` and `raw_batches` against
the `io_queue_depth` budget, matching the parameter's documented
"batches to keep in flight per split" contract.
## Test plan
- [x] Added
`test_raw_queue_depth_bounded_when_transform_is_the_bottleneck`
(deterministic, via synchronous I/O) asserting `raw_queue_depth <=
io_queue_depth * read_batch_size` even when the transform stage is the
bottleneck.
- [x] Full `python/python/tests/test_elastic_dataloader.py` suite passes
(190/190), confirming no change to sample ordering, counts, or
checkpoint/resumability behavior.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
udf() rejected a large_binary input or output, and large_binary nested in a list or struct, outside a Blob v2 payload. It is now a grammar primitive like large_utf8, and the shared type golden lists it.
## Why
Every MemWAL write and read resolves rows by the table's unenforced
primary key, but `enable_memwal` accepted a table without one. The spec
installed, and every MemWAL `merge_insert` then failed with "table has a
MemWAL index but no unenforced primary key" until MemWAL was disabled.
Lance's MemWAL initialization doesn't catch it: its key checks pass when
there is no key.
## What changes
`enable_memwal` refuses a spec on a table with no unenforced primary
key, for every sharding mode, before anything is committed. The check
runs against the dataset the spec is committed to; a key cannot be
removed once set, and an overwrite or restore of the table conflicts
with that commit. Servers that install specs through this method get the
check too.
The Rust doc on `Table::enable_memwal` now states the requirement; the
Python and TypeScript docs already did.
## Tests
- New `test_enable_memwal_requires_primary_key` (Rust and Python):
unsharded, bucket and identity specs are all refused without a key, a
refused spec leaves nothing installed, and the same table accepts a spec
once a key is set. The Rust test fails with the check disabled.
- Tests that installed a spec on a table without a key now declare one
first (6 Rust, 3 Python).
- `cargo nextest run -p lancedb --features remote --lib`: 1529 passed.
`cargo clippy -p lancedb --features remote --tests --examples -- -D
warnings` and `cargo fmt --check` are clean; no integration test or
example installs a spec.
- Python: `test_memwal_spec.py` and `test_merge_insert_lsm.py`, 43
passed; `ruff format` and `ruff check` are clean.
## What
Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.
### 1. MemWAL APIs drop "LSM"
The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.
| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |
TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.
Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.
### 2. Stats use SSTable and table shard
Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.
**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.
**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.
| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |
Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.
`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.
## Compatibility
Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:
- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies
## Validation
- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
`RemoteTable::set_unenforced_primary_key` returned `NotSupported`, so
the call failed against LanceDB Cloud and enterprise from every SDK —
Python and TypeScript both forward to it. That also blocked sharded LSM
writes, since the server rejects bucket/identity sharding on a table
that declares no unenforced primary key, and the SDK had no way to
declare one.
No server-side support was missing. The unenforced primary key is Lance
schema field metadata, and the existing `update_field_metadata` endpoint
writes exactly that, so the remote table now installs the key through
it. The commit layer behind that endpoint installs the position,
enforces immutability and runs `verify_primary_key()` — the same code a
native table reaches, so both paths agree on semantics and not just on
error messages.
## Changes
- `rust/lancedb/src/table/primary_key.rs`: the request validation
(single column, column exists, supported dtype, key not already set) and
the metadata edit (set the 1-indexed position, clear the legacy boolean
key) move into `validate()` and `install_edit()`, shared by both table
types so they cannot drift. Native behaviour is unchanged.
- `rust/lancedb/src/remote/table.rs`: real implementation —
`check_mutable()`, validate against the described schema, then one
`update_field_metadata` request.
- `python/python/lancedb/remote/table.py`: drop the "Not supported on
LanceDB Cloud." docstring.
No binding changes were needed; Python and TypeScript already forward to
the Rust `BaseTable` method.
## Tests
Three mocked-server tests in `rust/lancedb/src/remote/table.rs`:
- the happy path, asserting the request body carries
`lance-schema:unenforced-primary-key:position: "1"` and a null legacy
key on the right field path;
- the invalid requests a native table also rejects (empty, compound,
unknown column, unsupported dtype), each failing before any write
reaches the server;
- the already-set case, rejected from the schema the server already
reports.
`cargo test --features remote -p lancedb --lib`: 1186 passed, 0 failed.
`cargo clippy --features remote -p lancedb --lib --tests` is clean.
## Note for enterprise
Sophon pins the lancedb submodule at `v0.39.0-beta.6`, so this reaches
enterprise on the next bump.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011xV8EebEZDmDDV8hVBm6MD
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
\`inf_vector_column_query\`'s \`dim\` filter (added in #3423) is meant
to disambiguate between multiple vector columns of different sizes. When
a schema has exactly one vector column and the query's dimension doesn't
match it, the filter excluded that column too, raising "There is no
vector column in the data" instead of resolving the column and letting
the dimension mismatch surface at query execution.
This was latent until #4409 wired client-side vector-column inference
into remote search: a mismatched-dimension query against a
single-vector-column remote table now fails client-side with the wrong
error, before ever reaching the server's own dimension validation.
Fall back to the unfiltered candidate set when the dimension filter
leaves zero matches, and only raise when that is still ambiguous or
genuinely empty.
Add an authorization API to the LanceDB catalog. This is an API that can
be used to bind access control lists (ACLs) to individual catalog
objects. It can also be used to manage server-side roles, and introspect
principals and groups. The new API is implemented in both the Rust,
Python, and async Python styles and includes list_acls, add_acl,
delete_acl, list_role_bindings, add_role_binding, delete_role_binding,
get_principal, and get_group. This API implements the design principle
of forwards-compatibility. So unknown privilege, object, and subject
types do not cause errors, but are be supported as best as possible in
the library.
Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.
The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.
Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.
BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.
Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.
Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):
- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.
All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.
Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
Python `Decimal` values and Arrow decimal scalars raise
`NotImplementedError` in `table.update(values=...)`. Register a Decimal
converter that emits quoted fixed-point text so the native cast to the
target column preserves every digit, including decimal128 and decimal256
values. Non-finite Decimal values raise `ValueError` with a static
message.
Closes#4360.
Validation with the native extension built from the current source at
7f593359, on macOS arm64 and Python 3.12.14:
- All 33 new regression cases failed before the fix and passed
afterward. They cover Python and Arrow scalar updates, signs, zero,
exponents, full 38/76-digit precision, and a Decimal context precision
of 6.
- 16 existing converter, update, and Decimal expression tests passed.
- Repository Ruff formatting and lint passed; formatting left all 156
files unchanged.
- A native round-trip check confirmed that an unquoted SQL number rounds
a high-precision value while the quoted Decimal update preserves it
exactly.
The full suite, other Python versions, and remote API were not
exercised.
AI assistance: Codex helped implement and validate the Decimal
conversion and its regression tests.
Mohd Quamar TyagiandXuanwo
Madan Kumar
Vinayak Vathare
Zhewen TanandXuanwo
primorLee
lancedb-gatefixer[bot]
Lance Release
Jack Ye
Lei XuandClaude Opus 5.5
Tobias
Will JonesandClaude Opus 5.5
Weston PaceandClaude Sonnet 5
Wyatt Alt
Drew
Colin Patrick McCabe
SIDDARTHA REDDY
Yang Cen
Kevin Dong