## Summary
Fixes#4142: `lancedb` failed to compile without the `remote` feature.
`TerminalResult::decode` in `rust/lancedb/src/job.rs` constructed the
`Error::Http` variant unconditionally, but that variant is declared
behind
`#[cfg(feature = "remote")]` in `rust/lancedb/src/error.rs`. Since the
crate's default feature set is empty, a plain `lancedb = "0.38"`
dependency
failed with:
```
error[E0599]: no variant named `Http` found for enum `error::Error`
--> rust/lancedb/src/job.rs:102:40
```
The module itself is not remote-only (it carries local job support), so
instead of gating `pub mod job`, both `Error::Http` constructions now go
through a small `remote_job_error` helper that resolves to `Error::Http`
with the `remote` feature and falls back to `Error::Runtime` without it.
A `request_id` is only ever `Some` for remote jobs, so the fallback is
unreachable in practice without the feature.
## Testing
- Reproduced the reported failure on current `main`:
`cargo check -p lancedb --no-default-features` failed with the exact
E0599 errors from the issue before the change, and passes after it.
- `cargo check -p lancedb --features remote` passes with no errors and
no
new warnings.
- Added two regression tests in `rust/lancedb/src/job.rs`
(`decode_missing_result_uses_feature_appropriate_error`,
`decode_unparsable_result_uses_feature_appropriate_error`) asserting the
error variant matches the feature configuration. The test code compiles
under the fixed configuration (`cargo check --tests` passes), and the
branching logic was executed standalone against both feature cfgs
(3/3 green with and without `remote`). The full `cargo test` binary link
could not complete on the small build VM available here (the linker
stalled), so CI coverage of the new tests is appreciated.
- `cargo fmt --all -- --check` is clean.
## Disclosure
AI assistance was used while preparing this change (code search,
editing,
and verification commands were agent-driven); the diagnosis, fix design,
and all test results above were verified by actually running the builds.
`table_names()` enumerates with `read_dir`, which returns files as well
as directories, filtered only on the `.lance` extension — so a loose
`foo.lance` *file* at the database prefix is reported as a table.
`list_tables()` already applies the correct rule ("Only child
directories can be tables"), and
`test_listing_ignores_non_table_children` documents it, so the two
listing APIs currently disagree.
The practical damage is in `drop_all_tables()`, which is built on
`table_names`: with a loose `.lance` file present it deletes some real
tables, fails on the file with `os error 20 (NotADirectory)`, and then
fails the same way on every retry — the database can never be emptied
through the API, and the partial drop is not recoverable.
`table_names` now lists via `list_with_delimiter` and filters
`common_prefixes` through the same `table_name` helper `list_tables`
uses; the sort/`start_after`/`limit` handling is unchanged.
`drop_tables` additionally attempts every name before returning, still
reporting the first error unchanged (kept rather than an aggregate error
so the mapped exception types in the bindings stay exactly as they are).
Extended the non-table-children test to cover `table_names`, plus tests
for the drop behavior — all three fail without the fix. Full `cargo test
-p lancedb --lib --features remote` 1320 passed on the repo-pinned 1.97
toolchain; clippy/fmt clean.
`LanceHybridQueryBuilder._validate_query`'s ValueError concatenated two
adjacent string literals without a separator, rendering "...explicitly
for hybrid search.**But** not both.". Added the trailing space.
String-only change.
`VoyageAIReranker`'s `api_key` docstring said it falls back to
`OPENAI_API_KEY`, but the class actually reads `VOYAGE_API_KEY` (and
raises a "VOYAGE_API_KEY not set" error without it) — the docstring was
copy-pasted from the OpenAI reranker. Docstring-only fix.
This PR converts the uri parameter to string before calling urlparse in
get_uri_scheme and get_uri_location, allowing pathlib.Path objects to be
passed seamlessly without raising a TypeError.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Fixes#4033
`create_table(data=...)` without an explicit schema raised `IndexError:
index out of bounds` when a `vector`/`embedding`-named list column was
all-null. `_modal_list_size` infers the dimension with
`pc.mode(pc.list_value_length(arr))[0]`, and pyarrow's `mode` skips
nulls, so an all-null column produces a zero-length result and the `[0]`
subscript blows up.
The sibling `_infer_vector_dim` already guards the identical
`pc.mode(...)` with a `len == 0` check and returns `None`;
`_modal_list_size` just didn't. This makes `_modal_list_size` return
`Optional[int]` and return `None` when every value is null, and the call
site then leaves the column as a variable-length list instead of
inventing a fixed size — the dimension is genuinely unknowable when
there are no non-null rows.
Added a regression test that creates a table from an all-null
`list<float32>` vector column and asserts it succeeds, the column stays
a `list<float32>`, and the rows are queryable. It fails on `main` with
the `IndexError` and passes with this change.
Retries individual inputs one at a time when the OpenAI API rejects a
batch with BadRequestError, instead of discarding the whole batch.
Valid inputs now still get embeddings; only the actually-bad input
resolves to None, and the log names the exact failing input.
Fixes#1677
## Summary
Sync `Table.search()` declares tuple, `pyarrow.Array`, and
`pyarrow.ChunkedArray` query vectors as supported, and the async query
path already accepts them. The sync query builder only classified lists
and NumPy arrays as vectors, however, so these inputs were routed
through embedding/FTS resolution and then rejected. Arrow and tuple
inputs also did not contribute a query dimension when selecting among
multiple vector columns.
This change:
- recognizes the supported tuple and Arrow containers as sync vector
queries for both `auto` and explicit `vector` query types;
- converts them to the existing internal `float32` NumPy representation
at the sync builder boundary;
- infers dimensions from tuple, primitive Arrow, and list-valued Arrow
queries so schemas with differently sized vector columns select the
matching column.
The regression test uses two vector columns with dimensions 2 and 3. It
verifies tuple, Arrow array, and chunked Arrow queries through both sync
query modes and confirms the dimension-2 column is selected.
## Test plan
- `pytest
python/tests/test_table.py::test_sync_search_accepts_supported_vector_containers
-q` (6 passed)
- `pytest python/tests/test_util.py python/tests/test_query.py
python/tests/test_table.py -q -m "not slow and not s3_test"` (308
passed, 1 skipped, 1 deselected)
- `pytest python/tests -q -m "not slow and not s3_test"
--ignore=python/tests/docs` (981 passed, 25 skipped, 65 deselected)
- `ruff format --check python`
- `ruff check python`
- `cargo fmt --all --check`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- treat an OAuth `expires_in` value of zero as an explicit immediate
expiration
- refresh zero-lifetime tokens instead of caching them indefinitely
- add a regression test covering two consecutive header requests
## Tests
- `pytest python/tests/test_header_provider.py -q` (16 passed)
- `ruff format --check python/lancedb/remote/header.py
python/tests/test_header_provider.py`
- `ruff check python/lancedb/remote/header.py
python/tests/test_header_provider.py`
## What
Preserve unbounded plain-query semantics in the Rust Namespace
`QueryTable` adapter.
Plain queries without an explicit limit are now encoded with `i32::MAX`.
Explicit limits remain unchanged, while vector and FTS queries continue
to use the default top-k of 10.
## Root cause
The Rust Namespace adapter converted a missing limit using:
```rust
q.limit.unwrap_or(10)
```
This turned an unbounded plain query into an explicit `k=10`, silently
truncating the results returned by Namespace pushdown.
## Tests
Added regression coverage for unbounded and explicitly limited plain
queries, while verifying that vector and FTS queries retain their
default limit.
```shell
cargo fmt --all -- --check
cargo test --quiet --features remote -p lancedb \
test_convert_to_namespace_query --lib
```
close#3926.
## Summary
- move synchronous iterator preparation off LanceDB's singleton
background event loop
- keep the loop available when an input iterator executes a synchronous
table query
- cover query-backed iterators for both `Table.add` and `create_table`
## Root cause
The async implementations prepared input data on the singleton
background loop and peeked at the first item of iterators there. If that
iterator called a synchronous LanceDB query, `LOOP.run()` scheduled the
query on the same loop and blocked waiting for it, so neither operation
could progress.
## Validation
- `ruff format` on the changed Python files
- `ruff check .`
- `cd python && uv run --frozen --extra tests pytest
python/tests/test_table.py -q` (138 passed)
- the broader non-slow suite was attempted but hit a native segmentation
fault in the external DataFusion `register_table_provider` path; an
isolated control using the pre-change inline preparation behavior
reproduced the same unrelated crash
Fixes#2107
<!-- lance-gatekeeper-fix:v1 agent=8b4aaf556014ac0e457c9d63495697e4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a Python regression for filtered scans over deeply nested
list/struct values
- verify the public query path returns every nested sibling field
without an Arrow schema mismatch
## Root cause
Older Lance scanner code split filter projections into partial nested
arrays and did not recursively merge List<Struct> children, so a sibling
field could be omitted from the returned array while remaining in the
declared schema. That merge was fixed upstream in
lance-format/lance#4838, and the Lance revision currently pinned by
LanceDB contains that fix. This test makes the repair durable at the
LanceDB Python API boundary.
## Validation
- cd python && uv run --extra tests pytest
python/tests/test_query.py::test_filter_preserves_deeply_nested_struct_fields
-q
- cd python && uv run --extra tests pytest python/tests/test_query.py -q
- uv run --project python --extra tests --extra dev ruff check .
- uv run --project python --extra tests --extra dev ruff format --check
python/python/tests/test_query.py
Fixes#2217
<!-- lance-gatekeeper-fix:v1 agent=9ddc0efd1f13888b2a7a896881c6e68a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add regression coverage for an IVF_RQ index created through the Lance
Python API and consumed through LanceDB
- verify a fresh LanceDB handle can list and query the
externally-created index
## Root cause
Lance 4.0.0-beta.5 could serialize `rotate_mat_position` as null, while
the Lance 2.0.0 reader used by LanceDB 0.29.2 required a `u32`. Upstream
Lance fixed the incompatible IVF_RQ metadata handling and index
versioning, and the current LanceDB dependency includes that repair.
LanceDB was missing coverage for the reported mixed-API workflow,
allowing the compatibility regression to go undetected.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_index.py
-q` (24 passed)
- `uv run --project python --extra dev ruff format .`
- `uv run --project python --extra dev ruff check .`
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_index.py`
- `uv run --project python --extra dev ruff check
python/python/tests/test_index.py`
Fixes#3092
<!-- lance-gatekeeper-fix:v1 agent=7d8c849bd8421be65d32355c42a60b8a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- inspect the assembled x64 musl addon for the unresolved AVX-512
distance-table symbol before npm publication
- fail the publish lifecycle with an actionable error if the broken
symbol is present
- add regression coverage using the dynamic-symbol output from the
affected 0.27.x artifact
## Root cause
LanceDB 0.27.x could enable the Rust AVX-512 distance-table callsite
when the separate C kernel build did not emit its object, leaving
`sum_4bit_dist_table_32bytes_batch_avx512` undefined in the musl addon.
The current Lance dependency includes the upstream split-kernel cfg fix,
and the published 0.33.0 musl artifact no longer has the undefined
symbol, but the LanceDB publish path did not verify that property before
releasing an addon.
This adds a release guard around the corrected implementation so the
same invalid binary cannot be published again.
## Validation
- `pnpm lint`
- `pnpm run build`
- `pnpm test --runInBand` (680 passed, 5 skipped)
- `pnpm run docs`
- checked published artifacts: 0.27.1 is rejected for the unresolved
symbol; 0.33.0 is accepted
Fixes#3200
<!-- lance-gatekeeper-fix:v1 agent=4094b9b6de4b43665a9ff50fa35322fb
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Native `list_indices()` drops registered scalar plugin types absent from
LanceDB's fixed enum, and `index_stats()` reports them as `Unknown`.
Preserve their registered display names through
`IndexType::Custom(String)`.
Resolve these names through Lance's shared `IndexDetails` registry entry
point. Registered types such as MinHashLSH become visible; system and
unregistered types remain excluded. Existing builtin aliases retain
their canonical names. Rust consumers with exhaustive `IndexType`
matches need to handle the added variant.
Manual validation with [Lance
#10224](https://github.com/lance-format/lance/pull/10224) at
`fcc070f88a`: a downstream statistics-only plugin appears as
`Custom("Sketch")` in native listing and statistics. Its `load_index`
deliberately returns an error, and repeated calls do not construct
additional plugin instances, verifying metadata-only lookup through the
shared registry.
Expose optional `auto_refresh_enabled` and
`auto_refresh_interval_seconds` when creating materialized views from
Python and Rust. Omitted intervals remain unset: the server determines
when to refresh.
```python
view = db.create_materialized_view(
"recent_orders", "orders",
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
## Architecture and workflow
- Python synchronous, asynchronous, job-returning and namespace wrappers
forward both optional values through PyO3 to the Rust creation builder.
Both default to `None`; no client interval is selected.
- Remote creation calls `POST /v1/materialized_view/{id}/create` with
the defining query and only the supplied refresh properties. A `202
Accepted` response must contain a job ID. Existing job tracking waits
for the server's ordinary SQL CREATE job before synchronous creation
opens the view.
- The server validates and authorizes the CREATE, then uses the existing
SQL worker to populate the view and persist its refresh settings. The
Lance agent subsequently uses those settings to submit REFRESH SQL jobs.
- Explicit positive finite fractional seconds are preserved. An interval
alone does not enable refresh. Embedded connections retain manual
refresh, accept explicit disabled state, and reject requests for
automatic scheduling rather than ignore them.
Includes API documentation and coverage for omission, false,
fractional/invalid intervals, namespace forwarding, server job IDs and
invalid acceptance responses.
Requires the server REST-create endpoint to be deployed before SDK
release. Existing remote `with_no_data=True` remains unsupported.
## Context
`StreamingDataset`'s shipped split strategies (`split_random`,
`split_sequential`, `split_hash`, `split_calculated`) all share one
trait: every row in the base table is assigned to exactly one split,
exactly once. They only differ in *how* that one-to-one assignment is
chosen -- none of them can express "give me a resampled stream of N
rows, allowing repeats."
That's a distinct, common need in ML training: **the bootstrap** -- the
statistical technique (Efron, 1979) of estimating a dataset's properties
by resampling it *with replacement*. It's the same sense scikit-learn's
bagging classifiers mean by `bootstrap=True`. `SplitStrategy::Bootstrap`
adds exactly this to the data loader: sample rows uniformly with
replacement from the (optionally filtered) base table, independently per
split.
## Main use cases
- **Bagging / ensemble training** (e.g. random forests): each split
trains on an independent bootstrap resample of the training set.
- **Oversampling a rare class**: filter to the minority class, then
sample it up to whatever count balances it against the majority, instead
of exhausting its small set once per epoch.
- **Fixed-size training steps independent of dataset size**: want
exactly N batches per epoch regardless of whether the table has 50 rows
or 50 million? Set `num_samples` directly instead of it being implicitly
`len(dataset)`.
- **Weighted sampling without a weight column**: filter to "eligible"
rows (e.g. a windowed-sampling workload filtering to valid window-start
positions), then bootstrap the filtered set -- an item with more
eligible rows is naturally drawn proportionally more often, with no
explicit per-row weight needed.
## Example (an actual run, not a hypothetical one)
A 5-row base table (`row_id` 0..4), sampled via `Bootstrap {
num_samples: 8, sizes: Fixed(2), seed: Some(42) }`:
```
row_id split_id
0 0
2 0
4 0
4 0 <- row 4 repeated within split 0
4 1 <- row 4 also drawn in split 1 (independent draw)
4 1
1 1
2 1
```
8 output rows from a 5-row table -- not 5. Row 4 drawn 4 times total,
row 3 never drawn. That's with-replacement sampling actually behaving
like with-replacement sampling, not an even round-robin dressed up with
a new name.
## What changed
Adds `SplitStrategy::Bootstrap` (uniform sampling with replacement,
optionally filtered, independently seeded per split -- a given split's
content doesn't depend on how many other splits there are, the same
elastic-determinism property documented for the loader's other
strategies). No new `PermutationBuilder` Rust method was needed:
`with_split_strategy` already accepts any `SplitStrategy` value
generically. Python binding:
`PermutationBuilder.bootstrap_sample(num_samples, *, ratios=None,
counts=None, fixed=None, seed=None, split_names=None)`, mirroring
`split_random`'s existing shape.
Also adds a generic `split_strategy` parameter to `StreamingDataset`'s
constructor (a callable taking/returning a `PermutationBuilder`), since
it previously could only reach `split_random`/`split_sequential` even
though `split_hash`/`split_calculated` already shipped. This unblocks
`Bootstrap` and any future strategy without further constructor changes.
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
Cargo.lock: webpki-root-certs 1.0.10; webpki-roots 1.0.10 and
rustls-webpki 0.103.16 are already current.
docs/web-tests/rs/Cargo.lock: rustls-webpki 0.103.16. python/uv.lock:
certifi 2026.10.19, the only entry that changes.
The root crate drops the root CA whose intermediates were compromised
(Mozilla bug 2083449), and rustls-webpki 0.103.16 rejects chains that
carry the compromised eMudhra intermediate keys. certifi 2026.10.19 is
the release the certifi advisory names as fixed: it drops the same root
and keeps the issuer's other roots, so HTTPS connections that chain to
them keep working. Relative to the previous 2026.6.17 pin it also adds
Telia EC and RSA TLS Root CA v3 and SECOM TLS ECC and RSA Root CA 2024
and drops Atos TrustedRoot 2011.
The webpki-root-certs entry is written by hand with the crates.io
checksum.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Esteban Gutierrez <esteban@lancedb.com>
Co-authored-by: Claude Opus 5.1 <noreply@anthropic.com>
Pin the Rust workspace and Java lance-core dependency to [Lance
v14.0.0-beta.22](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.22),
including managed data file parts and descriptor reuse from
lance-format/lance#10176. This enables the next LanceDB preview release
to provide the tagged dependency needed by lancedb/sophon#8898. No
compatibility code changes are required.
Update rustls-webpki to 0.103.16 and webpki-roots to 1.0.10 in the
lockfile to resolve the two TLS advisories blocking dependency CI
(RUSTSEC-2026-0382 and RUSTSEC-2026-0381), without adding advisory
exceptions.
The Java dependency availability blocker is resolved: both the beta.22
POM and JAR return HTTP 200 from maven.lance.org. On `ea043bf93`, the
review reproducer (`cd java && ./mvnw -U -q test -pl lancedb-core
-Dspotless.skip=true`) passes all 37 tests, and [Java CI attempt
2](https://github.com/lancedb/lancedb/actions/runs/39556336205/attempts/2)
passes.
Lines 105-106 of AGENTS.md tell agents to add TypeScript methods in
`nodejs/src/table.ts` and `nodejs/src/native_table.ts`. Neither file
exists; the abstract `Table` class and the concrete `LocalTable` class
are both in `nodejs/lancedb/table.ts`. An agent following these steps
searches for files that are not there.
This updates both paths. Docs-only, no code change.
Suggested by LanceDB support (Daisuke) after I reported it by email.
## Summary
- Rework `docs/web/training/torch.mdx` for people who already know the
PyTorch `DataLoader`, built around `lancedb.streaming.StreamingDataset`.
- The quickstart trains a state-to-action policy on the real
`lance-format/lerobot-xvla-soft-fold` dataset loaded via `hf://`, using
a zero-copy Arrow-to-tensor transform.
- Add a short table mapping PyTorch idioms (`IterableDataset`,
`DistributedSampler`, `collate_fn`, `random_split`,
`StatefulDataLoader`) to their LanceDB equivalents.
- Add an Optimizations section: Transform (including an `ExtractFrames`
example using blob v2 and torchcodec), Distributed Training, and the
existing Output Formats content moved underneath.
## Notes
- The quickstart depends on #4602 (permutation building over tables with
schema metadata), which is merged.
- `ExtractFrames` takes a URI and opens the table lazily because a local
`LanceTable` cannot yet be pickled into DataLoader workers.
- The Permutation-based "Selecting columns" and "Using multiple
DataLoader workers" sections further down are unchanged from main.
## Test plan
- [x] Ran the quickstart, transform, `ExtractFrames` (main process and
forkserver workers) and distributed snippets locally.
- [x] Rendered the page with `mint dev` and checked for console errors.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Adds a user-facing realtime engine (RTE) page at
`docs/web/tables/realtime-engine.mdx`, listed under **Management** after
Branches. It is adapted from the internal WAL field guide
(lancedb/sophon#6213), with the deployment and operations material left
out.
The page covers:
- what RTE is and how it works (memtable → SSTables → compaction; reads
merge the base table with the fresh tier)
- getting started: `set_unenforced_primary_key`, indexes, `enable_rte` /
`ALTER TABLE ... SET RTE`, `merge_insert` upserts and queries, partial
updates, deletes, and `checkpoint_rte` / `disable_rte`
- performance: write throughput and shard scaling, and read latency with
and without RTE
- a note on operations not yet supported (time travel, branching,
`count_rows`, row IDs, `TABLESAMPLE`)
Also aligns related docs with current RTE naming and behavior:
- `sql/ddl/table.mdx`: `SET MEMWAL` / `UNSET MEMWAL` → `SET RTE` /
`UNSET RTE`
- `maintenance/background-jobs.mdx`: job names are `memwal_compaction`
and `memwal_catchup_<type>`
- `feature-engineering/materialized-views.mdx`: MemWAL → realtime engine
- Python `AsyncTable.enable_rte` /
`RTEWriteSpec.with_maintained_indexes` docstrings: by default, indexes
created after enabling RTE are maintained too
The examples are written directly in the page rather than generated from
`docs/web-tests`, because RTE needs an Enterprise server to run.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: XY Zhan <zhaner08@hotmail.com>
## Problem
`permutation_builder(table).execute()` and `StreamingDataset(table,
...)` (which builds a permutation at construction) panic with
`RustPanic` on any table whose Arrow schema has schema-level metadata:
```python
schema = pa.schema([pa.field("x", pa.int64())], metadata={b"huggingface": b"{}"})
t = lancedb.connect(tmp).create_table("t", pa.table({"x": list(range(100))}, schema=schema))
permutation_builder(t).split_random(fixed=2, seed=1).execute() # RustPanic
```
Every Hugging Face Lance dataset triggers this, e.g.
`hf://datasets/lance-format/lerobot-xvla-soft-fold/data` table `frames`
carries `source-column-name-map`.
## Root cause
`PermutationBuilder::add_config_metadata` replaced the stream schema's
metadata with the permutation config and then called
`batch.with_schema(schema).unwrap()`. The base-table scan's batches
carry the base schema metadata, and `RecordBatch::with_schema` requires
the target to be a superset of the current schema, metadata included. It
returned `SchemaError("target schema is not superset of current schema
...")`, and the `unwrap` turned that into a panic on a tokio worker.
## Fix
- The permutation table deliberately does **not** inherit base-table
schema metadata. That metadata describes columns the permutation table
(`row_id`, `split_id`) does not have, and its keys can collide with
config keys such as `split_names`. A merge would let a base table's
`split_names` leak into, or override, the permutation's split names.
Batches are rebuilt with `RecordBatch::try_new(schema, columns)`.
- Errors are propagated (`map_ok` → `and_then`) instead of `unwrap`.
- The same panic-on-worker pattern in the splitter's streaming closures
(`split.rs`: `try_new`/`filter(...).unwrap()`) now propagates errors
too. Nothing else in `dataloader/` used `with_schema(...).unwrap()`.
## Tests
- Rust: `test_permutation_over_table_with_schema_metadata` in
`builder.rs` builds a random split + shuffled permutation over a table
with `huggingface` and a colliding `split_names` key. It panics with the
original `SchemaError` on the old code and passes now.
- Python: `test_permutation_over_table_with_schema_metadata` in
`test_permutation.py` covers the repro and reads both splits through
`Permutations`/`Permutation.from_tables`.
`test_streaming_dataset_over_table_with_schema_metadata` in
`test_elastic_dataloader.py` covers `StreamingDataset` construction and
iteration.
- `cargo fmt`, `cargo clippy --features remote --tests --examples`, the
dataloader Rust tests (42), `test_permutation.py` (72), and
`test_elastic_dataloader.py` all pass.
`test_torch.py::test_remote_permutation_dataloader_multiprocessing`
fails locally with `take query result did not include ordering column
'__lancedb_take_row_offset'`. It uses `Permutation.identity`, which does
not go through `PermutationBuilder`, so this change does not cause it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Document explicit Managed Blob v2 opt-in, descriptor queries and payload
fetching, Function inputs and outputs, and whole-object cleanup. Explain
null versus empty values, snapshot ownership, packed-object retention,
and the distinction from the legacy large-binary Blob API.
The local ingestion and Enterprise Function examples have been executed
end to end. The Function lifecycle description depends on
lance-format/lance#10176 and the corresponding Enterprise migration. The
existing Functions page remains outside the published navigation until
launch.
Validation: the 347-file assembled site has no broken links. Mintlify
4.2.888 build validation reports four existing
`RsExampleBody`/`RsExampleTail` import warnings in the reranking and
vector-search pages; the same diagnostics reproduce with the unchanged
base versions of all three pages edited here.
The existing docs have 254 findings from our shared Vale rules.
Contributors would otherwise inherit these when editing the same lines.
Clean up the prose across 80 pages: correct product names and a repeated
word, simplify a few phrases, and replace dashes with punctuation that
fits each sentence. The pinned Vale scan now reports zero findings, with
no suppressions.
Closes#4246.
lancedb publishes no source distribution, so a platform without a
prebuilt wheel (#4104, #4239) has nothing to install. This adds an sdist
to the Python release.
## The sdist job
The release workflow builds an sdist with the maturin version of the
wheel jobs. On a tag, the publish job uploads it with the wheels: to
PyPI, and to Fury for beta tags. The job also runs on the pull request
dry run, which now starts on a change to `python/pyproject.toml`.
## Checks
- The job builds a wheel from the sdist with `--locked` and smoke-tests
it in a fresh venv.
- `ci/check_sdist.py` fails if a tracked file of the Python package is
missing from the tarball.
- `twine check --strict` checks that the metadata renders. `pydistcheck`
enforces PyPI's 100 MiB size limit.
- The unit tests of both check scripts run in the CI scripts workflow.
## Cargo.lock
maturin ships the repository lockfile, which pins crates that only the
dropped `nodejs` member uses, so `cargo --locked` fails
(PyO3/maturin#2609). The job first runs `cargo metadata --locked` on the
repository workspace, so a lockfile that has drifted from `Cargo.toml`
fails there. It then removes the nodejs-only pins and repacks the sdist.
`ci/check_cargo_lock.py` fails if the prune changes any package it
keeps, or any dependency of one. The published file is the repacked one.
## Behavior change
After the first release with an sdist, pip on a platform with no current
wheel builds the newest version from source, which needs a Rust
toolchain and `protoc`. Before, pip installed the last version with a
wheel for that platform.
Checks `docs/web` against the shared LanceDB style rules, so the same
mechanical mistakes reach review.
**Change.**
- Add `.vale.ini`, pinning the shared LanceDB Vale package by release
(`vale-lancedb-v0.1.0`), and ignore the synced `.vale/` directory.
- Add the Docs prose style workflow. On pull requests that change
`docs/web` pages or `.vale.ini`, it annotates only added or changed
lines and never fails on a style finding; an install, sync or
configuration error still fails it.
- Link the style guide from `CONTRIBUTING.md` and show how to check a
page locally.
Validation: Tested in [temporary PR
#4595](https://github.com/lancedb/lancedb/pull/4595). Vale flagged all
five deliberate style issues on the changed prose lines, left inline
code and unchanged lines alone, and kept the check green. [Passing
workflow
run](https://github.com/lancedb/lancedb/actions/runs/39299187786/job/118219554825).
BREAKING CHANGE: `SentenceTransformerEmbeddings.trust_remote_code` now
defaults to `False` (was `True`). When an embedding function is restored
from table metadata, `trust_remote_code=True` and custom endpoints
(OpenAI `base_url`, Ollama `host`/proxy, watsonx `url`) are only honored
if the process approved them; otherwise computing embeddings raises
`UntrustedEmbeddingConfigError`. Functions created in the same process
are approved automatically. To reopen such a table in a new process,
call `get_registry().allow_remote_code(model, revision=...)` or
`get_registry().allow_endpoint(url)` first.
## Summary
Embedding functions are rebuilt from table metadata, which anyone who
can write the table controls. Since sentence-transformers defaulted
`trust_remote_code` to `True`, opening and querying a crafted table ran
arbitrary code from a model repository (GHSA-x8v2-vj73-gfwm). Metadata
could likewise point `base_url` at another server, sending it queries
and the `OPENAI_API_KEY` from the environment.
The table can no longer grant itself trust:
```python
registry = get_registry()
registry.allow_remote_code("nomic-ai/nomic-embed-text-v1.5", revision="<commit>")
registry.allow_endpoint("http://my-ollama-host:11434")
```
Trust is keyed by model name (optionally pinned to a revision), so
approving one model doesn't let a table swap in another. Restored
functions that aren't approved are replaced by a placeholder that raises
on use without constructing the real function (`huggingface` loads its
model in `__init__`), so non-embedding operations on the table keep
working. `sentence-transformers` and `huggingface` gain a `revision`
parameter. Custom embedding functions can opt in by overriding the new
`remote_code_model()` / `endpoints()` hooks.
Not covered: TypeScript (its built-ins have neither remote code nor
configurable endpoints), Bedrock `assumed_role`, and third-party
functions that don't implement the hooks.
## Evidence
`test_embeddings_trust.py` covers restore with and without approval,
revision pinning, swapped model names, endpoint gating, and that the
table stays usable. Opening a table with crafted metadata and searching
it now gives:
```
UntrustedEmbeddingConfigError: Embedding function 'sentence-transformers' was restored from table metadata with settings this process has not approved:
- runs code from model 'attacker/model' at revision None. If you trust it, call get_registry().allow_remote_code('attacker/model', revision=None)
```
## Merge Risk
**Door:** One-way: changes a default and makes previously working
reopen-and-search flows require explicit approval.
**Blast Radius:** public-api (Python embeddings); users of remote-code
models or custom endpoints who reopen tables in a new process.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`cargo deny` fails on `main` (e.g. [run
39153525802](https://github.com/lancedb/lancedb/actions/runs/39153525802))
and on #4590 with three new `zip 7.4.0` advisories:
- RUSTSEC-2026-0367, RUSTSEC-2026-0368, RUSTSEC-2026-0369 (SUID/SGID
bits, symlink targets and permissions applied through symlinks during
archive extraction), all patched only in `zip >= 9.0.0`.
`zip` is in the lock only through `candle-core` (optional
local-embedding feature). Every published candle release pins `zip < 9`
(0.9.x/0.10.x `^7.2`, 0.11.0 `^8.6`, git main 8.6), so neither `cargo
update -p zip` nor a candle bump clears it. LanceDB never extracts
archives; the crate is only linked.
Same approach as #4551 (rustls): ignore with the reason recorded in
`deny.toml`. `cargo deny check advisories` passes locally with this
change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Summary
- `StreamingDataset`'s Stage 1 (I/O) queue (`raw_batches`) had no cap:
`_fill_io` only limited *in-flight* fetch futures (`io_pending`) to
`io_queue_depth`, but every completed fetch was unconditionally drained
into `raw_batches` and the freed slot immediately refilled, so I/O could
race arbitrarily far ahead of a slow transform stage with no bound on
buffered memory.
- `transform_queue_depth`'s admission check (`_try_submit_tx`) couldn't
compensate either, since it only counts `cooked` rows and in-flight
transform futures — it never looks at `raw_batches`.
- Fix: `_fill_io` now counts both `io_pending` and `raw_batches` against
the `io_queue_depth` budget, matching the parameter's documented
"batches to keep in flight per split" contract.
## Test plan
- [x] Added
`test_raw_queue_depth_bounded_when_transform_is_the_bottleneck`
(deterministic, via synchronous I/O) asserting `raw_queue_depth <=
io_queue_depth * read_batch_size` even when the transform stage is the
bottleneck.
- [x] Full `python/python/tests/test_elastic_dataloader.py` suite passes
(190/190), confirming no change to sample ordering, counts, or
checkpoint/resumability behavior.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
udf() rejected a large_binary input or output, and large_binary nested in a list or struct, outside a Blob v2 payload. It is now a grammar primitive like large_utf8, and the shared type golden lists it.
Old versions of the zip crate have been yanked due to some rather
serious vulnerabilities. However, it is a transitive dependency via
`candle` and they have not yet released a patch.
For now I think the best course of action is to continue to ship,
knowing that we will need to address this issue as soon as candle
upgrades.
I also updated rustls since we have passed the 48 hour mark and I
removed all deny exclusions that were no longer needed.
Creating a view now follows its source through each view's stored
coordinate, up to 32 levels, and refuses when the walk reaches the name
being created. One table open per level; no catalog scan.
Part of ENT-2922.
SIDDARTHA REDDY
Joaquin Hui Gomez
陈志谦
Mohd Quamar TyagiandXuanwo
Madan Kumar
Vinayak Vathare
Zhewen TanandXuanwo
primorLee
Li Zinuo
lancedb-gatefixer[bot]
Lance Release
Jack Ye
Colin Patrick McCabe
Yang Cen
IOCRYPOSTO
Lei XuandClaude Sonnet 5.5
LanceDB RobotandLuQQiu
Brendan Clement
Tobias
Will JonesandClaude Opus 5.5
Weston PaceandClaude Sonnet 5
Wyatt Alt
Bilal Atique