The existing docs have 254 findings from our shared Vale rules.
Contributors would otherwise inherit these when editing the same lines.
Clean up the prose across 80 pages: correct product names and a repeated
word, simplify a few phrases, and replace dashes with punctuation that
fits each sentence. The pinned Vale scan now reports zero findings, with
no suppressions.
Closes#4246.
lancedb publishes no source distribution, so a platform without a
prebuilt wheel (#4104, #4239) has nothing to install. This adds an sdist
to the Python release.
## The sdist job
The release workflow builds an sdist with the maturin version of the
wheel jobs. On a tag, the publish job uploads it with the wheels: to
PyPI, and to Fury for beta tags. The job also runs on the pull request
dry run, which now starts on a change to `python/pyproject.toml`.
## Checks
- The job builds a wheel from the sdist with `--locked` and smoke-tests
it in a fresh venv.
- `ci/check_sdist.py` fails if a tracked file of the Python package is
missing from the tarball.
- `twine check --strict` checks that the metadata renders. `pydistcheck`
enforces PyPI's 100 MiB size limit.
- The unit tests of both check scripts run in the CI scripts workflow.
## Cargo.lock
maturin ships the repository lockfile, which pins crates that only the
dropped `nodejs` member uses, so `cargo --locked` fails
(PyO3/maturin#2609). The job first runs `cargo metadata --locked` on the
repository workspace, so a lockfile that has drifted from `Cargo.toml`
fails there. It then removes the nodejs-only pins and repacks the sdist.
`ci/check_cargo_lock.py` fails if the prune changes any package it
keeps, or any dependency of one. The published file is the repacked one.
## Behavior change
After the first release with an sdist, pip on a platform with no current
wheel builds the newest version from source, which needs a Rust
toolchain and `protoc`. Before, pip installed the last version with a
wheel for that platform.
Checks `docs/web` against the shared LanceDB style rules, so the same
mechanical mistakes reach review.
**Change.**
- Add `.vale.ini`, pinning the shared LanceDB Vale package by release
(`vale-lancedb-v0.1.0`), and ignore the synced `.vale/` directory.
- Add the Docs prose style workflow. On pull requests that change
`docs/web` pages or `.vale.ini`, it annotates only added or changed
lines and never fails on a style finding; an install, sync or
configuration error still fails it.
- Link the style guide from `CONTRIBUTING.md` and show how to check a
page locally.
Validation: Tested in [temporary PR
#4595](https://github.com/lancedb/lancedb/pull/4595). Vale flagged all
five deliberate style issues on the changed prose lines, left inline
code and unchanged lines alone, and kept the check green. [Passing
workflow
run](https://github.com/lancedb/lancedb/actions/runs/39299187786/job/118219554825).
BREAKING CHANGE: `SentenceTransformerEmbeddings.trust_remote_code` now
defaults to `False` (was `True`). When an embedding function is restored
from table metadata, `trust_remote_code=True` and custom endpoints
(OpenAI `base_url`, Ollama `host`/proxy, watsonx `url`) are only honored
if the process approved them; otherwise computing embeddings raises
`UntrustedEmbeddingConfigError`. Functions created in the same process
are approved automatically. To reopen such a table in a new process,
call `get_registry().allow_remote_code(model, revision=...)` or
`get_registry().allow_endpoint(url)` first.
## Summary
Embedding functions are rebuilt from table metadata, which anyone who
can write the table controls. Since sentence-transformers defaulted
`trust_remote_code` to `True`, opening and querying a crafted table ran
arbitrary code from a model repository (GHSA-x8v2-vj73-gfwm). Metadata
could likewise point `base_url` at another server, sending it queries
and the `OPENAI_API_KEY` from the environment.
The table can no longer grant itself trust:
```python
registry = get_registry()
registry.allow_remote_code("nomic-ai/nomic-embed-text-v1.5", revision="<commit>")
registry.allow_endpoint("http://my-ollama-host:11434")
```
Trust is keyed by model name (optionally pinned to a revision), so
approving one model doesn't let a table swap in another. Restored
functions that aren't approved are replaced by a placeholder that raises
on use without constructing the real function (`huggingface` loads its
model in `__init__`), so non-embedding operations on the table keep
working. `sentence-transformers` and `huggingface` gain a `revision`
parameter. Custom embedding functions can opt in by overriding the new
`remote_code_model()` / `endpoints()` hooks.
Not covered: TypeScript (its built-ins have neither remote code nor
configurable endpoints), Bedrock `assumed_role`, and third-party
functions that don't implement the hooks.
## Evidence
`test_embeddings_trust.py` covers restore with and without approval,
revision pinning, swapped model names, endpoint gating, and that the
table stays usable. Opening a table with crafted metadata and searching
it now gives:
```
UntrustedEmbeddingConfigError: Embedding function 'sentence-transformers' was restored from table metadata with settings this process has not approved:
- runs code from model 'attacker/model' at revision None. If you trust it, call get_registry().allow_remote_code('attacker/model', revision=None)
```
## Merge Risk
**Door:** One-way: changes a default and makes previously working
reopen-and-search flows require explicit approval.
**Blast Radius:** public-api (Python embeddings); users of remote-code
models or custom endpoints who reopen tables in a new process.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`cargo deny` fails on `main` (e.g. [run
39153525802](https://github.com/lancedb/lancedb/actions/runs/39153525802))
and on #4590 with three new `zip 7.4.0` advisories:
- RUSTSEC-2026-0367, RUSTSEC-2026-0368, RUSTSEC-2026-0369 (SUID/SGID
bits, symlink targets and permissions applied through symlinks during
archive extraction), all patched only in `zip >= 9.0.0`.
`zip` is in the lock only through `candle-core` (optional
local-embedding feature). Every published candle release pins `zip < 9`
(0.9.x/0.10.x `^7.2`, 0.11.0 `^8.6`, git main 8.6), so neither `cargo
update -p zip` nor a candle bump clears it. LanceDB never extracts
archives; the crate is only linked.
Same approach as #4551 (rustls): ignore with the reason recorded in
`deny.toml`. `cargo deny check advisories` passes locally with this
change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Summary
- `StreamingDataset`'s Stage 1 (I/O) queue (`raw_batches`) had no cap:
`_fill_io` only limited *in-flight* fetch futures (`io_pending`) to
`io_queue_depth`, but every completed fetch was unconditionally drained
into `raw_batches` and the freed slot immediately refilled, so I/O could
race arbitrarily far ahead of a slow transform stage with no bound on
buffered memory.
- `transform_queue_depth`'s admission check (`_try_submit_tx`) couldn't
compensate either, since it only counts `cooked` rows and in-flight
transform futures — it never looks at `raw_batches`.
- Fix: `_fill_io` now counts both `io_pending` and `raw_batches` against
the `io_queue_depth` budget, matching the parameter's documented
"batches to keep in flight per split" contract.
## Test plan
- [x] Added
`test_raw_queue_depth_bounded_when_transform_is_the_bottleneck`
(deterministic, via synchronous I/O) asserting `raw_queue_depth <=
io_queue_depth * read_batch_size` even when the transform stage is the
bottleneck.
- [x] Full `python/python/tests/test_elastic_dataloader.py` suite passes
(190/190), confirming no change to sample ordering, counts, or
checkpoint/resumability behavior.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
udf() rejected a large_binary input or output, and large_binary nested in a list or struct, outside a Blob v2 payload. It is now a grammar primitive like large_utf8, and the shared type golden lists it.
Old versions of the zip crate have been yanked due to some rather
serious vulnerabilities. However, it is a transitive dependency via
`candle` and they have not yet released a patch.
For now I think the best course of action is to continue to ship,
knowing that we will need to address this issue as soon as candle
upgrades.
I also updated rustls since we have passed the 48 hour mark and I
removed all deny exclusions that were no longer needed.
Creating a view now follows its source through each view's stored
coordinate, up to 32 levels, and refuses when the walk reaches the name
being created. One table open per level; no catalog scan.
Part of ENT-2922.
## Why
Every MemWAL write and read resolves rows by the table's unenforced
primary key, but `enable_memwal` accepted a table without one. The spec
installed, and every MemWAL `merge_insert` then failed with "table has a
MemWAL index but no unenforced primary key" until MemWAL was disabled.
Lance's MemWAL initialization doesn't catch it: its key checks pass when
there is no key.
## What changes
`enable_memwal` refuses a spec on a table with no unenforced primary
key, for every sharding mode, before anything is committed. The check
runs against the dataset the spec is committed to; a key cannot be
removed once set, and an overwrite or restore of the table conflicts
with that commit. Servers that install specs through this method get the
check too.
The Rust doc on `Table::enable_memwal` now states the requirement; the
Python and TypeScript docs already did.
## Tests
- New `test_enable_memwal_requires_primary_key` (Rust and Python):
unsharded, bucket and identity specs are all refused without a key, a
refused spec leaves nothing installed, and the same table accepts a spec
once a key is set. The Rust test fails with the check disabled.
- Tests that installed a spec on a table without a key now declare one
first (6 Rust, 3 Python).
- `cargo nextest run -p lancedb --features remote --lib`: 1529 passed.
`cargo clippy -p lancedb --features remote --tests --examples -- -D
warnings` and `cargo fmt --check` are clean; no integration test or
example installs a spec.
- Python: `test_memwal_spec.py` and `test_merge_insert_lsm.py`, 43
passed; `ruff format` and `ruff check` are clean.
## What
Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.
### 1. MemWAL APIs drop "LSM"
The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.
| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |
TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.
Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.
### 2. Stats use SSTable and table shard
Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.
**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.
**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.
| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |
Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.
`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.
## Compatibility
Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:
- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies
## Validation
- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
`RemoteTable::set_unenforced_primary_key` returned `NotSupported`, so
the call failed against LanceDB Cloud and enterprise from every SDK —
Python and TypeScript both forward to it. That also blocked sharded LSM
writes, since the server rejects bucket/identity sharding on a table
that declares no unenforced primary key, and the SDK had no way to
declare one.
No server-side support was missing. The unenforced primary key is Lance
schema field metadata, and the existing `update_field_metadata` endpoint
writes exactly that, so the remote table now installs the key through
it. The commit layer behind that endpoint installs the position,
enforces immutability and runs `verify_primary_key()` — the same code a
native table reaches, so both paths agree on semantics and not just on
error messages.
## Changes
- `rust/lancedb/src/table/primary_key.rs`: the request validation
(single column, column exists, supported dtype, key not already set) and
the metadata edit (set the 1-indexed position, clear the legacy boolean
key) move into `validate()` and `install_edit()`, shared by both table
types so they cannot drift. Native behaviour is unchanged.
- `rust/lancedb/src/remote/table.rs`: real implementation —
`check_mutable()`, validate against the described schema, then one
`update_field_metadata` request.
- `python/python/lancedb/remote/table.py`: drop the "Not supported on
LanceDB Cloud." docstring.
No binding changes were needed; Python and TypeScript already forward to
the Rust `BaseTable` method.
## Tests
Three mocked-server tests in `rust/lancedb/src/remote/table.rs`:
- the happy path, asserting the request body carries
`lance-schema:unenforced-primary-key:position: "1"` and a null legacy
key on the right field path;
- the invalid requests a native table also rejects (empty, compound,
unknown column, unsupported dtype), each failing before any write
reaches the server;
- the already-set case, rejected from the schema the server already
reports.
`cargo test --features remote -p lancedb --lib`: 1186 passed, 0 failed.
`cargo clippy --features remote -p lancedb --lib --tests` is clean.
## Note for enterprise
Sophon pins the lancedb submodule at `v0.39.0-beta.6`, so this reaches
enterprise on the next bump.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011xV8EebEZDmDDV8hVBm6MD
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
The curation, SQL and operations pages contain outdated claims and
examples that do not reliably work as written.
This corrects those pages against the reviewed SDK and local server
evidence, including prepared statements, background jobs, cleanup and
monitoring. Integration guides are grouped with their examples in #4533.
Function and Secret-object documentation stays unpublished until its
release. The two Function pages are listed in `.mintignore` and left out
of the navigation, and the sections about them on other pages are MDX
comments.
Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after
#4533 is squash-merged.
Local assembly, Mint validation and link checks pass, and the assembled
site shows no Function or Secret-object content. A local release
rehearsal that uncomments the drafts and restores the navigation also
passes. Hold publication until this PR and lancedb/sophon#8688 are on
`main`.
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
Ingestion, search and integration examples are incomplete or
inconsistent across languages, and some guidance misstates SDK behavior.
This brings the guides and their example sources together. It includes
complete Python, TypeScript and Rust workflows, isolated provider and
reranker examples, the Superlinked corrections, and guidance on retries
that can duplicate writes.
Stacked on #4491. Each MDX file has one PR owner. Rebase onto main after
#4491 is squash-merged.
Local assembly, Mint validation and link checks pass. Example code is
unchanged from the previously validated package. Publication stays held
through integration.
Updates the Rust workspace dependencies and Java lance-core to [Lance
v14.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.18).
Mirrors Lance's pinned object_store patch to support the multipart write
mode API required by this release.
Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed.
---------
Co-authored-by: Lu Qiu <luqiujob@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Yang Cen <bubble-cal@outlook.com>
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
The introductory pages are inconsistent, and the old PR split asked
reviewers to review some pages twice.
This brings the final quickstart, connection guide, architecture and
Concepts pages together. It clarifies OSS and Enterprise, simplifies
naming rules and includes the navigation and diagram fixes from #4489.
Function and Secret details on the Concepts pages are MDX comments until
their release, and CONTRIBUTING.md explains how to write docs before a
release.
Includes #4532. Ingestion, search and integrations are in #4533; data
workflows, SQL and operations are in #4534. Each MDX file has one PR
owner.
Local assembly, Mint validation and link checks pass. The Function pages
stay published from `main` until #4534 hides them, so hold publication
until the complete stack is integrated.
The \`deny\` job fails on main and on every open PR (for example #4539)
because RustSec published four advisories against rustls 0.23.45:
- RUSTSEC-2026-0346: \`Debug\` impls for client session values expose
TLS secrets
- RUSTSEC-2026-0347: buffered application data sent to the outer server
after ECH rejection
- RUSTSEC-2026-0348: Brotli certificate decompression can allocate up to
1 GiB before authentication
- RUSTSEC-2026-0349: unauthenticated ServerHello can downgrade later
connections to classical key exchange
All four are fixed in rustls 0.23.46, published 2026-10-14 09:54 UTC.
The 48-hour dependency age policy does not allow that upgrade until
2026-10-16 09:54 UTC, so this mirrors lance-format/lance#9985 and
ignores the four advisories with that reason. rustls reaches us only
through lance-io -> aws-config -> aws-smithy-http-client.
It also ignores RUSTSEC-2024-0449 (\`unsafe-libyaml\` unmaintained),
reached through lance-tokenizer -> lindera -> serde_yaml_ng. No
maintained upgrade exists, so this one stays until lindera moves off
serde_yaml_ng.
Follow-up once the window passes: drop the four rustls entries and run
\`cargo update -p rustls\`.
Validation: \`cargo deny --all-features check advisories bans licenses
sources\` (the CI invocation) passes locally with cargo-deny 0.20.2:
\`advisories ok, bans ok, licenses ok, sources ok\`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
\`inf_vector_column_query\`'s \`dim\` filter (added in #3423) is meant
to disambiguate between multiple vector columns of different sizes. When
a schema has exactly one vector column and the query's dimension doesn't
match it, the filter excluded that column too, raising "There is no
vector column in the data" instead of resolving the column and letting
the dimension mismatch surface at query execution.
This was latent until #4409 wired client-side vector-column inference
into remote search: a mismatched-dimension query against a
single-vector-column remote table now fails client-side with the wrong
error, before ever reaching the server's own dimension validation.
Fall back to the unfiltered candidate set when the dimension filter
leaves zero matches, and only raise when that is still ambiguous or
genuinely empty.
## Summary
The Python wheel builds used different protoc versions: v24.4 on Linux,
v21.12 on Windows, and whatever Homebrew shipped on macOS. None of them
verified the download. A new composite action,
`.github/actions/setup-protoc`, now holds one pin, v36.2, with a SHA-256
for each platform's zip.
- macOS and Windows: the action downloads protoc, checks its hash, and
installs it.
- Linux: protoc has to be installed inside the manylinux container, so
the build reads the version and hash from the action's outputs and
checks the hash there.
protobuf does not publish checksum files for protoc. The pinned hashes
are the digests GitHub recorded for the v36.2 release assets. Older
releases such as v24.4 have none, which is why this moves to v36.2.
protoc only runs at build time, through `prost-build` in the Lance
crates. The Linux protoc binary is statically linked, so it runs in the
manylinux 2_17 container.
The action is also added to the path triggers of `pypi-publish.yml` and
`python.yml`, so a change to the pin runs the wheel builds.
Follow-up to
https://github.com/lancedb/lancedb/pull/4522#issuecomment-6160848344.
The sdist job in that PR can use the same action once this merges.
## Evidence
Running the action's install script locally on linux-aarch_64 installs
`libprotoc 36.2`, and a wrong hash fails with `protoc checksum
mismatch`. `cargo check -p lancedb` with `PROTOC` set to protoc 36.2
builds the Lance protos. The PyPI Publish dry run on this PR covers the
Linux, macOS and Windows builds.
## Merge Risk
**Door:** Two-way. CI only.
**Blast Radius:** Python wheel builds.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add an authorization API to the LanceDB catalog. This is an API that can
be used to bind access control lists (ACLs) to individual catalog
objects. It can also be used to manage server-side roles, and introspect
principals and groups. The new API is implemented in both the Rust,
Python, and async Python styles and includes list_acls, add_acl,
delete_acl, list_role_bindings, add_role_binding, delete_role_binding,
get_principal, and get_group. This API implements the design principle
of forwards-compatibility. So unknown privilege, object, and subject
types do not cause errors, but are be supported as best as possible in
the library.
WAL queries have the ability to bypass WAL by using `use_lsm(bool)` to
determine whether to explicitly use or not use it. This is exposed on
many of the entry-points but not on the `BaseTableAdapter`. This PR adds
that function to allow WAL use configuration.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
## Why
lance-format/lance#9803 makes the MemWAL's memtable index kinds
pluggable: a `MemIndexRegistry` on the writer config decides which
base-table index kinds are maintained in memory, so a kind Lance does
not build in can be added by registering a plugin. LanceDB opens the
MemWAL writer itself for `merge_insert`, and checks a named maintained
set itself, both against Lance's built-ins only. So a kind registered
outside Lance could not be named in
`LsmWriteSpec::with_maintained_indexes`, and under "maintain every
index" it was skipped.
## What changes
- `NativeTable::set_mem_index_registry(registry)` sets the memtable
index kinds a table maintains. The default is Lance's built-ins, so
nothing changes unless a caller sets one. The registry checks a named
maintained set (`validate_maintained_indexes_with`), is recorded on
`initialize_mem_wal`, and goes on the writer config `merge_insert`
opens. `lancedb::table` re-exports `MemIndexRegistry` and Lance's plugin
traits (`mem_index`), so a caller needs no direct `lance` dependency.
- The LSM vector read no longer checks resident memtables at all, and
the `carries_vector_index` helper goes with it. Lance already compares
every row of a memtable whose index can't answer the search, so no rows
are missed. It still requires the base vector index to be maintained.
## Limits
- The registry belongs to the table handle. Clones of a handle and its
branch handles share it; a handle opened separately starts from the
built-ins. Set it before the handle's first LSM write: a writer already
open keeps the kinds it opened with.
- The maintained set is stored with the table but the registry is not.
Once a spec names an index only a registered plugin maintains, an LSM
write from a handle without that registry is refused with Lance's "no
registered plugin maintains" error rather than skipping the index. That
includes the Python and Node bindings, which have no way to set a
registry yet.
- Rust only. A registry holds Rust plugin objects, which the Python,
TypeScript and Java bindings cannot construct, so they get no setter;
their documented behaviour (built-in kinds only) is unchanged.
- Per-index plugin settings (`ShardWriterConfig::index_overrides`, from
lance-format/lance#9805) are not exposed; writers use each plugin's
defaults.
## Open decision
Whether the registry should also be settable on the connection, so every
table a connection opens carries it. This PR keeps it per table; a
connection-level option would sit on top of `set_mem_index_registry`
without changing it.
## Depends on
Lance v14.0.0-beta.13, which `main` already uses. It includes
lance-format/lance#9803 and #9805 (pluggable memtable index kinds) and
#9857 (flushed generations keep the table's field ids).
## Testing
- `lsm_maintains_an_index_kind_from_the_tables_registry`: a stand-in
plugin maintains the base table's bitmap index as a B-tree. Naming the
index is refused without the registry and accepted with it; a separately
opened handle without the registry is refused an LSM write. Rows written
through `merge_insert` are found by id before and after the flush. Each
generation the shard manifests publish carries a bitmap index on
`region`, and a filter on it is answered through that index.
- `lsm_default_set_maintains_a_registry_kind_created_while_writing`:
under "maintain every index", a bitmap index created after the writer
opened is picked up by that writer and flushed. A clone of the table
handle shares the registry.
- `lsm_vector_search_compares_every_row_when_no_resident_index_answers`:
with a vector plugin that answers no search, LSM vector search still
returns the right rows, including a second page.
- `cargo test -p lancedb --features remote --lib` passes; `cargo clippy
-p lancedb --features remote --tests -- -D warnings` and `cargo fmt
--all -- --check` are clean.
---------
Co-authored-by: XYZhan <xuanyu.zhan@lancedb.com>
Gatefixer
Lance Release
LanceDB RobotandLuQQiu
Brendan Clement
Tobias
Will JonesandClaude Opus 5.5
Weston PaceandClaude Sonnet 5
Wyatt Alt
Bilal Atique
XY Zhan
Dan RammerandClaude Opus 5
Jack Ye
Drew
Colin Patrick McCabe
SIDDARTHA REDDY
Daniel Peng
