Creating a view now follows its source through each view's stored
coordinate, up to 32 levels, and refuses when the walk reaches the name
being created. One table open per level; no catalog scan.
Part of ENT-2922.
## Why
Every MemWAL write and read resolves rows by the table's unenforced
primary key, but `enable_memwal` accepted a table without one. The spec
installed, and every MemWAL `merge_insert` then failed with "table has a
MemWAL index but no unenforced primary key" until MemWAL was disabled.
Lance's MemWAL initialization doesn't catch it: its key checks pass when
there is no key.
## What changes
`enable_memwal` refuses a spec on a table with no unenforced primary
key, for every sharding mode, before anything is committed. The check
runs against the dataset the spec is committed to; a key cannot be
removed once set, and an overwrite or restore of the table conflicts
with that commit. Servers that install specs through this method get the
check too.
The Rust doc on `Table::enable_memwal` now states the requirement; the
Python and TypeScript docs already did.
## Tests
- New `test_enable_memwal_requires_primary_key` (Rust and Python):
unsharded, bucket and identity specs are all refused without a key, a
refused spec leaves nothing installed, and the same table accepts a spec
once a key is set. The Rust test fails with the check disabled.
- Tests that installed a spec on a table without a key now declare one
first (6 Rust, 3 Python).
- `cargo nextest run -p lancedb --features remote --lib`: 1529 passed.
`cargo clippy -p lancedb --features remote --tests --examples -- -D
warnings` and `cargo fmt --check` are clean; no integration test or
example installs a spec.
- Python: `test_memwal_spec.py` and `test_merge_insert_lsm.py`, 43
passed; `ruff format` and `ruff check` are clean.
## What
Renames the MemWAL surface in two passes, across Rust, Python,
TypeScript, and Java, including the wire format.
### 1. MemWAL APIs drop "LSM"
The write path these APIs drive is Lance's MemWAL; "LSM" described its
shape rather than naming it. Turning it on and off now reads as enabling
a feature rather than installing a spec.
| Before | After |
|---|---|
| `set_lsm_write_spec` | `enable_memwal` |
| `unset_lsm_write_spec` | `disable_memwal` |
| `get_lsm_write_spec` | `get_memwal_spec` |
| `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` /
`compact_memwal` / `checkpoint_memwal` |
| `get_lsm_stats` | `get_memwal_stats` |
| `close_lsm_writers` | `close_memwal_writers` |
| `use_lsm` (query, merge insert) | `use_memwal` |
| `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` |
| Java `LanceDbTableLsm` | `LanceDbTableMemWal` |
TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`,
...); the napi methods set `js_name` so the generated names keep the
`MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so
`compact_memwal` is not confused with `compact_files` / `optimize`,
which compact the base table.
Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm`
modules) are unchanged.
### 2. Stats use SSTable and table shard
Aligns the stats surface with the MemWAL naming settled upstream in
lance-format/lance#7943 and #7957, where the persisted unit became an
**SSTable**.
**Generation → SSTable.** A `GenerationStats` describes one flushed
MemTable, which is an SSTable. The generation *number* is kept — an
SSTable is identified by its generation — so only the noun moved.
**Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names
only the hash sharding *transform*, so it was already wrong for
`identity` and `year` sharding, which produce shards and no buckets at
all.
| Before | After |
|---|---|
| `GenerationStats` | `SsTableStats` |
| `BucketStats` | `TableShardStats` |
| `stats.buckets` | `stats.table_shards` |
| `BucketStats.generations` | `TableShardStats.sstables` |
| `include_generation_rows` | `include_sstable_rows` |
| `newest_generation` | `newest_sstable_generation` |
| `outstanding_generations` | `outstanding_sstables` |
Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed
generations" → SSTables, "bucket" (meaning a shard) → table shard.
`MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard
status value are **unchanged** — those name the sharding transform or a
server value, not the shard.
## Compatibility
Breaking, for an experimental API. The wire format changes with the
SDKs, so this must roll out together with the matching server change:
- Routes: `/set_lsm_write_spec/` → `/enable_memwal/`,
`/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` →
`/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` →
`/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/`
- Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` →
`memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables`
- Request fields: `include_generation_rows` → `include_sstable_rows`;
`use_lsm` → `use_memwal` on query and merge-insert bodies
## Validation
- Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467
lib tests and all integration test targets pass; `cargo clippy` clean
- Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`,
`test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table`
— 655 passed; `ruff format` / `ruff check` clean
- TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593
passed; `biome check` clean; `docs/src/js` regenerated with `npm run
docs`
- Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned
google-java-format does not run on JDK 21)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
`RemoteTable::set_unenforced_primary_key` returned `NotSupported`, so
the call failed against LanceDB Cloud and enterprise from every SDK —
Python and TypeScript both forward to it. That also blocked sharded LSM
writes, since the server rejects bucket/identity sharding on a table
that declares no unenforced primary key, and the SDK had no way to
declare one.
No server-side support was missing. The unenforced primary key is Lance
schema field metadata, and the existing `update_field_metadata` endpoint
writes exactly that, so the remote table now installs the key through
it. The commit layer behind that endpoint installs the position,
enforces immutability and runs `verify_primary_key()` — the same code a
native table reaches, so both paths agree on semantics and not just on
error messages.
## Changes
- `rust/lancedb/src/table/primary_key.rs`: the request validation
(single column, column exists, supported dtype, key not already set) and
the metadata edit (set the 1-indexed position, clear the legacy boolean
key) move into `validate()` and `install_edit()`, shared by both table
types so they cannot drift. Native behaviour is unchanged.
- `rust/lancedb/src/remote/table.rs`: real implementation —
`check_mutable()`, validate against the described schema, then one
`update_field_metadata` request.
- `python/python/lancedb/remote/table.py`: drop the "Not supported on
LanceDB Cloud." docstring.
No binding changes were needed; Python and TypeScript already forward to
the Rust `BaseTable` method.
## Tests
Three mocked-server tests in `rust/lancedb/src/remote/table.rs`:
- the happy path, asserting the request body carries
`lance-schema:unenforced-primary-key:position: "1"` and a null legacy
key on the right field path;
- the invalid requests a native table also rejects (empty, compound,
unknown column, unsupported dtype), each failing before any write
reaches the server;
- the already-set case, rejected from the schema the server already
reports.
`cargo test --features remote -p lancedb --lib`: 1186 passed, 0 failed.
`cargo clippy --features remote -p lancedb --lib --tests` is clean.
## Note for enterprise
Sophon pins the lancedb submodule at `v0.39.0-beta.6`, so this reaches
enterprise on the next bump.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_011xV8EebEZDmDDV8hVBm6MD
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
The curation, SQL and operations pages contain outdated claims and
examples that do not reliably work as written.
This corrects those pages against the reviewed SDK and local server
evidence, including prepared statements, background jobs, cleanup and
monitoring. Integration guides are grouped with their examples in #4533.
Function and Secret-object documentation stays unpublished until its
release. The two Function pages are listed in `.mintignore` and left out
of the navigation, and the sections about them on other pages are MDX
comments.
Stacked on #4533. Each MDX file has one PR owner. Rebase onto main after
#4533 is squash-merged.
Local assembly, Mint validation and link checks pass, and the assembled
site shows no Function or Secret-object content. A local release
rehearsal that uncomments the drafts and restores the navigation also
passes. Hold publication until this PR and lancedb/sophon#8688 are on
`main`.
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
Ingestion, search and integration examples are incomplete or
inconsistent across languages, and some guidance misstates SDK behavior.
This brings the guides and their example sources together. It includes
complete Python, TypeScript and Rust workflows, isolated provider and
reranker examples, the Superlinked corrections, and guidance on retries
that can duplicate writes.
Stacked on #4491. Each MDX file has one PR owner. Rebase onto main after
#4491 is squash-merged.
Local assembly, Mint validation and link checks pass. Example code is
unchanged from the previously validated package. Publication stays held
through integration.
Updates the Rust workspace dependencies and Java lance-core to [Lance
v14.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.18).
Mirrors Lance's pinned object_store patch to support the multipart write
mode API required by this release.
Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed.
---------
Co-authored-by: Lu Qiu <luqiujob@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Yang Cen <bubble-cal@outlook.com>
This PR is part of the documentation revamp. Once the full set of PRs is
merged, we’ll publish the combined site to staging for review before
updating docs.lancedb.com.
The introductory pages are inconsistent, and the old PR split asked
reviewers to review some pages twice.
This brings the final quickstart, connection guide, architecture and
Concepts pages together. It clarifies OSS and Enterprise, simplifies
naming rules and includes the navigation and diagram fixes from #4489.
Function and Secret details on the Concepts pages are MDX comments until
their release, and CONTRIBUTING.md explains how to write docs before a
release.
Includes #4532. Ingestion, search and integrations are in #4533; data
workflows, SQL and operations are in #4534. Each MDX file has one PR
owner.
Local assembly, Mint validation and link checks pass. The Function pages
stay published from `main` until #4534 hides them, so hold publication
until the complete stack is integrated.
The \`deny\` job fails on main and on every open PR (for example #4539)
because RustSec published four advisories against rustls 0.23.45:
- RUSTSEC-2026-0346: \`Debug\` impls for client session values expose
TLS secrets
- RUSTSEC-2026-0347: buffered application data sent to the outer server
after ECH rejection
- RUSTSEC-2026-0348: Brotli certificate decompression can allocate up to
1 GiB before authentication
- RUSTSEC-2026-0349: unauthenticated ServerHello can downgrade later
connections to classical key exchange
All four are fixed in rustls 0.23.46, published 2026-10-14 09:54 UTC.
The 48-hour dependency age policy does not allow that upgrade until
2026-10-16 09:54 UTC, so this mirrors lance-format/lance#9985 and
ignores the four advisories with that reason. rustls reaches us only
through lance-io -> aws-config -> aws-smithy-http-client.
It also ignores RUSTSEC-2024-0449 (\`unsafe-libyaml\` unmaintained),
reached through lance-tokenizer -> lindera -> serde_yaml_ng. No
maintained upgrade exists, so this one stays until lindera moves off
serde_yaml_ng.
Follow-up once the window passes: drop the four rustls entries and run
\`cargo update -p rustls\`.
Validation: \`cargo deny --all-features check advisories bans licenses
sources\` (the CI invocation) passes locally with cargo-deny 0.20.2:
\`advisories ok, bans ok, licenses ok, sources ok\`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
\`inf_vector_column_query\`'s \`dim\` filter (added in #3423) is meant
to disambiguate between multiple vector columns of different sizes. When
a schema has exactly one vector column and the query's dimension doesn't
match it, the filter excluded that column too, raising "There is no
vector column in the data" instead of resolving the column and letting
the dimension mismatch surface at query execution.
This was latent until #4409 wired client-side vector-column inference
into remote search: a mismatched-dimension query against a
single-vector-column remote table now fails client-side with the wrong
error, before ever reaching the server's own dimension validation.
Fall back to the unfiltered candidate set when the dimension filter
leaves zero matches, and only raise when that is still ambiguous or
genuinely empty.
## Summary
The Python wheel builds used different protoc versions: v24.4 on Linux,
v21.12 on Windows, and whatever Homebrew shipped on macOS. None of them
verified the download. A new composite action,
`.github/actions/setup-protoc`, now holds one pin, v36.2, with a SHA-256
for each platform's zip.
- macOS and Windows: the action downloads protoc, checks its hash, and
installs it.
- Linux: protoc has to be installed inside the manylinux container, so
the build reads the version and hash from the action's outputs and
checks the hash there.
protobuf does not publish checksum files for protoc. The pinned hashes
are the digests GitHub recorded for the v36.2 release assets. Older
releases such as v24.4 have none, which is why this moves to v36.2.
protoc only runs at build time, through `prost-build` in the Lance
crates. The Linux protoc binary is statically linked, so it runs in the
manylinux 2_17 container.
The action is also added to the path triggers of `pypi-publish.yml` and
`python.yml`, so a change to the pin runs the wheel builds.
Follow-up to
https://github.com/lancedb/lancedb/pull/4522#issuecomment-6160848344.
The sdist job in that PR can use the same action once this merges.
## Evidence
Running the action's install script locally on linux-aarch_64 installs
`libprotoc 36.2`, and a wrong hash fails with `protoc checksum
mismatch`. `cargo check -p lancedb` with `PROTOC` set to protoc 36.2
builds the Lance protos. The PyPI Publish dry run on this PR covers the
Linux, macOS and Windows builds.
## Merge Risk
**Door:** Two-way. CI only.
**Blast Radius:** Python wheel builds.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add an authorization API to the LanceDB catalog. This is an API that can
be used to bind access control lists (ACLs) to individual catalog
objects. It can also be used to manage server-side roles, and introspect
principals and groups. The new API is implemented in both the Rust,
Python, and async Python styles and includes list_acls, add_acl,
delete_acl, list_role_bindings, add_role_binding, delete_role_binding,
get_principal, and get_group. This API implements the design principle
of forwards-compatibility. So unknown privilege, object, and subject
types do not cause errors, but are be supported as best as possible in
the library.
WAL queries have the ability to bypass WAL by using `use_lsm(bool)` to
determine whether to explicitly use or not use it. This is exposed on
many of the entry-points but not on the `BaseTableAdapter`. This PR adds
that function to allow WAL use configuration.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
## Why
lance-format/lance#9803 makes the MemWAL's memtable index kinds
pluggable: a `MemIndexRegistry` on the writer config decides which
base-table index kinds are maintained in memory, so a kind Lance does
not build in can be added by registering a plugin. LanceDB opens the
MemWAL writer itself for `merge_insert`, and checks a named maintained
set itself, both against Lance's built-ins only. So a kind registered
outside Lance could not be named in
`LsmWriteSpec::with_maintained_indexes`, and under "maintain every
index" it was skipped.
## What changes
- `NativeTable::set_mem_index_registry(registry)` sets the memtable
index kinds a table maintains. The default is Lance's built-ins, so
nothing changes unless a caller sets one. The registry checks a named
maintained set (`validate_maintained_indexes_with`), is recorded on
`initialize_mem_wal`, and goes on the writer config `merge_insert`
opens. `lancedb::table` re-exports `MemIndexRegistry` and Lance's plugin
traits (`mem_index`), so a caller needs no direct `lance` dependency.
- The LSM vector read no longer checks resident memtables at all, and
the `carries_vector_index` helper goes with it. Lance already compares
every row of a memtable whose index can't answer the search, so no rows
are missed. It still requires the base vector index to be maintained.
## Limits
- The registry belongs to the table handle. Clones of a handle and its
branch handles share it; a handle opened separately starts from the
built-ins. Set it before the handle's first LSM write: a writer already
open keeps the kinds it opened with.
- The maintained set is stored with the table but the registry is not.
Once a spec names an index only a registered plugin maintains, an LSM
write from a handle without that registry is refused with Lance's "no
registered plugin maintains" error rather than skipping the index. That
includes the Python and Node bindings, which have no way to set a
registry yet.
- Rust only. A registry holds Rust plugin objects, which the Python,
TypeScript and Java bindings cannot construct, so they get no setter;
their documented behaviour (built-in kinds only) is unchanged.
- Per-index plugin settings (`ShardWriterConfig::index_overrides`, from
lance-format/lance#9805) are not exposed; writers use each plugin's
defaults.
## Open decision
Whether the registry should also be settable on the connection, so every
table a connection opens carries it. This PR keeps it per table; a
connection-level option would sit on top of `set_mem_index_registry`
without changing it.
## Depends on
Lance v14.0.0-beta.13, which `main` already uses. It includes
lance-format/lance#9803 and #9805 (pluggable memtable index kinds) and
#9857 (flushed generations keep the table's field ids).
## Testing
- `lsm_maintains_an_index_kind_from_the_tables_registry`: a stand-in
plugin maintains the base table's bitmap index as a B-tree. Naming the
index is refused without the registry and accepted with it; a separately
opened handle without the registry is refused an LSM write. Rows written
through `merge_insert` are found by id before and after the flush. Each
generation the shard manifests publish carries a bitmap index on
`region`, and a filter on it is answered through that index.
- `lsm_default_set_maintains_a_registry_kind_created_while_writing`:
under "maintain every index", a bitmap index created after the writer
opened is picked up by that writer and flushed. A clone of the table
handle shares the registry.
- `lsm_vector_search_compares_every_row_when_no_resident_index_answers`:
with a vector plugin that answers no search, LSM vector search still
returns the right rows, including a second page.
- `cargo test -p lancedb --features remote --lib` passes; `cargo clippy
-p lancedb --features remote --tests -- -D warnings` and `cargo fmt
--all -- --check` are clean.
---------
Co-authored-by: XYZhan <xuanyu.zhan@lancedb.com>
Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.
The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.
Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.
BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.
Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.
Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):
- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.
All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.
Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core to
[v14.0.0-beta.11](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.11).
Adapts the LSM resident-index guards to Lance's query-based index API
and adds regression coverage for index presence and type checks.
Validation: workspace Clippy with all features and warnings denied, Rust
formatting, and targeted LSM query unit tests.
`MergeInsertBuilder::execute` now takes any `Scannable`, and the source
is routed so Lance's merge join can pick its build side from statistics:
* Rescannable sources become a provider reporting their exact row count.
* One-shot sources are collected up to `source_collect_threshold_bytes`
(default 64MiB). If they end first they are merged from memory;
otherwise the prefix is re-chained, spilled for retries, and reported
as an inexact lower bound.
The threshold can be set per merge or as a connection default.
BREAKING CHANGE: `MergeInsertBuilder::execute` and
`BaseTable::merge_insert` take a `Scannable` instead of
`Box<dyn RecordBatchReader + Send>`. Pass a `Box<dyn RecordBatchReader +
Send>` (e.g. `Box::new(reader) as Box<dyn RecordBatchReader + Send>`), a
`RecordBatch`, or a `Vec<RecordBatch>`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
The Java SDK pinned `jackson-databind` 2.17.1, and `jackson-core` and
the other Jackson modules came in transitively via `arrow-vector` and
`lance-namespace-apache-client` at 2.16.x–2.17.x, all affected by known
CVEs.
This imports `jackson-bom` 2.18.11 in the parent POM, and declares
`jackson-core`, `jackson-annotations`, `jackson-datatype-jsr310`, and
`jackson-jaxrs-json-provider` as direct (BOM-managed) dependencies of
`lancedb-core`. The direct declarations are needed because the parent's
dependency management doesn't apply to a consumer's transitive
resolution; without them, Maven's nearest-wins mediation still hands
downstream apps the older versions from `arrow-vector`.
## Evidence
`mvn dependency:list` in a separate consumer project depending only on
`lancedb-core`:
| Artifact | Before | After |
|---|---|---|
| jackson-core | 2.16.0 | 2.18.11 |
| jackson-annotations | 2.16.0 | 2.18.11 |
| jackson-databind | 2.17.1 | 2.18.11 |
| jackson-datatype-jsr310 | 2.16.0 | 2.18.11 |
| jackson-jaxrs-base / json-provider | 2.17.1 | 2.18.11 |
| jackson-module-jaxb-annotations | 2.17.1 | 2.18.11 |
## Merge Risk
**Door:** Two-way. Dependency versions only, and they stay within
Jackson 2.x.
**Blast Radius:** `java` dependencies. Downstream users of
`lancedb-core` now get Jackson 2.18.11 transitively.
Fixes OSS-2360, OSS-2337
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Fixes the Dependabot alerts for Rust crates that have a patched release.
- `Cargo.lock`: aws-smithy-json 0.62.5 → 0.62.7, quinn-proto 0.11.14 →
0.11.19, xxhash-rust 0.8.15 → 0.8.19.
- `docs/web-tests/rs/Cargo.lock`: rustls 0.23.35 → 0.23.45,
rustls-webpki 0.103.13 → 0.103.15, serde_with 3.16.1 → 3.22.0, rand
0.8.5 → 0.8.8 and 0.9.2 → 0.9.5, xxhash-rust 0.8.15 → 0.8.19.
- `rust/lancedb/Cargo.toml`: the AWS SDK crates' default features
include the legacy `rustls` feature, which pulls in rustls 0.21 and the
vulnerable rustls-webpki 0.101.7 via hyper 0.14.
`aws-sdk-bedrockruntime` (behind the optional `bedrock` feature) and the
`aws-sdk-s3`/`dynamodb`/`kms` dev-dependencies now use `default-features
= false` with `default-https-client` + `rt-tokio`, the same rustls 0.23
client `aws-config` already uses. Minimum versions are raised to the
first releases that have `default-https-client`. This removes rustls
0.21, rustls-webpki 0.101, hyper 0.14 and h2 0.3 from the lockfile.
Not fixed: fast-float 0.2.0 has no patched release. It comes from
polars-arrow via the optional `polars` feature, pinned to `<0.40`;
removing it needs a polars upgrade.
## Merge Risk
**Door:** two-way; lockfile and dependency-feature changes only.
**Blast Radius:** `bedrock` embedding function's HTTP client, and
S3/DynamoDB/KMS integration tests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
LanceDB's remote HTTP client (reqwest) and gRPC client (tonic) used
rustls with the `ring` crypto provider, while the AWS SDK and
object_store use aws-lc-rs. This moves both clients to aws-lc-rs:
- `tonic`: `tls-ring` → `tls-aws-lc`.
- `reqwest`: 0.12 → 0.13. reqwest 0.12 has no aws-lc-rs feature; 0.13's
`rustls` feature uses it. 0.13 was already in the lockfile through
object_store. No code changes were needed.
- `macos-system-configuration` is replaced by its 0.13 name,
`system-proxy`, which also reads the system proxy on Windows.
- `query` is now an opt-in feature in 0.13; it's used for request query
parameters.
- Certificate verification moves from `rustls-native-certs` to
`rustls-platform-verifier`, which uses the OS verifier on macOS and
Windows.
`ring` is still compiled in, because `lance-namespace-impls` and
`lance-namespace-reqwest-client` enable it through reqwest 0.12
(lance-format/lance#9855, lance-format/lance-namespace#369). Both
clients here pick aws-lc-rs explicitly, so they don't depend on which
providers are compiled in.
## Evidence
A temporary example that connects the remote client to real HTTPS hosts:
```
https://example.com: Http error: ... 404 Not Found: <!doctype html>... # TLS handshake OK
https://expired.badssl.com: Hit retry limit ... connect_failures=3/3 # bad cert rejected
```
## Merge Risk
**Door:** two-way; dependency feature changes only.
**Blast Radius:** `remote` TLS and certificate verification (Python,
Node, and Rust LanceDB Cloud clients).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Python `Decimal` values and Arrow decimal scalars raise
`NotImplementedError` in `table.update(values=...)`. Register a Decimal
converter that emits quoted fixed-point text so the native cast to the
target column preserves every digit, including decimal128 and decimal256
values. Non-finite Decimal values raise `ValueError` with a static
message.
Closes#4360.
Validation with the native extension built from the current source at
7f593359, on macOS arm64 and Python 3.12.14:
- All 33 new regression cases failed before the fix and passed
afterward. They cover Python and Arrow scalar updates, signs, zero,
exponents, full 38/76-digit precision, and a Decimal context precision
of 6.
- 16 existing converter, update, and Decimal expression tests passed.
- Repository Ruff formatting and lint passed; formatting left all 156
files unchanged.
- A native round-trip check confirmed that an unquoted SQL number rounds
a high-precision value while the quoted Decimal update preserves it
exactly.
The full suite, other Python versions, and remote API were not
exercised.
AI assistance: Codex helped implement and validate the Decimal
conversion and its regression tests.
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.
The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
Centralize the entity name validation for tables, views, functions,
namespaces, secrets, and databases. This helps in several ways. One is
that percent-encoding is no longer needed in most cases (the exception
is database names, which can contain slashes). Another good thing is
that LLMs looking at this repository can now understand what the naming
rules are, since they will be the same here as on the server side.
Finally, this will help with the upcoming authZ code, since it also
needs to serialize these names, and needs to know that they don't
contain (among other things) colons.
Moves Function timestamps on the wire from RFC 3339 strings to epoch
milliseconds.
## What
- `FunctionVersion.created_at` (an RFC 3339 string) becomes
`created_at_millis` (milliseconds since the epoch, as an integer).
That's the shape Secrets, jobs and `FunctionErrorRecord` already use.
- Rust: `FunctionVersion::created_at() -> &str` becomes
`created_at_millis() -> i64`.
- Python: `FunctionVersion.created_at: str` becomes `created_at_millis:
int`.
- Fixtures and the mock server now send `1787270400000`, the old
`2026-08-21T00:00:00Z`.
Node.js doesn't expose `FunctionVersion`, so it doesn't change.
## Compatibility
Breaking, and it must ship together with the matching server change: a
server that still sends `created_at` fails to deserialize on this
client. Functions aren't deployed yet, so nothing is migrated.
## Tests
- `cargo test --features remote --test first_class_function_slice1`: 10
passed
- `cargo test --features remote --lib function`: 51 passed
- `pytest python/tests/test_first_class_function*.py`: 100 passed
- clippy, `cargo fmt --check`, ruff check and ruff format are clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BY5xcEGEAa1E11VG55Kd1w
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Covers lance-format/lance#9750's `TopKLateMaterialization` rule through
lancedb's DataFusion `TableProvider`, now that main is on lance
v14.0.0-beta.7: SQL `ORDER BY ... LIMIT` over `BaseTableAdapter` with
the rule registered returns the same rows as without it, the plan
contains a `Take`, and the output schema metadata stays erased. No
certification of `MetadataEraserExec` is needed because DataFusion
pushes the top-k sort below it.
ENT-2777
`FTS()` rejects JSON columns because validation only considers their
`LargeBinary` storage type. Validate the complete Arrow field in
embedded and remote creation so `lance.json` columns reach Lance's
existing JSON-aware inverted index, while ordinary binary fields remain
rejected.
JSON searches use path-qualified triples such as `meta.tag,str,nature`;
the API documentation describes this syntax and its field-scoping
behavior. Remote creation preserves the supplied FTS configuration.
Two integration limits remain: malformed triples can still panic in the
pinned Lance dependency (tracked by lance-format/lance#8812 and the
pending fix in lance-format/lance#8813), and the opt-in server test has
not run against a real deployment because no remote database is
configured. That test covers index completion, statistics, search
results, and index use; HTTP mock coverage only verifies the client
request.
Refs #3891.
Will Jones
Wyatt Alt
Bilal Atique
XY Zhan
Dan RammerandClaude Opus 5
Lance Release
Brendan Clement
Jack Ye
lancedb-gatefixer[bot]
Drew
Colin Patrick McCabe
SIDDARTHA REDDY
Daniel Peng
Kevin Dong
Jonathan HsiehandClaude Opus 5.5
Xuanwo