Updates the Rust workspace dependencies and Java lance-core to [Lance
v14.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.18).
Mirrors Lance's pinned object_store patch to support the multipart write
mode API required by this release.
Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, and `git diff --check` passed.
---------
Co-authored-by: Lu Qiu <luqiujob@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Yang Cen <bubble-cal@outlook.com>
Add an authorization API to the LanceDB catalog. This is an API that can
be used to bind access control lists (ACLs) to individual catalog
objects. It can also be used to manage server-side roles, and introspect
principals and groups. The new API is implemented in both the Rust,
Python, and async Python styles and includes list_acls, add_acl,
delete_acl, list_role_bindings, add_role_binding, delete_role_binding,
get_principal, and get_group. This API implements the design principle
of forwards-compatibility. So unknown privilege, object, and subject
types do not cause errors, but are be supported as best as possible in
the library.
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core to
[v14.0.0-beta.11](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.11).
Adapts the LSM resident-index guards to Lance's query-based index API
and adds regression coverage for index presence and type checks.
Validation: workspace Clippy with all features and warnings denied, Rust
formatting, and targeted LSM query unit tests.
## Summary
Fixes the Dependabot alerts for Rust crates that have a patched release.
- `Cargo.lock`: aws-smithy-json 0.62.5 → 0.62.7, quinn-proto 0.11.14 →
0.11.19, xxhash-rust 0.8.15 → 0.8.19.
- `docs/web-tests/rs/Cargo.lock`: rustls 0.23.35 → 0.23.45,
rustls-webpki 0.103.13 → 0.103.15, serde_with 3.16.1 → 3.22.0, rand
0.8.5 → 0.8.8 and 0.9.2 → 0.9.5, xxhash-rust 0.8.15 → 0.8.19.
- `rust/lancedb/Cargo.toml`: the AWS SDK crates' default features
include the legacy `rustls` feature, which pulls in rustls 0.21 and the
vulnerable rustls-webpki 0.101.7 via hyper 0.14.
`aws-sdk-bedrockruntime` (behind the optional `bedrock` feature) and the
`aws-sdk-s3`/`dynamodb`/`kms` dev-dependencies now use `default-features
= false` with `default-https-client` + `rt-tokio`, the same rustls 0.23
client `aws-config` already uses. Minimum versions are raised to the
first releases that have `default-https-client`. This removes rustls
0.21, rustls-webpki 0.101, hyper 0.14 and h2 0.3 from the lockfile.
Not fixed: fast-float 0.2.0 has no patched release. It comes from
polars-arrow via the optional `polars` feature, pinned to `<0.40`;
removing it needs a polars upgrade.
## Merge Risk
**Door:** two-way; lockfile and dependency-feature changes only.
**Blast Radius:** `bedrock` embedding function's HTTP client, and
S3/DynamoDB/KMS integration tests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
LanceDB's remote HTTP client (reqwest) and gRPC client (tonic) used
rustls with the `ring` crypto provider, while the AWS SDK and
object_store use aws-lc-rs. This moves both clients to aws-lc-rs:
- `tonic`: `tls-ring` → `tls-aws-lc`.
- `reqwest`: 0.12 → 0.13. reqwest 0.12 has no aws-lc-rs feature; 0.13's
`rustls` feature uses it. 0.13 was already in the lockfile through
object_store. No code changes were needed.
- `macos-system-configuration` is replaced by its 0.13 name,
`system-proxy`, which also reads the system proxy on Windows.
- `query` is now an opt-in feature in 0.13; it's used for request query
parameters.
- Certificate verification moves from `rustls-native-certs` to
`rustls-platform-verifier`, which uses the OS verifier on macOS and
Windows.
`ring` is still compiled in, because `lance-namespace-impls` and
`lance-namespace-reqwest-client` enable it through reqwest 0.12
(lance-format/lance#9855, lance-format/lance-namespace#369). Both
clients here pick aws-lc-rs explicitly, so they don't depend on which
providers are compiled in.
## Evidence
A temporary example that connects the remote client to real HTTPS hosts:
```
https://example.com: Http error: ... 404 Not Found: <!doctype html>... # TLS handshake OK
https://expired.badssl.com: Hit retry limit ... connect_failures=3/3 # bad cert rejected
```
## Merge Risk
**Door:** two-way; dependency feature changes only.
**Blast Radius:** `remote` TLS and certificate verification (Python,
Node, and Rust LanceDB Cloud clients).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.
The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
Updates the Rust workspace Lance dependencies and Java lance-core from
14.0.0-beta.1 to
[v14.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.6),
and refreshes Cargo.lock. No compatibility fixes were required.
Validation passed: `cargo clippy --quiet --workspace --tests
--all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff
--check`.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Updates the Rust workspace and Java Lance dependencies to
[v13.1.0-beta.1](https://github.com/lance-format/lance/releases/tag/v13.1.0-beta.1)
and refreshes Cargo.lock. Removes a redundant `pub(crate)` declaration
in the private Node.js blob module to satisfy the required Clippy check.
Validation: `cargo clippy --quiet --workspace --tests --all-features --
-D warnings`, `cargo fmt --all --quiet`, `pnpm build`, and `pnpm test
__test__/blob.test.ts --runInBand` (26 tests passed) in `nodejs`.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Updates the Rust workspace dependencies, Cargo lockfile, and Java
`lance-core` to [Lance
v13.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.18).
It includes lance-format/lance#9574, which bounds the memory an IVF
partition split uses when it re-reads raw vectors during
`optimize_indices`, and lance-format/lance#9582, which keeps the mTLS
REST namespace client on rustls.
Lance's new combined-fields FTS query variant (introduced by
v13.0.0-beta.17) is handled the same way as in #4336: the remote
capability check and Rust/Node bindings accept it, with a regression
test for the remote request payload, and the Python conversion reports
that this query type is not yet supported. This PR supersedes #4336.
## Summary
Materialize a source-preserving dedup result from an indexed, pinned
snapshot with a short declaration. The default greedy policy keeps the
smallest unassigned snapshot row ID and removes only its directly
matching neighbors: For A < B < C, an A–B–C chain without an A–C edge
keeps A and C.
```python
from lancedb import vector_dedup
view = db.create_materialized_view(
"images_clean",
vector_dedup("images", column="phash", distance_threshold=4),
)
cleaned = view.table
```
The Python source expression captures the source version once at
creation and
waits for the materialized result by default. Use `dataset_version=...`
for an
explicit snapshot, `with_no_data=True` for declaration only, or
`create_materialized_view_async(...)` for a job handle. AsyncConnection
supports
the same expression. It uses the existing MV REST endpoints and registry
jobs,
without opening a SQL query connection. Source names are exact
root-namespace
table names. Additional select/where/limit options are rejected for this
source.
The SQL declaration remains available:
```sql
CREATE MATERIALIZED VIEW images_clean AS
SELECT * FROM vector_dedup('images', 7, 'phash', 4);
REFRESH MATERIALIZED VIEW images_clean;
```
This is the host-facing SQL entry point. The MV builder now accepts a
typed native vector source; the Rust SDK also supports the same
definition through `MaterializedViewDefinition::from_sql` and
`prepare_definition`; `view.refresh().execute()` executes it locally.
Hosts distribute its units using their existing job infrastructure. The
raw `vector_duplicate_pairs` source remains available separately when
callers want the pair relation.
## Implementation and contract
- Use actual Rust Lance `13.0.0-beta.15` (`db53dcf367`); the native
partition primitive computes pairs from index representations without
source-vector aggregation or a Python pair loop.
- Extend partitioned MV refresh with two dependent waves. Physical
segment/partition tasks stream pairs through a bounded external sort,
select direct representatives and write per-source-fragment bitmap
artifacts. Source-fragment tasks start after every selection receipt
completes and stream retained original rows, including blob payloads,
into uncommitted output fragments.
- Reuse normal blob write coercion and row-mask scans, including stable
row-ID translation after compaction. Null vectors, isolated rows and
source deletions are preserved correctly. Conflicting representative
decisions across overlapping index scopes fail before publication.
- Expose `dependency_units()` and `write_refresh_partition_with_inputs`
alongside the existing plan/commit API. Receipts bind
snapshot/configuration/policy/destination generation. Only a complete
accepted set can publish; unfinished units restart, completed units can
be reused. There is no pair-position checkpoint.
- Use MV format 4 for dedup results and format 3 for native pairs;
Python and TypeScript metadata readers recognize both. No source
deletion or source row-ID migration occurs.
Only same-segment/same-partition edges are covered. Quantized scores use
symmetric represented-vector distances. Result materialization requires
V2 data storage and is initially a native MV source, not an ordinary
SELECT table function. Compressed vertex masks stay in memory; edge
sorting spills to local disk. Unreferenced artifacts use normal Lance
retention/cleanup and must be retained for the job's recovery window.
## Validation
AWS Linux validation:
- Native pair/result integration matrix passed: multiple
segments/partitions, exact thresholds, empty output, cross-batch data,
single-process pair parity, source preservation, lost receipts and
publication guards.
- Blob/null-vector/pinned-deletion tests passed with both row-ID modes,
including compaction that changes physical row addresses.
- Representative policy, forced external-sort spill and
conflicting/corrupt artifact tests passed.
- Full MV regression suite: 153 passed, 2 pre-existing ignored. Rust
formatting and Clippy passed.
- Python MV suite: 47 passed; TypeScript MV suite: 11 passed. Native
bindings were rebuilt in project-local environments; formatting, lint
and TypeScript docs passed.
- Rust declaration doctests: 3 passed, including the complete local
dedup example.
- Companion host validation: 41 MV job tests passed (1 pre-existing
ignored), full workspace Clippy passed, and the complete process-level
CLI E2E suite passed (3 tests). Recovery exercises both selection and
materialization waves.
- A real CIFAR-10 pHash run used five actual registry-worker owners,
completed 32 selection + 24 materialization tasks, and retained 49,846
of 50,000 rows with identical original values and image payloads.
Host workflow performance measurements are recorded in the companion
host PR. Cross-platform CI is still running. The previous remote-test
bootstrap failed before Cargo tests with `401 Bad credentials`; the
workflow fix described below is being validated by CI. No deployment or
merge is performed.
## Python source validation
AWS tests cover automatic and explicit snapshots, physical/stable row
IDs, local
and remote creation, sync/job/async-connection APIs, delayed refresh,
strict
options and identifier quoting. The native Python extension was rebuilt
through
the project's uv/maturin backend. Ruff, Pyright and the full MkDocs
API-reference
build passed; both new public symbols are rendered.
A real 50,000-image FEv2 run disabled both Python SQL entry points and
supplied an
unreachable SQL endpoint. The new API still materialized the same 49,846
retained
rows and original image payloads. Three recorded creation/refresh jobs
completed
56 tasks each on all five registry-worker owners, and remained on source
snapshot
3 after source latest advanced to 4. No worker or server wire-protocol
changes
were needed for this client entry point.
Previous CI authentication failure:
https://github.com/lancedb/lancedb/actions/runs/36334679276/job/108663151265.
## Review and CI fixes
- Validate all JSON merge source fields before projection, rejecting
unknown and duplicate names at every struct level. Missing/reordered
fields and ordinary JSON merges remain supported. A checked-in 12-case
integration matrix covers root fields, structs, lists, large lists,
fixed-size lists and map values; rejected writes leave the table version
and row count unchanged.
- Match namespace connection MV method signatures to
`MaterializedViewSource` in all four sync/async methods, fixing the
Python 3.10 import failure from runtime override checks.
- Restore the remote-test script's encrypted-CLI path: use the Actions
token for artifact download and pass `SOPHON_READ_TOKEN` under the
variable the decryption script consumes. The previous workflow omitted
that variable and attempted the private release path, failing with HTTP
401. On the final head, remote CI passes: 1,464 Rust unit tests, the
integration targets and doctests pass (3 unit tests ignored).
AWS validation for these fixes: both review reproducers failed before
their fixes and pass afterward; 55 merge tests, 29 cast tests and the
standalone 7-case admission regression pass. Python 3.10.21 imports
successfully; 90 MV/namespace tests, 14 JSON tests and four real
namespace dedup cases pass. After `794b97a0`, the native binding was
rebuilt again and the JSON suite rerun. Rust formatting/Clippy, Ruff and
Pyright pass. Sophon's locked test compilation passes against the final
SDK source for all seven affected crates. Extended runners, logs and
benchmark artifacts remain local.
Gatekeeper recommends approval on `794b97a0`, and the `breaking-change`
label is set. Final CI: **49 checks passed, 3 expected skips, no
failures or pending checks**, including remote tests, Python 3.10,
minimum dependencies and Windows ARM Rust tests. All wheel builds pass
after retrying the Linux matrix affected by transient AlmaLinux
repository-metadata 404s. All code-review threads are resolved. The PR
remains open for maintainer review and merge.
---------
Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
Co-authored-by: Lance Release <lance-dev@lancedb.com>
Update the Rust workspace Lance dependencies and Java lance-core from
v13.0.0-beta.12 to
[v13.0.0-beta.14](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.14),
including the refreshed Cargo lockfile.
No compatibility fixes were required; `cargo clippy --quiet --workspace
--tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and
`git diff --check` passed.
Update the Rust workspace Lance dependencies and Java lance-core from
v13.0.0-beta.8 to
[v13.0.0-beta.12](https://github.com/lance-format/lance/releases/tag/v13.0.0-beta.12),
refreshing Cargo.lock.
No compatibility fixes were required; validation passed with `cargo
clippy --quiet --workspace --tests --all-features -- -D warnings`,
`cargo fmt --all --quiet`, and `git diff --check`.
---------
Co-authored-by: Lu Qiu <luqiujob@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Gatefixer
LanceDB RobotandDaniel Rammer
Lance Release
Colin Patrick McCabe
Will JonesandClaude Opus 5.5
Jack Ye
dependabot[bot]
Xuanwo
