Upgrade the Rust Lance dependency, Java lance-core, and Python test
dependency from beta.17 to the officially published
[v15.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.18)
(`079fd77683a7acf056b0912f6ecda938e39ee120`). This provides a compatible
SDK for the newer Lance source already adopted downstream. Rust registry
versions and checksums are unchanged; the Python lock uses the six
published beta.18 wheels.
### Compatibility
The adopted release includes [Lance
#10863](https://github.com/lance-format/lance/pull/10863):
`FilteredReadOptionsProto.has_fragment_ids` distinguishes an explicit
empty fragment selection. Older executors ignore this field, so upgrade
executors before relying on the corrected empty-selection behavior. Rust
callers constructing this generated struct need the new field. [Lance
#10112](https://github.com/lance-format/lance/pull/10112) also replaces
the internal `hamming_distance_batch` API with
`hamming_distance_batch_into`.
The Linux Python matrix timeout is raised from 30 to 60 minutes: the
Python 3.13 cold native build and both test passes completed, but the
job hit its limit during cleanup. Build profiles, test commands, and
coverage are unchanged.
### Validation
- Three native FRI/ANN regressions passed: deferred-compaction refined
RQ, deferred index remapping, and approximate-query builder.
- `cargo clippy --quiet --locked --features remote --tests --examples --
-D warnings` passed with pinned Rust 1.97.0.
- Locked metadata, formatting, Python Makefile-scope Ruff checks, and
`uv lock --check --offline` passed.
- YAML, whitespace, end-of-file, and typo checks passed for the
timeout-only workflow change.
### Publication and CI
Java beta.18 is now published. After [Lance
#10968](https://github.com/lance-format/lance/pull/10968) merged, the
[normal publication
workflow](https://github.com/lance-format/lance/actions/runs/43146352505)
built all three JNI targets and published successfully; the Maven jar is
available. The upstream Python publication run was cancelled after its
Windows ARM job hit the one-hour timeout; six other wheels were
published and are the actual artifacts recorded in this lock.
Cross-platform SDK CI is still required. No performance claim is made by
this dependency update.
Update the Rust workspace, Java dependency, and Python test dependency
to [Lance
v15.0.0-beta.17](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.17).
This includes the FRI cache and vector-index changes required by current
Sophon and retains the RaBitQ prewarm fixes. Lockfiles use the published
source and test wheels without unrelated registry version bumps.
The dependency dry run also exposed CI resource limits. Use the existing
larger x64 runner for GNU Node and compatibility-wheel builds; allow 60
minutes for Mac Python tests, 120 minutes for the compatibility wheel,
and 150 minutes for the Mac release wheel. Disable the CLI toolchain
action's automatic target cache so only the subsequent downloads-only
cache runs. Release optimization, the compatibility wheel's manylinux
container and x86-64-v2 baseline, and test coverage stay unchanged.
Validation:
- Three native SDK regressions pass: refined RQ queries after deferred
compaction, deferred index remapping, and ANN approximation planning.
- Pinned Rust 1.97.0: `cargo clippy --quiet --locked --features remote
--tests --examples -- -D warnings` passes.
- Locked offline Rust metadata, Python lock validation, Ruff
lint/format, and diff checks pass. The Python lock contains six
published beta.17 wheels.
- Repository YAML, whitespace and typo checks pass for the workflow
changes. Product-source checks are reused for the subsequent CI-only
commits.
- At head `07033dc6`, all 65 applicable pre-review CI checks pass,
including the compatibility wheel, Mac CLI, GNU Node, Mac Python tests,
release wheels, and Java checks. Six checks are skipped by tag/main-only
conditions or draft-only app behavior; the automated review and security
checks run after the Ready transition.
The upstream Java publisher succeeded and its beta.17 Maven artifact is
available. Six upstream Python wheel builds/uploads succeeded; its
Windows ARM job was cancelled. These checks do not establish Sophon
performance or aggregate-memory acceptance.
Update the Rust workspace, Java lance-core, and Python test dependency
to [Lance
v15.0.0-beta.16](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.16).
Synchronize `python/uv.lock` with the published beta.16 wheels for
Linux, macOS, and Windows.
Run the native Linux x64 wheel build on an eight-core runner, matching
the existing ARM build allocation, after its optimized build repeatedly
exceeded the job deadline on the standard runner.
The dependency update passed Clippy, formatting, and CI-script checks;
the Python lockfile refresh/check passes. Fresh CI is running for the
runner adjustment. Java dependency-resolution failures caused solely by
publication lag are accepted for this upgrade.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
Update Rust and Java Lance dependencies to
[v15.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.15),
pin Python tests to published pylance 15.0.0b15, and refresh both
lockfiles; no compatibility fixes were needed.
Validation passed: workspace Clippy with all features/tests and warnings
denied, cargo fmt, uv lock --check, and all 35 CI-script tests.
The [upstream wheel
workflow](https://github.com/lance-format/lance/actions/runs/42669437482)
canceled Windows ARM64 and manylinux_2_28 x86_64 builds after an hour,
so the lockfile contains the five published variants (Linux x86_64
remains covered by manylinux2014); Windows ARM64 is unavailable until
upstream publishes its wheel and the lockfile is refreshed.
Update the Rust workspace Lance crates and Python test dependency to
[Lance
v15.0.0-beta.14](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.14),
which includes JSON cache reuse across request stores and stable-row-ID
JSON index support. The Python lock contains all seven published wheels.
The independent Java module retains beta.8, as in the previous beta.13
update; its separate beta.14 publisher repair is outside this Rust
consumer upgrade.
Preserve all CI coverage while reclaiming completed workspace lancedb
artifacts before the package-level feature-test build, which otherwise
exhausted the Linux runner disk. Give only the Linux ARM64 release-wheel
job two hours after its one-hour timeout, retaining the same runner,
fat-LTO release profile and build flags.
Validation: 2,280 Rust tests passed (79 existing ignores), strict
workspace/all-feature Clippy, Rust formatting, and remote example checks
passed on unchanged Rust source and dependency lock. Python dependency
guards, lock validation, locked sync, and the published macOS wheel
import passed. The new Windows ARM64 wheel was downloaded and its
SHA-256 verified. Workflow YAML/shell checks and independent review
passed. Root Python Ruff/format checks still fail on unchanged
documentation examples; no suppressions or unrelated cleanup were added.
The CI repairs require a complete new run before readiness. This
dependency update does not establish corrected Sophon performance;
downstream native acceptance and the paired S3 replay remain separate.
Update the Rust workspace Lance dependencies to
[v15.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.13),
synchronize the Python test dependency to published pylance 15.0.0b13,
and refresh both lockfiles; no compatibility fixes were needed.
The Java SDK stays on `lance-core` 15.0.0-beta.8, as in the beta.9 and
beta.10 updates: Maven Central has no `lance-core` artifact for
15.0.0-beta.9 through beta.13.
Validation passed: workspace Clippy with all features and warnings
denied, Cargo formatting, Python lockfile checks, and all 35 CI-script
tests. Repository-wide Ruff reports 163 diagnostics, all verified
identical on unchanged HEAD; unrelated formatter changes were reverted.
---------
Co-authored-by: Vivek <vbharathan2008@gmail.com>
Lance beta.11 calibrates lower `search_effort` tiers by metric and
requested `k`. With the Rust dependency and Python/Node execution tests
already on main via #4881, document the tiers across Rust, Python, Node
and the search guide, extend the remote request matrix to cover `0.1`
and `0.25`, and align the Python test package with beta.11.
`0.1` and `0.25` target empirical mean partition-routing recall of 80%
and 90% for L2, cosine and dot through `k=100000`. These are not
per-query or final Recall@k guarantees. Default Auto is unchanged;
Hamming and larger `k` remain uncalibrated, and remote queries require
corresponding server support. This PR makes no additional performance
claim. See [Lance
#10695](https://github.com/lance-format/lance/pull/10695) for
measurements and limitations.
Validation: both strict workspace Clippy configurations, Rust
formatting, 13 Rust effort tests, 208 Python query tests, 41 Node query
tests, native binding builds, TypeScript, Node lint/generated docs,
Python lint/format, full lockfile check and two dependency guards passed
on `a8b09170`. All validated Rust/Python/Node source and dependency
files are byte-identical at `2a82d07c`; the only subsequent change
restores the Java POM to main. Root Ruff has 59 existing diagnostics and
six formatting files, all byte-identical to the base; the Python package
passes.
The Java REST/Namespace client has no changed API here, and its
dependency remains identical to main. The separate [Java beta
publication failure](https://github.com/lance-format/lance/issues/10773)
is not claimed resolved or validated. Official SDK release, automatic
mirror, Sophon adoption and measured consumer acceptance remain
follow-up stages.
## Summary
Bumps dependencies that Aikido flags with critical advisories, where a
patched version is available without code changes.
| Package | Where | From → To | Ships to users |
|---|---|---|---|
| jackson (BOM) | `java/pom.xml` | 2.18.11 → 2.18.12 | Yes, via the
published pom |
| zerovec / zerovec-derive | `Cargo.lock` | 0.11.6 → 0.11.8 / 0.11.3 →
0.11.7 | Yes, linked into the wheels and Node binaries |
| pillow | `python/uv.lock` | 12.3.0 → 13.0.0 | No, dev lockfile |
| pyarrow | `python/uv.lock` | 24.0.0 → 26.0.1 | No, dev lockfile |
| pyjwt, urllib3, anyio, cryptography | `python/uv.lock` | → 2.15.1,
2.8.0, 4.16.0, 50.0.2 | No, dev lockfile |
| pillow | `docs/web-tests/blob-v2/py/requirements.txt` | 12.1.1 →
13.0.0 | No |
pyarrow was held at 24.0.0 by a `pyarrow<25` pin in the `tests` extra.
It came in with #3675 without a stated reason, so this PR removes it.
pillow 13 and pyarrow 26 require Python 3.11+, so the lockfile keeps
pillow 12.3.0 and pyarrow 25.0.1 on Python 3.10, and those remain
flagged.
The sharp findings in the Node examples and docs are not addressed here.
They come from the `@huggingface/transformers@3.0.2` optional
dependency, which needs a major upgrade to 4.3+.
## Evidence
With the pyarrow pin removed and pyarrow 26.0.1 installed, the Python
suite (`-m "not slow and not s3_test"`) passed locally on Python 3.12:
3319 passed, 164 skipped. That run was on main as of #4879; this branch
is based on a slightly newer main.
## Merge Risk
**Door:** Two-way. Patch and minor version bumps that a revert undoes.
**Blast Radius:** Java consumers (jackson patch release), native
binaries (zerovec patch release), Python test environments.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Update the Rust workspace Lance dependencies to
[v15.0.0-beta.10](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.10),
which contains the released MinHash signature scanning API needed by
downstream consumers. Pin Python tests to the published `15.0.0b10`
wheels, including macOS, and refresh both lockfiles. Java remains on
beta.8.
No SDK source compatibility changes were needed. All 25 Lance Git
packages resolve to release commit
`f56a08d40b5630ce1124535c4c3b1406d5ceedf1`; third-party package versions
are unchanged.
Validation on macOS ARM64 with pinned Rust 1.97.0: strict workspace
Clippy with default and all features, Rust formatting, 32 CI-script
tests, three count tests, one WAL/vector regression, and the Python
lockfile check passed. The native test binary emitted a macOS
compact-unwind size warning; both test commands completed successfully
without changing linker settings. Downstream release and integrated
Sophon acceptance remain separate.
Update Rust Lance dependencies to
[v15.0.0-beta.9](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.9),
including the scoped stable-row-ID conversion fix. Python tests use
published beta.9 wheels on Linux/Windows while macOS retains beta.4.
Java stays on beta.8, matching the original base, so this update does
not depend on beta.9 Maven publication or macOS Java packaging.
The dependency updater preserves workspace inheritance, the WAL test
wrapper uses the new StoredBatch module, and Linux/remote Rust CI uses
its installed mold linker. The Python lockfile also records the
already-declared OCI extra. CLI CI now caches Cargo downloads under a
separate key instead of restoring compiled targets: the previous runner
exhausted its disk during cache restoration, before compilation. Its
test matrix and commands are unchanged.
This branch includes #4820's row-freshness changes: successful rows
become current while skipped rows retain their values and remain
eligible for retry. Repair identities, replay handling and Restore
fencing protect publication completion.
BREAKING CHANGE: `StalenessPlan::row_dirty` is a new public field. Rust
callers using exhaustive struct literals must initialize it.
Validation:
- All reported non-Java checks passed on
`f5fa66fcbcf31a8eb188935f728735560c77c873`, including Rust
lint/Linux/remote, Python, NodeJS and CLI checks. Current head
`c660bc0ac9fd5920ad4b64be9c19f732ef4c56a5` preserves those product
sources, manifests, lockfiles and compiler/test commands; only the Java
pin restoration and CLI cache configuration differ. Current-head CI is
tracked separately.
- The complete Java directory is identical to the original PR base. The
scope correction reuses [Xuanwo's prepared and validated
commit](https://github.com/lancedb/lancedb/pull/4814#issuecomment-6626088466);
no new Java build was run.
- CLI workflow YAML parses successfully, its inputs match the pinned
rust-cache action schema, and the existing jobs, matrix and commands are
unchanged. Hosted execution must confirm the cache-restoration repair.
- Existing validation includes the updater regressions and 32 CI-script
tests, dependency-policy checks, count/WAL coverage, and #4820's 42
freshness tests, strict core Clippy and Rust doctest. Root Ruff
previously reported unchanged documentation-example issues;
Python-package Ruff passed.
This PR adopts the existing released Lance beta.9. A subsequent official
SDK release and normal Sophon dependency adoption remain separate
delivery steps.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
## Problem
The Python test extra pins `pylance==15.0.0b7` for every platform. On macOS, pip cannot resolve that pin: the configured `lance-format` index publishes b7 wheels for Linux and Windows but not macOS, while public PyPI returns HTTP 404 for the version and the `lancedb` index contains no 15.x release. The macOS Python dependency installation consequently cannot resolve the pin.
## Change
- Keep the exact `pylance==15.0.0b7` test pin on non-macOS platforms.
- Use `pylance==15.0.0b4` on macOS. This is the newest release in this range with a macOS wheel (`cp310-abi3-macosx_11_0_arm64`).
- Refresh `python/uv.lock` for the platform-specific resolutions.
The production Rust/Lance inputs are unchanged: `Cargo.lock` is byte-identical to `main` and remains on Lance `15.0.0-beta.7`. No tests, assertions, coverage, or CI checks are removed or weakened.
## Validation
- Confirmed the wheel identity from the configured index and matched its published SHA-256 to `uv.lock`; wheel metadata reports `Name: pylance`, `Version: 15.0.0b4`.
- Verified the b4 APIs used by the Python tests (`blob_field`, `blob_array`, `BlobType`, `Blob`, `dataset`, `write_dataset`, `FFILanceTableProvider`, and `DirectoryNamespace`). The Python test suite does not reference the b7-only `mark_sstables_as_compacted`, `mem_wal_dir_id`, `mem_wal_index_details`, `clustering_id`, `LanceFragment`, `IndexSegment`, `UpdateConfig`, or `max_source_fragments` changes.
- Verified scalar read/write interoperability in both directions between b4 and the beta.7 LanceDB native binding, plus beta.7 reads of a b4-written blob dataset with byte and null fidelity.
- `uv lock --check` with both configured Fury indexes and `unsafe-best-match`.
- CI dependency tests.
- Ruff 0.9.9 format and lint checks.
- Targeted Python tests: 911 passed, 22 skipped.
- Full Python suite (`-m "not slow and not s3_test"`): 2858 passed, 157 skipped, 70 deselected.
Updates Rust workspace and Java Lance dependencies to
[v15.0.0-beta.7](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.7),
and pins Python tests to the published `pylance==15.0.0b7` release with
refreshed Cargo and uv lockfiles. No compatibility fixes were required.
Validation: workspace Clippy with all features and warnings denied,
Cargo formatting, uv lock consistency, and all 18 CI-script tests pass;
repository-wide Ruff diagnostics are pre-existing.
---------
Co-authored-by: Lu Qiu <lu@lancedb.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Updates Rust and Java Lance dependencies to
[v15.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.6),
pins Python tests to published pylance 15.0.0b6, and refreshes both
lockfiles.
Aligns Arrow 59, DataFusion 55, and PyO3 0.29; adapts execution plans
and MemWAL directory handling, updates the ABI metadata test, and
removes the obsolete Lance Arrow patch now fixed upstream.
Validation: workspace clippy with all features/tests and warnings
denied, cargo fmt, Python/CI Ruff checks, uv lock check, 16 CI-script
tests, and targeted Python regressions (54 passed, 28 skipped);
repository-wide Ruff retains existing documentation-example failures,
and the upstream macOS wheel build failed while Linux and Windows wheels
are published.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Upgrade Lance to 15.0.0-beta.3 so `Dataset::concat_data_file_parts`
returns a `Send` future and downstream workers can run it on Tokio. The
upstream fix is https://github.com/lance-format/lance/pull/10370.
Keep the vendored Arrow validity fix; upstream Arrow sources are
unchanged between beta.2 and beta.3.
**Blocked on the corrected Lance release:**
https://github.com/lance-format/lance/pull/10370 restores the `Send`
contract required by downstream Function workers. Refresh this upgrade
with that release before merging.
Upgrade the Rust, Java, and Python Lance dependencies to
https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.2. This
brings in bounded vector-index consolidation without retraining
(https://github.com/lance-format/lance/pull/10335) and deletion
filtering for index entries whose fragment coverage was pruned
(https://github.com/lance-format/lance/pull/10325).
Synchronize the vendored `lance-arrow` with the release and retain its
offset-safe validity patch. The existing `nullable_struct_merge`
regression still panics with the unpatched release, so removing that
patch would regress nullable nested queries. Refresh both lockfiles,
including the `zstd-sys` version required by the new Lance encoding
dependency.
Update the custom memtable bitmap test adapter to train on row
addresses, matching the new bitmap contract; remove its obsolete row-ID
conversion.
Preserve the Blob APIs' existing missing-row error messages when newer
Lance rejects a missing fragment earlier in the read path. Retain
upstream attribution in vendored sources while satisfying the repository
license-header check.
Validation of the compatibility fixes: all 58 Rust Blob integration
tests and 219 Python Blob tests pass (one Python test skipped). Strict
workspace Clippy, formatting, and diff checks pass.
Fixes#4248.
The preview index still contains pylance 1.0.0b6. uv's default
first-index resolution can select it, or backtrack to an old LanceDB
release when installing the extra. Meanwhile, the extra still permits
pylance 5 even though current main bundles Lance 14.
Raise the optional dependency to `pylance>=14.0.0b12`, the version
already pinned in the test extra, and keep the lockfile metadata
consistent. Document both Fury indexes and uv's `unsafe-best-match`
strategy in the Python README and the source-owned quickstart
(`docs/web/quickstart.mdx`). Explain the index trust requirement and
recommend installing the extra to enforce the version floor.
Add CI checks that reject older Lance majors and ensure the pylance and
tests extras remain compatible. Run these checks when either dependency
manifest changes, so a future Lance major bump cannot silently leave the
Python minimum behind.
### Validation
- Recreated the reported install on Ubuntu 26.04 / CPython 3.12.15 with
uv 0.12.23: `lancedb==0.40.0b5` plus the old index selects
`pylance==1.0.0b6`. Reproduced both the missing `blob_array` import and
the unexpected `blob_handling` scanner argument using a real blob table.
- Resolved the modified `python/pyproject.toml` with the pylance extra
against both indexes: selects `pylance==14.0.0b18`, matching the current
Rust pin.
- Ran both documented pip and uv installations in isolated environments:
`lancedb==0.41.0b7`, `pylance==14.0.0b18`; blob-array construction and a
blob-bytes round trip through `to_pandas(blob_mode="bytes")` pass. Both
package dependency checks pass.
- New regression fails on the original dependency bound (27
rejected-version cases incorrectly accepted), then passes with the fix.
Full CI-script suite: 3 tests pass.
- `uv lock --check` with both indexes passes; focused Ruff checks pass;
`git diff --check` passes.
- Repository-wide Ruff reports 49 existing errors in untouched
`docs/web-tests` files. Unrelated formatting changes were discarded.
Runtime smoke checks used released native wheels; source dependency
resolution was tested directly from the modified pyproject. No Rust or
Python runtime code changes.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
Cargo.lock: webpki-root-certs 1.0.10; webpki-roots 1.0.10 and
rustls-webpki 0.103.16 are already current.
docs/web-tests/rs/Cargo.lock: rustls-webpki 0.103.16. python/uv.lock:
certifi 2026.10.19, the only entry that changes.
The root crate drops the root CA whose intermediates were compromised
(Mozilla bug 2083449), and rustls-webpki 0.103.16 rejects chains that
carry the compromised eMudhra intermediate keys. certifi 2026.10.19 is
the release the certifi advisory names as fixed: it drops the same root
and keeps the issuer's other roots, so HTTPS connections that chain to
them keep working. Relative to the previous 2026.6.17 pin it also adds
Telia EC and RSA TLS Root CA v3 and SECOM TLS ECC and RSA Root CA 2024
and drops Atos TrustedRoot 2011.
The webpki-root-certs entry is written by hand with the crates.io
checksum.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Esteban Gutierrez <esteban@lancedb.com>
Co-authored-by: Claude Opus 5.1 <noreply@anthropic.com>
Lance supports `search_effort`, but LanceDB cannot pass it through its
query APIs. Expose the parameter in Rust, Python sync/async and
TypeScript, including remote requests and hybrid query conversion, using
the [Lance
v14.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.13)
dependency already on main.
The default is `None` (Auto, internally 0.5). Any explicit effort,
including 0.5, overrides fixed `nprobes` in either setter order;
explicit minimum/maximum bounds remain effective. Invalid values are
rejected in the Rust core. Hybrid conversion now retains the vector
probe parameters and handles an inferred FTS column without panicking.
Namespace queries with explicit effort execute locally because the
namespace wire model cannot carry it. MemWAL LSM queries reject it
explicitly because Lance's LSM scanner does not yet expose this option.
Maturin is included in the dev extra so the documented build command
works in the project environment.
BREAKING CHANGE: The public Rust `VectorQueryRequest` struct now
includes `search_effort: Option<f64>`. Downstream code that initializes
every field explicitly must add `search_effort: None` or use
`..Default::default()`. Exhaustive destructuring patterns must include
the new field or `..`. Query-builder calls and struct literals that
already use `..Default::default()` require no source changes. The new
effort setters in Python and TypeScript are additive. This PR does not
change the persisted file format.
Using explicit effort with remote tables requires server support for
`search_effort` and its precedence over `nprobes`; the client wire tests
do not establish that a deployed server supports it. Server-side support
is tracked in lancedb/sophon#8607.
Validation on AWS after merging current main (Lance beta.13 / SDK
beta.5):
- 14 Rust regressions and the API doctest pass.
- 60 parameter regressions, the full 47-test namespace suite, and three
API doctests pass. The parameter selection includes some namespace
cases.
- 40 TypeScript query tests and 54 remote tests pass; build, lint and
generated docs pass.
- Rust Clippy, Python package lint/format, Pyright and lockfile checks
pass. Repository-wide Ruff reports 49 lint errors and 11 formatting
failures in unchanged `docs/web-tests` files; those files are
byte-identical to main and outside this change.
All 1,103 Git-tracked files match the AWS validation checkout. No
benchmark scripts or result artifacts are included, and no performance
improvement is claimed.
Uses lance-format/lance#9798. Server-side support is in
lancedb/sophon#8607.
Keep IVF probe fields unset by default so Lance supplies adaptive
behavior. Preserve `nprobes`, `minimum_nprobes`, and `maximum_nprobes`
independently through Rust, Python, namespace, and remote request
construction, with compatibility fields serialized only when explicitly
configured.
The final Rust namespace and LSM propagation update depends on the Lance
release containing lance-format/lance#9184.
Updates the Rust workspace Lance dependencies and Java lance-core from
14.0.0-beta.1 to
[v14.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.6),
and refreshes Cargo.lock. No compatibility fixes were required.
Validation passed: `cargo clippy --quiet --workspace --tests
--all-features -- -D warnings`, `cargo fmt --all --quiet`, and `git diff
--check`.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Closes#4045
## What changed
- replace the unpublished pylance 9.0.0rc1 test-extra pin with the
published 9.0.0 release
- restore dependency resolution for editable installs using the tests
extra
## Validation
- downloaded pylance==9.0.0 from PyPI with pip --no-deps
- parsed python/pyproject.toml with tomllib and verified the tests extra
- git diff --check
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
LanceDB's Python SDK now requires Pydantic `>=2.7.4,<3` and uses the v2
APIs throughout. This removes dual-version behavior from schema
conversion, query serialization, embedding models, and Function wire
models while preserving their existing public and canonical-wire
behavior.
The minimum-dependencies CI job pins Pydantic 2.7.4 so the declared
compatibility floor remains covered.
## Root cause
`Table.to_polars()` disabled PyArrow predicate pushdown by selecting the
non-PyArrow Polars scan callback. Polars 1.32.3 invokes that callback
with `batch_size` both positionally and through its partial, so
collecting the returned lazy frame raises `TypeError:
_scan_pyarrow_dataset_impl() got multiple values for argument
batch_size`.
## Fix
- Keep the compatible PyArrow callback path.
- Add an identity `map_batches` barrier so predicates stay in Polars
instead of reaching the LanceDB adapter as unsupported PyArrow
expressions.
- Extend the tested Polars range through 1.32.3 and retain lazy-frame
regression coverage.
## Validation
- `python/tests/test_table.py::test_polars` with Polars 1.32.3
- `python/tests/test_table.py::test_polars` with the locked Polars 1.3.0
baseline
- `ruff format --check` on the changed Python files
- `ruff check .`
- `uv lock --check`
Fixes#2619
<!-- lance-gatekeeper-fix:v1 agent=0d42bcda944ac42765b25f2c19ff729f
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
In the current LanceDB usage implementation, there is no way to check
whether a table or namespace already exists. This PR introduces the
namespace_exists and table_exists methods to determine the existence of
tables and namespaces.
useage like this:
```
# check table exists
db.table_exists(table_id=['xxx'])
# check namespace exists
db.namespace_exists(namespace_id=['xxx'])
```
fixes: #3419
---------
Signed-off-by: farmer <farmerchillax@outlook.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
BREAKING CHANGE: splits generated by the permutation data loader will
not be the same, due to a change in hash function.
Updates the Lance dependencies and Java lance-core to
[v9.0.0-rc.1](https://github.com/lance-format/lance/releases/tag/v9.0.0-rc.1).
Includes the required DataFusion 54 and Lance file-reader compatibility
updates.
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds client-side support for analyze_plan distributed metrics modes
across Rust, Python, and TypeScript clients. Defaults to aggregate for
backward compatibility and sends the remote distributed_metrics
parameter only when a non-default mode is requested.
Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.
## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.
## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.
## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.
Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.
## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This follows the Rust-side Tantivy removal by deleting the remaining
Python Tantivy runtime, tests, and packaging references.
It also turns the legacy Python-only Tantivy parameters into explicit
errors and stops reading legacy `_indices/fts` directories so Python FTS
is fully native-only.
1. Refactored every client (Rust core, Python, Node/TypeScript) so
“namespace” usage is explicit: code now keeps namespace paths
(namespace_path) separate from namespace clients (namespace_client).
Connections propagate the client, table creation routes through it, and
managed versioning defaults are resolved from namespace metadata. Python
gained LanceNamespaceDBConnection/async counterparts, and the
namespace-focused tests were rewritten to match the clarified API
surface.
2. Synchronized the workspace with Lance 5.0.0-beta.3 (see
https://github.com/lance-format/lance/pull/6186 for the upstream
namespace refactor), updating Cargo/uv lockfiles and ensuring all
bindings align with the new namespace semantics.
3. Added a namespace-backed code path to lancedb.connect() via new
keyword arguments (namespace_client_impl, namespace_client_properties,
plus the existing pushdown-ops flag). When those kwargs are supplied,
connect() delegates to connect_namespace, so users can opt into
namespace clients without changing APIs. (The async helper will gain
parity in a later change)
Without this fix, if user directly use the native table to do operations
like `add_columns`, even if it is configured to use namespace db
connection, it is not really propagated through.
The fix is to bring lancedb's python binding up to date and do a similar
implementation as https://github.com/lance-format/lance/pull/5968, and
make sure the namespace is fully propagated through all the related
calls.
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Yang Cen
LanceDB RobotandJack Ye
Will JonesandClaude Opus 5.5
KarthikSandXuanwo
Shiduo LiandXuanwo
lancedb-gatefixer[bot]
Farmer.ChillaxandClaude Opus 4.7