Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.
## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.
## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.
## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.
Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.
## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps Lance to v9.0.0-beta.19, which includes lance-format/lance#7687
for side-effect-free DirectoryNamespace read paths.
This fixes root-level read-only table opens that previously could
trigger `__manifest` creation through directory namespace construction,
including Hugging Face bucket reads with read-only tokens. A LanceDB
regression test now covers root listing operations without creating
`__manifest`.
Fixes#3633.
Updates LanceDB's Lance dependencies to v9.0.0-beta.18.\n\nThis
refreshes the Rust workspace lockfile and Java lance-core version using
the repository update script. Triggering Lance tag:
https://github.com/lancedb/lance/releases/tag/v9.0.0-beta.18
lancedb's public API forces downstream crates to construct foreign types
— `RecordBatch`/arrays/builders for `Table::add(...)` (arrow), and
`datafusion_expr::Expr` for `only_if_expr`/`expr_projection`/merge
filters. The required version must exactly match lancedb's internal
arrow/datafusion line, but nothing on the API surface makes that
visible. Drift surfaces only as confusing trait/type errors:
```text
error[E0277]: the trait bound `RecordBatch: Scannable` is not satisfied
= note: there are multiple different versions of crate `arrow_array` in the dependency graph
```
This re-exports the crates lancedb already pins, so consumers can rely
on a single, guaranteed-matching line via a discoverable import path
instead of declaring their own (potentially mismatched) direct
dependency.
- `lancedb::arrow::{arrow, arrow_array, arrow_buffer, arrow_cast,
arrow_data, arrow_ipc, arrow_ord, arrow_schema, arrow_select}` —
previously only `arrow_schema` was re-exported. `arrow-buffer` is
promoted from a transitive to a direct dependency.
- `lancedb::datafusion` — `Expr` is a first-class part of the query and
merge APIs (`only_if_expr`, `expr_projection`,
`QueryFilter::Datafusion`, `when_matched_update_all_expr`), and
`ExecutionPlan` is returned from `create_plan`.
This follows DataFusion's own precedent of re-exporting `arrow`. The
coupling already exists via the trait/impl bounds — this surfaces it
rather than hiding it behind an `E0277`.
Closes#3575🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updates Lance Rust workspace dependencies and Java lance-core to
v9.0.0-beta.10.
No compatibility code changes were required; clippy and rustfmt passed
after installing the missing runner components.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.10
Updates LanceDB's Lance dependencies to v9.0.0-beta.2 across the Rust
workspace and Java lance-core dependency.\n\nNo compatibility fixes were
required; clippy and formatting pass after installing the missing
toolchain components on the runner. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.2
Updates LanceDB's Lance dependencies from v8.0.0-beta.17 to
v8.0.0-beta.19.
This includes the Rust workspace Lance crates, Cargo.lock refresh, and
Java lance-core version bump. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.19
Updates the Lance Rust workspace dependencies and Java lance-core
dependency to v8.0.0-beta.17.
No LanceDB compatibility code changes were required; validation passed
with cargo clippy and cargo fmt. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.17
Updates LanceDB's Lance dependencies to v8.0.0-beta.14.\n\nThis
refreshes the Rust workspace lockfile and Java lance-core version; no
compatibility code changes were required. Triggering Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.14
Updates Lance dependencies to v8.0.0-beta.11 and refreshes the Rust and
Java lock/config files. This also adapts namespace external manifest
store call sites to the new table-root-aware constructor required by
Lance. Triggering tag:
https://github.com/lancedb/lance/releases/tag/v8.0.0-beta.11
Updates LanceDB Lance dependencies from v8.0.0-beta.5 to v8.0.0-beta.6
and refreshes Cargo metadata.
No compatibility fixes were required; Java lance-core was bumped to
8.0.0-beta.6 as well.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.6
Updates Lance dependencies from v8.0.0-beta.4 to v8.0.0-beta.5 across
the Rust workspace and Java lance-core version.
No compatibility code changes were required; clippy and rustfmt pass
after installing the missing runner components.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.5
Updates Lance dependencies to v8.0.0-beta.2 across the Rust workspace
and Java lance-core metadata.
The update was generated with ci/update_lance_dependency.py and required
no compatibility code changes.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.2
## ⛔ Merge blocker: legal review required
This bump pulls in a new transitive **dev/profiling** dependency chain
`inferno v0.11.21` → `pprof v0.15.0` → `lance-testing`, and `inferno` is
licensed **CDDL-1.0** (copyleft). To get `cargo-deny` green, `CDDL-1.0`
was added to the `deny.toml` allow list.
**Do not merge until legal has reviewed and signed off on allowing
CDDL-1.0.** The dependency is dev/test-only and not distributed, but the
allow-list addition still requires legal approval per our policy.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This updates the workspace Lance dependencies from `v7.1.0-beta.4` to
`v7.2.0-beta.3` and refreshes `Cargo.lock`.
The lockfile now points at Lance commit
`7c070f760fa8e24c8015cb2afbd22c5e6b7898e8` and includes the transitive
dependency updates required by the new beta.
Update the Rust workspace Lance git dependencies and Java lance-core
dependency to v7.2.0-beta.1.
This keeps LanceDB aligned with the latest Lance beta release and
refreshes the Cargo lockfile for the new Lance dependency graph.
## Summary
- Bump lance dependency from `v7.0.0-beta.13` to `v7.0.0-rc.1`
- Remove PK constraint from `LsmWriteSpec::Bucket` docs and
`Table::set_lsm_write_spec` docs
- Remove test assertions that expected rejection when no PK is set or
when bucket column != PK
Closes https://github.com/lance-format/lance/issues/6917
## Summary
Split out from #3354
Adds `LsmWriteSpec` and `Table::set_lsm_write_spec` /
`unset_lsm_write_spec` to
install and clear the spec that selects Lance's MemWAL LSM-style write
path for
`merge_insert`.
`LsmWriteSpec` offers three sharding strategies, all built on Lance's
`InitializeMemWalBuilder`:
- `LsmWriteSpec::bucket(column, num_buckets)` — hash-bucket sharding by
the
single-column unenforced primary key.
- `LsmWriteSpec::identity(column)` — identity sharding by the raw value
of a
scalar column.
- `LsmWriteSpec::unsharded()` — a single MemWAL shard.
Each can be refined with `with_maintained_indexes(...)` (indexes the
MemWAL
keeps up to date as rows are appended) and
`with_writer_config_defaults(...)`
(default `ShardWriter` configuration recorded in the MemWAL index, so
every
writer starts from the same defaults). All variants require the table to
have
an unenforced primary key.
- `set_lsm_write_spec` installs the spec by initializing the MemWAL
index;
`unset_lsm_write_spec` removes it (dropping the MemWAL index), reverting
to
the standard `merge_insert` path. `unset` is idempotent.
- Bindings: Python (`LsmWriteSpec.bucket` / `.identity` / `.unsharded`,
`set_lsm_write_spec` / `unset_lsm_write_spec`) and TypeScript
(`setLsmWriteSpec` with `specType` `"bucket"` / `"identity"` /
`"unsharded"`). `RemoteTable` returns `NotSupported`.
The actual `merge_insert` LSM dispatch and `ShardWriter` write path are
a
follow-up — this PR only installs and clears the spec.
Adds `.github/dependabot.yml` enabling weekly cargo update PRs for the
root workspace, which produces the Rust binaries we ship: the Node.js
and Python native extensions. The `rust/lancedb` library crate shares
the same lockfile — its consumers pick versions themselves, but bumping
transitive deps here keeps the shipped binaries current.
Also removes the misleading `exclude = ["python"]` line from the root
`Cargo.toml`: `python` is listed in `members`, and `cargo metadata`
confirms it's a workspace member, so the exclude was dead code that
implied the opposite.
Minor/patch updates are grouped to reduce PR noise.
Part of #3292. Only covers the cargo ecosystem; pip, npm, and
github-actions can follow.
1. Refactored every client (Rust core, Python, Node/TypeScript) so
“namespace” usage is explicit: code now keeps namespace paths
(namespace_path) separate from namespace clients (namespace_client).
Connections propagate the client, table creation routes through it, and
managed versioning defaults are resolved from namespace metadata. Python
gained LanceNamespaceDBConnection/async counterparts, and the
namespace-focused tests were rewritten to match the clarified API
surface.
2. Synchronized the workspace with Lance 5.0.0-beta.3 (see
https://github.com/lance-format/lance/pull/6186 for the upstream
namespace refactor), updating Cargo/uv lockfiles and ensuring all
bindings align with the new namespace semantics.
3. Added a namespace-backed code path to lancedb.connect() via new
keyword arguments (namespace_client_impl, namespace_client_properties,
plus the existing pushdown-ops flag). When those kwargs are supplied,
connect() delegates to connect_namespace, so users can opt into
namespace clients without changing APIs. (The async helper will gain
parity in a later change)
Bumps all lance-* workspace dependencies from `4.0.0-rc.3` (git source)
to the stable `4.0.0` release on crates.io, removing the `git`/`tag`
overrides.
No code changes were required — compiles and passes clippy cleanly.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Update all 14 lance crates from `3.0.0-rc.3` (git source) to `3.0.0`
(crates.io release)
- Remove git/tag source references since 3.0.0 is published on crates.io
## Test plan
- [x] `cargo check --features remote --tests --examples` passes
- [x] `cargo clippy --features remote --tests --examples` passes
- [ ] CI passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This PR migrates all Rust crates in the workspace to Rust 2024 edition
and addresses the resulting compatibility updates. It also fixes all
clippy warnings surfaced by the workspace checks so the codebase remains
warning-free under the current lint configuration.
Context:
- Scope: workspace edition bump (`2021` -> `2024`) plus follow-up
refactors required by new edition and clippy rules.
- Validation: `cargo fmt --all` and `cargo clippy --quiet --features
remote --tests --examples -- -D warnings` both pass.
Will Jones
Xuanwo
LanceDB Robot
Jack Ye
Heng Ge
Weston Pace