feat: expose Lance metrics via OpenTelemetry in Python and Node (#3609)

Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.

## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.

## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.

## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.

Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.

## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Will Jones
2026-07-09 15:36:03 -07:00
committed by GitHub
parent 22bf091de1
commit 285add40dd
23 changed files with 1658 additions and 21 deletions
+5
View File
@@ -47,6 +47,10 @@ repository = "https://github.com/lancedb/lancedb"
pylance = [
"pylance>=5.0.0b5",
]
# A library only needs the OpenTelemetry API; the application supplies and
# configures the SDK (the actual exporter/reader). See
# https://opentelemetry.io/docs/languages/python/instrumentation/
otel = ["opentelemetry-api"]
tests = [
"aiohttp>=3.9.0",
"boto3>=1.28.57",
@@ -61,6 +65,7 @@ tests = [
"pylance>=5.0.0b5",
"requests>=2.31.0",
"datafusion>=52,<53",
"opentelemetry-sdk>=1.30.0",
]
dev = [
"ruff>=0.3.0",