`mkdocs build --strict` only catches references it cannot resolve. A bare
anchor such as `[limit][]` or `[vector search][search]` is matched by
autorefs against any heading on the site, so six of them silently linked
into the JavaScript reference instead. The relative links in
`permutation.py` and `remote/errors.py` pointed at in-page anchors and
paths that do not exist.
Targets that still exist here or in an imported inventory now use
mkdocstrings references; the guide pages deleted in #2770 use their
lancedb.com URLs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`mkdocs build` emitted 61 warnings on main, and rendering the previously
undocumented classes in this PR pushed that to 158. That backlog is what
blocks turning on strict mode (#3707), so clear it here rather than leave
it worse than we found it.
Most of it was one systematic false positive: griffe cannot see the
generated `__init__` of a pydantic dataclass, so every documented
parameter looked unknown. `warn_unknown_params` turns that check off.
The rest were real docstring bugs, in 15 docstrings:
* Prose trailing a `Parameters` section is read as parameter names, which
invented parameters called `The`, `you` and `To`. Moved into `Notes` or
the summary.
* numpydoc only reads a type when the colon has spaces around it. Where
the documented name is a pydantic attribute rather than a signature
parameter, griffe has no signature to fall back on and the type was
dropped. Affects nine embedding classes.
* `num_partitions, default sqrt(num_rows)` and friends parse as a list of
names, rendering a bogus `default` parameter.
* One parameter indented five spaces instead of four.
`nodejs/CONTRIBUTING.md` links to the repo-root CONTRIBUTING.md, which
does not resolve once typedoc copies the file into `docs/src/js/_media/`;
an absolute URL works from both places.
`mkdocs build --strict` now exits 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four packages are now rendered by a single mkdocstrings directive each,
driven by the module's `__all__`, instead of a hand-maintained list of
symbols. These were where most of the drift was: 7 of 12 rerankers and
14 of 17 embedding functions had never been listed.
`lancedb.embeddings` had no `__all__`; without one mkdocstrings renders
no members at all for a re-export package, so one is added.
AGENTS.md gains a section describing how the reference page is wired up
and how to check a docs build locally, plus a step in the "adding a new
method on Table" checklist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Python API reference page had drifted from the public API. Branch
management (`Branches` / `AsyncBranches`, which own `diff` and `merge`),
structured full-text query classes, take queries, blob helpers,
namespace connections, most rerankers and embedding functions, the
PyTorch dataloader, and several other public symbols were never listed,
so they did not appear in the rendered docs.
Also fixes docstring cross-references that pointed at guide pages which
have since moved off this site, and at unresolvable relative targets
(`[Table](Table)`, `[PyArrow Table](pyarrow.Table)`).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_read_consistency_interval` asserted that a table opened with a
100ms `read_consistency_interval` still read stale data immediately
after a concurrent write. The cache timestamp is set when the table is
opened and reads within the interval do not refresh it, so that
assertion only held if the intervening open/count/commit/count sequence
finished within 100ms of real wall-clock time. On a loaded CI runner it
did not: the TTL expired, `count_rows` refreshed synchronously, and the
test failed with `left: 1, right: 0`. This broke the Rust workflow on
`main` at 0bc08160 (a Python-only commit).
This pins the `background_cache` mock clock once `table2` has seeded its
cache, and advances it explicitly in place of `tokio::time::sleep`, so
the test controls when the interval elapses. Same approach as #3547.
With the clock pinned there is no real sleep left to be imprecise, so
the `cfg(not(target_os = "windows"))` guard is dropped and the test now
runs on Windows too.
Verified by inserting a stall before the write: 120ms reproduces the
original failure deterministically, and with this change the test still
passes with a 500ms stall.
Fixes#3712
Standard GitHub-hosted runners are free on public repos, so all Actions
spend here is on the `*-8x-*` / `4x` larger runners. Measured over 30
days at current (post-Jan-2026) larger-runner rates, that is ~$1,400/mo,
and `npm-publish` is ~70% of it.
## Changes
**Fat LTO was forcing builds onto large runners.** `[profile.release]`
in `.cargo/config.toml` sets `lto = "fat"` with `codegen-units = 1`,
which is single-threaded and the peak-memory step. The macOS
`npm-publish` build was 111 of its 113 minutes in one `napi build` step,
making it the critical path of the whole publish pipeline. The ThinLTO
override already applied to Windows now covers macOS too, and both
Windows builds move from `windows-2025-8x-x64` to the free standard
`windows-2025`.
**The npm-publish cargo cache never existed.** There are zero caches
with its key prefix. The key was static, so `actions/cache` (which only
writes on a miss) could never refresh it, and a multi-GB release
`target/` per target could never fit the repo's 10 GB budget anyway. Now
caches only the crate registry, keyed on `Cargo.lock`. The docker builds
also mounted `.cargo/registry/*` while the cache saved `.cargo-cache`,
so containers re-downloaded the registry every run.
**Cache eviction thrash.** Repo cache usage is 10.4 GB against GitHub's
10 GB cap, so every PR run evicted main's warm entries. `rust.yml` and
`nodejs.yml` now restore everywhere but only save from `main`.
**npm-publish moves to nightly + tags** instead of every push to main
(~90/month). The cross-compiled targets do need watching, so
`report-failure` now fires on scheduled runs, and dedupes onto an
existing open issue rather than filing one per night.
**rust.yml aarch64-pc-windows-msvc** cross-compiled its tests and then
skipped them, paying full codegen and link cost for a compile check.
`windows-11-arm` is now GA and free on public repos, so it builds and
tests natively. Its test step also passes `--target` — without it cargo
used `target/ci/` rather than `target/<triple>/ci/` and rebuilt the
entire dependency graph a second time.
**pypi-publish.yml had no concurrency group**, so force-pushes left a
~74 minute Windows job running.
## What is cost vs. wall-clock
| Change | Cost | Wall-clock |
|---|---|---|
| Windows npm-publish → free runners | **−$570/mo** | slower per job
(8→4 cores) |
| npm-publish nightly | **−$125/mo** | — |
| pypi-publish concurrency | small | — |
| macOS ThinLTO | $0 (already free) | **−~50 min** per release |
| rust aarch64 Windows native | $0 (already free) | **−~25 min** |
| rust `--target` on test step | $0 | large, avoids a second full build
|
| rust-cache `save-if` | small | faster via real cache hits |
## Risks
- The two Windows builds now have 4 cores instead of 8 and ~14 GB of
free disk. If they fail, it is most likely disk rather than memory;
fallback is `windows-2025-4x-x64`, which still halves that line.
- `windows-11-arm` has a thinner toolset (choco/vcpkg/protoc under
emulation) and this enables a test step that has never run, so it may
surface real aarch64 failures. That is the point, but it is the change
most likely to need iteration.
- ThinLTO applies to published macOS and Windows binaries, typically
within a few percent of fat LTO. Linux release builds are untouched.
## Follow-ups
- `python.yml` `pydantic1x` (37 min) and `Doctest` (33 min) each rebuild
the extension from source via `pip install -e .` with no Rust cache;
they should consume the wheel the `linux` job already builds. Worth
~$235/mo and ~70 min of compute per run. Separate PR.
- The three `ubuntu-2404-8x-x64` npm-publish builds (~$420/mo at the old
cadence) are the remaining large-runner spend;
`aarch64-unknown-linux-gnu` could run natively on free
`ubuntu-24.04-arm`. Worth doing after this lands so the ThinLTO change
can be validated first.
- The wheel composite actions declare `python-minor-version` as required
but never use it, and every caller omits it (actionlint warns).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Python was versioned and tagged separately from the Rust, Java, and
Node.js SDKs, and had drifted three minor versions ahead (0.36 vs 0.33).
Users had no way to tell which Python version corresponded to which Rust
or Node release, and the gap had no meaning behind it.
This unifies the two tracks so there is one version and one tag for all
four SDKs.
## Version
The shared version is set to `0.37.0-beta.0`. Python continues its own
sequence (highest published: 0.36 → 0.37) while Rust, Java, and Node.js
jump 0.33 → 0.37 to meet it. Picking Python's next minor means Python
users see no discontinuity at all, and only the other SDKs skip forward.
Note that `main` trails the `release/v0.32` branch on both lines (main
is at 0.32.0-beta.3 / 0.35.0-beta.3; the release branch carries
0.33.0-beta.0 / 0.36.0-beta.0), so 0.37 is chosen to clear the highest
tag on either branch. Every index stays monotonic:
| index | publishes | last published | next |
|---|---|---|---|
| PyPI | stable only | 0.34.0 | 0.37.0 |
| Fury | previews | 0.36.0b0 | 0.37.0-beta.1 |
| npm | both | 0.33.0-beta.0 | 0.37.0-beta.1 |
| crates.io | stable only | 0.31.0 | 0.37.0 |
| Maven | both | 0.33.0-beta.0 | 0.37.0-beta.1 |
A one-time jump for three SDKs, versus explaining the offset
indefinitely.
## Mechanism
* `python/.bumpversion.toml` is removed. `python/Cargo.toml` — the
source of the Python package version, since `pyproject.toml` declares
`dynamic = ["version"]` — becomes a tracked file of the root config. Its
`cargo update -p lancedb-python` pre-commit hook is dropped as
redundant: `ci/update_lockfiles.sh` already refreshes every workspace
member version in `Cargo.lock`.
* `pypi-publish.yml` triggers on `v*` instead of `python-v*`, so one tag
releases all four packages. `ci/bump_version.sh` and
`make-release-commit.yml` lose their now-dead tag-prefix and
per-language plumbing, including the `python` / `other` dispatch inputs.
* The two byte-identical GH release jobs in `npm-publish.yml` and
`pypi-publish.yml` are replaced by a single `gh-release.yml`. One
release per tag, named `LanceDB vX.Y.Z`, instead of separate "Python
LanceDB" and "Node/Rust LanceDB" releases for the same commit.
The trade-off: there is no longer a way to ship a Python-only patch
without also releasing crates.io, Maven, and npm. That is the cost of
making drift structurally impossible.
## Beta releases marked "Latest" (#3666)
Both GH release jobs used:
```yaml
prerelease: ${{ contains('beta', github.ref) }}
```
The arguments are reversed. `contains(search, item)` asks whether
*`search`* contains *`item`*, so this evaluated "does the literal string
`'beta'` contain `refs/tags/python-v0.35.0-beta.2`?" — always `false`.
Every beta was published as a full release, and GitHub awards "Latest"
to the newest non-prerelease.
The new workflow derives the flag from the parsed version rather than
the raw ref, and sets `make_latest` explicitly:
```yaml
prerelease: ${{ steps.extract_version.outputs.prerelease }}
make_latest: ${{ steps.extract_version.outputs.prerelease == 'false' }}
```
npm was never affected (`--tag preview` uses correct bash), and PyPI
already excludes pre-releases from resolution.
This only fixes releases published from here on. Already-published betas
need a one-time backfill:
```shell
gh api --paginate /repos/lancedb/lancedb/releases \
--jq '.[] | select(.prerelease == false) | select(.tag_name | test("beta")) | .id' \
| xargs -I{} gh api -X PATCH /repos/lancedb/lancedb/releases/{} -F prerelease=true
```
## Verification
Ran `ci/bump_version.sh` end-to-end against this branch with the release
tooling installed:
* `preview` → tags `v0.37.0-beta.1` (previous tag `v0.33.0-beta.0`
detected, `pre_n` bump)
* `stable` → tags `v0.37.0`
* Both paths update `.bumpversion.toml`, `rust/lancedb/Cargo.toml`,
`nodejs/Cargo.toml`, `python/Cargo.toml`, `nodejs/package.json`, the 7
`nodejs/npm/*/package.json` files, both Java poms, and
`docs/src/java/java.md` together
* `check_breaking_changes.py` resolves the last stable as `v0.31.0`, so
the minor-version gate passes
All five touched workflows parse as valid YAML and the pre-commit hooks
pass.
## Notes for review
* This targets `main` only, so it takes effect at the next
release-branch cut. The in-flight `release/v0.32` branch still carries
`v0.33.0-beta.0` / `python-v0.36.0-beta.0`; if we want the imminent
stable to be 0.37.0, this needs to be applied there too.
* Historical `python-v*` tags are left alone. The changelog builder
scans `^v`, which does not match them, so the first unified release's
notes will compute `fromTag` from the Rust/Node line only — a one-time
gap in the Python-side changelog.
* Pre-existing and not addressed here: `ci/update_lockfiles.sh --amend`
amends the commit that `bump-my-version` has already tagged, so the
lockfile update lands outside the tag on stable releases.
Fixes#3666
## Problem
On the remote (LanceDB Cloud) write path, each write partition is
uploaded as a **single** `/insert?upload_id=...` request that stays open
until the whole partition has been streamed and the server has written
it to object storage. For large bulk ingests a partition can be many GB,
so a single request can run longer than the client read timeout (default
300s), surfacing as:
```
lancedb.remote.errors.HttpError: operation timed out
```
The server already supports staging **multiple** parts under one
`upload_id` (each `/insert` writes a separate transaction that
`complete` merges atomically), but the client never used that — it sent
one part per partition.
## Change
Split each partition into multiple parts of at most
`max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded
as its own `/insert?upload_id=...&upload_part_id=...` request. This
bounds how long any single request stays open, independent of total data
size or write parallelism.
Key properties:
- **Still streamed, not buffered.** Each part's body is driven through a
bounded channel while the request is in flight (`futures::join!` of a
producer + the send), so peak memory stays at a couple of batches per
partition regardless of the part size. Backpressure from a
slow/throttled server still propagates upstream.
- **Correct part accounting.** An empty partition still sends exactly
one (schema-only) part so `complete` has a transaction to commit; a size
cut landing exactly on the end of input does not emit a trailing empty
part.
- **Multipart only.** The single-request (non-multipart) path is
unchanged.
## Config
New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable
via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable.
**Default 1 GiB** (`Some(0)` disables splitting → one request per
partition). Python users pick up the default/env automatically through
the remote client.
## Tests
- `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each
batch in its own part → N requests, each carrying the shared `upload_id`
and a distinct `upload_part_id`.
- `test_multipart_single_part_when_under_budget`: a large budget keeps
the partition in a single request.
- Verified end-to-end against a live remote table: a forced-chunked
multipart add (many parts) assembles to the correct row count.
Related to ENT-1883.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
`table.add(dataset)` with a `pyarrow.dataset.Dataset` OOMs the client
during bulk ingestion of wide rows (e.g. embedding columns), even
against a remote table where the upload itself is streaming.
The cause is in `to_scannable`: a `Dataset` is scanned with pyarrow's
default scanner settings (`batch_size=131072` rows,
`batch_readahead=16`, `fragment_readahead=4`). pyarrow's internal
threads prefetch that read-ahead window independently of LanceDB's
backpressure, so for wide rows a large fraction of the dataset is held
in memory. On the remote path this is then multiplied across the
multipart write partitions (one in-flight batch per partition, up to
CPU-core count).
Reproduced on a 10 GB / 1.55M-row dataset with two 768-dim float32
embeddings: peak client RSS ~11.7 GB for the scan alone (6.8 GB after
consuming a *single* batch), ~15.4 GB for the full remote `add()`.
## Fix
`to_scannable` now sizes the scanner from an estimate of bytes-per-row
derived from the schema:
- **Narrow datasets keep pyarrow's defaults** (empty scanner kwargs) —
no throughput regression. The bound only engages above ~410 bytes/row.
- **Wide rows** get a smaller `batch_size` (~16 MiB/batch) and reduced
read-ahead (`batch_readahead=2`, `fragment_readahead=1`) so peak
in-flight memory stays near a ~1 GiB budget. Read-ahead (not just batch
size) has to drop, because pyarrow pins whole row-group buffers.
On the 10 GB dataset this drops peak client RSS to ~1.4 GB, and it stays
flat as the dataset grows. The `Dataset`/`LanceDataset` scannables
remain rescannable (retry-safe).
## Also: expose `write_parallelism` on `add()`
`AddDataBuilder::write_parallelism` already existed in Rust but was not
exposed in Python. This PR forwards it through the async, sync, and
remote `add()` methods, so users can cap the number of parallel write
partitions (each buffers data in flight) to trade throughput for memory
on large uploads.
## Tests
- `test_scannable.py`: bytes-per-row estimation; narrow → defaults; wide
→ bounded; `Dataset` reader streams bounded batches and stays
rescannable.
- `test_table.py`: `write_parallelism` on sync and async `add()`, and
that `write_parallelism=0` is rejected.
Fixes ENT-1883
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`Dataset::index_statistics()` loads index files and does meaningful CPU
work to serialize low-level info. Most fields
`NativeTable::index_stats()` needs are available from manifest metadata
via `Dataset::describe_indices()`, which is much cheaper.
`NativeTable::index_stats()` now:
- Calls `describe_indices()` filtered by name; returns `Ok(None)` if no
match.
- Parses `distance_type` from `description.details()` JSON (the
`VectorIndexDetails` proto stored in the manifest by recent Lance
versions).
- Falls back to `index_statistics()` only for vector indices where
`details()` returns no `distance_type` — this handles older Lance
datasets that didn't write `VectorIndexDetails`.
- `Unknown` index types (e.g. Lance's internal `FragReuseIndex`) are
explicitly filtered out of `list_indices` rather than erroring.
## Test plan
- [x] `test_create_scalar_index` — asserts `index_type`,
`distance_type`, and `num_unindexed_rows > 0` after adding rows
post-index
- [x] `test_create_fm_index`, `test_create_bitmap_index`,
`test_create_label_list_index` — added `index_stats` assertions
- [x] IvfPq, IvfHnswPq, IvfHnswSq, IvfHnswFlat tests assert
`distance_type == Some(L2)`
- [x] `test_list_indices_skip_frag_reuse` — FragReuseIndex is filtered
by the Unknown guard in `list_indices`
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CODEOWNERS file added in #3312 automatically requests reviewers on
every PR — the `*` default owner routes all changes to two reviewers.
This is mostly noise for contributors, and we prefer a single requested
reviewer per PR.
Remove the file.
Reverts #3312.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `build - aarch64-pc-windows-msvc` node build job (and, marginally,
the x86_64 one) had started hitting `rustc-LLVM ERROR: out of memory`
while linking the `lancedb-nodejs` cdylib — most recently surfaced by
#3526, which adds the goosefs backend (and its tonic/prost gRPC subtree)
to the default node binary.
The peak-memory step is the fat-LTO codegen (`lto=fat`,
`codegen-units=1` from `.cargo/config.toml`), which merges the whole
crate graph into a single LLVM module and runs single-threaded. It
therefore neither parallelizes across cores nor fits in the 16 GB of the
standard `windows-latest` runner as the dependency graph grows.
This PR:
- Moves both `*-pc-windows-msvc` node build jobs to
`windows-2025-8x-x64` (more memory + cores).
- Overrides the release profile to ThinLTO for just these jobs, via
`CARGO_PROFILE_RELEASE_LTO=thin` /
`CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16` in `pre_build`. ThinLTO
parallelizes the cross-module optimization across the runner's cores and
keeps peak memory well under the limit. Scoped so Python wheels and Rust
release builds keep fat LTO.
The larger runner alone would clear the OOM but waste the added cores on
the single-threaded fat-LTO tail; ThinLTO is what makes the extra cores
actually reduce wall-clock and gives durable memory headroom for future
dependency growth.
Tradeoff: ThinLTO can leave a small runtime-perf gap vs fat LTO for the
node native binary, but it recovers most of it and is a common release
configuration.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bridges Lance's internal `metrics`-crate instrumentation (object store
request counts, bytes, latency, errors, and throttles) into
OpenTelemetry, in both the Python and Node bindings, with a shared
adapter in the Rust core. This is the LanceDB counterpart to
lance-format/lance#7537.
## Rust core (`rust/lancedb`)
Two new, **off-by-default** features:
- `metrics` — re-exports the [`metrics`](https://docs.rs/metrics) crate
as `lancedb::metrics` and turns on Lance's object-store instrumentation.
Install any `metrics`-compatible recorder to collect them.
- `metrics-otel` — adds `lancedb::metrics_otel`, a pull-based adapter
that installs a process-global recorder aggregating into lock-free
cumulative storage and exposes a snapshot/catalog API
(`register_metrics_recorder`, `metrics_catalog`, `snapshot_metrics`,
`MetricPoint`/`MetricValue`/`MetricKind`/`MetricDescription`). Both
bindings build on this.
## Python
`lancedb.otel.instrument_lancedb_metrics()` registers each metric as an
OpenTelemetry observable instrument on the given (or global)
`MeterProvider`. Available via the `otel` extra (`pip install
lancedb[otel]`), which pulls in only `opentelemetry-api` — the
application supplies and configures the SDK.
## Node
`instrumentLanceDbMetrics()` provides the equivalent wiring against
`@opentelemetry/api`. This is the only public entry point; the
underlying recorder/catalog/snapshot functions stay internal.
Because OpenTelemetry has no asynchronous histogram instrument,
histograms are exported Prometheus-style as `<name>_bucket` (with an
`le` attribute), `<name>_count`, and `<name>_sum`. Only `_sum` carries
the histogram's unit; `_bucket` and `_count` observe cumulative counts
and are unitless. The adapter is enabled by default in the Python and
Node builds, and off by default in the Rust crate.
## Notes
- Requires Lance ≥ `v9.0.0-beta.19`, which ships the object-store
metrics APIs (upstream lance-format/lance#7537, now merged). `main` is
already on beta.19, so this is a single feature commit with no
dependency bump.
- Tests: 8 Rust unit tests, 3 Python tests, 2 Node tests, all covering
the end-to-end object-store-metrics → OpenTelemetry path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BREAKING CHANGE: When passing multiple where clauses to a query, they
now stack instead of replacing the previous filter.
Previously, calling `where`/`only_if` more than once on a query silently
replaced the previous filter, so only the last filter was applied. This
was
surprising and could return rows that an earlier filter should have
excluded.
This implements the alternative suggested in
https://github.com/lancedb/lancedb/pull/3514#issuecomment-4664901580:
instead of
rejecting a second filter, repeated filters are combined with a logical
AND
(`(previous) AND (new)`).
The combination happens in the Rust core (`QueryBase::only_if` and
`only_if_expr`), so it applies to all SDKs at once (Rust, Python async,
and
TypeScript). The Python sync query builder keeps its own filter state,
so it
combines filters in the binding layer as well.
SQL string and expression filters are combined within their own
representation.
When the two representations are mixed, the expression is lowered to SQL
(via
`expr_to_sql_string`) and the filters are combined as SQL strings, so
chaining
`where` works regardless of which form each filter takes.
Fixes#2649
## Tests
- Rust: `cargo test --features remote -p lancedb --lib query`
- Python: `uv run --extra tests pytest python/tests/test_query.py`
- TypeScript: `pnpm test __test__/query.test.ts`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lancedb's public API forces downstream crates to construct foreign types
— `RecordBatch`/arrays/builders for `Table::add(...)` (arrow), and
`datafusion_expr::Expr` for `only_if_expr`/`expr_projection`/merge
filters. The required version must exactly match lancedb's internal
arrow/datafusion line, but nothing on the API surface makes that
visible. Drift surfaces only as confusing trait/type errors:
```text
error[E0277]: the trait bound `RecordBatch: Scannable` is not satisfied
= note: there are multiple different versions of crate `arrow_array` in the dependency graph
```
This re-exports the crates lancedb already pins, so consumers can rely
on a single, guaranteed-matching line via a discoverable import path
instead of declaring their own (potentially mismatched) direct
dependency.
- `lancedb::arrow::{arrow, arrow_array, arrow_buffer, arrow_cast,
arrow_data, arrow_ipc, arrow_ord, arrow_schema, arrow_select}` —
previously only `arrow_schema` was re-exported. `arrow-buffer` is
promoted from a transitive to a direct dependency.
- `lancedb::datafusion` — `Expr` is a first-class part of the query and
merge APIs (`only_if_expr`, `expr_projection`,
`QueryFilter::Datafusion`, `when_matched_update_all_expr`), and
`ExecutionPlan` is returned from `create_plan`.
This follows DataFusion's own precedent of re-exporting `arrow`. The
coupling already exists via the trait/impl bounds — this surfaces it
rather than hiding it behind an `E0277`.
Closes#3575🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The server now serializes an index's `created_at` as an RFC 3339 string
(e.g. `"2026-06-18T21:37:36.637Z"`), but the client deserializer only
accepted a unix timestamp in milliseconds. This caused `list_indices` to
fail with:
```
Failed to parse list_indices response: invalid type: string "2026-06-18T21:37:36.637Z", expected a unix timestamp in milliseconds
```
This PR replaces the fixed millisecond deserializer with a custom one
that accepts both an RFC 3339 string (current server) and a
unix-millisecond integer (legacy deployments), so the client works
against any server version.
It also improves the `IndexConfig` repr in the Python bindings.
Previously it printed only three fields (`Index(FTS, columns=["text"],
name="text_idx")`), hiding the metadata that `list_indices` returns. It
now renders every populated field, omitting any that are `None`. Each
value is valid Python — integer counts use `_` thousands separators and
`created_at` uses the `datetime` repr — so values round-trip. The real
repr is a single line; it's wrapped here for readability:
```python
>>> table.list_indices()
[IndexConfig(
name="text_idx",
index_type="FTS",
columns=["text"],
index_uuid="aefd3e00-2f95-4bdc-92ac-06de84442bf1",
type_url="/lance.table.InvertedIndexDetails",
created_at=datetime.datetime(2026, 6, 18, 21, 37, 36, 637000, tzinfo=datetime.timezone.utc),
num_indexed_rows=2,
size_bytes=3_669,
num_segments=1,
index_version=1,
index_details={
'lance_tokenizer': None,
'base_tokenizer': 'simple',
'language': 'English',
'with_position': False,
'max_token_length': 40,
'lower_case': True,
'stem': True,
'remove_stop_words': True,
'custom_stop_words': None,
'ascii_folding': True,
'min_ngram_length': 3,
'max_ngram_length': 3,
'prefix_only': False,
},
)]
```
Fixes#3556🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Create release commit" workflow (`make-release-commit.yml`) has
failed on its last two runs; no release tags have been created since
June 4. Since this workflow creates the tag that the cargo/npm/pypi/java
publish workflows trigger off of, all recent releases are effectively
blocked.
The workflow installs `bump-my-version` unpinned. Version `1.4.0` added
a check that refuses to run `pre_commit_hooks` containing shell syntax
(pipes, `&&`, `if`, variable expansion) unless `allow_shell_hooks =
true` is set. Both bumpversion configs use such hooks:
- `python/.bumpversion.toml` — updates `Cargo.lock` after the bump
(fails first)
- `.bumpversion.toml` — runs `mvn versions:set` for the Java packages
The job dies at the version-bump step with:
> Hook '…' contains shell syntax (pipes, redirects, or variable
expansion). Set `allow_shell_hooks = true` in your configuration to
enable shell execution…
This sets `allow_shell_hooks = true` in both configs to restore the
previous behavior.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`RemoteTable::list_indices` currently makes one `/index/list/` call plus
one `/index/{name}/stats/` call per index just to recover `index_type`.
When the server returns `index_type` directly in the `/index/list/`
response, all enriched fields are used and the per-index stats fan-out
is skipped entirely. When `index_type` is absent (legacy servers), the
existing stats fallback runs as before. This is content-based: no
version header required.
## Changes
- `RemoteTable::parse_index_list_response` replaces the old split
between enriched and legacy parsers. A single struct deserializes both
old and new response shapes, with all fields except `index_name` and
`columns` optional. `index_type` acts as the sentinel: present → use
enriched fields directly; absent → call `/index/{name}/stats/`.
## Tests
Added `test_list_indices_enriched` covering:
- All enriched fields populated correctly when `index_type` is in the
list response
- Optional fields absent from the response deserialize as `None`
- Stats endpoint is **not** called (panics if hit), verifying the
fan-out is eliminated
Existing `test_list_indices` and `test_list_indices_nested_field_paths`
exercise the legacy path unchanged.
## Depends on
- #3497 (expand `IndexConfig`) — already merged
- Server-side enriched response support
Closes#3494
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Surfaces the rich per-index metadata added in #3497 to the Python and
Node.js language bindings. Closes#3495.
New optional fields exposed on `IndexConfig` in both bindings:
- `index_uuid` / `indexUuid` — UUID of the first index segment
- `type_url` / `typeUrl` — protobuf type URL for the index
- `created_at` / `createdAt` — creation timestamp (milliseconds since
Unix epoch)
- `num_indexed_rows` / `numIndexedRows` — rows covered by the index
- `num_unindexed_rows` / `numUnindexedRows` — rows not yet indexed
- `size_bytes` / `sizeBytes` — total index file size in bytes
- `num_segments` / `numSegments` — number of index segments
- `index_version` / `indexVersion` — on-disk format version
- `index_details` / `indexDetails` — type-specific JSON details string
All fields are `None`/`undefined` for remote tables (which don't yet
surface this metadata through the server response).
## Changes
- `python/src/index.rs`: extend `IndexConfig` pyclass; update `From`
impl; update `__getitem__`
- `python/python/lancedb/_lancedb.pyi`: add type hints for new fields
- `python/python/tests/test_table.py`: new `test_index_config_fields`
test
- `nodejs/src/table.rs`: extend `IndexConfig` napi struct; update `From`
impl
- `nodejs/__test__/table.test.ts`: new test; update existing `toEqual`
assertions to `expect.objectContaining` to accommodate new fields
## Test plan
- [x] Python: `uv run --extra tests pytest
python/tests/test_table.py::test_index_config_fields`
- [x] Node.js: `pnpm test __test__/table.test.ts`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
`IndexConfig` (returned by `Table::list_indices`) previously exposed
only `name`, `index_type`, and `columns`. Lance's `describe_indices`
provides richer per-index info cheaply (reads manifest metadata, often
cached), so this surfaces it.
Adds these `Option<T>` fields to `lancedb::index::IndexConfig`,
populated in `NativeTable::list_indices` from the `IndexDescription`:
- `index_uuid`: uuid of the first segment
- `type_url`: protobuf type URL (`IndexDescription::type_url`)
- `created_at`: minimum creation time across segments
- `num_indexed_rows`: approximate rows indexed across segments
- `num_unindexed_rows`: table row count minus `num_indexed_rows`
- `size_bytes`: total size of index files across segments
- `num_segments`: number of segments making up the index
- `index_version`: on-disk index format version (first segment)
- `index_details`: index-type-specific details as JSON
This field set mirrors the lance-namespace `IndexContent` contract
(lance-format/lance-namespace#348) so client and server agree on the
same shape. Note these are populated **locally** via `describe_indices`
— `NativeTable::list_indices` reads the dataset directly and does not
depend on the namespace spec change.
`RemoteTable` leaves the new fields `None` until a follow-up wires them
through the server response (#3494). Bindings exposure will also be a
follow up: #3495
Existing `list_indices` tests in `rust/lancedb/src/table.rs` are
extended to assert the new fields.
Fixes#3492🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
BREAKING CHANGE: direct Rust users lose the `IndexStatistics::loss`
field. Python and Node.js consumers are unaffected in practice for
remote tables (the value was always `None`/absent), but the attribute is
gone for local tables too.
`IndexStatistics::loss` was local-only — LanceDB Cloud never returned
it, so
`RemoteTable::index_stats` always set `loss: None`. It's vestigial; this
removes it.
- Remove `loss` from `IndexStatistics` and the internal `IndexMetadata`
in `rust/lancedb/src/index.rs`, plus the summing logic in
`NativeTable::index_stats`.
- Drop `loss` from the Python and Node.js bindings (and their
tests/docs).
Fixes#3493🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bumps the pinned toolchain in `rust-toolchain.toml` from 1.94.0 to
1.95.0.
Fixes new lints surfaced by clippy on 1.95.0:
- `manual_checked_ops` — fragment size mean in `table.rs` uses
`checked_div`
- `explicit_counter_loop` — shuffle test loop in `shuffle.rs`
No rustc warnings were introduced.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Transitions `LanceTable` and `RemoteTable` to use the unified
`create_index()` API matching `AsyncTable`
- Deprecates `create_scalar_index()` and `create_fts_index()` with
deprecation warnings
- Adds detection logic to distinguish legacy vs new API calls
- Adds `@overload` decorators for type checker compatibility
- Adds `accelerator` parameter to IVF config classes for GPU support
**New API:**
```python
table.create_index("vec", config=IvfPq(distance_type="l2"))
table.create_index("col", config=BTree())
table.create_index("text_col", config=FTS(with_position=True))
```
**Legacy API (deprecated):**
```python
table.create_index("l2", vector_column_name="vec") # emits DeprecationWarning
table.create_scalar_index("col", index_type="BTREE") # deprecated
table.create_fts_index("text_col") # deprecated
```
Fixes#2879🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
manylinux2_17 reached EOL in 2024 and pyarrow stopped publishing 2_17
wheels long ago. We already build manylinux2_28 wheels, so drop the 2_17
matrix entries.
Fixes#3452
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The python-v0.32.0 publish run failed on every build matrix entry. Three
independent issues:
1. **Mac and Windows**: `pypa/gh-action-pypi-publish` only runs on
Linux, but was being called inline from each build job.
2. **Linux (all arches)**: `pypa/gh-action-pypi-publish` derives its
docker image name from `github.action_repository`, which is empty when
the action is invoked from inside a composite action
(actions/runner#2473 — pypa's own `action.yml` references this bug). It
falls back to `github.repository`, generating
`docker://ghcr.io/lancedb/lancedb:<tag>`, which doesn't exist →
`denied`. Only the ARM matrix entry surfaced this because it failed
first and cancel-cascaded the rest.
3. **Windows**: `upload-artifact` in `build_windows_wheel` pointed at
`python\target\wheels`, but maturin writes to the workspace-root
`target/wheels`. The artifact was always empty. Also, `pypi-publish.yml`
passed a `vcpkg_token` input that the composite doesn't declare.
## Changes
- Build jobs (linux/mac/windows) now upload their wheels as
`actions/upload-artifact` artifacts.
- New Linux `publish` job downloads all wheel artifacts and runs the
Fury or PyPA publish step directly (not via a composite), so
`github.action_repository` resolves correctly.
- Delete the unused `upload_wheel` composite action.
- Drop the broken upload-artifact step inside `build_windows_wheel`.
- Remove the bogus `vcpkg_token` input.
- Fury upload now loops over all wheels instead of just the first.
- Bump `actions/checkout`, `actions/upload-artifact`,
`actions/download-artifact` to current major versions (Node 24) to clear
deprecation warnings.
- Bump Windows job timeout 60 → 90 minutes; previous run was
cancel-timing-out on a 60m cap.
- Use `rust-lld` as the Windows MSVC linker via
`CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER`. `link.exe` is
single-threaded and the long pole on Windows builds.
Fixes#3445
## Test plan
- [x] Open this PR — `paths` filter triggers a dry-run build on all
three platforms.
- [x] Verify all three builds produce wheels.
- [x] Confirm the `pypa/gh-action-pypi-publish` container actually
starts (the actions/runner#2473 bug) via the `publish-dry-run` job
pointed at TestPyPI.
- [x] **REMOVE BEFORE MERGE**: drop the `publish-dry-run` job and the
now-redundant `actions/upload-artifact` runs on PRs (currently always-on
so the dry-run has wheels to publish).
- [ ] After merge, cherry-pick onto `python-v0.32.0` and force-push the
tag to re-trigger the publish.
When a table is created with `pa.json_()` (PyArrow's JSON extension
type),
it is stored internally as `lance.json` (LargeBinary with `lance.json`
extension metadata). Calling `table.add()` with `pa.json_()` data failed
with:
```
RuntimeError: lance error: Append with different schema:
`data` should have type json but type was large_binary
```
`build_field_exprs` in `rust/lancedb/src/table/datafusion/cast.rs` saw
that
the input field (`Utf8` with `arrow.json` metadata) differed from the
table
field (`LargeBinary` with `lance.json` metadata). Since
`can_cast_types(Utf8, LargeBinary)` is true, it inserted a DataFusion
`Utf8 → LargeBinary` cast. That cast preserved the input field's
`arrow.json`
extension metadata instead of adopting the table's `lance.json`
metadata, so
lance-core detected a schema mismatch and rejected the append.
This adds a special case in `build_field_exprs`: when the input is
`arrow.json` and the table field is `lance.json`, the expression is
passed
through unchanged. Lance-core's write path already handles the
`arrow.json → lance.json` conversion (including JSONB encoding), so no
DataFusion cast is needed.
Fixes#3144
Continues #3291 from a fork (the original author's branch could not be
pushed to). The original commits are preserved; an additional commit
fixes
the CI failures on that PR — formatting, a missing trait import, and
read-back assertions that assumed binary storage when a lance.json
column
is read back as `Utf8`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: yunju.lly <yunju.lly@antgroup.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously `check_lance_release.py` used `git/refs/tags` with
`--paginate --jq`, which drops the last page in some `gh` versions. The
7.x Lance tags all landed on the final (partial) page, causing the
script to report `v6.0.1` as the latest and never triggering an update.
Switch to the releases API with `per_page=20`, which returns the 20 most
recent releases sorted newest-first — one API call, no pagination
needed.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The `test_resolve_user_id_*` tests in `remote/client.rs` mutate the
process-global `LANCEDB_USER_ID` and `LANCEDB_USER_ID_ENV_KEY`
environment variables. cargo runs tests in a binary across multiple
threads, so one test's `remove_var` can race another's `set_var` between
when it's set and when `resolve_user_id()` reads it.
This surfaced as an intermittent failure of
`test_resolve_user_id_from_env_key` on Windows CI:
```
assertion `left == right` failed
left: None
right: Some("custom-env-user-id")
```
Annotates the five env-mutating tests with `serial_test`'s
`#[serial(user_id_env)]` so they run serially with respect to each
other.
Should be backported to `release/v0.28` (CI for #3421 hit this same
flake).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Switch the nodejs bindings and examples package from npm to pnpm 11 to
pick up its stronger supply-chain defaults:
- `minimumReleaseAge` defaults to 1 day, so newly-published (potentially
compromised) versions aren't resolved into installs for at least 24h.
- Install lifecycle scripts (`preinstall`/`install`/`postinstall`) are
no longer run for arbitrary transitive deps; only an explicit allowlist
may run them, and unapproved scripts cause install to fail
(`strictDepBuilds: true`).
- Audit uses GHSA IDs and `--fix=update` to add patched versions to
`minimumReleaseAgeExclude`.
This is the same class of protection that would have blunted the recent
TanStack/`@uipath`/etc. compromise discussed in the [Aikido
write-up](https://www.aikido.dev/blog/mini-shai-hulud-is-back-tanstack-compromised).
## Changes
- Replace `nodejs/package-lock.json` and
`nodejs/examples/package-lock.json` with `pnpm-lock.yaml`.
- Pin pnpm via `packageManager: pnpm@11.1.1` in both `package.json`s.
- Add `pnpm-workspace.yaml` with the four build-script packages we
actually need: `@biomejs/biome`, `onnxruntime-node`, `protobufjs`,
`sharp`. Everything else is blocked from running install scripts.
- Update package.json scripts (`npm run X` → `pnpm X`).
- Update workflows: `.github/workflows/nodejs.yml`,
`.github/workflows/npm-publish.yml`, and
`.github/workflows/codex-fix-ci.yml` — install pnpm via
`pnpm/action-setup@v4` and switch `setup-node` caches to
`pnpm-lock.yaml`.
- Refresh `nodejs/AGENTS.md`, `nodejs/CLAUDE.md`, and
`nodejs/CONTRIBUTING.md`.
`docs/package-lock.json` is **not** touched — out of scope for this PR.
## Test plan
- [ ] `Lint` job (lint Rust/TS + examples lint) passes on CI.
- [ ] `Linux (NodeJS 18/20)` build+test passes, including the examples
test step.
- [ ] `macos` build+test passes.
- [ ] `NPM Publish` workflow's PR dry-run completes (build matrix + test
matrix + dry `npm publish`).
- [ ] No new install-script approvals are required at install time.
## Follow-ups
- `update_package_lock_run_nodejs.yml` references a composite action
path that doesn't exist
(`./.github/workflows/update_package_lock_nodejs`); it was already
broken pre-PR. We may want to either delete this workflow or rewrite it
for pnpm in a follow-up.
- Consider migrating `docs/` to pnpm in a separate PR.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a `deny.toml` at the workspace root and a `deny` CI job that runs
`cargo deny check` on every PR. Catches yanked crates, license drift,
banned or wildcard dependencies, unapproved sources, and new RUSTSEC
advisories.
As part of wiring this up:
- Updated `aws-lc-rs` 1.13.0 → 1.16.3 / `aws-lc-sys` 0.28.0 → 0.40.0 to
clear four 2026 AWS-LC advisories (timing side-channel, PKCS7 bypass,
CRL scope). Removed the `=0.28.0` workaround pin; the original build
failure no longer reproduces.
- Updated `bytes`, `zlib-rs`, `rand`, `rustls-webpki`, `lz4_flex` to
clear their current advisories.
- Marked `lancedb-nodejs` and `lancedb-python` as `publish = false` and
pinned `lzma-sys` from `*` to `0.1` so `bans.wildcards = "deny"` can
be enforced.
10 remaining advisories have no safe upgrade available (transitive via
opendal, lance, datafusion, async-openai, aws-sdk on the legacy rustls
0.21 chain). Each is ignored in `deny.toml` with a per-entry rationale
and a link to the RUSTSEC advisory. New advisories still fail CI.
Fixes#3297
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Replaces `LANCEDB_PYPI_API_TOKEN` (long-lived token) with OIDC trusted
publishing via `pypa/gh-action-pypi-publish`
- Adds `id-token: write` permission to linux/mac/windows jobs
- Removes `twine`-based upload and the `pypi_token` input from
`upload_wheel` composite action
- Enables PEP 740 Sigstore attestations on published wheels as a bonus
After merging, rotate/revoke the `LANCEDB_PYPI_API_TOKEN` secret.
Closes#3294🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds `.github/dependabot.yml` enabling weekly cargo update PRs for the
root workspace, which produces the Rust binaries we ship: the Node.js
and Python native extensions. The `rust/lancedb` library crate shares
the same lockfile — its consumers pick versions themselves, but bumping
transitive deps here keeps the shipped binaries current.
Also removes the misleading `exclude = ["python"]` line from the root
`Cargo.toml`: `python` is listed in `members`, and `cargo metadata`
confirms it's a workspace member, so the exclude was dead code that
implied the opposite.
Minor/patch updates are grouped to reduce PR noise.
Part of #3292. Only covers the cargo ecosystem; pip, npm, and
github-actions can follow.
Adds `permissions: contents: read` to the 10 workflows that had no
top-level permissions block. Workflows that already declared
permissions, or individual jobs that need elevated permissions (`issues:
write`, `pull-requests: write`, `contents: write`), are left unchanged.
Affected workflows: `dev.yml`, `java-publish.yml`, `java.yml`,
`license-header-check.yml`, `nodejs.yml`, `pypi-publish.yml`,
`python.yml`, `rust.yml`, `update_package_lock_run.yml`,
`update_package_lock_run_nodejs.yml`
Bumps the Rust toolchain to 1.94.0 (latest installed) to unblock CI
failures caused by the AWS SDK's MSRV requirement. No lint fixes were
needed.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Bumps all lance-* workspace dependencies from `4.0.0-rc.3` (git source)
to the stable `4.0.0` release on crates.io, removing the `git`/`tag`
overrides.
No code changes were required — compiles and passes clippy cleanly.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The test added in #3190 unconditionally imports `PIL`, which is an
optional dependency. This causes CI failures in environments where
Pillow isn't installed (`ModuleNotFoundError: No module named 'PIL'`).
Use `pytest.importorskip` to skip gracefully when Pillow is unavailable.
Fixes CI failure on main.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Namespace tests expected `RuntimeError` for table-not-found and
namespace-not-empty cases, but `lance_namespace` raises
`TableNotFoundError` and `NamespaceNotEmptyError` which inherit from
`Exception`, not `RuntimeError`.
- Updated `pytest.raises` to use the correct exception types.
## Test plan
- [x] CI passes on `test_namespace.py`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
Adds progress reporting for `table.add()` so users can track large write
operations. The progress callback is available in Rust, Python (sync and
async), and through the PyO3 bindings.
### Usage
Pass `progress=True` to get an automatic tqdm bar:
```python
table.add(data, progress=True)
# 100%|██████████| 1000000/1000000 [00:12<00:00, 82345 rows/s, 45.2 MB/s | 4/4 workers]
```
Or pass a tqdm bar for more control:
```python
from tqdm import tqdm
with tqdm(unit=" rows") as pbar:
table.add(data, progress=pbar)
```
Or use a callback for custom progress handling:
```python
def on_progress(p):
print(f"{p['output_rows']}/{p['total_rows']} rows, "
f"{p['active_tasks']}/{p['total_tasks']} workers, "
f"done={p['done']}")
table.add(data, progress=on_progress)
```
In Rust:
```rust
table.add(data)
.progress(|p| println!("{}/{:?} rows", p.output_rows(), p.total_rows()))
.execute()
.await?;
```
### Details
- `WriteProgress` struct in Rust with getters for `elapsed`,
`output_rows`, `output_bytes`, `total_rows`, `active_tasks`,
`total_tasks`, and `done`. Fields are private behind getters so new
fields can be added without breaking changes.
- `WriteProgressTracker` tracks progress across parallel write tasks
using a mutex for row/byte counts and atomics for active task counts.
- Active task tracking uses an RAII guard pattern (`ActiveTaskGuard`)
that increments on creation and decrements on drop.
- For remote writes, `output_bytes` reflects IPC wire bytes rather than
in-memory Arrow size. For local writes it uses in-memory Arrow size as a
proxy (see TODO below).
- tqdm postfix displays throughput (MB/s) and worker utilization
(active/total).
- The `done` callback always fires, even on error (via `FinishOnDrop`),
so progress bars are always finalized.
### TODO
- Track actual bytes written to disk for local tables. This requires
Lance to expose a progress callback from its write path. See
lance-format/lance#6247.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Similar to https://github.com/lancedb/lancedb/pull/3062, we can write in
parallel to remote tables if the input data source is large enough.
We take advantage of new endpoints coming in server version 0.4.0, which
allow writing data in multiple requests, and the committing at the end
in a single request.
To make testing easier, I also introduce a `write_parallelism`
parameter. In the future, we can expose that in Python and NodeJS so
users can manually specify the parallelism they get.
Closes#2861
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Move away from buildjet, which is shutting down runners for GHA [^1]
* Add `Cargo.lock` to build jobs, so when we upgrade locked dependencies
we check the builds actually pass. CI started failing because
dependencies were changed in #3116 without running all build jobs.
* Add fixes for aws-lc-rs build in NodeJS.
[^1]: https://buildjet.com/for-github-actions/blog/we-are-shutting-down
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When we write data with `add()`, we can input data to the table's
schema. However, we were using "safe" mode, which propagates errors as
nulls. For example, if you pass `u64::max` into a field that is a `u32`,
it will just write null instead of giving overflow error. Now it
propagates the overflow. This is the same behavior as other systems like
DuckDB.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
When input data is sufficiently large, we automatically split up into
parallel writes using a round-robin exchange operator. We sample the
first batch to determine data width, and target size of 1 million rows
or 2GB, whichever is smaller.
This hooks up a new writer implementation for the `add()` method. The
main immediate benefit is it allows streaming requests to remote tables,
and at the same time allowing retries for most inputs.
In NodeJS, we always convert the data to `Vec<RecordBatch>`, so it's
always retry-able.
For Python, all are retry-able, except `Iterator` and
`pa.RecordBatchReader`, which can only be consumed once. Some, like
`pa.datasets.Dataset` are retry-able *and* streaming.
A lot of the changes here are to make the new DataFusion write pipeline
maintain the same behavior as the existing Python-based preprocessing,
such as:
* casting input data to target schema
* rejecting NaN values if `on_bad_vectors="error"`
* applying embedding functions.
In future PRs, we'll enhance these by moving the embedding calls into
DataFusion and making sure we parallelize them. See:
https://github.com/lancedb/lancedb/issues/3048
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Upgrades `@napi-rs/cli` from v2 to v3, `napi`/`napi-derive` Rust
crates to 3.x
- Fixes a bug
([napi-rs#1170](https://github.com/napi-rs/napi-rs/issues/1170)) where
the CLI failed to locate the built `.node` binary when a custom Cargo
target directory is set (via `config.toml`)
## Changes
**package.json / CLI**:
- `napi.name` → `napi.binaryName`, `napi.triples` → `napi.targets`
- Removed `--no-const-enum` flag and fixed output dir arg
- `napi universal` → `napi universalize`
**Rust API migration**:
- `#[napi::module_init]` → `#[napi_derive::module_init]`
- `napi::JsObject` → `Object`, `.get::<_, T>()` → `.get::<T>()`
- `ErrorStrategy` removed; `ThreadsafeFunction` now takes an explicit
`Return` type with `CalleeHandled = false` const generic
- `JsFunction` + `create_threadsafe_function` replaced by typed
`Function<Args, Return>` + `build_threadsafe_function().build()`
- `RerankerCallbacks` struct removed (`Function<'env,...>` can't be
stored in structs); `VectorQuery::rerank` now accepts the function
directly
- `ClassInstance::clone()` now returns `ClassInstance`, fixed with
explicit deref
- `Vec<u8>` in `#[napi(object)]` now maps to `Array<number>` in v3;
changed to `Buffer` to preserve the TypeScript `Buffer` type
**TypeScript**:
- `inner.rerank({ rerankHybrid: async (_, args) => ... })` →
`inner.rerank(async (args) => ...)`
- Header provider callback wrapped in `async` to match stricter typed
constructor signature
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
`DatasetConsistencyWrapper::update()` only stored datasets with a
strictly newer
version. This caused `migrate_manifest_paths_v2` to silently drop its
update since
the migration renames files without bumping the dataset version. The
subsequent
`uses_v2_manifest_paths()` call would then return the stale cached
dataset.
Changed the version check from `>` to `>=` so same-version updates are
accepted.
## Test plan
- [x] Existing `test_create_table_v2_manifest_paths_async` Python test
should pass
- [x] Existing `should be able to migrate tables to the V2 manifest
paths` NodeJS test should pass
- [x] All dataset wrapper unit tests pass locally
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This updates `DatasetConsistencyWrapper` to block less:
1. `DatasetConsistencyWrapper::get()` just returns `Arc<Dataset>` now,
instead of a guard that blocks writes.
`DatasetConsistencyWrapper::get_mut()` is gone; now write methods just
use `get()` and then later call `update()` with the new version. This
means a given table handle can do concurrent reads **and** writes.
2. In weak consistency mode, will check for dataset updates in the
background, instead of blocking calls to `get()`.
---------
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Caches the schema of remote tables and invalidates the cache when:
1. After 30 second TTL
2. When we do an operation that changes schema (e.g. add_columns) or
checks out a different version (e.g. checkout_version)
3. When we get a 400, 404, or 500 reponse
If the schema is retrieved close to the TTL, we optimistically fetch the
schema in the background. This means a continuous stream of queries will
never have the schema fetch on the critical path.
Closes#3014
---------
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
BREAKING CHANGE: Arbitrary `impl RecordBatchReader` is no longer
accepted, it must be made into `Box<dyn RecordBatchReader>`.
This PR replaces `IntoArrow` with a new trait `Scannable` to define
input row data. This provides the following advantages:
1. **We can implement `Scannable` for more types than `IntoArrow`, such
as `RecordBatch` and `Vec<RecordBatch>`.** The `IntoArrow` trait was
implemented for arbitrary `T: RecordBatchReader`, and the Rust compiler
would prevent us from implementing it for foreign types like
`RecordBatch` because (theoretically) those types might implement
`RecordBatchReader` in the future. That's why we implement `Scannable`
for `Box<dyn RecordBatchReader>` instead; since it's a concrete type it
doesn't block implementing for other foreign types.
2. **We can potentially replay `Scannable` values**. Previously, we had
to choose between buffering all data in memory and supporting retries of
writes. But because `Scannable` things can optionally support
re-scanning, we now have a way of supporting retries while also
streaming.
3. **`Scannable` can provide hints like `num_rows`, which can be used to
schedule parallel writers.** Without knowing the total number of rows,
it's difficult to know whether it's worth writing multiple files in
parallel.
We don't yet fully take advantage of (2) and (3) yet, but will in future
PRs. For (2), in order to be ready to leverage this, we need to hook the
`Scannable` implementation up to Python and NodeJS bindings. Right now
they always pass down a stream, but we want to make sure they support
retries when possible. And for (3), this will need to be hooked up to
#2939 and to a pipeline for running pre-processing steps (like embedding
generation).
## Other changes
* Moved `create_table` and `add_data` into their own modules. I've
created a follow up issue to split up `table.rs` further, as it's by far
the largest file: https://github.com/lancedb/lancedb/issues/2949
* Eliminated the `HAS_DATA` generic for `CreateTableBuilder`. I didn't
see any public-facing places where we differentiated methods, which is
why I felt this simplification was okay.
* Added an `Error::External` variant and integrated some conversions to
allow certain errors to pass through transparently. This will fully work
once we upgrade Lance and get to take advantage of changes in
https://github.com/lance-format/lance/pull/5606
* Added LZ4 compression support for write requests to remote endpoints.
I checked and this has been supported on the server for > 1 year.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
The `report-failure` jobs in npm, cargo, and pypi publish workflows
checked for
`release` or `workflow_dispatch` events, but these workflows are
triggered by tag
pushes where `github.event_name` is `push`. The condition was never
true, so failure
notifications were silently skipped.
- Use `startsWith(github.ref, 'refs/tags/...')` to match actual tag
triggers
- Add `failure()` to only notify on actual failures
This matches the pattern already used by `java-publish.yml`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Implements `InsertExec` and `RemoteInsertExec` to support running
inserts in DataFusion.
## Context
In https://github.com/lancedb/lancedb/pull/2929, I've prototyped moving
the insert pipeline into DataFusion. This will enable parallelism at two
levels:
1. Running preprocessing, such as casting the input schema or computing
embeddings
2. Writing out files
This PR is just the first part of running the actual writes. In the end,
the plans might look like:
```
InsertExec
RepartitionExec num_partitions=<write_parallelism>
ProjectionExec vector=compute_embedding()
RepartitionExec num_partitions=<num_cpus>
DataSourceExec
```
where `num_cpus` is used to take advantage of all cores, while
`write_parallelism` might be less than `num_cpus` if there are too few
rows to want to split writes across `num_cpus` files.
Later PRs will move the preprocessing steps into DataFusion, and then
hook this up to the `Table::add()` implementations.
## Relation to future SQL work
We eventually plan on having the Remote SDK go through a FlightSQL
endpoint. Then for most queries we will send just the SQL string to the
server, and not run any sort of DataFusion plan on the client.
However, I think writes will be a little special, especially bulk writes
where we need to upload large streams of data and likely want
parallelism. So we'll have different code paths for writes, and I think
using DataFusion makes sense, especially as long as we are doing the
pre-processing on the client side still.
## Summary
- PR #2957 changed the permutation builder to only select `_rowid` from
the base table, but `Splitter::project()` for hash and calculated splits
replaced the selection entirely, dropping `_rowid`.
- Include `_rowid` in the column selections for hash and calculated
split projections.
- Fix a Python test that queried the permutation table for base table
columns no longer materialized.
Fixes the `test_split_hash`, `test_split_hash_with_discard`,
`test_split_calculated`, `test_shuffle_combined_with_splits`, and
`test_filter_with_splits` failures in `test_permutation.py`.
## Test plan
- [x] `cargo test -p lancedb -- permutation` (22 passed)
- [x] `pytest python/tests/test_permutation.py` (46 passed)
- [x] `npm test __test__/permutation.test.ts` (20 passed)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Realized our MSRV check was inert because `rust-toolchain.toml` was
overriding the Rust version. We set the `RUSTUP_TOOLCHAIN` environment
variable, which overrides that.
Also needed to update to MSRV 1.88 (due to dependencies like Lance and
DataFusion) and fix some clippy warnings.
BREAKING CHANGE: removes `aws`, `dynamodb`, `azure`, `gcs`, `oss`,
`huggingface` from default Rust features. They can be enabled by users
as needed.
They are still enabled for Python and NodeJS, since those users don't
control the compilation of artifacts.
Closes#2911
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Documentation**
- Expanded Python API reference with new entries for table metadata,
tagging, remote client configuration, and index statistics.
- Added documentation for new classes and modules in both synchronous
and asynchronous sections, including `FragmentStatistics`,
`FragmentSummaryStats`, `Tags`, `AsyncTags`, `IndexStatistics`, and
remote configuration options.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
* Add `ci` profile for smaller build caches. This had a meaningful
impact in Lance, and I expect a similar impact here.
https://github.com/lancedb/lance/pull/5236
* Get caching working in Rust. Previously was not working due to
`workspaces: rust`.
* Get caching working in NodeJs lint job. Previously wasn't working
because we installed the toolchain **after** we called `- uses:
Swatinem/rust-cache@v2`, which invalidates the cache locally.
* Fix broken pytest from async io transition
(`pytest.PytestRemovedIn9Warning`)
* Altered `get_num_sub_vectors` to handle bug in case of 4-bit PQ. This
was cause of `rust future panicked: unknown error`. Raised an issue
upstream to change panic to error:
https://github.com/lancedb/lance/issues/5257
* Call `npm run docs` to fix doc issue.
* Disable flakey Windows test for consistency. It's just an OS-specific
timer issue, not our fault.
* Fix Windows absolute path handling in namespaces. Was causing CI
failure `OSError: [WinError 123] The filename, directory name, or volume
label syntax is incorrect: `
This will make it easier to see if there are underlying problems. We
should see the actual object store HTTP request error within the error
chain after this.
## Summary
- Created custom GitHub Action that creates issues when workflow jobs
fail
- Added report-failure jobs to cargo-publish.yml, java-publish.yml,
npm-publish.yml, and pypi-publish.yml
- Issues are created automatically with workflow name, failed job names,
and run URL
## Test plan
- Workflows will only create issues on actual release or
workflow_dispatch events
- Can be tested by triggering workflow_dispatch on a publish workflow
Based on lancedb/lance#4873🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
This PR adds the missing `name` parameter to `create_scalar_index` and
`create_fts_index` methods in the Python SDK, which was inadvertently
omitted when it was added to `create_index` in PR #2586.
## Changes
- Add `name: Optional[str] = None` parameter to abstract
`Table.create_scalar_index` and `Table.create_fts_index` methods
- Update `LanceTable` implementation to accept and pass the `name`
parameter to the underlying Rust layer
- Update `RemoteTable` implementation to accept and pass the `name`
parameter
- Enhanced tests to verify custom index names work correctly for both
scalar and FTS indices
- When `name` is not provided, default names are generated (e.g.,
`{column}_idx`)
## Test plan
- [x] Added test cases for custom names in scalar index creation
- [x] Added test cases for custom names in FTS index creation
- [x] Verified existing tests continue to pass
- [x] Code formatting and linting checks pass
This ensures API consistency across all index creation methods in the
LanceDB Python SDK.
Fixes#2616🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
Enables two new parameters when building indices:
* `name`: Allows explicitly setting a name on the index. Default is
`{col_name}_idx`.
* `train` (default `True`): When set to `False`, an empty index will be
immediately created.
The upgrade of Lance means there are also additional behaviors from
https://github.com/lancedb/lance/commit/cd76a993b8b65877135cb9cfb47448a761ac7f7b:
* When a scalar index is created on a Table, it will be kept around even
if all rows are deleted or updated.
* Scalar indices can be created on empty tables. They will default to
`train=False` if the table is empty.
---------
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Adds tests to ensure that users can paginate through simple scan, FTS,
and vector search results using `limit` and `offset`.
Tests upstream work: https://github.com/lancedb/lance/pull/4318Closes#2459
## Summary
- Enhanced vector column detection to use substring matching instead of
exact matching
- Now detects columns with names containing "vector" or "embedding"
(case-insensitive)
- Added integer vector support to Node.js implementation (matching
Python)
- Comprehensive test coverage for both float and integer vector types
## Changes
### Python (`python/python/lancedb/table.py`)
- Updated `_infer_target_schema()` to use substring matching with helper
function `_is_vector_column()`
- Preserved original field names instead of forcing "vector"
- Consolidated duplicate logic for better maintainability
### Node.js (`nodejs/lancedb/arrow.ts`)
- Enhanced type inference with `nameSuggestsVectorColumn()` helper
function
- Added `isAllIntegers()` function with performance optimization (checks
first 10 elements)
- Implemented integer vector support using `Uint8` type (matching
Python)
- Improved type safety by removing `any` usage
### Tests
- **Python**: Added
`test_infer_target_schema_with_vector_embedding_names()` in
`test_util.py`
- **Node.js**: Added comprehensive test case in `arrow.test.ts`
- Both test suites cover various naming patterns and integer/float
vector types
## Examples of newly supported column names:
- `user_vector`, `text_embedding`, `doc_embeddings`
- `my_vector_field`, `embedding_model`
- `VECTOR_COL`, `Vector_Mixed` (case-insensitive)
- Both float and integer arrays are properly converted to fixed-size
lists
## Test plan
- [x] All existing tests pass (backward compatibility maintained)
- [x] New tests pass for both Python and Node.js implementations
- [x] Integer vector detection works correctly in Node.js
- [x] Code passes linting and formatting checks
- [x] Performance optimized for large vector arrays
Fixes#2546🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
- Adds an overall `timeout` parameter to `TimeoutConfig` that limits the
total time for the entire request
- Can be set via config or `LANCE_CLIENT_TIMEOUT` environment variable
- Exposed in Python and Node.js bindings
- Includes comprehensive tests
## Test plan
- [x] Unit tests for Rust TimeoutConfig
- [x] Integration tests for Python bindings
- [x] Integration tests for Node.js bindings
- [x] All existing tests pass
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
- Exposes `Session` in Python and Typescript so users can set the
`index_cache_size_bytes` and `metadata_cache_size_bytes`
* The `Session` is attached to the `Connection`, and thus shared across
all tables in that connection.
- Adds deprecation warnings for table-level cache configuration
🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Add support for providing a custom `Session` when connecting to a
`ListingDatabase`. This allows users to configure object store
registries, caching, and other session-related settings while
maintaining full backward compatibility.
## Usage Example
```rust
use std::sync::Arc;
use lancedb::connect;
let custom_session = Arc::new(lance::session::Session::default());
let db = connect("/path/to/database")
.session(custom_session)
.execute()
.await?;
```
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Fixes#2515 by implementing comprehensive support for missing columns in
Arrow table inputs when using embedding functions.
### Problem
Previously, when an Arrow table was passed to `fromDataToBuffer` with
missing columns and a schema containing embedding functions, the system
would fail because `applyEmbeddingsFromMetadata` expected all columns to
be present in the table.
🤖 Generated with [Claude Code](https://claude.ai/code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
Make sure we only update the latest version if it's actually newer. This
is important if there are concurrent queries, as they can take different
amounts of time.
## Summary
Fixes issue #2465 where FTS explain plans only showed basic `LanceScan`
instead of detailed execution plans with FTS query details, limits, and
offsets.
## Root Cause
The `FTSQuery::explain_plan()` and `analyze_plan()` methods were missing
the `.full_text_search()` call before calling explain/analyze plan,
causing them to operate on the base query without FTS context.
## Changes
- **Fixed** `explain_plan()` and `analyze_plan()` in `src/query.rs` to
call `.full_text_search()`
- **Added comprehensive test coverage** for FTS explain plans with
limits, offsets, and filters
- **Updated existing tests** to expect correct behavior instead of buggy
behavior
## Before/After
**Before (broken):**
```
LanceScan: uri=..., projection=[...], row_id=false, row_addr=false, ordered=true
```
**After (fixed):**
```
ProjectionExec: expr=[id@2 as id, text@3 as text, _score@1 as _score]
Take: columns="_rowid, _score, (id), (text)"
CoalesceBatchesExec: target_batch_size=1024
GlobalLimitExec: skip=2, fetch=4
MatchQuery: query=test
```
## Test Plan
- [x] All new FTS explain plan tests pass
- [x] Existing tests continue to pass
- [x] FTS queries now show proper execution plans with MatchQuery,
limits, filters
Closes#2465🤖 Generated with [Claude Code](https://claude.ai/code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Added new test cases to verify explain plan output for full-text
search, vector queries with pagination, and queries with filters.
* **Bug Fixes**
* Improved the accuracy of explain plan and analysis output for
full-text search queries, ensuring the correct query details are
reflected.
* **Refactor**
* Enhanced the formatting and hierarchical structure of execution plans
for hybrid queries, providing clearer and more detailed plan
representations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated the release workflow to explicitly check out the main branch
during the publishing process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated release workflow to set a specific Git user name and email for
automated commits during the package publishing process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated the release workflow to include an additional step for
improved process reliability. No changes to user-facing functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Adds a script to change the lance dependency easily. To make this
change, I just had to run:
```bash
python ci/set_lance_version.py stable
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added a script to automate updating the Lance package version in
project dependencies.
- **Chores**
- Updated workflows to improve lockfile management and automate updates
during releases and publishing.
- Switched Lance dependencies from git-based references to fixed version
numbers for improved stability.
- Enhanced lockfile update script with an option to amend commits and
quieter output.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Follow up to #2416
Forgot to do `git add`.
Also need to delete old actions updating package lock.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Removed legacy workflows related to updating package lock files.
- Improved the update lockfiles script to ensure updated lockfiles are
always included in amended commits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated workflow to ignore changes in the `Cargo.lock` file during
documentation checks, reducing unnecessary workflow failures.
- Enhanced release process by adding automated lockfile updates for
Node.js and Rust components.
- Removed an obsolete package-lock update job from the publishing
workflow to streamline releases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Downgraded version numbers for the Node.js, Python, and Rust packages.
No other user-facing changes were made.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Provides the ability to set a timeout for merge insert. The default
underlying timeout is however long the first attempt takes, or if there
are multiple attempts, 30 seconds. This has two use cases:
1. Make the timeout shorter, when you want to fail if it takes too long.
2. Allow taking more time to do retries.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added support for specifying a timeout when performing merge insert
operations in Python, Node.js, and Rust APIs.
- Introduced a new option to control the maximum allowed execution time
for merge inserts, including retry timeout handling.
- **Documentation**
- Updated and added documentation to describe the new timeout option and
its usage in APIs.
- **Tests**
- Added and updated tests to verify correct timeout behavior during
merge insert operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Will JonesandClaude Opus 5
