The concurrent-append and drop-all DynamoDB tests shared a database
prefix. A timeout in the first could leave its table visible to the
second. Give each test its own prefix while retaining five independent
concurrent appends, the six-row assertion, and all drop/recreate checks.
The final two-prefix diff passed both tests in hosted LocalStack across
Node 22, 24, 26, and Node 24 experimental, without timing
instrumentation ([run
43245851764](https://github.com/lancedb/lancedb/actions/runs/43245851764),
head `5896c419`). Concurrency and the 5000 ms budget are unchanged. This
fixes cascading fixture contamination; the original timeout remains
unexplained, and these passing runs do not prove its root cause was
fixed.
## Problem
The existing Function `column_status` surface cannot decode the Agent
scheduler's `queued` state or the `failure_threshold_reached` and
`binding_revision_missing` blocked reasons. It also preserves only the
latest refresh job, so a later cancellation can hide an earlier actual
failure.
## Change
- Adds the `queued`, `failure_threshold_reached`, and
`binding_revision_missing` wire states in Rust, Python, and Node. The
wire change is additive, while Rust callers must update exhaustive enum
matches and public struct literals.
- Adds optional `last_failure_job` alongside `last_refresh_job`. The
latest cancellation and the most recent actual failure retain separate
meanings.
- Reuses the canonical Rust `JobFailure` and existing Python/Node
`JobFailureInfo` models for nested job failure details instead of
introducing a parallel status error model.
- Maps the existing public `sql_job_id` field through Node's
`JobFailureInfo`.
- Keeps `column_status`, `column_statuses`, and `wait_fresh` as the
existing API surfaces. There is no new endpoint, row-error list, durable
ledger, or unknown-enum fallback.
- Documents `queued` versus `running` and the separate latest-attempt
and latest-failure jobs in the existing hidden freshness release draft.
The new status values are additive but cross-service: `queued`, the two
new blocked reasons, and `last_failure_job` require matching Agent/QN
and SDK deployments before clients can rely on them. This SDK follow-up
does not change the normal `7cc3904a` dependency pin or claim whole
ENT-3487 completion.
## Validation
The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`,
based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The
final product docs commit changes generated TypeDoc only.
- Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust
blobs are unchanged from their tested `788ef85b` source through
`c9bac25e`.
- Python: the PyO3 extension built successfully, and the selected native
PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The
test requests pinned version 42.
- Node: the addon and TypeScript builds passed, Biome passed, and the
selected Jest transport test passed 1/0 with 57 filtered tests. Addon
SHA-256:
`c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`.
- Python extension SHA-256:
`36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`.
- TypeDoc was generated from the native `native.d.ts` produced by that
Node build. A second generation produced no remaining diff.
The public diff contains only product, test, and generated API
documentation changes. Native runner scripts and preserved
compile-failure logs remain private validation evidence.
Agent/QN production mapping, mixed-version rollout, and the remaining
ENT-3487 server-side work are tracked separately.
## Problem
Function-backed columns need an exact `fresh`, `stale`,
`never_computed`, and
`failed` partition for the table version a caller is reading. The
existing
error listing is capped, and composing it with job and schema listings
cannot
prove which rows currently failed.
## Behavior
- Adds a strict freshness reader beside the existing permissive planner.
It
distinguishes an absent certificate from legacy evidence that omits
failures, and from a current certificate carrying failure coverage.
- Extends the immutable freshness sidecar with per-fragment known and
failed
row bitmaps. Partial or `WHERE` publications cover only observed rows;
untouched rows stay unknown. Deleted rows do not block classification.
- Adds failure-aware whole-fragment and row-repair writers. A rejected
older
attempt cannot overwrite failure evidence already recorded for a newer
input
signature.
- Adds `column_status`, `column_statuses`, and `wait_fresh` to Python
and the
corresponding Node methods. The client calls the server's exact status
endpoint, validates the four-way partition and one-version guarantee,
and
preserves the complete Function version, last-job, and automatic-refresh
metadata.
- Adds an MDX-commented draft usage section to the Enterprise
computed-column
refresh guide, covering Python, Rust, and TypeScript and remaining
hidden
until the server integration ships.
- Sends `column`, `branch`, and exact `version` selectors to the status
route.
A pinned handle rejects a nonempty response from any other table
version;
latest handles keep the existing read-freshness headers.
- Retries `wait_fresh` only for HTTP 503 with
`function_column_status_finalization_pending`. Existing failed rows
remain
visible and do not keep the wait pending; other errors propagate.
- Models the shared scheduler projection, including `up_to_date`, real
`blocked` states, their reason, failure streak, and effective cooldown.
## Validation
This is the SDK side of ENT-3563. Sophon's failure-coverage publication
and
status endpoint are a separate integration; deployed end-to-end and
whole-ticket
acceptance remain pending.
- Rust: 9 exact freshness/status tests passed, plus 1 exact
latest-handle
freshness-header test. The pinned transport test was extended with a
wrong-version response and rerun at `ebd6649a` (1 passed). Coverage
includes untouched partial rows,
selected-row failure clearing, cross-generation evidence, typed and
non-retried errors, `up_to_date`, blocked scheduler fields, and
exact-version
requests.
- Python: 2 native PyO3 remote-transport tests passed together,
including
typed retry; one local facade test reaches the existing unsupported
boundary.
- Node: the NAPI addon built successfully, TypeScript and Biome passed,
and the
selected native remote-transport test passed 1/0, including
exact-version
selection and blocked scheduler metadata.
- `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile,
Biome, and
TypeScript checks passed.
Validation is mapped by the relevant unchanged source rather than
treated as one
final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header
and Python
remote tests at `31e68b7d`, the local Python test and Node build at
`6f213584`, and
the final TypeScript/Jest delta at `4f2d5c27`. The final product tree
excludes those five private EC2 runner files and adds
six generated TypeDoc files plus one hidden draft usage section in the
computed-column refresh guide. The Node delta corrected a private
TypeScript
command and reused the unchanged native addon.
## Summary
Adds an optional `max_concurrency` setting to Function registration and
explicit refresh:
- A registered Function version can define the default maximum number of
user-code calls in flight for one logical execution.
- A refresh can override that default for that execution only.
- The value must be a positive `u32` and is independent from
`resource_budget`; it is not an account-wide quota.
- Omitting the option preserves the previous request payload. Local
embedded refreshes reject this Enterprise-only option.
Rust, Python sync/async, PyO3, and the existing Node refresh surface
forward the option. Node has no Function registration surface today, so
no registration API is added there. The generated TypeDoc pages are
included for the new public option.
## Validation
The native-tested source covers Function version decoding, registration
validation and serialization, the Flight SQL registration ticket and
returned version, remote refresh forwarding, local rejection, and Node's
numeric boundary before conversion. Exact stage results are Rust `1+3`
plus four remote cases at `1` each; Python `14/1/49/24`; Node `57`
remote and `1` local rejection. Ruff, standard repository Pyright, and
Biome passed.
The tested Python extension was
`21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the
tested Node addon and generated native declaration were
`f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and
`4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The
final 24-file diff preserves the 22 tested
product/test/fixture/source-doc blobs and adds only two generated
TypeDoc pages.
## Summary
- Add the canonical Rust `FunctionWorkerResourceBudget` and
`RefreshColumnOptions::resource_budget()` for explicit Enterprise
Function refreshes.
- Forward the nested budget through Python sync/async, PyO3, Rust remote
requests, Node NAPI, and TypeScript.
- Require positive CPU and memory values; normalize an omitted nested
GPU count to numeric zero; reject the nested budget when combined with
the top-level `gpu_count` shorthand.
- Reject `resource_budget` on LanceDB Embedded refresh paths instead of
silently ignoring it.
- Document the Rust, Python, and TypeScript APIs, including that SQL
servers currently require all three budget fields.
The Rust, Python, and TypeScript additions use `cpu_millis` /
`cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` /
`gpuCount`. The top-level GPU shorthand keeps its existing behavior and
remains mutually exclusive with the complete nested budget.
## Scope and compatibility
This change configures only the explicit refresh call. It does not
configure `add_columns` automatic refresh and does not introduce a
fleet-wide call-cap interface.
A LanceDB Enterprise server must expose the nested `resource_budget`
backfill contract. LanceDB Embedded continues to support its existing
expression refresh defaults and rejects the new complete budget
explicitly.
## Validation
Validated against the final signed product source:
- Rust: resource-budget remote serialization test `2 passed`; validation
test `1 passed`.
- Python: actual PyO3 build; remote-request matrix `22 passed`;
invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff
check and format clean.
- Repository-standard Pyright entry point: `0 errors`, within the
repository's configured include set.
- Node: native build and Biome lint passed; the existing remote-request
test `1 passed`, including fractional, negative, overflow, NaN, and
infinity nested GPU cases; the existing local-rejection test `1 passed`.
- TypeDoc generation and generated-document cleanliness passed.
- The native-tested product source differs from this head only in two
semantically equivalent Node range checks required by Clippy. Current CI
compiles and lints the final blobs. No dependency or lockfile changes
are included.
## Summary
`@lancedb/lancedb` pins the optional dependency
`@huggingface/transformers` at 3.0.2, which requires `sharp ^0.33.5`.
Every npm install of LanceDB therefore pulled in a sharp version with
critical advisories (GHSA-wq5f-xc86-pv6w, GHSA-rgj7-g3m4-5g8c). No 3.x
release of transformers allows a patched sharp, so this upgrades to
4.3.2, which requires `sharp ^0.35.4`. A fresh install resolves that
range to 0.35.5, the first release that fixes both advisories.
- `nodejs/package.json`: `@huggingface/transformers` 3.0.2 → 4.3.2.
- `nodejs/pnpm-workspace.yaml`: remove the `sharp: ^0.35.4` override. It
only covered our own install, not consumers, and transformers now
requires that range itself.
- `nodejs/examples`: transformers 3.0.2 → 4.3.2, sharp 0.33.5 → 0.35.5.
- Both lockfiles resolve sharp to 0.35.5; `pnpm audit` reports no sharp
advisories in either.
`TransformersEmbeddingFunction` needed no code changes. The transformers
types are only used in private fields, so the published typings don't
change.
`docs/web-tests/ts` and `docs/web-tests/blob-v2/ts` are left alone: they
resolve transformers through a published `@lancedb/lancedb`, so they
pick this up after a release.
## Evidence
Embeddings from the built-in `huggingface` embedding function (default
`Xenova/all-MiniLM-L6-v2`) on 4.3.2, compared with the same mean pooling
on 3.0.2, for three sentences:
```
transformers 4.3.2: 3 x 384
text 0: max|diff|=6.62e-07 cosine=1.00000000
text 1: max|diff|=6.26e-07 cosine=1.00000000
text 2: max|diff|=7.33e-07 cosine=1.00000000
```
End to end through a table, `search("a feline sat down")` returns `["The
cat sat on the mat.", "A dog barked at the mailman."]`.
## Merge Risk
**Door:** Two-way. A revert restores the 3.0.2 pin.
**Blast Radius:** Node `huggingface` embedding function, and install
footprint of the optional dependency (onnxruntime-node 1.19.2 → 1.30.0,
which still downloads the CUDA provider on Linux x64 by default, as
1.19.2 did).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The server's backfill request takes a row filter, failure limits,
per-call timeouts and worker sizing, and SQL exposes all of them through
`REFRESH COLUMN ... WHERE ... WITH (...)`, but the typed SDKs could send
only batch_size, GPU placement and the retry and error policies. Anyone
needing a filtered refresh or a bounded skip policy had to drop to SQL.
This adds `where`, `max_failed_fraction`, `max_failed_rows`,
`row_timeout_seconds`, `init_timeout_seconds`, `num_workers` and
`max_executor_processes_per_worker` to Python, Rust and Node, passed
through unchanged. `RefreshColumnOptions` now serializes straight into
the request body, so the three SDKs share one producer. Node gets a
`RefreshColumnOptions` object, which also carries the options it lacked
before. A LanceDB Embedded refresh rejects every option other than
batch_size and GPU placement, naming the option.
Lance beta.11 calibrates lower `search_effort` tiers by metric and
requested `k`. With the Rust dependency and Python/Node execution tests
already on main via #4881, document the tiers across Rust, Python, Node
and the search guide, extend the remote request matrix to cover `0.1`
and `0.25`, and align the Python test package with beta.11.
`0.1` and `0.25` target empirical mean partition-routing recall of 80%
and 90% for L2, cosine and dot through `k=100000`. These are not
per-query or final Recall@k guarantees. Default Auto is unchanged;
Hamming and larger `k` remain uncalibrated, and remote queries require
corresponding server support. This PR makes no additional performance
claim. See [Lance
#10695](https://github.com/lance-format/lance/pull/10695) for
measurements and limitations.
Validation: both strict workspace Clippy configurations, Rust
formatting, 13 Rust effort tests, 208 Python query tests, 41 Node query
tests, native binding builds, TypeScript, Node lint/generated docs,
Python lint/format, full lockfile check and two dependency guards passed
on `a8b09170`. All validated Rust/Python/Node source and dependency
files are byte-identical at `2a82d07c`; the only subsequent change
restores the Java POM to main. Root Ruff has 59 existing diagnostics and
six formatting files, all byte-identical to the base; the Python package
passes.
The Java REST/Namespace client has no changed API here, and its
dependency remains identical to main. The separate [Java beta
publication failure](https://github.com/lance-format/lance/issues/10773)
is not claimed resolved or validated. Official SDK release, automatic
mirror, Sophon adoption and measured consumer acceptance remain
follow-up stages.
Downstream workspaces adopting Lance 15.0.0-beta.11 currently also
resolve beta.10 because the SDK pins every Lance crate exactly. This
produces incompatible Lance types across the SDK and the consumer.
Update the workspace pins and lockfile to the published
[v15.0.0-beta.11](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.11).
This release includes payload-free list length reads and preserves those
read modes across serialized plans (lance-format/lance#10759,
lance-format/lance#10766). Existing feature flags and unrelated locked
packages are unchanged.
The release also includes the metric/k-dependent search-effort
calibration from lance-format/lance#10695. Update the Node and Python
execution tests to its exact probe budgets (for example, effort 0.25
with L2, k=8 and Auto=6 now probes 4 partitions), and cover the 0.1
tier. Recall, caller bounds and probe-precedence assertions remain in
place.
Validation:
- `cargo fmt --all --check`
- `cargo clippy --quiet --features remote --tests --examples`
- Downstream metadata fixture with SDK source and the released Lance
beta.11 path overrides resolves exactly one Lance version.
- `cargo test --quiet --features remote --tests`: 2,271 passed, 0
failed, 79 existing ignored tests.
- Node native/TypeScript build, lint and API documentation generation
passed; 41 query tests passed.
- Python native binding build and package Ruff checks passed; 208 query
tests passed, 1 slow test deselected, 5 existing deprecation warnings.
- Rust results remain applicable: the follow-up changes only binding
tests, with product source and dependency graph unchanged from the
tested dependency update.
Repository-root Ruff currently reports 59 lint findings and 6 formatting
failures in untouched documentation snippets. Their blobs match the PR
base; the changed Python test and the entire Python package pass
lint/format checks.
Passing a record batch created by an ESM `apache-arrow` import to
`RRFReranker.rerankHybrid` throws an IPC footer parse error. The SDK's
CommonJS Arrow writer receives data types from another module instance.
Normalize single-record-batch input with the existing table sanitizer
before writing IPC. Extend the Arrow 15–21 round-trip test with
caller-created batches and nulls, and add a native ESM reranker
regression covering 64-bit row IDs, scores, and an empty FTS side.
Validation:
- Reproduced with published `0.42.0-beta.3`; CommonJS and explicitly
normalized ESM controls pass.
- Same native module and inputs: all 8 targeted cases fail with the
parent serializer and pass with this change.
- Local native build, TypeScript build, lint, and formatting pass.
- Current-head CI passes on Linux Node 22/24/26, macOS, and Arrow 19/20.
The macOS run passes 1,643 tests; Linux Node 24 also passes docs and
example checks.
Function refresh failures and skipped-row diagnostics lose the traceback
when decoded by the SDK. Expose the server-provided optional `traceback`
on job failure details and Function error records in Rust, Python, and
Node.js, while keeping existing compact error messages.
Responses from older servers remain supported when the field is absent.
Tracebacks may be truncated by the server. The Rust `Error::JobFailed`
variant now stores `Box<JobFailure>` so the added diagnostics do not
enlarge every `Result` error payload.
`Table::optimize()` previously returned `NotSupported` for remote
tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE
TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL.
Only `OptimizeAction::Compact` without a custom `remap_options` or
non-default `CompactionOptions` is supported remotely.
`OptimizeAction::All`, `::Prune`, and `::Index` all return a clear
`NotSupported` error instead of silently doing the wrong thing. A
branch-scoped table handle submits a `WITH (branch = '<branch>')`
clause; the statement doesn't validate the branch exists itself, so an
unknown branch surfaces as an async job failure rather than an error
from this call.
The server's SQL job adapter doesn't expose compaction metrics, so
`OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a
placeholder rather than real numbers — it must still be `Some`, since
callers (including the Python/Node bindings) unwrap it unconditionally
on success. This mirrors how `refresh_materialized_view_async` already
handles the same limitation for its own metrics.
The Python and Node `optimize()` convenience methods call a new shared
`Table::optimize_full`, which runs `Compact` and then, on a remote
table, skips `Prune` and `Index` instead of attempting and failing on
them — returning zeroed prune stats rather than submitting a compaction
job and then raising. Python's `RemoteTable.optimize()` also now
forwards `compaction_options` to the Rust layer, so a non-default value
is rejected before submission instead of being silently dropped.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Extend the existing job progress regression to cover Function worker
counts, GPU capacity and wait reasons, and known or absent warm-reuse
evidence through `Job.progress()` and `Job.wait()`. The callback
assertions also preserve an unknown warm state instead of turning it
into `false`.
Validation on EC2: all 56 remote SDK tests, Biome lint/format,
TypeScript build, and documentation generation pass. Native code and
dependencies match the previously validated build.
Refs ENT-3486.
TypeScript job handles now expose structured progress through
`progress()` and an optional synchronous callback to `wait()`. Snapshots
preserve unknown fields and missing counters, and job listings include
progress and pause state. Existing `status()` strings and no-argument
`wait()` behavior remain compatible.
Wait callbacks observe the existing backend poller. Observer exceptions
are logged without changing the job outcome. A best-effort ETA is
available only from compatible samples with positive progress, and is
omitted during initialization, queueing, or pause.
Part of ENT-3500.
---------
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-12-104.us-east-2.compute.internal>
On remote tables, `create_index` returns as soon as the build is scheduled. Until it finishes, a vector index is missing from `list_indices`, a scalar or FTS index shows 0 indexed rows, and queries don't use the index. The docs didn't say any of this, so it was easy to assume the index was ready.
This adds that to the `create_index` and `list_indices` docs in Python (`Table` and `AsyncTable`) and TypeScript, and points to `wait_timeout` / `waitTimeoutSeconds`, `wait_for_index`, and `create_index_async` for blocking until the index is ready. It also explains that completion tracking is tied to the submitted build only when the server returns a job ID, while legacy name-based waits can return early when replacing an existing index.
Closes#4405
Add a central documentation link registry (`lancedb::docs::DocLink`) in
the Rust core that builds `docs.lancedb.com` URLs with UTM parameters
(`utm_source=lancedb_sdk`, `utm_medium=warning|error`,
`utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any
`#anchor` fragment and without user-identifying data.
- Initialize `utm_content` per language binding (`python`, `nodejs`,
`cli`, defaulting to `rust`) and expose the link builder to Python so
`DBConnection` and `Job` stubs reuse the core definitions.
- Append published documentation links to Enterprise-only `NotSupported`
/ `NotImplementedError` errors across database, job, catalog, and table
operations.
- Emit rate-limited maintenance warnings (once per table per process
when `warn` logging is enabled) when opening a native table with at
least 100 small fragments (`< 100k` physical rows each) or at least
1,000,000 unindexed rows on an optimizable user index, reading only
manifest metadata without opening data or `_indices/` files, and
document the thresholds in the compaction and reindexing guides.
- Verify published/in-repo `docs/web/` targets and `#anchor` fragments
in unit tests and include extracted SDK links in the scheduled
`docs-link-check` workflow.
Linear: OSS-2772
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing.
Co-authored-by: Leo <leo@insight.co>
Renaming or dropping an embedding source or vector column leaves schema metadata pointing at missing fields, causing later `add()` calls to fail.
Reject these operations in local schema evolution before committing any changes. The guard reads Python/Node embedding column metadata and native Rust `lancedb::column_definitions` without loading embedding models. It supports explicit and default Rust vector-column names and respects quoted column paths. Unrelated column changes and nullability updates remain supported.
The restriction is documented in the Python, Rust, and TypeScript APIs. Regression coverage includes both SDK metadata formats, native Rust definitions with explicit and default destinations, atomic rejection of batched changes, and continued embedding generation after reopening.
Fixes#4763
Validation at the exact PR head and GitHub test-merge:
- Hosted macOS Python wheel suite: 3061 passed, 162 skipped, 70 deselected, including all 8 sync and 4 async embedding schema-evolution cases.
- Hosted Windows Rust suite: 1900 library tests passed, 8 embedding integration tests passed, and 107 doctests passed, including the native explicit/default destination and binding guard cases.
- Hosted Linux Python 3.10 rerun: 3084 passed, 159 skipped, 50 deselected; the previously observed remote-fork timeout passed on the replacement job.
- All 10 applicable exact-head workflow runs completed successfully. The existing main-only Windows ARM job was skipped by its workflow condition.
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.
- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.
Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.
Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
Update the Rust workspace and Java SDK dependencies to [Lance
v15.0.0-beta.8](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.8),
preserve the optional `manifest_size` field in remote version responses,
and verify directory-namespace renames preserve registration and data
both within and across namespaces. Update the Rust and TypeScript API
documentation to reflect directory-namespace rename support.
Rust callers constructing `Version` literals must now provide
`manifest_size` (`None` when unavailable). Java is intentionally pinned
to beta.8 even while Maven artifacts are still publishing;
dependency-resolution CI failures caused solely by that publishing delay
are accepted for this upgrade. Python package pins remain unchanged.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
References #3211.
The core `col()` camelCase integration coverage requested by #3211 is already on `main` from #4493. This PR adds the adjacent raw-SQL table-surface regressions that were previously uncovered:
- unquoted `where("camelCase = 1")`
- `update` and `delete` with unquoted camelCase predicates
- `mergeInsert` with a camelCase join key, including matched-update and not-matched-insert input
- `addColumns` `valueSql` referencing a camelCase column
- `createIndex` on a camelCase column
These are characterization tests for the existing local-backend behavior. They do not add `col()` coverage or require a live remote service.
## Summary
- add a regression test for object-form `createTable` on two independent local Node.js connections
- verify both connections use the same table name while preserving each database's rows
## Scope
This test covers the public local `connect(tmpDir)` path. It does not exercise LanceDB Cloud or remote create requests, and it does not establish a remote root cause or a complete fix for #952.
References #952
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.
Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.
OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.
The supported-flows guide documents the Rust-only device PKCE option and
its default.
## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.
## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.
---------
Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
Stacked on #4675.
Remote view create, describe, list, and drop now execute SQL. CREATE returns the stored definition, resolution defaults, and Arrow schema from the same execution; DESCRIBE VIEW returns that metadata and SHOW VIEWS supports scoped pagination.
`drop_view_async()` returns a `job_type="query"` handle whose single `job.id` is the SQL query UUID. Waiting drains the query result once and caches completion; `describe_query(job.id)` works on the submitting connection. Completion means catalog removal, with physical cleanup continuing separately. Query handles have no registry history and cannot be reopened through `open_job()`.
## Summary
Fixes the problems a page-by-page review of the docs found before
launch, after checking each one against the current source and, where
noted, the code. The matching fixes for the dataset and Geneva pages,
which live in `lancedb/docs`, are in lancedb/docs#402.
**Tables**
- `integrations/embedding/cohere`: each input-type description was split
across two table rows and wrapped in literal quotes
(`"`search_document`"`). Now one row per type.
- `integrations/embedding/gemini`: removes the same stray quotes from
the task-type table.
- `integrations/ai/langchain`: the MMR search table's `kwargs` row was
missing a cell, so its description sat under Type.
- Six reranker pages (answerdotai, cohere, colbert, jina, openai,
voyageai): closes the backtick before `_relevance_score`.
- `indexing/vector-index`: finishes a cell cut off at "with scalar".
- `sql/dql/dedupe`: labels the empty header cell in the resolution
table.
**Corrections**
- `curation/sampling`: sampling works in LanceDB Embedded through the
Python SDK, so the page is available in both, with the SQL sections
marked Enterprise.
- `feature-engineering/index`, `add-computed-column`: automatic refresh
is on by default for each computed column and takes effect once a
deployment enables its scheduler. `refresh-column` already said so;
these two implied it was off.
- `indexing/scalar-index`, `indexing/index`: `LABEL_LIST` accelerates
`array_has_all`/`array_has_any`; `array_contains_*` doesn't exist. Also
merges scalar-index's two "nested fields" sections into one.
- `integrations/embedding/ibm`: lists the four current models from
`CURRENT_MODELS` (it listed a deprecated one) and explains that the
default stays a legacy ID so existing tables reload with their model.
- `integrations/reranking/openai`: the class is `OpenaiReranker`, not
`OpenAI()`.
- `integrations/ai/synthetic-data-kit`: the CLI is `synthetic-data-kit`
throughout, each step runs one command, each step passes the file it
writes to the next, and `save-as` uses `-f`/`-o`.
- `integrations/data/dlt`: `secrets.toml` defines `api_token`, which the
pipeline reads, and the run command runs the file the page creates.
- `search/vector-search`: the multivector walkthrough connected to
Enterprise in step 1 and a local path in step 2. It now runs locally
throughout, with a note on connecting to Enterprise, and is renamed so
it doesn't read as a duplicate of the short section above.
**Structure and wording**
- `integrations/embedding/nvidia`: adds headings, and drops test-suite
details aimed at contributors.
- `integrations/ai/genkit`: makes the retrieval test a real heading and
describes the query step.
- `integrations/data/voxel51`: replaces a raw MkDocs `!!! note` with a
`<Note>`.
- `sql/ddl/table`, `sql/ddl/job`: "Errors and limits" is a real heading,
not bold text.
- `curation/windowing`: sidebar title matches the curation overview
("Rank and Deduplicate").
- `deployment/storage`, `basics/agentic-skills`: fixes a garbled
sentence and a stale reference to the quickstart.
**Checked:** no new Vale alerts on the changed pages. Assembled together
with lancedb/docs#402, the site passes `mint validate` and `mint
broken-links --check-anchors --check-redirects`. Every changed page was
reviewed in a local preview.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Updates Rust and Java Lance dependencies to
[v15.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.6),
pins Python tests to published pylance 15.0.0b6, and refreshes both
lockfiles.
Aligns Arrow 59, DataFusion 55, and PyO3 0.29; adapts execution plans
and MemWAL directory handling, updates the ABI metadata test, and
removes the obsolete Lance Arrow patch now fixed upstream.
Validation: workspace clippy with all features/tests and warnings
denied, cargo fmt, Python/CI Ruff checks, uv lock check, 16 CI-script
tests, and targeted Python regressions (54 passed, 28 skipped);
repository-wide Ruff retains existing documentation-example failures,
and the upstream macOS wheel build failed while Linux and Windows wheels
are published.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The Node SDK's vector index options only take `numPartitions`. The Rust index builders (`target_partition_size` in `rust/lancedb/src/index/vector.rs`) and the Python SDK already support a target partition size, so this adds `targetPartitionSize` to `IvfPqOptions`, `IvfRqOptions`, `IvfFlatOptions`, `HnswPqOptions` and `HnswSqOptions` and passes it to their builders. When both are set, `numPartitions` is used, as it is for Rust and Python callers.
The reason to want it: an index built from a partition count records no target partition size, so `table.optimize()` splits and joins it against the index type's default and can merge small partitions down to one or two. Building with a target records it in the index details, and optimize keeps to it. https://github.com/lance-format/lance/pull/10154 fixes this on the Lance side for indexes built from a partition count; this lets Node users set the size directly.
The new tests build IvfPq, IvfFlat, IvfRq, IvfHnswPq and IvfHnswSq indexes with `targetPartitionSize` and check the size recorded in the index details, and check that `numPartitions` wins when both are set. Each option's doc comment states that index type's default target. Docs regenerated with `pnpm run docs`.
## Summary
Adopts the product terminology guide across the docs site pages, SDK
docstrings and error messages. "LanceDB OSS" and local mode become
**LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB
Distributed**, which LanceDB Enterprise provides. Where free vs paid is
the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB
Enterprise". This is a first trial of the Embedded name, so reviewers
can judge it in context.
- **Embedded vs Distributed** for where execution runs (`optimize()`,
consistency, branching, maintenance). Main pages define it once:
"LanceDB Enterprise runs LanceDB Distributed: …"
- **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://`
region, the brute-force KNN limit) and marked legacy. "Cloud and
Enterprise" becomes Enterprise or Distributed.
- **Plan executor** becomes "executor" or the fleet name.
- **Error messages** change, e.g. `optimize is not supported for remote
tables.` → `... in LanceDB Distributed.` Tests that match on them are
updated.
- **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`,
`DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the
terms in prose.
Unchanged: identifiers (`availability.oss`, the `remote` module and
feature, `RemoteTable`, `db://`), heading anchors, internal code
comments, and Geneva.
## Evidence
- Rust lib tests for the changed messages and the Python tests in
`test_remote_db.py`, `test_materialized_views.py` and `test_table.py`
pass locally.
- Not run locally:
- The Node tests: `remote.test.ts` and `table.test.ts` have updated
assertions.
- Vale.
- The `experimental_functions`-marked materialized-view tests.
- `docs/src` API reference was hand-edited to match the docstrings
rather than regenerated.
## Merge Risk
**Door:** two-way; prose and message text only.
**Blast Radius:** `docs`, `error-messages`: callers matching on old
error strings would break.
Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon
(Enterprise pages).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Remote list_indices fills num_indexed_rows / num_unindexed_rows from the
server's enriched response and only leaves them unset for servers that
do not report them. The Node and Rust documentation previously implied
those fields were always unavailable for remote tables.
Synchronize the generated TypeScript API reference with the Node source
comments.
Closes#4301
Co-authored-by: desty <lee.ji.myung@gmail.com>
#2349 changed the default retryable status codes to `[409, 429, 500, 502, 503, 504]`, but only the Rust docstring was updated. The Python and TypeScript bindings — and the generated TypeScript API reference — still document the old `[429, 500, 502, 503]`.
The actual default is set in `ResolvedRetryConfig::try_from` in `rust/lancedb/src/remote/retry.rs`, and `rust/lancedb/src/remote/client.rs` already documents it correctly. This brings the other three in line.
Docs only — no behavior change.
Split out from #3963 per the "one problem at a time" guidance in `CONTRIBUTING.md`.
## Summary
Fixes#3211 by exposing the shared Rust/DataFusion expression builder through Node bindings. Case-sensitive references such as `col("firstName")` can be composed and passed to `where()`, while SQL strings remain supported.
The expression-object representation follows [the design requested on #3290](https://github.com/lancedb/lancedb/pull/3290#discussion_r3632371998). That earlier PR remains open; no claim is made that the issue was unclaimed or that maintainers have selected this implementation.
## Review corrections
- Construct the named native struct correctly and parse casts through `arrow_schema::DataType`.
- Preserve numeric values: use Int64 only for JavaScript safe integers; retain Float64 otherwise. Explicit `litInt` rejects unsupported numbers with a documented `RangeError`.
- Correct test syntax/API calls and align SQL assertions with the shared dialect, including reserved and case-sensitive names.
- Use typed, isolated temporary test tables with cleanup, and cover an actual large-number filter threshold.
- Document the public methods and commit generated API documentation.
Synced with current main `0bceaaf3` through a normal merge. The diff is limited to seven Node source/test/config files and eleven generated documentation files; no Rust core change.
## Validation
[Linux validation run](https://github.com/Ganesh1729-ui/lancedb/actions/runs/38244794288) on Node 24 / Rust 1.97.0 with the repository's `ci` profile and `remote,lancedb/metrics-otel` features:
- Rebuilt native addon, TypeScript build, Biome, Rust formatting and native Clippy pass.
- Expression/query/Arrow/table suites: **1,129 passed**; expression suite repeated: **42 passed**.
- Using the original JavaScript numeric dispatch with the rebuilt addon: **8 regressions fail, 2 controls pass**.
- Strict TypeDoc generation passes. Final local typing/docs/lint checks pass; the PR source equals the validated source except for the separate validation workflow, which is excluded from this PR.
Full provider/model/S3, other feature combinations and OS/Node matrices were not run in this follow-up. Upstream checks and maintainer review remain separate.
## AI assistance
AI was used for implementation, review and testing. No prior human review is claimed.
The remote catalog could only drop an empty database: it always asked
the server for a restricted drop.
`DropDatabaseRequest::cascade`, `drop_database(..., cascade=True)` in
Python and `dropDatabase(name, { cascade: true })` in TypeScript ask the
server to drop everything in the database first. The name stays taken
until the server finishes cleaning up the dropped objects' storage. A
restricted drop remains the default.
Xuanwo
Lance Release
Jack Ye
Sarwar Bhuiyan
Bruno Ramirez
Will JonesandClaude Opus 5.5
Wyatt Alt
Yang Cen
Gaétan
LeoandLeo
lancedb-gatefixer[bot]
TaylorandBrendan Clement
LanceDB RobotandJack Ye
Prince Jain
ryan-haynesandRyan Haynes
geoffgscott
JiMyung Lee
Joshua Snider
J Ganesh Kumar Reddy