Table existence checks should not depend on listing every table or
searching only the first 100 names. Use a direct lookup for `name in db`
and implement `table_exists(table_id)` on synchronous local connections,
matching the remote and asynchronous APIs. Native namespace connections
use their initialized Rust connection for existence checks; custom
namespace clients retain their existing callback. Missing tables return
`False`; other lookup errors propagate.
```python
if "events" in db:
db.drop_table("events")
exists = db.table_exists(["analytics", "events"])
all_names = list(db.list_tables())
```
Clarify that `.tables` contains one page and update deprecation guidance
with safe replacements. Add regression coverage for lookups beyond the
first page, nested namespaces, native and custom namespace clients,
missing tables, permission errors, and lookups that do not list tables.
Include an unpublished draft for the table-management guide until the
new SDK support ships.
We weren’t explicit about the job states returned by the Python SDK.
This documents those values and clarifies the difference between
checking the current status and waiting for completion.
This PR addresses [OSS-2913](https://linear.app/lancedb/issue/OSS-2913)
([#4845](https://github.com/lancedb/lancedb/issues/4845)) and
[ENT-2909](https://linear.app/lancedb/issue/ENT-2909):
`table.optimize()` on an Enterprise table exposes SQL parser errors when
the deployment does not support its generated SQL. The SDK generates the
statement itself, but previously propagated unsupported grammar as a
generic SQL failure. Parser errors and unknown-option errors reported as
`InvalidArgument` now become `NotSupported` (`NotImplementedError` in
Python), retaining the original diagnostic and request ID.
## Problem
The existing Function `column_status` surface cannot decode the Agent
scheduler's `queued` state or the `failure_threshold_reached` and
`binding_revision_missing` blocked reasons. It also preserves only the
latest refresh job, so a later cancellation can hide an earlier actual
failure.
## Change
- Adds the `queued`, `failure_threshold_reached`, and
`binding_revision_missing` wire states in Rust, Python, and Node. The
wire change is additive, while Rust callers must update exhaustive enum
matches and public struct literals.
- Adds optional `last_failure_job` alongside `last_refresh_job`. The
latest cancellation and the most recent actual failure retain separate
meanings.
- Reuses the canonical Rust `JobFailure` and existing Python/Node
`JobFailureInfo` models for nested job failure details instead of
introducing a parallel status error model.
- Maps the existing public `sql_job_id` field through Node's
`JobFailureInfo`.
- Keeps `column_status`, `column_statuses`, and `wait_fresh` as the
existing API surfaces. There is no new endpoint, row-error list, durable
ledger, or unknown-enum fallback.
- Documents `queued` versus `running` and the separate latest-attempt
and latest-failure jobs in the existing hidden freshness release draft.
The new status values are additive but cross-service: `queued`, the two
new blocked reasons, and `last_failure_job` require matching Agent/QN
and SDK deployments before clients can rely on them. This SDK follow-up
does not change the normal `7cc3904a` dependency pin or claim whole
ENT-3487 completion.
## Validation
The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`,
based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The
final product docs commit changes generated TypeDoc only.
- Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust
blobs are unchanged from their tested `788ef85b` source through
`c9bac25e`.
- Python: the PyO3 extension built successfully, and the selected native
PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The
test requests pinned version 42.
- Node: the addon and TypeScript builds passed, Biome passed, and the
selected Jest transport test passed 1/0 with 57 filtered tests. Addon
SHA-256:
`c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`.
- Python extension SHA-256:
`36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`.
- TypeDoc was generated from the native `native.d.ts` produced by that
Node build. A second generation produced no remaining diff.
The public diff contains only product, test, and generated API
documentation changes. Native runner scripts and preserved
compile-failure logs remain private validation evidence.
Agent/QN production mapping, mixed-version rollout, and the remaining
ENT-3487 server-side work are tracked separately.
## Problem
Function-backed columns need an exact `fresh`, `stale`,
`never_computed`, and
`failed` partition for the table version a caller is reading. The
existing
error listing is capped, and composing it with job and schema listings
cannot
prove which rows currently failed.
## Behavior
- Adds a strict freshness reader beside the existing permissive planner.
It
distinguishes an absent certificate from legacy evidence that omits
failures, and from a current certificate carrying failure coverage.
- Extends the immutable freshness sidecar with per-fragment known and
failed
row bitmaps. Partial or `WHERE` publications cover only observed rows;
untouched rows stay unknown. Deleted rows do not block classification.
- Adds failure-aware whole-fragment and row-repair writers. A rejected
older
attempt cannot overwrite failure evidence already recorded for a newer
input
signature.
- Adds `column_status`, `column_statuses`, and `wait_fresh` to Python
and the
corresponding Node methods. The client calls the server's exact status
endpoint, validates the four-way partition and one-version guarantee,
and
preserves the complete Function version, last-job, and automatic-refresh
metadata.
- Adds an MDX-commented draft usage section to the Enterprise
computed-column
refresh guide, covering Python, Rust, and TypeScript and remaining
hidden
until the server integration ships.
- Sends `column`, `branch`, and exact `version` selectors to the status
route.
A pinned handle rejects a nonempty response from any other table
version;
latest handles keep the existing read-freshness headers.
- Retries `wait_fresh` only for HTTP 503 with
`function_column_status_finalization_pending`. Existing failed rows
remain
visible and do not keep the wait pending; other errors propagate.
- Models the shared scheduler projection, including `up_to_date`, real
`blocked` states, their reason, failure streak, and effective cooldown.
## Validation
This is the SDK side of ENT-3563. Sophon's failure-coverage publication
and
status endpoint are a separate integration; deployed end-to-end and
whole-ticket
acceptance remain pending.
- Rust: 9 exact freshness/status tests passed, plus 1 exact
latest-handle
freshness-header test. The pinned transport test was extended with a
wrong-version response and rerun at `ebd6649a` (1 passed). Coverage
includes untouched partial rows,
selected-row failure clearing, cross-generation evidence, typed and
non-retried errors, `up_to_date`, blocked scheduler fields, and
exact-version
requests.
- Python: 2 native PyO3 remote-transport tests passed together,
including
typed retry; one local facade test reaches the existing unsupported
boundary.
- Node: the NAPI addon built successfully, TypeScript and Biome passed,
and the
selected native remote-transport test passed 1/0, including
exact-version
selection and blocked scheduler metadata.
- `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile,
Biome, and
TypeScript checks passed.
Validation is mapped by the relevant unchanged source rather than
treated as one
final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header
and Python
remote tests at `31e68b7d`, the local Python test and Node build at
`6f213584`, and
the final TypeScript/Jest delta at `4f2d5c27`. The final product tree
excludes those five private EC2 runner files and adds
six generated TypeDoc files plus one hidden draft usage section in the
computed-column refresh guide. The Node delta corrected a private
TypeScript
command and reused the unchanged native addon.
## Summary
Adds an optional `max_concurrency` setting to Function registration and
explicit refresh:
- A registered Function version can define the default maximum number of
user-code calls in flight for one logical execution.
- A refresh can override that default for that execution only.
- The value must be a positive `u32` and is independent from
`resource_budget`; it is not an account-wide quota.
- Omitting the option preserves the previous request payload. Local
embedded refreshes reject this Enterprise-only option.
Rust, Python sync/async, PyO3, and the existing Node refresh surface
forward the option. Node has no Function registration surface today, so
no registration API is added there. The generated TypeDoc pages are
included for the new public option.
## Validation
The native-tested source covers Function version decoding, registration
validation and serialization, the Flight SQL registration ticket and
returned version, remote refresh forwarding, local rejection, and Node's
numeric boundary before conversion. Exact stage results are Rust `1+3`
plus four remote cases at `1` each; Python `14/1/49/24`; Node `57`
remote and `1` local rejection. Ruff, standard repository Pyright, and
Biome passed.
The tested Python extension was
`21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the
tested Node addon and generated native declaration were
`f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and
`4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The
final 24-file diff preserves the 22 tested
product/test/fixture/source-doc blobs and adds only two generated
TypeDoc pages.
## Depends on merged #4951
Phase 1 merged as `4d22c0ce2b6825c880c818e737b079db494b4053`, so this PR
now targets `main`. It reuses the canonical
`FunctionWorkerResourceBudget` introduced there rather than copying the
type.
For Function `add_columns` declarations, Python and Rust now persist
`auto_refresh_resource_budget` in computed-column metadata and send it
as `auto_refresh_resource_budget` in the remote request. CPU
milliseconds and memory bytes are required positive integers. Omitting
`gpu_count` is accepted as zero and the canonical outgoing JSON still
contains `"gpu_count": 0`. Unknown fields, missing CPU or memory, wrong
types, zero CPU or memory, negative values, and overflow are rejected.
The same budget is applied to every public output binding of a
multi-output Function declaration. Existing metadata is retained, and
SQL-expression declarations or local execution reject the budget instead
of silently ignoring it. Node is unchanged because its current
`addColumns` surface only accepts SQL computed expressions and has no
Function declaration path.
This change does not implement fleet call concurrency limits or enforce
the budget in the runtime. The Enterprise server must support the nested
Function worker resource-budget contract before these values can act as
execution limits.
## Validation
- Python extension build passed.
- Remote Function declaration and local unsupported-path rejection each
passed.
- Canonical metadata round-trip passed, including omitted GPU decoding
to zero and explicit `gpu_count: 0` on encode.
- Function multi-output remote wire test passed with `--features
remote`.
- Repository-scoped Python type check passed with 0 errors.
- All runtime source blobs match the native-tested source. The
post-native deltas are one documentation example line and an existing
routing-test stub update for the new optional argument; the affected
parameterized test passed 3/3 against the existing extension build. No
private test runners are part of this PR.
## Problem
Function catalog lookup required an exact version string even though
callers
commonly need the latest currently visible immutable revision. On the
current
SDK, resolving that version required an out-of-band catalog query before
calling
`get_function`.
## Change
- Make Python `get_function(name)` default to latest while accepting
`int` or
`str` for exact lookup.
- Normalize integer versions to their decimal string form and reject
`bool`
explicitly. Existing string validation and errors remain unchanged.
- Thread `None` through the rebuilt PyO3 binding and route it to the new
Rust
`Connection::get_function_latest`.
- Resolve latest through the existing catalog `DESCRIBE FUNCTION` path
without
`WITH (version = ...)`. The returned value is one exact immutable
version, not
a mutable alias.
- Preserve the existing Rust exact-version API.
## Validation
On the designated EC2 host with the rebuilt extension:
- `test_local_function_catalog_operations_are_not_supported`: **1
passed**.
This crosses the Python connection, rebuilt PyO3, and embedded Rust
`NotSupported` boundary with the omitted-version call.
- Async and sync catalog-path tests: **2 passed**, 117 deselected.
- Remote SQL Function creation/description integration test: **1
passed**.
- An earlier Rust invocation with the wrong module path matched **0
tests**;
it is retained as harness evidence and is not counted as a pass.
## Scope
This is the Function lookup SDK slice. It does not include the separate
atomic
rebind/publication server work tracked under ENT-3418.
## Summary
- Let local plain queries over row-addressable legacy v1 and v2 blob
columns use `blob_mode="bytes"` without importing `pylance` through
pandas, batch, or Arrow conversion.
- Preserve aliases, filters, ordering, limits, null and empty payloads,
empty result schemas, and the caller's `_rowid` choice.
- Build the hidden row-id query separately so reusing an async builder
for lazy reads or non-blob projections cannot retain an automatic
`_rowid`.
- Keep byte reads closed for blobs inside lists, remote or namespace
pushdown, and query shapes whose result rows cannot be matched back to
the source. Blob v2 lazy auto-row-id behavior is unchanged.
Fixes#4645
## Validation
- 26 focused Python tests passed with the test subprocess blocking all
`lance` imports. The persisted fixture writes and reopens a storage-v2.1
table, then covers sync and async pandas, batches, and Arrow conversion
plus async builder reuse.
- The existing unsupported vector and hybrid reranker regressions now
assert the current eligibility diagnostic, and the legacy table-level
Arrow regression now asserts successful bytes, schema, values, and
hidden row IDs. Both focused tests passed.
- The runtime receipt records no installed `pylance` distribution and no
importable `lance` module. The Python-only change was verified with an
existing compatible 0.42.0b9 extension; the relevant query and fetch
APIs are unchanged between that source and this 0.42.0b10 base.
- Ruff format and lint pass for every changed Python file.
## Summary
- Add the canonical Rust `FunctionWorkerResourceBudget` and
`RefreshColumnOptions::resource_budget()` for explicit Enterprise
Function refreshes.
- Forward the nested budget through Python sync/async, PyO3, Rust remote
requests, Node NAPI, and TypeScript.
- Require positive CPU and memory values; normalize an omitted nested
GPU count to numeric zero; reject the nested budget when combined with
the top-level `gpu_count` shorthand.
- Reject `resource_budget` on LanceDB Embedded refresh paths instead of
silently ignoring it.
- Document the Rust, Python, and TypeScript APIs, including that SQL
servers currently require all three budget fields.
The Rust, Python, and TypeScript additions use `cpu_millis` /
`cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` /
`gpuCount`. The top-level GPU shorthand keeps its existing behavior and
remains mutually exclusive with the complete nested budget.
## Scope and compatibility
This change configures only the explicit refresh call. It does not
configure `add_columns` automatic refresh and does not introduce a
fleet-wide call-cap interface.
A LanceDB Enterprise server must expose the nested `resource_budget`
backfill contract. LanceDB Embedded continues to support its existing
expression refresh defaults and rejects the new complete budget
explicitly.
## Validation
Validated against the final signed product source:
- Rust: resource-budget remote serialization test `2 passed`; validation
test `1 passed`.
- Python: actual PyO3 build; remote-request matrix `22 passed`;
invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff
check and format clean.
- Repository-standard Pyright entry point: `0 errors`, within the
repository's configured include set.
- Node: native build and Biome lint passed; the existing remote-request
test `1 passed`, including fractional, negative, overflow, NaN, and
infinity nested GPU cases; the existing local-rejection test `1 passed`.
- TypeDoc generation and generated-document cleanliness passed.
- The native-tested product source differs from this head only in two
semantically equivalent Node range checks required by Clippy. Current CI
compiles and lints the final blobs. No dependency or lockfile changes
are included.
Add an automated prompt catalog section to the agentic skills page. The
combined docs build fills its insertion marker from the evaluation
source, avoiding a manually maintained prompt list. Keep the evaluation
context and fixture guidance in this page so docs reviewers can edit it
independently of the generator. Existing workflow examples remain in
place.
Part of UAX-192.
Depends on https://github.com/lancedb/docs/pull/434; merge that docs
pipeline change first. Validation: combined assembly and Mintlify
validation pass.
Unsupported remote blob materialization errors now point to row-ID
selection, fetch_blobs and fetch_blob_files. Document descriptor
structs, fetched bytes and Function BlobReader inputs, and show a
bounded subset-copy recipe that keeps the source unchanged between
selection and fetching.
Adds guidance assertions and copy coverage preserving payloads, empty
blobs and nulls.
No new blob materialization API is introduced.
Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
Python refresh APIs currently reject `publication_interval_rows`, so
callers cannot request incremental Function output publication through
the SDK. Add the optional typed parameter through the sync/async Python
APIs, PyO3, and Rust `RefreshColumnOptions`, and serialize it as a
top-level Enterprise refresh option.
An omitted value preserves the existing wire request and whole-fragment
default. Positive values set a publication threshold; zero and negative
Python values fail before a request is sent. Embedded refreshes reject
the Enterprise-only option. Documentation describes a threshold rather
than an exact number of rows per publication.
Validation on the designated development host:
- A matching baseline build reproduced the missing public keyword
argument in both sync and async submission tests.
- The updated PyO3 build passed 54 focused Python tests; Rust payload,
invalid-option, and default-wire tests each passed. Rust formatting and
Python Ruff checks passed.
- The single signed delivery commit differs from that tested
implementation only in a Rustdoc correction describing Embedded
rejection; formatting passed again. No dependency pins or lockfiles
changed.
The matching SDK is prepared for the separate server validation of
failure/cancellation retention and missing-only refill. Those end-to-end
results are still pending and are not implied by these SDK tests.
Refs [ENT-3733](https://linear.app/lancedb/issue/ENT-3733).
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.
```python
job = view.refresh_async(
cpu_millis=8000,
memory_bytes=16 * 1024**3,
gpu_count=2,
gpu_class="h100",
)
job.wait()
```
```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
gpu_count = 2, gpu_class = 'h100');
```
The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.
Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.
The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.
Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
Adds Python where= examples next to the SQL ones.
Also fixes two examples that fail:
- Date filters now use date '2026-01-01'. A bare string fails on DATE
and TIMESTAMP columns (see [Dates, Timestamps,
Decimals](https://docs.lancedb.com/search/filtering#dates-timestamps-decimals)).
- The first example reads results with table.search().to_arrow(),
because table.to_arrow() raises on Enterprise.
where= needs lancedb 0.42.0 or later.
The Rust example downloads in the ingestion, search, reranking and SQL
guides point to `/static/code/*.zip` URLs that return HTTP 404 on the
published docs site. Replace all 15 references across 13 pages with
public GitHub downloads pinned to commit
`2af85bea8999c8cd23ec01ff62ba110c54652c10`.
The three archives already exist in this repository. Pinning their URLs
preserves the exact source, dependency manifests and lockfiles used by
the existing `cargo run --locked` instructions. Only link destinations
change.
Validation:
- All three public download URLs return HTTP 200 with ZIP content.
Downloaded bytes match the repository archives and the deployed docs
source archives exactly, and ZIP integrity checks pass.
- Verified that the diff contains exactly 15 URL replacements in 13
pages. No example code, run instructions, manifests or lockfiles
changed.
- `git diff --check` passes.
- `mint broken-links` reports the same 16 existing missing internal
links on both the unchanged `main` baseline and this branch.
- `mint validate` reports four existing missing snippet exports
(`RsExampleBody` and `RsExampleTail` in the reranking and vector-search
pages); the unchanged baseline has the same warnings. The combined docs
build and internal-link check pass with the equivalent download-link
fix.
The server's backfill request takes a row filter, failure limits,
per-call timeouts and worker sizing, and SQL exposes all of them through
`REFRESH COLUMN ... WHERE ... WITH (...)`, but the typed SDKs could send
only batch_size, GPU placement and the retry and error policies. Anyone
needing a filtered refresh or a bounded skip policy had to drop to SQL.
This adds `where`, `max_failed_fraction`, `max_failed_rows`,
`row_timeout_seconds`, `init_timeout_seconds`, `num_workers` and
`max_executor_processes_per_worker` to Python, Rust and Node, passed
through unchanged. `RefreshColumnOptions` now serializes straight into
the request body, so the three SDKs share one producer. Node gets a
`RefreshColumnOptions` object, which also carries the options it lacked
before. A LanceDB Embedded refresh rejects every option other than
batch_size and GPU placement, naming the option.
Lance beta.11 calibrates lower `search_effort` tiers by metric and
requested `k`. With the Rust dependency and Python/Node execution tests
already on main via #4881, document the tiers across Rust, Python, Node
and the search guide, extend the remote request matrix to cover `0.1`
and `0.25`, and align the Python test package with beta.11.
`0.1` and `0.25` target empirical mean partition-routing recall of 80%
and 90% for L2, cosine and dot through `k=100000`. These are not
per-query or final Recall@k guarantees. Default Auto is unchanged;
Hamming and larger `k` remain uncalibrated, and remote queries require
corresponding server support. This PR makes no additional performance
claim. See [Lance
#10695](https://github.com/lance-format/lance/pull/10695) for
measurements and limitations.
Validation: both strict workspace Clippy configurations, Rust
formatting, 13 Rust effort tests, 208 Python query tests, 41 Node query
tests, native binding builds, TypeScript, Node lint/generated docs,
Python lint/format, full lockfile check and two dependency guards passed
on `a8b09170`. All validated Rust/Python/Node source and dependency
files are byte-identical at `2a82d07c`; the only subsequent change
restores the Java POM to main. Root Ruff has 59 existing diagnostics and
six formatting files, all byte-identical to the base; the Python package
passes.
The Java REST/Namespace client has no changed API here, and its
dependency remains identical to main. The separate [Java beta
publication failure](https://github.com/lance-format/lance/issues/10773)
is not claimed resolved or validated. Official SDK release, automatic
mirror, Sophon adoption and measured consumer acceptance remain
follow-up stages.
Remote Python `drop_function` currently discards the dependent-view
warnings returned by SQL DROP. Both the synchronous and coroutine APIs
now emit those warnings through `warnings.warn` while preserving their
boolean result and executing DROP once. The server support is already
merged in lancedb/sophon#9820.
The Rust decoder reads warnings across every result batch, including an
empty first batch. Missing warning columns remain compatible with older
servers; malformed or null warning lists are rejected. The hidden
binding bridge carries the warnings to Python. The separate
`drop_function_async` API and its `(bool, Job)` contract are unchanged.
Python warning filters can raise after the catalog mutation has
completed.
Validation: a compiled PyO3/public-server reproduction failed before the
fix and passed afterward; the public async path also returned `true`
with the expected dependent-view warning. Final source passes two
focused Rust tests, two Python wrapper tests, strict targeted Clippy,
formatting, and Ruff. Three decoder tests cover batch/null/type and
older-server behavior. The public runs preceded a two-file
simplification of the completed Job projection; focused final-source
checks cover that delta. Current-main integration is conflict-free, and
its overlapping changes concern independent retry-policy APIs/tests.
Documentation now covers warning filters, the post-removal
warnings-as-errors behavior, and silent compatibility with older
servers. The follow-up changes only prose and a Python docstring; the
runtime validation above remains applicable.
Refs ENT-3659. Whole-issue independent acceptance remains pending.
## Summary
Bumps dependencies that Aikido flags with critical advisories, where a
patched version is available without code changes.
| Package | Where | From → To | Ships to users |
|---|---|---|---|
| jackson (BOM) | `java/pom.xml` | 2.18.11 → 2.18.12 | Yes, via the
published pom |
| zerovec / zerovec-derive | `Cargo.lock` | 0.11.6 → 0.11.8 / 0.11.3 →
0.11.7 | Yes, linked into the wheels and Node binaries |
| pillow | `python/uv.lock` | 12.3.0 → 13.0.0 | No, dev lockfile |
| pyarrow | `python/uv.lock` | 24.0.0 → 26.0.1 | No, dev lockfile |
| pyjwt, urllib3, anyio, cryptography | `python/uv.lock` | → 2.15.1,
2.8.0, 4.16.0, 50.0.2 | No, dev lockfile |
| pillow | `docs/web-tests/blob-v2/py/requirements.txt` | 12.1.1 →
13.0.0 | No |
pyarrow was held at 24.0.0 by a `pyarrow<25` pin in the `tests` extra.
It came in with #3675 without a stated reason, so this PR removes it.
pillow 13 and pyarrow 26 require Python 3.11+, so the lockfile keeps
pillow 12.3.0 and pyarrow 25.0.1 on Python 3.10, and those remain
flagged.
The sharp findings in the Node examples and docs are not addressed here.
They come from the `@huggingface/transformers@3.0.2` optional
dependency, which needs a major upgrade to 4.3+.
## Evidence
With the pyarrow pin removed and pyarrow 26.0.1 installed, the Python
suite (`-m "not slow and not s3_test"`) passed locally on Python 3.12:
3319 passed, 164 skipped. That run was on main as of #4879; this branch
is based on a slightly newer main.
## Merge Risk
**Door:** Two-way. Patch and minor version bumps that a revert undoes.
**Blast Radius:** Java consumers (jackson patch release), native
binaries (zerovec patch release), Python test environments.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Expose `FunctionRetryPolicy` on Function declarations and refresh
overrides, together with `error_policy` and recorded attempt counts.
Python uses the Rust validation rules before sending a request; local
refreshes reject these Enterprise options.
The policy bounds calls and backoff, supports
class/message/provider-code matching, and can disable retries with
`max_attempts=1`. Batch retry and skip operate on the whole failed call.
Requires Enterprise server support for the new policy.
Validation: rebuilt the Python extension; 47 targeted Python tests and
10 Rust tests passed. Python `make check`, Rust formatting, and `cargo
clippy --features remote --tests --examples -- -D warnings` passed. The
additional repository-root Ruff sweep reports existing errors under
`docs/web-tests`; the Python workspace checks pass.
Draft user guidance covers the supporting server requirement, safe
repetition of partial work and whole-batch exhaustion. The guide
sections stay hidden until release. Vale 3.24.0 passed for the added
draft prose; no paired deployed SDK/server retry acceptance was run.
Companion server: https://github.com/lancedb/sophon/pull/9812.
BREAKING CHANGE: Rust callers constructing `FunctionRegistrationRequest`
or `FunctionErrorRecord` with struct literals must supply the new
`retry_policy` or `attempts` field, respectively.
- Removes the stale `/enterprise` redirect that hijacked the real
LanceDB Enterprise page, sending it to the Embedded vs Enterprise
comparison page instead.
- Removes the Merge page from the Data Ingestion nav (confusing parity
story between Embedded and Enterprise); moves RTE Write up to right
below Delete and renames it **RTE Write (Upsert)** to take over that
slot.
- Removes the Early access badge from Tiered Index Cache and restores
its full sidebar title (was shortened to "Index Cache" to fit the
badge).
- Moves Changes below Dedupe in Data Curation.
- Matches Advanced Conflict Resolution's sidebar title to its page title
(previously just "Conflict Resolution" in the sidebar).
- Marks Job Framework as Enterprise-only.
- Distributed RTE Write diagram now shows the `metrics` table split
across multiple shards, illustrating the per-table sharding described in
the page's own text.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_018PXMCbRH6i5w47TsjeEUMT
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The refresh guide and SQL reference do not warn that a `WHERE` refresh
recomputes filled rows, and the guide describes resources as per-worker
allocations. Clarify the aggregate Function budget and expression-only
`num_workers`, show the output-null predicate with its stale-value and
successful-null limitations in both references, and document
refresh-level `error_policy='skip'`. Link both guides to the current SQL
options route.
Validation: Vale 3.24.0 found no issues on changed lines; `git diff
--check` and the new link targets passed. Mint 4.2.888 reports the same
four existing `RsExampleBody`/`RsExampleTail` import warnings in
`reranking/index.mdx` and `search/vector-search.mdx` on both pristine
base `5e38120b` and this branch. No additional warnings. Behavior was
checked against the owning Sophon implementation; no runtime code
changed or runtime tests were run.
Companion to https://github.com/lancedb/sophon/pull/9806 for ENT-3279.
Page visibility and saved documentation editions are unchanged. DOC-90
retains publication and matching-shipped-release verification.
## Summary
- **Define Storage Integration**: un-gates the Python (and Rust, for S3)
`create_storage_integration` examples, drafted against #4618 and held
back until it shipped -- it's now in v0.42.0-beta.0+, and the signature
matches the drafts exactly.
- **Naming rules**: drops the `Branch` row and its "Learn more" link,
consistent with de-emphasizing branches elsewhere in this nav.
- **Distributed RTE Write**: removes the REST path, the
`__lance_mem_wal` index name, and "WAL node" terminology in favor of an
abstract framing -- a query node routes writes to RTE nodes, each of
which can host shards from many different tables. Both diagram variants
are updated to match.
- **Data Page Cache / Tiered Index Cache / Cache Prewarm**: drop
Helm-specific and exact-percentage detail (disk-budget math,
`startup_prewarm_block_query_ready`/`healthPort`/failure-policy table)
in favor of a short feature → architecture → why-it-matters shape, each
closing with a contact-sales pointer. Data Page Cache's on-demand
prewarm section is removed outright.
## Note
I wasn't able to get a clean visual render of the updated Distributed
RTE Write diagram in local preview -- Mermaid isn't rendering in
headless Chrome in this environment for *any* page right now, confirmed
against an unmodified reference page showing the identical symptom. The
diagram source follows the exact pattern already shipped and working in
the 21 diagrams from the earlier brand-theming pass, and `mint
validate`/`mint broken-links` both pass clean.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
`take_row_ids` used an `IN` filter, so the backend could reorder rows
and collapse duplicate IDs. That can silently misalign training rows
with labels or sample weights.
Restore the requested order and duplicate occurrences in the shared Rust
take path for native and remote tables, and retain that information
through Python query serialization. Missing or deleted IDs are omitted.
Explicit Rust `order_by` overrides request order, and pagination applies
after restoring occurrences. Empty projections and `_rowid` projection
aliases retain their output schema.
RTE tables reject unsupported takes with the existing base-table
guidance before checking the internal row-ID projection used for
ordering. An explicit `use_rte(False)` reads the base table with the
same requested order and occurrences.
A pre-existing fixture defect also blocked CI:
`test_table_add_in_threadpool` responded without reading uploads,
allowing a connection reset during response decoding. This PR includes a
fix confined to that test's handler: consume each request body before
responding and verify one create row plus ten inserted rows. This was
not a regression in the take implementation. Production write retries,
concurrency and HTTP client code are unchanged.
The selective sampling guide scopes the row-ID guarantees to SDK
releases containing this fix, preserves the distinct offset-take
semantics, and explains that ordered takes collect and combine matching
rows before output batching. It recommends smaller ID requests and
narrow projections because `max_batch_length` does not cap working
memory. This uses the existing REST query protocol and requires a new
SDK release; no deployed service was changed.
Validation:
- Docs follow-up: the changed page compiles with the Mint 4.2.888 MDX
compiler; Typos 1.26.0 passes and Vale 3.24.0 has no findings on changed
lines. Full-site Mint validation reports four existing snippet-import
warnings on the reranking/vector-search pages, identical with the
original page restored in a baseline comparison.
- The applied threadpool test passed locally after the fixture change.
The identical candidate previously passed 10 trials with the ordinary
reader and 10 with an unbuffered reader; each trial verifies all
uploaded rows. Those checks used macOS arm64/Python 3.14/default
features, not the full failing CI environment. The normal new-head macOS
Python 3.13/experimental-functions job remains the required platform
check.
- Python package Ruff format/check passed. Repository-root Ruff still
reports 59 existing errors and six formatting differences in unchanged
`docs/web-tests` files; none are in changed files.
- At `a569d3f0`, the RTE diagnostic regression was reproduced and fixed;
5 targeted cases and strict workspace Clippy passed. The fixture-only
follow-up changes no production Rust/Python code, so that evidence is
reused.
- Earlier validation at `6bcc0878`: 439 Rust query/remote-table tests
and 24 Python take/getitems tests passed; the focused 49-test suite also
covered offset dataloaders. A local 9,216-row, four-fragment
reproduction returned 200 seeded random batches of 64 rows with zero
ordering or label-alignment mismatches. The ordering and projection
implementation is unchanged by the follow-ups.
- No live Enterprise run was performed.
Fixes [ENT-3469](https://linear.app/lancedb/issue/ENT-3469).
The client registers Functions, but the decorator that packages one was
called `udf` and produced a `UdfDefinition`. Rename them to `function`
and `FunctionDefinition`, and say "function" instead of "UDF" in the
docstrings, error messages and docs around them.
`udf` and `UdfDefinition` stay as deprecated aliases that emit a
`DeprecationWarning`. The source packager strips either spelling of the
decorator, so `@udf` and `@function` produce the same artifact and
digest, and the wire contract is unchanged. DataFusion and Lance UDF
types, and Geneva's `@udf` in the integration docs, are unrelated and
untouched.
Labels early access features and hides features we aren't documenting
yet.
**Early access label:** Functions (4 pages, plus the GPU refresh,
checkpointing and adaptive execution pages), Secrets (3 pages), Tiered
Index Cache, Blob V2, and merge insert, which is labelled for Enterprise
only. Two sidebar titles are shorter so the tag fits.
**Hidden:** branching, JSON columns and geospatial.
- The JSON functions page and the agent branch-experiments tutorial go
in `.mintignore`.
- `tables/branching` stays reachable but out of the nav, as #4859 left
it, because SDK error messages link to it.
- Other pages' mentions of these features become draft comments, so
they're easy to restore.
**Secrets page:** drops references to manual KMS setup, which the
control plane handles. It keeps the encryption-at-rest facts, states the
Azure value-size limit inline, and points a 501 error to support.
## Summary
- Hides `tools/cli.mdx` from the sidebar and from sitemap/search via
`.mintignore`, unlinking its one reference in `basics/sql.mdx`. No SDK
doc-link depends on this page.
- Removes `tables/branching.mdx` from the Management nav group, but
leaves it reachable (no `.mintignore` entry): the SDK's
`diff_branch()`/`cherry_pick()` error messages link to
`/tables/branching#diff-a-branch` and `#merge-a-branch`, and
`rust/lancedb/src/docs.rs`'s link-resolution test asserts those paths
aren't `.mintignore`'d.
- Adds a Console row to the comparison table, linking to the companion
change's new `enterprise/console` page.
## Companion change
lancedb/sophon#9783 adds `enterprise/console.mdx` and inserts it into
the "Getting Started" nav group this PR touches.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The Search group covers full-text and hybrid search as well as vector
search, and full-text search is lexical, not semantic. This renames the
group to "Search" in the sidebar and in the Embedded vs Enterprise
comparison table.
`mint validate` and `mint broken-links --check-anchors` pass on the
assembled site.
Function refresh failures and skipped-row diagnostics lose the traceback
when decoded by the SDK. Expose the server-provided optional `traceback`
on job failure details and Function error records in Rust, Python, and
Node.js, while keeping existing compact error messages.
Responses from older servers remain supported when the field is absent.
Tracebacks may be truncated by the server. The Rust `Error::JobFailed`
variant now stores `Box<JobFailure>` so the added diagnostics do not
enlarge every `Result` error payload.
## Summary
- Adds a new top-level "Introduction" nav section before "Getting
Started": LanceDB, LanceDB Embedded, LanceDB Enterprise, Embedded vs
Enterprise.
- Trims the landing page's combined
Architecture/Embedded/Enterprise/Compare block down to a short pointer
section (keeping its `#architecture` anchor), moving that content into
two new pages: `basics/lancedb-embedded.mdx` and
`basics/embedded-vs-enterprise.mdx`.
- `basics/embedded-vs-enterprise.mdx` consolidates the
feature-comparison table and the Enterprise overview's own comparison
content (previously duplicated across multiple pages and, within this
page, repeated across a dimension table, several narrative subsections,
and a usage-differences list) into one comparison page with each section
covering a distinct angle.
- Repoints the `/enterprise` and `/enterprise/deployment` redirects at
`/basics/embedded-vs-enterprise`, since the previous destination only
resolves once the Enterprise overlay is applied and fails
`rust/lancedb/src/docs.rs`'s link-resolution test against this repo's
own `docs/web` tree.
## Companion change
lancedb/sophon#9718 trims the Enterprise overview page to drop the
content that moved here, and relocates its nav insertion into the new
Introduction group.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
Follow-up to the Mermaid brand-theming PR, addressing review feedback:
- Diagram text was hard to read at the previous 15px font size; bumped
to 18px, light and dark alike.
- The dark-mode orange accent border was brighter and more saturated
than its fill or the default border, reading as too loud against the
dark background. It now reuses the same muted orange already used for
the light-mode accent border, which was fine as is.
## Companion change
lancedb/sophon#9719 makes the same two changes across its own diagrams.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Remote SQL failures currently become generic runtime errors, and HTTP
table lookup can discard server error metadata. Preserve transport
status, current request ID, SQL job ID and worker request ID across Rust
and Python. Select typed column, table, reference and schema exceptions
only when an explicit server code matches the transport status.
Remote `HttpError` and `RuntimeError` handlers remain compatible. Failed
SQL jobs retain `JobFailedError` with a typed SQL cause when available.
Missing metadata stays absent; message text never determines the
exception class. Local missing-column errors retain their type and
qualified-field suggestions, and `JobFailure.source` documents
structured remote Flight SQL failures.
This branch integrates main at `a9ad4997`, including its released Lance
15.0.0-beta.9 dependency update and remote optimize support. The only
textual conflict was two independent additions to `test_remote_db.py`;
both test groups and all original assertions are retained. The feature
has no dependency or lockfile delta against current main.
SDK runtime validation on `03ee633a` (runtime source unchanged in
`f3f2d0b8`), macOS ARM64, pinned Rust 1.97.0 and the frozen uv Python
3.14 environment:
- Rebuilt the native extension from the integrated source. All 22
targeted Python tests passed: native local errors, HTTP/Flight metadata,
legacy sync/async table-not-found behavior, and the three main
remote-optimize regressions.
- All 52 Rust SQL, metadata/source and nested-field tests passed,
including the new main optimize cases.
- Root and Python-binding Rust Clippy with `-D warnings`, Rust
formatting, Python package Ruff and conflict-file formatting passed.
- Repository-wide Ruff reports 59 existing documentation findings, and
its format check identifies six documentation files. Every affected file
is byte-identical to current main; these checks are not reported as
passing.
- Direct integration review and AST comparison confirm both conflicting
test groups were preserved. Earlier exception-constructor/pickling
evidence remains applicable to the unchanged Python error modules.
The unlocked build exhausted disk again on a runner with the same label
but only 9.4 GiB free on a 72 GiB root filesystem. Keep the existing
runner and dependency caches, serialize Cargo compilation, and remove
only completed test/bench executables reported by Cargo. This job never
runs those binaries; normal binaries, libraries and build-script outputs
remain available. Fresh resolution, the ci profile, all features, and
test/bench compilation and linking are retained.
Three probes using Cargo and target-path validation pass with `-D
warnings`: all fixture targets link while ordinary outputs remain
usable; an undefined symbol still produces Cargo exit 101 and its linker
diagnostic; files outside the target directory are never removed. The
[current-head build-no-lock
job](https://github.com/lancedb/lancedb/actions/runs/41911613851/job/126860364065)
passed on `f3f2d0b8`: all compilation/linking completed, 20 completed
test executables were removed, and `target` retained about 7.3 GiB. This
successful run had a 145 GiB root with about 78 GiB free afterward; it
was not an identical rerun of the failed 72 GiB image. The driver tests
also passed in this job. All previous-head jobs ended naturally before
the push; new-head CI and review gates still apply.
Typed remote input exceptions require the server metadata contract.
Older servers retain generic transport errors and existing missing-table
compatibility. The optimize tests use mock servers and do not establish
downstream server acceptance.
BREAKING CHANGE: Rust callers with exhaustive matches on
`lancedb::Error` must handle the new `FieldNotFound` and `Sql` variants.
Local `Error::ColumnNotFound` failures now raise
`lancedb.exceptions.ColumnNotFoundError`, a `ValueError` subclass.
Python callers that previously caught only `RuntimeError` for this local
failure must catch `ColumnNotFoundError` or `ValueError`.
`Table::optimize()` previously returned `NotSupported` for remote
tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE
TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL.
Only `OptimizeAction::Compact` without a custom `remap_options` or
non-default `CompactionOptions` is supported remotely.
`OptimizeAction::All`, `::Prune`, and `::Index` all return a clear
`NotSupported` error instead of silently doing the wrong thing. A
branch-scoped table handle submits a `WITH (branch = '<branch>')`
clause; the statement doesn't validate the branch exists itself, so an
unknown branch surfaces as an async job failure rather than an error
from this call.
The server's SQL job adapter doesn't expose compaction metrics, so
`OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a
placeholder rather than real numbers — it must still be `Some`, since
callers (including the Python/Node bindings) unwrap it unconditionally
on success. This mirrors how `refresh_materialized_view_async` already
handles the same limitation for its own metrics.
The Python and Node `optimize()` convenience methods call a new shared
`Table::optimize_full`, which runs `Compact` and then, on a remote
table, skips `Prune` and `Index` instead of attempting and failing on
them — returning zeroed prune stats rather than submitting a compaction
job and then raising. Python's `RemoteTable.optimize()` also now
forwards `compaction_options` to the Rust layer, so a non-default value
is rejected before submission instead of being silently dropped.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Lance Release
Jack Ye
Brendan Clement
Bruno Ramirez
Xuanwo
Lucia Cerchie
Sarwar Bhuiyan
Taylor
Wyatt Alt
Yang Cen
Will JonesandClaude Opus 5.5
lancedb-gatefixer[bot]