Function jobs can now export their recorded execution timeline with
`job.export_timeline(path)` or `await job.export_timeline(path)`. The
SDK fetches a fresh Chrome Trace Event recording that opens in Perfetto,
including worker split spans and timestamped job phases. Batch counts
and latency statistics remain details rather than invented batch
intervals.
```python
job = db.open_job(job_id)
job.export_timeline("refresh.trace.json")
async_job = await async_db.open_job(job_id)
await async_job.export_timeline("refresh.trace.json")
```
The export does not wait for completion or cancel the job. Running jobs
return partial recordings. Local operations and SQL query handles remain
unsupported, and servers must support the timeline endpoint. The
destination is touched only after a successful fetch; network and
filesystem errors propagate. Rust freshness wrappers forward the
operation without changing the submitting table's read baseline.
Update the Rust workspace, Java dependency, and Python test dependency
to [Lance
v15.0.0-beta.17](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.17).
This includes the FRI cache and vector-index changes required by current
Sophon and retains the RaBitQ prewarm fixes. Lockfiles use the published
source and test wheels without unrelated registry version bumps.
The dependency dry run also exposed CI resource limits. Use the existing
larger x64 runner for GNU Node and compatibility-wheel builds; allow 60
minutes for Mac Python tests, 120 minutes for the compatibility wheel,
and 150 minutes for the Mac release wheel. Disable the CLI toolchain
action's automatic target cache so only the subsequent downloads-only
cache runs. Release optimization, the compatibility wheel's manylinux
container and x86-64-v2 baseline, and test coverage stay unchanged.
Validation:
- Three native SDK regressions pass: refined RQ queries after deferred
compaction, deferred index remapping, and ANN approximation planning.
- Pinned Rust 1.97.0: `cargo clippy --quiet --locked --features remote
--tests --examples -- -D warnings` passes.
- Locked offline Rust metadata, Python lock validation, Ruff
lint/format, and diff checks pass. The Python lock contains six
published beta.17 wheels.
- Repository YAML, whitespace and typo checks pass for the workflow
changes. Product-source checks are reused for the subsequent CI-only
commits.
- At head `07033dc6`, all 65 applicable pre-review CI checks pass,
including the compatibility wheel, Mac CLI, GNU Node, Mac Python tests,
release wheels, and Java checks. Six checks are skipped by tag/main-only
conditions or draft-only app behavior; the automated review and security
checks run after the Ready transition.
The upstream Java publisher succeeded and its beta.17 Maven artifact is
available. Six upstream Python wheel builds/uploads succeeded; its
Windows ARM job was cancelled. These checks do not establish Sophon
performance or aggregate-memory acceptance.
## Problem
The existing Function `column_status` surface cannot decode the Agent
scheduler's `queued` state or the `failure_threshold_reached` and
`binding_revision_missing` blocked reasons. It also preserves only the
latest refresh job, so a later cancellation can hide an earlier actual
failure.
## Change
- Adds the `queued`, `failure_threshold_reached`, and
`binding_revision_missing` wire states in Rust, Python, and Node. The
wire change is additive, while Rust callers must update exhaustive enum
matches and public struct literals.
- Adds optional `last_failure_job` alongside `last_refresh_job`. The
latest cancellation and the most recent actual failure retain separate
meanings.
- Reuses the canonical Rust `JobFailure` and existing Python/Node
`JobFailureInfo` models for nested job failure details instead of
introducing a parallel status error model.
- Maps the existing public `sql_job_id` field through Node's
`JobFailureInfo`.
- Keeps `column_status`, `column_statuses`, and `wait_fresh` as the
existing API surfaces. There is no new endpoint, row-error list, durable
ledger, or unknown-enum fallback.
- Documents `queued` versus `running` and the separate latest-attempt
and latest-failure jobs in the existing hidden freshness release draft.
The new status values are additive but cross-service: `queued`, the two
new blocked reasons, and `last_failure_job` require matching Agent/QN
and SDK deployments before clients can rely on them. This SDK follow-up
does not change the normal `7cc3904a` dependency pin or claim whole
ENT-3487 completion.
## Validation
The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`,
based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The
final product docs commit changes generated TypeDoc only.
- Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust
blobs are unchanged from their tested `788ef85b` source through
`c9bac25e`.
- Python: the PyO3 extension built successfully, and the selected native
PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The
test requests pinned version 42.
- Node: the addon and TypeScript builds passed, Biome passed, and the
selected Jest transport test passed 1/0 with 57 filtered tests. Addon
SHA-256:
`c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`.
- Python extension SHA-256:
`36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`.
- TypeDoc was generated from the native `native.d.ts` produced by that
Node build. A second generation produced no remaining diff.
The public diff contains only product, test, and generated API
documentation changes. Native runner scripts and preserved
compile-failure logs remain private validation evidence.
Agent/QN production mapping, mixed-version rollout, and the remaining
ENT-3487 server-side work are tracked separately.
## Problem
Function-backed columns need an exact `fresh`, `stale`,
`never_computed`, and
`failed` partition for the table version a caller is reading. The
existing
error listing is capped, and composing it with job and schema listings
cannot
prove which rows currently failed.
## Behavior
- Adds a strict freshness reader beside the existing permissive planner.
It
distinguishes an absent certificate from legacy evidence that omits
failures, and from a current certificate carrying failure coverage.
- Extends the immutable freshness sidecar with per-fragment known and
failed
row bitmaps. Partial or `WHERE` publications cover only observed rows;
untouched rows stay unknown. Deleted rows do not block classification.
- Adds failure-aware whole-fragment and row-repair writers. A rejected
older
attempt cannot overwrite failure evidence already recorded for a newer
input
signature.
- Adds `column_status`, `column_statuses`, and `wait_fresh` to Python
and the
corresponding Node methods. The client calls the server's exact status
endpoint, validates the four-way partition and one-version guarantee,
and
preserves the complete Function version, last-job, and automatic-refresh
metadata.
- Adds an MDX-commented draft usage section to the Enterprise
computed-column
refresh guide, covering Python, Rust, and TypeScript and remaining
hidden
until the server integration ships.
- Sends `column`, `branch`, and exact `version` selectors to the status
route.
A pinned handle rejects a nonempty response from any other table
version;
latest handles keep the existing read-freshness headers.
- Retries `wait_fresh` only for HTTP 503 with
`function_column_status_finalization_pending`. Existing failed rows
remain
visible and do not keep the wait pending; other errors propagate.
- Models the shared scheduler projection, including `up_to_date`, real
`blocked` states, their reason, failure streak, and effective cooldown.
## Validation
This is the SDK side of ENT-3563. Sophon's failure-coverage publication
and
status endpoint are a separate integration; deployed end-to-end and
whole-ticket
acceptance remain pending.
- Rust: 9 exact freshness/status tests passed, plus 1 exact
latest-handle
freshness-header test. The pinned transport test was extended with a
wrong-version response and rerun at `ebd6649a` (1 passed). Coverage
includes untouched partial rows,
selected-row failure clearing, cross-generation evidence, typed and
non-retried errors, `up_to_date`, blocked scheduler fields, and
exact-version
requests.
- Python: 2 native PyO3 remote-transport tests passed together,
including
typed retry; one local facade test reaches the existing unsupported
boundary.
- Node: the NAPI addon built successfully, TypeScript and Biome passed,
and the
selected native remote-transport test passed 1/0, including
exact-version
selection and blocked scheduler metadata.
- `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile,
Biome, and
TypeScript checks passed.
Validation is mapped by the relevant unchanged source rather than
treated as one
final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header
and Python
remote tests at `31e68b7d`, the local Python test and Node build at
`6f213584`, and
the final TypeScript/Jest delta at `4f2d5c27`. The final product tree
excludes those five private EC2 runner files and adds
six generated TypeDoc files plus one hidden draft usage section in the
computed-column refresh guide. The Node delta corrected a private
TypeScript
command and reused the unchanged native addon.
## Summary
Adds an optional `max_concurrency` setting to Function registration and
explicit refresh:
- A registered Function version can define the default maximum number of
user-code calls in flight for one logical execution.
- A refresh can override that default for that execution only.
- The value must be a positive `u32` and is independent from
`resource_budget`; it is not an account-wide quota.
- Omitting the option preserves the previous request payload. Local
embedded refreshes reject this Enterprise-only option.
Rust, Python sync/async, PyO3, and the existing Node refresh surface
forward the option. Node has no Function registration surface today, so
no registration API is added there. The generated TypeDoc pages are
included for the new public option.
## Validation
The native-tested source covers Function version decoding, registration
validation and serialization, the Flight SQL registration ticket and
returned version, remote refresh forwarding, local rejection, and Node's
numeric boundary before conversion. Exact stage results are Rust `1+3`
plus four remote cases at `1` each; Python `14/1/49/24`; Node `57`
remote and `1` local rejection. Ruff, standard repository Pyright, and
Biome passed.
The tested Python extension was
`21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the
tested Node addon and generated native declaration were
`f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and
`4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The
final 24-file diff preserves the 22 tested
product/test/fixture/source-doc blobs and adds only two generated
TypeDoc pages.
## Depends on merged #4951
Phase 1 merged as `4d22c0ce2b6825c880c818e737b079db494b4053`, so this PR
now targets `main`. It reuses the canonical
`FunctionWorkerResourceBudget` introduced there rather than copying the
type.
For Function `add_columns` declarations, Python and Rust now persist
`auto_refresh_resource_budget` in computed-column metadata and send it
as `auto_refresh_resource_budget` in the remote request. CPU
milliseconds and memory bytes are required positive integers. Omitting
`gpu_count` is accepted as zero and the canonical outgoing JSON still
contains `"gpu_count": 0`. Unknown fields, missing CPU or memory, wrong
types, zero CPU or memory, negative values, and overflow are rejected.
The same budget is applied to every public output binding of a
multi-output Function declaration. Existing metadata is retained, and
SQL-expression declarations or local execution reject the budget instead
of silently ignoring it. Node is unchanged because its current
`addColumns` surface only accepts SQL computed expressions and has no
Function declaration path.
This change does not implement fleet call concurrency limits or enforce
the budget in the runtime. The Enterprise server must support the nested
Function worker resource-budget contract before these values can act as
execution limits.
## Validation
- Python extension build passed.
- Remote Function declaration and local unsupported-path rejection each
passed.
- Canonical metadata round-trip passed, including omitted GPU decoding
to zero and explicit `gpu_count: 0` on encode.
- Function multi-output remote wire test passed with `--features
remote`.
- Repository-scoped Python type check passed with 0 errors.
- All runtime source blobs match the native-tested source. The
post-native deltas are one documentation example line and an existing
routing-test stub update for the new optional argument; the affected
parameterized test passed 3/3 against the existing extension build. No
private test runners are part of this PR.
## Problem
Function catalog lookup required an exact version string even though
callers
commonly need the latest currently visible immutable revision. On the
current
SDK, resolving that version required an out-of-band catalog query before
calling
`get_function`.
## Change
- Make Python `get_function(name)` default to latest while accepting
`int` or
`str` for exact lookup.
- Normalize integer versions to their decimal string form and reject
`bool`
explicitly. Existing string validation and errors remain unchanged.
- Thread `None` through the rebuilt PyO3 binding and route it to the new
Rust
`Connection::get_function_latest`.
- Resolve latest through the existing catalog `DESCRIBE FUNCTION` path
without
`WITH (version = ...)`. The returned value is one exact immutable
version, not
a mutable alias.
- Preserve the existing Rust exact-version API.
## Validation
On the designated EC2 host with the rebuilt extension:
- `test_local_function_catalog_operations_are_not_supported`: **1
passed**.
This crosses the Python connection, rebuilt PyO3, and embedded Rust
`NotSupported` boundary with the omitted-version call.
- Async and sync catalog-path tests: **2 passed**, 117 deselected.
- Remote SQL Function creation/description integration test: **1
passed**.
- An earlier Rust invocation with the wrong module path matched **0
tests**;
it is retained as harness evidence and is not counted as a pass.
## Scope
This is the Function lookup SDK slice. It does not include the separate
atomic
rebind/publication server work tracked under ENT-3418.
## Summary
- Let local plain queries over row-addressable legacy v1 and v2 blob
columns use `blob_mode="bytes"` without importing `pylance` through
pandas, batch, or Arrow conversion.
- Preserve aliases, filters, ordering, limits, null and empty payloads,
empty result schemas, and the caller's `_rowid` choice.
- Build the hidden row-id query separately so reusing an async builder
for lazy reads or non-blob projections cannot retain an automatic
`_rowid`.
- Keep byte reads closed for blobs inside lists, remote or namespace
pushdown, and query shapes whose result rows cannot be matched back to
the source. Blob v2 lazy auto-row-id behavior is unchanged.
Fixes#4645
## Validation
- 26 focused Python tests passed with the test subprocess blocking all
`lance` imports. The persisted fixture writes and reopens a storage-v2.1
table, then covers sync and async pandas, batches, and Arrow conversion
plus async builder reuse.
- The existing unsupported vector and hybrid reranker regressions now
assert the current eligibility diagnostic, and the legacy table-level
Arrow regression now asserts successful bytes, schema, values, and
hidden row IDs. Both focused tests passed.
- The runtime receipt records no installed `pylance` distribution and no
importable `lance` module. The Python-only change was verified with an
existing compatible 0.42.0b9 extension; the relevant query and fetch
APIs are unchanged between that source and this 0.42.0b10 base.
- Ruff format and lint pass for every changed Python file.
## Summary
- Add the canonical Rust `FunctionWorkerResourceBudget` and
`RefreshColumnOptions::resource_budget()` for explicit Enterprise
Function refreshes.
- Forward the nested budget through Python sync/async, PyO3, Rust remote
requests, Node NAPI, and TypeScript.
- Require positive CPU and memory values; normalize an omitted nested
GPU count to numeric zero; reject the nested budget when combined with
the top-level `gpu_count` shorthand.
- Reject `resource_budget` on LanceDB Embedded refresh paths instead of
silently ignoring it.
- Document the Rust, Python, and TypeScript APIs, including that SQL
servers currently require all three budget fields.
The Rust, Python, and TypeScript additions use `cpu_millis` /
`cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` /
`gpuCount`. The top-level GPU shorthand keeps its existing behavior and
remains mutually exclusive with the complete nested budget.
## Scope and compatibility
This change configures only the explicit refresh call. It does not
configure `add_columns` automatic refresh and does not introduce a
fleet-wide call-cap interface.
A LanceDB Enterprise server must expose the nested `resource_budget`
backfill contract. LanceDB Embedded continues to support its existing
expression refresh defaults and rejects the new complete budget
explicitly.
## Validation
Validated against the final signed product source:
- Rust: resource-budget remote serialization test `2 passed`; validation
test `1 passed`.
- Python: actual PyO3 build; remote-request matrix `22 passed`;
invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff
check and format clean.
- Repository-standard Pyright entry point: `0 errors`, within the
repository's configured include set.
- Node: native build and Biome lint passed; the existing remote-request
test `1 passed`, including fractional, negative, overflow, NaN, and
infinity nested GPU cases; the existing local-rejection test `1 passed`.
- TypeDoc generation and generated-document cleanliness passed.
- The native-tested product source differs from this head only in two
semantically equivalent Node range checks required by Clippy. Current CI
compiles and lints the final blobs. No dependency or lockfile changes
are included.
Unsupported remote blob materialization errors now point to row-ID
selection, fetch_blobs and fetch_blob_files. Document descriptor
structs, fetched bytes and Function BlobReader inputs, and show a
bounded subset-copy recipe that keeps the source unchanged between
selection and fetching.
Adds guidance assertions and copy coverage preserving payloads, empty
blobs and nulls.
No new blob materialization API is introduced.
Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
Printing a Job or a bound status method could dump hundreds of lines of
serialized Function initialization. Bound nested payload previews in
both sync and async handles while preserving cached identity, state,
failure and result information. Full spec/result values remain available
through their properties, with no additional backend calls.
Adds regression coverage for large payloads, bound method display,
unobserved handles and completed local jobs.
The broader method/property API design remains separate.
Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
Function source packaging rejected unsupported global values without
identifying the variable or explaining the supported captures. Include
the global name even for a nested tuple failure, describe the supported
literal values and remedy, and document the capture boundary.
Adds function/class regression cases for Arrow types, nested tuples,
dictionaries and non-finite floats, plus a supported nested literal
control.
New serialization support remains separate.
Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
Expose the durable SQL registry id separately from the connection-scoped
query UUID. Rust and Python query handles and descriptions expose
optional `job_id`; result readers preserve it in schema metadata under
`x-lancedb-sql-job-id`, without adding business columns. Clients can
reopen that id with `open_job`.
Poll success and failure metadata update the same handle. A failure
before the first successful poll retains its job id and original
structured error. The cancellation documentation distinguishes the
Flight waiter from `cancel_job(job_id)`.
Validation on the designated EC2: the two focused error interoperability
tests and Ruff passed on `a8798e81`; delivery has the identical tree.
The preceding 51 Rust SQL tests and earlier 31 native Python SQL tests
are reused for unchanged coverage. Matching-runtime independent
acceptance remains pending.
Companion server change: https://github.com/lancedb/sophon/pull/10049.
Refs ENT-3562.
Update the Rust workspace, Java lance-core, and Python test dependency
to [Lance
v15.0.0-beta.16](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.16).
Synchronize `python/uv.lock` with the published beta.16 wheels for
Linux, macOS, and Windows.
Run the native Linux x64 wheel build on an eight-core runner, matching
the existing ARM build allocation, after its optimized build repeatedly
exceeded the job deadline on the standard runner.
The dependency update passed Clippy, formatting, and CI-script checks;
the Python lockfile refresh/check passes. Fresh CI is running for the
runner adjustment. Java dependency-resolution failures caused solely by
publication lag are accepted for this upgrade.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
Python refresh APIs currently reject `publication_interval_rows`, so
callers cannot request incremental Function output publication through
the SDK. Add the optional typed parameter through the sync/async Python
APIs, PyO3, and Rust `RefreshColumnOptions`, and serialize it as a
top-level Enterprise refresh option.
An omitted value preserves the existing wire request and whole-fragment
default. Positive values set a publication threshold; zero and negative
Python values fail before a request is sent. Embedded refreshes reject
the Enterprise-only option. Documentation describes a threshold rather
than an exact number of rows per publication.
Validation on the designated development host:
- A matching baseline build reproduced the missing public keyword
argument in both sync and async submission tests.
- The updated PyO3 build passed 54 focused Python tests; Rust payload,
invalid-option, and default-wire tests each passed. Rust formatting and
Python Ruff checks passed.
- The single signed delivery commit differs from that tested
implementation only in a Rustdoc correction describing Embedded
rejection; formatting passed again. No dependency pins or lockfiles
changed.
The matching SDK is prepared for the separate server validation of
failure/cancellation retention and missing-only refill. Those end-to-end
results are still pending and are not implied by these SDK tests.
Refs [ENT-3733](https://linear.app/lancedb/issue/ENT-3733).
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.
```python
job = view.refresh_async(
cpu_millis=8000,
memory_bytes=16 * 1024**3,
gpu_count=2,
gpu_class="h100",
)
job.wait()
```
```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
gpu_count = 2, gpu_class = 'h100');
```
The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.
Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.
The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.
Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
Update Rust and Java Lance dependencies to
[v15.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.15),
pin Python tests to published pylance 15.0.0b15, and refresh both
lockfiles; no compatibility fixes were needed.
Validation passed: workspace Clippy with all features/tests and warnings
denied, cargo fmt, uv lock --check, and all 35 CI-script tests.
The [upstream wheel
workflow](https://github.com/lance-format/lance/actions/runs/42669437482)
canceled Windows ARM64 and manylinux_2_28 x86_64 builds after an hour,
so the lockfile contains the five published variants (Linux x86_64
remains covered by manylinux2014); Windows ARM64 is unavailable until
upstream publishes its wheel and the lockfile is refreshed.
Update the Rust workspace Lance crates and Python test dependency to
[Lance
v15.0.0-beta.14](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.14),
which includes JSON cache reuse across request stores and stable-row-ID
JSON index support. The Python lock contains all seven published wheels.
The independent Java module retains beta.8, as in the previous beta.13
update; its separate beta.14 publisher repair is outside this Rust
consumer upgrade.
Preserve all CI coverage while reclaiming completed workspace lancedb
artifacts before the package-level feature-test build, which otherwise
exhausted the Linux runner disk. Give only the Linux ARM64 release-wheel
job two hours after its one-hour timeout, retaining the same runner,
fat-LTO release profile and build flags.
Validation: 2,280 Rust tests passed (79 existing ignores), strict
workspace/all-feature Clippy, Rust formatting, and remote example checks
passed on unchanged Rust source and dependency lock. Python dependency
guards, lock validation, locked sync, and the published macOS wheel
import passed. The new Windows ARM64 wheel was downloaded and its
SHA-256 verified. Workflow YAML/shell checks and independent review
passed. Root Python Ruff/format checks still fail on unchanged
documentation examples; no suppressions or unrelated cleanup were added.
The CI repairs require a complete new run before readiness. This
dependency update does not establish corrected Sophon performance;
downstream native acceptance and the paired S3 replay remain separate.
Update the Rust workspace Lance dependencies to
[v15.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.13),
synchronize the Python test dependency to published pylance 15.0.0b13,
and refresh both lockfiles; no compatibility fixes were needed.
The Java SDK stays on `lance-core` 15.0.0-beta.8, as in the beta.9 and
beta.10 updates: Maven Central has no `lance-core` artifact for
15.0.0-beta.9 through beta.13.
Validation passed: workspace Clippy with all features and warnings
denied, Cargo formatting, Python lockfile checks, and all 35 CI-script
tests. Repository-wide Ruff reports 163 diagnostics, all verified
identical on unchanged HEAD; unrelated formatter changes were reverted.
---------
Co-authored-by: Vivek <vbharathan2008@gmail.com>
The server's backfill request takes a row filter, failure limits,
per-call timeouts and worker sizing, and SQL exposes all of them through
`REFRESH COLUMN ... WHERE ... WITH (...)`, but the typed SDKs could send
only batch_size, GPU placement and the retry and error policies. Anyone
needing a filtered refresh or a bounded skip policy had to drop to SQL.
This adds `where`, `max_failed_fraction`, `max_failed_rows`,
`row_timeout_seconds`, `init_timeout_seconds`, `num_workers` and
`max_executor_processes_per_worker` to Python, Rust and Node, passed
through unchanged. `RefreshColumnOptions` now serializes straight into
the request body, so the three SDKs share one producer. Node gets a
`RefreshColumnOptions` object, which also carries the options it lacked
before. A LanceDB Embedded refresh rejects every option other than
batch_size and GPU placement, naming the option.
Lance beta.11 calibrates lower `search_effort` tiers by metric and
requested `k`. With the Rust dependency and Python/Node execution tests
already on main via #4881, document the tiers across Rust, Python, Node
and the search guide, extend the remote request matrix to cover `0.1`
and `0.25`, and align the Python test package with beta.11.
`0.1` and `0.25` target empirical mean partition-routing recall of 80%
and 90% for L2, cosine and dot through `k=100000`. These are not
per-query or final Recall@k guarantees. Default Auto is unchanged;
Hamming and larger `k` remain uncalibrated, and remote queries require
corresponding server support. This PR makes no additional performance
claim. See [Lance
#10695](https://github.com/lance-format/lance/pull/10695) for
measurements and limitations.
Validation: both strict workspace Clippy configurations, Rust
formatting, 13 Rust effort tests, 208 Python query tests, 41 Node query
tests, native binding builds, TypeScript, Node lint/generated docs,
Python lint/format, full lockfile check and two dependency guards passed
on `a8b09170`. All validated Rust/Python/Node source and dependency
files are byte-identical at `2a82d07c`; the only subsequent change
restores the Java POM to main. Root Ruff has 59 existing diagnostics and
six formatting files, all byte-identical to the base; the Python package
passes.
The Java REST/Namespace client has no changed API here, and its
dependency remains identical to main. The separate [Java beta
publication failure](https://github.com/lance-format/lance/issues/10773)
is not claimed resolved or validated. Official SDK release, automatic
mirror, Sophon adoption and measured consumer acceptance remain
follow-up stages.
Downstream workspaces adopting Lance 15.0.0-beta.11 currently also
resolve beta.10 because the SDK pins every Lance crate exactly. This
produces incompatible Lance types across the SDK and the consumer.
Update the workspace pins and lockfile to the published
[v15.0.0-beta.11](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.11).
This release includes payload-free list length reads and preserves those
read modes across serialized plans (lance-format/lance#10759,
lance-format/lance#10766). Existing feature flags and unrelated locked
packages are unchanged.
The release also includes the metric/k-dependent search-effort
calibration from lance-format/lance#10695. Update the Node and Python
execution tests to its exact probe budgets (for example, effort 0.25
with L2, k=8 and Auto=6 now probes 4 partitions), and cover the 0.1
tier. Recall, caller bounds and probe-precedence assertions remain in
place.
Validation:
- `cargo fmt --all --check`
- `cargo clippy --quiet --features remote --tests --examples`
- Downstream metadata fixture with SDK source and the released Lance
beta.11 path overrides resolves exactly one Lance version.
- `cargo test --quiet --features remote --tests`: 2,271 passed, 0
failed, 79 existing ignored tests.
- Node native/TypeScript build, lint and API documentation generation
passed; 41 query tests passed.
- Python native binding build and package Ruff checks passed; 208 query
tests passed, 1 slow test deselected, 5 existing deprecation warnings.
- Rust results remain applicable: the follow-up changes only binding
tests, with product source and dependency graph unchanged from the
tested dependency update.
Repository-root Ruff currently reports 59 lint findings and 6 formatting
failures in untouched documentation snippets. Their blobs match the PR
base; the changed Python test and the entire Python package pass
lint/format checks.
Remote Python `drop_function` currently discards the dependent-view
warnings returned by SQL DROP. Both the synchronous and coroutine APIs
now emit those warnings through `warnings.warn` while preserving their
boolean result and executing DROP once. The server support is already
merged in lancedb/sophon#9820.
The Rust decoder reads warnings across every result batch, including an
empty first batch. Missing warning columns remain compatible with older
servers; malformed or null warning lists are rejected. The hidden
binding bridge carries the warnings to Python. The separate
`drop_function_async` API and its `(bool, Job)` contract are unchanged.
Python warning filters can raise after the catalog mutation has
completed.
Validation: a compiled PyO3/public-server reproduction failed before the
fix and passed afterward; the public async path also returned `true`
with the expected dependent-view warning. Final source passes two
focused Rust tests, two Python wrapper tests, strict targeted Clippy,
formatting, and Ruff. Three decoder tests cover batch/null/type and
older-server behavior. The public runs preceded a two-file
simplification of the completed Job projection; focused final-source
checks cover that delta. Current-main integration is conflict-free, and
its overlapping changes concern independent retry-policy APIs/tests.
Documentation now covers warning filters, the post-removal
warnings-as-errors behavior, and silent compatibility with older
servers. The follow-up changes only prose and a Python docstring; the
runtime validation above remains applicable.
Refs ENT-3659. Whole-issue independent acceptance remains pending.
## Summary
Bumps dependencies that Aikido flags with critical advisories, where a
patched version is available without code changes.
| Package | Where | From → To | Ships to users |
|---|---|---|---|
| jackson (BOM) | `java/pom.xml` | 2.18.11 → 2.18.12 | Yes, via the
published pom |
| zerovec / zerovec-derive | `Cargo.lock` | 0.11.6 → 0.11.8 / 0.11.3 →
0.11.7 | Yes, linked into the wheels and Node binaries |
| pillow | `python/uv.lock` | 12.3.0 → 13.0.0 | No, dev lockfile |
| pyarrow | `python/uv.lock` | 24.0.0 → 26.0.1 | No, dev lockfile |
| pyjwt, urllib3, anyio, cryptography | `python/uv.lock` | → 2.15.1,
2.8.0, 4.16.0, 50.0.2 | No, dev lockfile |
| pillow | `docs/web-tests/blob-v2/py/requirements.txt` | 12.1.1 →
13.0.0 | No |
pyarrow was held at 24.0.0 by a `pyarrow<25` pin in the `tests` extra.
It came in with #3675 without a stated reason, so this PR removes it.
pillow 13 and pyarrow 26 require Python 3.11+, so the lockfile keeps
pillow 12.3.0 and pyarrow 25.0.1 on Python 3.10, and those remain
flagged.
The sharp findings in the Node examples and docs are not addressed here.
They come from the `@huggingface/transformers@3.0.2` optional
dependency, which needs a major upgrade to 4.3+.
## Evidence
With the pyarrow pin removed and pyarrow 26.0.1 installed, the Python
suite (`-m "not slow and not s3_test"`) passed locally on Python 3.12:
3319 passed, 164 skipped. That run was on main as of #4879; this branch
is based on a slightly newer main.
## Merge Risk
**Door:** Two-way. Patch and minor version bumps that a revert undoes.
**Blast Radius:** Java consumers (jackson patch release), native
binaries (zerovec patch release), Python test environments.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Functions that import a module unavailable on the client currently
package silently even when they declare no pip/conda dependencies. Emit
an advisory warning for these unresolved imports so callers can declare
the required package or confirm that the base image supplies it.
Any pip/conda declaration suppresses this new advisory because
distribution names need not match import names. The warning does not
claim that the module is absent remotely and follows normal Python
warning filters, including warnings-as-errors.
Validation includes the original `import scipy.ndimage` with `pip=()`
through the public decorator/packager path, plus declared-package and
existing local-source controls. This does not establish remote image
availability or deployment acceptance.
Refs ENT-3448.
Expose `FunctionRetryPolicy` on Function declarations and refresh
overrides, together with `error_policy` and recorded attempt counts.
Python uses the Rust validation rules before sending a request; local
refreshes reject these Enterprise options.
The policy bounds calls and backoff, supports
class/message/provider-code matching, and can disable retries with
`max_attempts=1`. Batch retry and skip operate on the whole failed call.
Requires Enterprise server support for the new policy.
Validation: rebuilt the Python extension; 47 targeted Python tests and
10 Rust tests passed. Python `make check`, Rust formatting, and `cargo
clippy --features remote --tests --examples -- -D warnings` passed. The
additional repository-root Ruff sweep reports existing errors under
`docs/web-tests`; the Python workspace checks pass.
Draft user guidance covers the supporting server requirement, safe
repetition of partial work and whole-batch exhaustion. The guide
sections stay hidden until release. Vale 3.24.0 passed for the added
draft prose; no paired deployed SDK/server retry acceptance was run.
Companion server: https://github.com/lancedb/sophon/pull/9812.
BREAKING CHANGE: Rust callers constructing `FunctionRegistrationRequest`
or `FunctionErrorRecord` with struct literals must supply the new
`retry_policy` or `attempts` field, respectively.
Bare Rust test runners do not install the `lancedb` Python package, so
the typed-error tests fail with `ModuleNotFoundError`. Load the
checked-in exception modules once for the embedded interpreter,
preserving exception-class identity across parallel tests and the
existing type/cause assertions.
Rust CI now watches the binding sources and the Python exception modules
embedded by these tests, so those changes exercise the macOS Rust test
jobs.
Catalog lookup failures lose their structured resource identity at the
Rust/Python SDK boundaries, leaving callers with generic errors or empty
correlation fields. Decode the existing Flight SQL error metadata and
preserve resource kind/name, namespace, error code, and request/job IDs
in the existing error types.
Direct `get_function` calls unwrap named Function or namespace failures.
Generic SQL job failures remain `JobFailedError` with the typed resource
error as their cause; native deferred commands correctly leave durable
job IDs null. Existing unstructured SQL classifications, including
`FailedPrecondition`, remain supported.
Validation: focused Rust/Python regressions and a native-extension run
against a real Sophon process passed all five Function/Secret
resource-error cases. The final one-commit candidate passes `cargo check
-p lancedb-python` and formatting; earlier runtime evidence applies
after removal of an unused tuple member. Independent deployed acceptance
remains pending for ENT-3373.
Server companion: https://github.com/lancedb/sophon/pull/9804.
Integration validation: all 14 DocLinks pass on the merge candidate with
#4869. The subsequent main documentation routing change was inspected;
it does not alter the typed-error paths. The inherited pin-utils
cargo-deny finding is tracked in a separate dependency repair.
`take_row_ids` used an `IN` filter, so the backend could reorder rows
and collapse duplicate IDs. That can silently misalign training rows
with labels or sample weights.
Restore the requested order and duplicate occurrences in the shared Rust
take path for native and remote tables, and retain that information
through Python query serialization. Missing or deleted IDs are omitted.
Explicit Rust `order_by` overrides request order, and pagination applies
after restoring occurrences. Empty projections and `_rowid` projection
aliases retain their output schema.
RTE tables reject unsupported takes with the existing base-table
guidance before checking the internal row-ID projection used for
ordering. An explicit `use_rte(False)` reads the base table with the
same requested order and occurrences.
A pre-existing fixture defect also blocked CI:
`test_table_add_in_threadpool` responded without reading uploads,
allowing a connection reset during response decoding. This PR includes a
fix confined to that test's handler: consume each request body before
responding and verify one create row plus ten inserted rows. This was
not a regression in the take implementation. Production write retries,
concurrency and HTTP client code are unchanged.
The selective sampling guide scopes the row-ID guarantees to SDK
releases containing this fix, preserves the distinct offset-take
semantics, and explains that ordered takes collect and combine matching
rows before output batching. It recommends smaller ID requests and
narrow projections because `max_batch_length` does not cap working
memory. This uses the existing REST query protocol and requires a new
SDK release; no deployed service was changed.
Validation:
- Docs follow-up: the changed page compiles with the Mint 4.2.888 MDX
compiler; Typos 1.26.0 passes and Vale 3.24.0 has no findings on changed
lines. Full-site Mint validation reports four existing snippet-import
warnings on the reranking/vector-search pages, identical with the
original page restored in a baseline comparison.
- The applied threadpool test passed locally after the fixture change.
The identical candidate previously passed 10 trials with the ordinary
reader and 10 with an unbuffered reader; each trial verifies all
uploaded rows. Those checks used macOS arm64/Python 3.14/default
features, not the full failing CI environment. The normal new-head macOS
Python 3.13/experimental-functions job remains the required platform
check.
- Python package Ruff format/check passed. Repository-root Ruff still
reports 59 existing errors and six formatting differences in unchanged
`docs/web-tests` files; none are in changed files.
- At `a569d3f0`, the RTE diagnostic regression was reproduced and fixed;
5 targeted cases and strict workspace Clippy passed. The fixture-only
follow-up changes no production Rust/Python code, so that evidence is
reused.
- Earlier validation at `6bcc0878`: 439 Rust query/remote-table tests
and 24 Python take/getitems tests passed; the focused 49-test suite also
covered offset dataloaders. A local 9,216-row, four-fragment
reproduction returned 200 seeded random batches of 64 rows with zero
ordering or label-alignment mismatches. The ordering and projection
implementation is unchanged by the follow-ups.
- No live Enterprise run was performed.
Fixes [ENT-3469](https://linear.app/lancedb/issue/ENT-3469).
The client registers Functions, but the decorator that packages one was
called `udf` and produced a `UdfDefinition`. Rename them to `function`
and `FunctionDefinition`, and say "function" instead of "UDF" in the
docstrings, error messages and docs around them.
`udf` and `UdfDefinition` stay as deprecated aliases that emit a
`DeprecationWarning`. The source packager strips either spelling of the
decorator, so `@udf` and `@function` produce the same artifact and
digest, and the wire contract is unchanged. DataFusion and Lance UDF
types, and Geneva's `@udf` in the integration docs, are unrelated and
untouched.
Clarify `FunctionVersion` forward compatibility: unknown top-level
fields are accepted during decoding and omitted from canonical JSON,
while all defined fields and the open `image.descriptor` content retain
their values. Extend the existing Rust and Python canonical round-trip
tests to assert the complete projected object, including future
discriminator values.
This documents and tests the existing behavior; it does not change
runtime serialization or promise lossless retention of unknown top-level
fields.
Related: ENT-2176.
Function refresh failures and skipped-row diagnostics lose the traceback
when decoded by the SDK. Expose the server-provided optional `traceback`
on job failure details and Function error records in Rust, Python, and
Node.js, while keeping existing compact error messages.
Responses from older servers remain supported when the field is absent.
Tracebacks may be truncated by the server. The Rust `Error::JobFailed`
variant now stores `Box<JobFailure>` so the added diagnostics do not
enlarge every `Result` error payload.
Function catalog responses include `batch_size`, `row_timeout_seconds`,
and `result_stability`, but the Rust and Python `FunctionVersion` models
silently discarded them. Preserve these optional execution controls
through decoding, typed access, and canonical serialization; versions
without the fields retain their existing representation.
The regression exercises raw SQL transport responses and public Python
sync/async get/list paths. Live Sophon catalog acceptance remains
pending under ENT-2073 and the related FunctionVersion round-trip
requirement in ENT-2176.
Update the Rust workspace Lance dependencies to
[v15.0.0-beta.10](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.10),
which contains the released MinHash signature scanning API needed by
downstream consumers. Pin Python tests to the published `15.0.0b10`
wheels, including macOS, and refresh both lockfiles. Java remains on
beta.8.
No SDK source compatibility changes were needed. All 25 Lance Git
packages resolve to release commit
`f56a08d40b5630ce1124535c4c3b1406d5ceedf1`; third-party package versions
are unchanged.
Validation on macOS ARM64 with pinned Rust 1.97.0: strict workspace
Clippy with default and all features, Rust formatting, 32 CI-script
tests, three count tests, one WAL/vector regression, and the Python
lockfile check passed. The native test binary emitted a macOS
compact-unwind size warning; both test commands completed successfully
without changing linker settings. Downstream release and integrated
Sophon acceptance remain separate.
Function error records expose physical row locations, which can change
when a table is compacted. Expose the server-provided optional stable
`row_id` through Rust and Python so callers can locate the same row
after compaction. Document stable-ID retries and retain physical
addresses for tables without stable row IDs. Older server responses
continue to deserialize with `row_id=None`.
The Python remote response test and Rust remote deserialization test
passed on verification SDK `4e1725fa352035e2079dfd235b8271cd05adf433`,
including decoding a legacy response without the new field. This branch
applies the same SDK patch to current main; intervening main changes
leave the affected error types and assertions unchanged. Formatting and
lint checks passed after correcting the retry examples to use the
`table.name` property. In a separate Linux server experiment, a real
skipped Function row retained `_rowid=506` after compaction changed its
physical address. The SDK field requires a server that emits stable row
IDs; this PR does not change server dependencies.
Remote SQL failures currently become generic runtime errors, and HTTP
table lookup can discard server error metadata. Preserve transport
status, current request ID, SQL job ID and worker request ID across Rust
and Python. Select typed column, table, reference and schema exceptions
only when an explicit server code matches the transport status.
Remote `HttpError` and `RuntimeError` handlers remain compatible. Failed
SQL jobs retain `JobFailedError` with a typed SQL cause when available.
Missing metadata stays absent; message text never determines the
exception class. Local missing-column errors retain their type and
qualified-field suggestions, and `JobFailure.source` documents
structured remote Flight SQL failures.
This branch integrates main at `a9ad4997`, including its released Lance
15.0.0-beta.9 dependency update and remote optimize support. The only
textual conflict was two independent additions to `test_remote_db.py`;
both test groups and all original assertions are retained. The feature
has no dependency or lockfile delta against current main.
SDK runtime validation on `03ee633a` (runtime source unchanged in
`f3f2d0b8`), macOS ARM64, pinned Rust 1.97.0 and the frozen uv Python
3.14 environment:
- Rebuilt the native extension from the integrated source. All 22
targeted Python tests passed: native local errors, HTTP/Flight metadata,
legacy sync/async table-not-found behavior, and the three main
remote-optimize regressions.
- All 52 Rust SQL, metadata/source and nested-field tests passed,
including the new main optimize cases.
- Root and Python-binding Rust Clippy with `-D warnings`, Rust
formatting, Python package Ruff and conflict-file formatting passed.
- Repository-wide Ruff reports 59 existing documentation findings, and
its format check identifies six documentation files. Every affected file
is byte-identical to current main; these checks are not reported as
passing.
- Direct integration review and AST comparison confirm both conflicting
test groups were preserved. Earlier exception-constructor/pickling
evidence remains applicable to the unchanged Python error modules.
The unlocked build exhausted disk again on a runner with the same label
but only 9.4 GiB free on a 72 GiB root filesystem. Keep the existing
runner and dependency caches, serialize Cargo compilation, and remove
only completed test/bench executables reported by Cargo. This job never
runs those binaries; normal binaries, libraries and build-script outputs
remain available. Fresh resolution, the ci profile, all features, and
test/bench compilation and linking are retained.
Three probes using Cargo and target-path validation pass with `-D
warnings`: all fixture targets link while ordinary outputs remain
usable; an undefined symbol still produces Cargo exit 101 and its linker
diagnostic; files outside the target directory are never removed. The
[current-head build-no-lock
job](https://github.com/lancedb/lancedb/actions/runs/41911613851/job/126860364065)
passed on `f3f2d0b8`: all compilation/linking completed, 20 completed
test executables were removed, and `target` retained about 7.3 GiB. This
successful run had a 145 GiB root with about 78 GiB free afterward; it
was not an identical rerun of the failed 72 GiB image. The driver tests
also passed in this job. All previous-head jobs ended naturally before
the push; new-head CI and review gates still apply.
Typed remote input exceptions require the server metadata contract.
Older servers retain generic transport errors and existing missing-table
compatibility. The optimize tests use mock servers and do not establish
downstream server acceptance.
BREAKING CHANGE: Rust callers with exhaustive matches on
`lancedb::Error` must handle the new `FieldNotFound` and `Sql` variants.
Local `Error::ColumnNotFound` failures now raise
`lancedb.exceptions.ColumnNotFoundError`, a `ValueError` subclass.
Python callers that previously caught only `RuntimeError` for this local
failure must catch `ColumnNotFoundError` or `ValueError`.
`Table::optimize()` previously returned `NotSupported` for remote
tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE
TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL.
Only `OptimizeAction::Compact` without a custom `remap_options` or
non-default `CompactionOptions` is supported remotely.
`OptimizeAction::All`, `::Prune`, and `::Index` all return a clear
`NotSupported` error instead of silently doing the wrong thing. A
branch-scoped table handle submits a `WITH (branch = '<branch>')`
clause; the statement doesn't validate the branch exists itself, so an
unknown branch surfaces as an async job failure rather than an error
from this call.
The server's SQL job adapter doesn't expose compaction metrics, so
`OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a
placeholder rather than real numbers — it must still be `Some`, since
callers (including the Python/Node bindings) unwrap it unconditionally
on success. This mirrors how `refresh_materialized_view_async` already
handles the same limitation for its own metrics.
The Python and Node `optimize()` convenience methods call a new shared
`Table::optimize_full`, which runs `Compact` and then, on a remote
table, skips `Prune` and `Index` instead of attempting and failing on
them — returning zeroed prune stats rather than submitting a compaction
job and then raising. Python's `RemoteTable.optimize()` also now
forwards `compaction_options` to the Rust layer, so a non-default value
is rejected before submission instead of being silently dropped.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Jack Ye
Yang Cen
Xuanwo
BHAWESHBHASKAR
Lance Release
LanceDB RobotandJack Ye
Sarwar Bhuiyan
Bruno Ramirez
Brendan ClementandClaude
Wyatt Alt
Will JonesandClaude Opus 5.5
lancedb-gatefixer[bot]