Ambiguous gateway failures can occur after a write commits, so replaying
the operation can append duplicate rows. Non-idempotent writes now retry
only known precommit failures. Multipart retries reuse the same upload
ID, part ID, and encoded bytes; ambiguous completion returns an
unknown-outcome error without abort or replay. A definite 409/429
completion rejection triggers best-effort cleanup using the call's retry
budget.
Replayable parts and the shared byte-based partition target default to
64 MiB. Dictionary-bearing batches are compacted before size estimation
and again when actual compressed IPC framing requires further splitting:
keys are remapped without changing their types or schema, view backing
buffers are collected, and ListView/LargeListView child ranges are
compacted as an interval union so overlapping views retain sharing.
Parts are prepared on demand; a single row and its schema can still
exceed the target. Multipart progress starts at zero and counts each
acknowledged IPC body once. Rust write builders and Python/TypeScript
`add` expose per-call retry overrides. Atomic `create_table` pre-encodes
to a temporary file and rejects compressed IPC over 256 MiB before
sending any request; the schema-first import guidance now includes
Python, TypeScript, and Rust examples. The shared partition target also
affects local automatic partitioning.
At `cb2a79473bd1b51b950c9fa3e9f247f929c8ac3b`, 70 targeted Rust cases
pass on source verified against the committed four-file patch. With a 16
KiB target and 128 unique 1 KiB dictionary values, the original ordinary
dictionary reproducer drops from 128 requests / 16,942,080 bytes to 16 /
143,488; ListView and LargeListView drop from 128 / 17,056,768 to 16 /
149,632. Encoded HTTP bodies are decoded to check values, nulls, slices,
metadata, changing dictionaries, and overlapping views; additional cases
cover framing-driven splitting, oversized rows, stable retry bytes/IDs,
acknowledged progress, and atomic create. Real HTTP fault tests commit
Blob payloads to a local Lance table and verify byte equality, no
duplicate append, and zero abort/replay after ambiguous completion.
The earlier 811 remote and 7 progress cases at
`f36d217110a25f6b501c0255df9aaa60e3c8000a`, Linux Python remote
database/options results (437 cases plus 8 fixture consumers), Node
retry-binding evidence, and Rust/TypeScript documentation example checks
remain historical validation. Python/Node extensions were not rebuilt
for the dictionary follow-up; these results do not establish current
binding integration or fresh cross-platform/minimum-dependency CI
acceptance.
A separate real Query Node lifecycle experiment used SDK
`b8efe7d2d63647d4962789ca5fc042b59efe073d` and Sophon
`58b7895c7c287810fff4e89f13226fc5fa60d9ef`. After the actual listener
forced a connection closed, a surviving QN accepted a retry with
identical upload ID, part ID, and IPC bytes. Ordinary insert and
complete response-loss scenarios sent no replay or abort; fresh reads
observed exactly one commit and exact Blob bytes in all three scenarios.
This local experiment used a 1-second drain grace, two 32 KiB payloads,
and one distinct staged part. It validates acknowledgement-loss recovery
at those revisions, not concurrent overlapping attempts, large uploads,
physical orphan cleanup, the later dictionary encoding change, or the
original sixth-insert EOF.
The part target is not a process memory limit: source/ancestor buffers
and temporary compaction allocations may coexist. Struct/FixedSizeList
parent nulls can retain hidden child keys; this is not complete
logical-reachability GC for every Arrow combination, and retained hidden
values may affect wire size. The current local Lance 15.0.0-beta.17
rejects ListView/LargeListView persistence; the tests establish remote
client IPC encoding behavior, not native or Cloud persistence support.
Live Cloud, full workspace/Clippy, the original ordinary-insert
disconnect root cause, and Sophon's asynchronous transaction cleanup
behavior are not established. Cancelling during staging retains the
existing server-session cleanup limitation. Function reader delivery is
separate.
Remote Blob reads retain and concatenate intermediate IPC buffers across
batches and HTTP requests, increasing peak memory. Decode fragmented IPC
bodies using their declared length, assemble Blob values once across
batches/requests, and collect large-object Range responses into
known-size buffers. Impossible IPC/Range allocations return errors
instead of panicking. Closing a remote Blob file releases cached bytes
immediately, including while a cross-window read is waiting for the
network.
Rust and Python expose configurable Blob read buffers. Python sync/async
`fetch_blob_files` and `open_blob_file(s)` accept `buffer_size`; the
default is 512 KiB and zero disables prefetch. Explicit ranges bypass
prefetch. New PE batch-endpoint adoption is separate.
Malformed-length and retained-cache regressions fail on the previous
implementation and pass after the correction. Real HTTP tests cover
buffer propagation, close during blocked input, batch/request
aggregation, and large-object fallback. The final extension was rebuilt
before Python validation. Non-Blob Arrow arrays may still require
alignment copies; a contiguous multi-batch result still requires its
final allocation.
## Benchmark
Peak client RSS increment in KiB, using
`python/benchmarks/bench_fetch_blobs_rss.py`, independent clients and
real loopback HTTP. Both extensions were rebuilt from clean, unchanged
source with the same toolchain/configuration; loaded artifact hashes
were checked. Baseline `42f11d13a2755e44b63876b5a5edd93901291150`, after
`0a8a76de5286949d80368f95a5c529adf422c187`.
| Equivalent fetch workload | Payload | Before | After |
| --- | ---: | ---: | ---: |
| Single response, one batch | 48 MiB | 63,812 | 63,896 |
| Multiple responses, one batch per response | 144 MiB | 423,104 |
195,628 |
| Single response, 12 rows per batch | 48 MiB | 130,520 | 71,348 |
| Multiple responses, 12 rows per batch | 144 MiB | 363,144 | 172,748 |
| One oversized row, Range fallback | 70 MiB | 182,816 | 85,548 |
These are one observation per workload and build, not a statistical
distribution or memory upper bound. The single-batch workload shows no
improvement. Cloud/Sophon deployment, credentialed S3 and long-running
stress have not been exercised.
GPU Functions that use a scalar callable currently invoke the callable
once per row. The SDK documents Arrow-vectorized callables, but silently
accepts a scalar GPU callable, which makes it easy to assume that the
runtime will batch model work automatically.
Warn during Function construction when a GPU callable does not use the
documented vectorized form. A callable is recognized as vectorized when
every input is annotated as `pyarrow.Array`, or when a single input is
annotated as `pyarrow.RecordBatch`. The warning explains the resulting
per-row invocation and shows how to opt into batch execution. CPU
Functions and the vectorized GPU forms do not emit it.
The warning is advisory and does not change registration serialization,
the artifact, the runtime adapter, or execution behavior. It is one part
of the ENT-3522 documentation-and-warning alternative; it does not claim
that a single A10G meets the original batch throughput requirement.
Validation on the designated EC2 host reused the compatible existing
native extension from `ent3563-sdk-freshness-9336cfb`; this does not
claim a new native package or GPU performance run. The final head has a
clean source tree and ran:
```bash
PYTHONPATH="$sdk/python/python" /home/ec2-user/.venvs/ent3563-0a839fa/bin/python -m pytest -q python/python/tests/test_first_class_function_slice2.py -k 'gpu_marker_uses_gpu_runtime or gpu_scalar_warning_names_the_vectorized_alternative'
/home/ec2-user/.venvs/ent3563-0a839fa/bin/ruff format --check python/python/lancedb/functions.py python/python/tests/test_first_class_function_slice2.py
/home/ec2-user/.venvs/ent3563-0a839fa/bin/ruff check python/python/lancedb/functions.py python/python/tests/test_first_class_function_slice2.py
```
The actual result was `2 passed, 123 deselected`; both Ruff commands
passed. The original 2,000-row single-A10G throughput requirement
remains a separate ENT-3522 acceptance gate.
A plain `import ctypes.util` in a Function currently reports “dynamic
namespace access” without explaining the module restriction or a way
forward. Give blocked module imports a distinct error that names the
modules, explains the reconstructed namespace constraint, and recommends
an importable helper supplied through `code=`, `pip=`, or `conda=`.
Dynamic namespace operations also get an actionable explanation.
Document the full restrictions and the helper alternative in the
decorator and Python Functions reference. Add regression coverage for
direct imports, `from` imports, global module aliases, class Functions,
the deprecated `@udf` alias, and helpers imported normally or shipped
with `code=`. Existing import restrictions remain in place.
For example, move `ctypes.util.find_library` into
`native_diagnostics/__init__.py`, then use:
```python
import native_diagnostics
from lancedb import function
@function(code=[native_diagnostics])
def probe_library(name: str) -> str:
return native_diagnostics.find_library(name)
```
Fixes ENT-3408.
Add `exist_ok=False` to synchronous and asynchronous Python
`create_secret`. Passing `True` uses `CREATE SECRET IF NOT EXISTS`,
preserving an existing credential. The default continues to fail on an
existing name; rotation stays explicit through `alter_secret`.
```python
db.create_secret("api-key", value, exist_ok=True)
await async_db.create_secret("api-key", value, exist_ok=True)
```
The Rust API gains an awaitable builder with `.exist_ok(true)` while
existing `create_secret(...).await` calls keep working. Includes Python
SQL round-trip coverage and Rust coverage for direct and deferred SQL
responses.
Restore typed results for reopened Function and SQL-expression refresh jobs while preserving their handles and cached descriptions.
Fixes https://linear.app/lancedb/issue/ENT-3441.
Align Python catalog connection inputs with database connections, warn when region is ignored, and preserve remote storage settings in opened databases.
Expose `FunctionVersion.runtime_digest` through Rust and Python, and add
`PythonRuntimeSpec.digest` so clients can compare declared environment
inputs before registering a Function. Dependency pins, environment
variables, Python version, and GPU settings affect this digest; package
ordering does not.
Clarify that matching `source_digest` alone is insufficient for reuse.
Older servers/images leave `runtime_digest` unset, which cannot
establish a match. Both digests and the other registration inputs must
match before reusing a version.
```python
request = definition.registration_request
same_source = version.source_digest == request.artifact.digest
same_runtime = version.runtime_digest == request.runtime.digest
```
Add shared runtime digest fixtures and coverage for environment-only
changes and catalog decoding. Local builds and tests were skipped as
requested.
Upgrade the Rust Lance dependency, Java lance-core, and Python test
dependency from beta.17 to the officially published
[v15.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.18)
(`079fd77683a7acf056b0912f6ecda938e39ee120`). This provides a compatible
SDK for the newer Lance source already adopted downstream. Rust registry
versions and checksums are unchanged; the Python lock uses the six
published beta.18 wheels.
### Compatibility
The adopted release includes [Lance
#10863](https://github.com/lance-format/lance/pull/10863):
`FilteredReadOptionsProto.has_fragment_ids` distinguishes an explicit
empty fragment selection. Older executors ignore this field, so upgrade
executors before relying on the corrected empty-selection behavior. Rust
callers constructing this generated struct need the new field. [Lance
#10112](https://github.com/lance-format/lance/pull/10112) also replaces
the internal `hamming_distance_batch` API with
`hamming_distance_batch_into`.
The Linux Python matrix timeout is raised from 30 to 60 minutes: the
Python 3.13 cold native build and both test passes completed, but the
job hit its limit during cleanup. Build profiles, test commands, and
coverage are unchanged.
### Validation
- Three native FRI/ANN regressions passed: deferred-compaction refined
RQ, deferred index remapping, and approximate-query builder.
- `cargo clippy --quiet --locked --features remote --tests --examples --
-D warnings` passed with pinned Rust 1.97.0.
- Locked metadata, formatting, Python Makefile-scope Ruff checks, and
`uv lock --check --offline` passed.
- YAML, whitespace, end-of-file, and typo checks passed for the
timeout-only workflow change.
### Publication and CI
Java beta.18 is now published. After [Lance
#10968](https://github.com/lance-format/lance/pull/10968) merged, the
[normal publication
workflow](https://github.com/lance-format/lance/actions/runs/43146352505)
built all three JNI targets and published successfully; the Maven jar is
available. The upstream Python publication run was cancelled after its
Windows ARM job hit the one-hour timeout; six other wheels were
published and are the actual artifacts recorded in this lock.
Cross-platform SDK CI is still required. No performance claim is made by
this dependency update.
Reject synchronous and asynchronous generator Functions, including class
`__call__` methods, during decoration so invalid outputs never reach
remote execution. Give actionable errors for lambdas and decorated
wrappers such as `functools.lru_cache`. Defaulted callable inputs remain
unsupported; the error now names the parameter and explains how to use
class initialization fields.
```python
@function
def double(value: float) -> float:
yield value * 2
# TypeError: generator Functions are not supported; return a value instead of
# yielding, or return a list for multiple values
```
The deprecated `@udf` alias shares the validation. Adds regression
coverage for both decorators, class methods, explicit schemas, and the
supported list-return alternative.
Fixes https://linear.app/lancedb/issue/ENT-3539
Table existence checks should not depend on listing every table or
searching only the first 100 names. Use a direct lookup for `name in db`
and implement `table_exists(table_id)` on synchronous local connections,
matching the remote and asynchronous APIs. Native namespace connections
use their initialized Rust connection for existence checks; custom
namespace clients retain their existing callback. Missing tables return
`False`; other lookup errors propagate.
```python
if "events" in db:
db.drop_table("events")
exists = db.table_exists(["analytics", "events"])
all_names = list(db.list_tables())
```
Clarify that `.tables` contains one page and update deprecation guidance
with safe replacements. Add regression coverage for lookups beyond the
first page, nested namespaces, native and custom namespace clients,
missing tables, permission errors, and lookups that do not list tables.
Include an unpublished draft for the table-management guide until the
new SDK support ships.
Function dependency typos currently reach the environment build before
failing. Validate pip requirements with
`packaging.requirements.Requirement` and conda requirements with
Rattler's MatchSpec parser when constructing a function definition,
before source packaging or registration. Errors identify the original
entry and its index.
```python
@lancedb.function(pip=["numpy", "torch>=2.0,,<3"])
def score(value: float) -> float:
return value
# ValueError: invalid pip[1] requirement 'torch>=2.0,,<3': ...
```
Preserves pip extras, markers, named references, bare archive/git URLs,
and conda channel/build specifications. The existing native extension
exposes the conda parser, so no additional Python package is required.
Adds regression coverage for invalid and supported forms, including the
deprecated `udf` alias.
## Summary
`RefreshColumnResult` exposed the canonical refresh counters without
documenting their unit. This change describes them as logical rows
planned for one refresh: `rows_assigned` counts successful rows written,
`rows_failed` counts rows whose computation failed, and `rows_remaining`
counts planned rows neither assigned nor terminally skipped. Rows
already covered by `rows_failed` are not counted again as remaining.
Retry attempts and user-code calls do not change that logical-row scope.
The change also documents the source and published versions and the
deprecated `rows_filled` alias.
The existing skip-policy example is now explicit: a refresh over 2,001
logical rows with 1,891 assigned, 110 failed, and 0 remaining uses 2,001
as the failure-rate denominator.
## Behavior
Documentation only. No field names, Python or native behavior, REST wire
format, dependencies, or pins change.
## Validation
- `git diff --check`
- `uvx ruff format --check`
- `uvx ruff check`
- Imported this Python source with a compatible prebuilt ABI3 extension
and asserted the public class and property documentation, the affected
source version, and the native extension identity.
Function jobs can now export their recorded execution timeline with
`job.export_timeline(path)` or `await job.export_timeline(path)`. The
SDK fetches a fresh Chrome Trace Event recording that opens in Perfetto,
including worker split spans and timestamped job phases. Batch counts
and latency statistics remain details rather than invented batch
intervals.
```python
job = db.open_job(job_id)
job.export_timeline("refresh.trace.json")
async_job = await async_db.open_job(job_id)
await async_job.export_timeline("refresh.trace.json")
```
The export does not wait for completion or cancel the job. Running jobs
return partial recordings. Local operations and SQL query handles remain
unsupported, and servers must support the timeline endpoint. The
destination is touched only after a successful fetch; network and
filesystem errors propagate. Rust freshness wrappers forward the
operation without changing the submitting table's read baseline.
Update the Rust workspace, Java dependency, and Python test dependency
to [Lance
v15.0.0-beta.17](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.17).
This includes the FRI cache and vector-index changes required by current
Sophon and retains the RaBitQ prewarm fixes. Lockfiles use the published
source and test wheels without unrelated registry version bumps.
The dependency dry run also exposed CI resource limits. Use the existing
larger x64 runner for GNU Node and compatibility-wheel builds; allow 60
minutes for Mac Python tests, 120 minutes for the compatibility wheel,
and 150 minutes for the Mac release wheel. Disable the CLI toolchain
action's automatic target cache so only the subsequent downloads-only
cache runs. Release optimization, the compatibility wheel's manylinux
container and x86-64-v2 baseline, and test coverage stay unchanged.
Validation:
- Three native SDK regressions pass: refined RQ queries after deferred
compaction, deferred index remapping, and ANN approximation planning.
- Pinned Rust 1.97.0: `cargo clippy --quiet --locked --features remote
--tests --examples -- -D warnings` passes.
- Locked offline Rust metadata, Python lock validation, Ruff
lint/format, and diff checks pass. The Python lock contains six
published beta.17 wheels.
- Repository YAML, whitespace and typo checks pass for the workflow
changes. Product-source checks are reused for the subsequent CI-only
commits.
- At head `07033dc6`, all 65 applicable pre-review CI checks pass,
including the compatibility wheel, Mac CLI, GNU Node, Mac Python tests,
release wheels, and Java checks. Six checks are skipped by tag/main-only
conditions or draft-only app behavior; the automated review and security
checks run after the Ready transition.
The upstream Java publisher succeeded and its beta.17 Maven artifact is
available. Six upstream Python wheel builds/uploads succeeded; its
Windows ARM job was cancelled. These checks do not establish Sophon
performance or aggregate-memory acceptance.
## Problem
The existing Function `column_status` surface cannot decode the Agent
scheduler's `queued` state or the `failure_threshold_reached` and
`binding_revision_missing` blocked reasons. It also preserves only the
latest refresh job, so a later cancellation can hide an earlier actual
failure.
## Change
- Adds the `queued`, `failure_threshold_reached`, and
`binding_revision_missing` wire states in Rust, Python, and Node. The
wire change is additive, while Rust callers must update exhaustive enum
matches and public struct literals.
- Adds optional `last_failure_job` alongside `last_refresh_job`. The
latest cancellation and the most recent actual failure retain separate
meanings.
- Reuses the canonical Rust `JobFailure` and existing Python/Node
`JobFailureInfo` models for nested job failure details instead of
introducing a parallel status error model.
- Maps the existing public `sql_job_id` field through Node's
`JobFailureInfo`.
- Keeps `column_status`, `column_statuses`, and `wait_fresh` as the
existing API surfaces. There is no new endpoint, row-error list, durable
ledger, or unknown-enum fallback.
- Documents `queued` versus `running` and the separate latest-attempt
and latest-failure jobs in the existing hidden freshness release draft.
The new status values are additive but cross-service: `queued`, the two
new blocked reasons, and `last_failure_job` require matching Agent/QN
and SDK deployments before clients can rely on them. This SDK follow-up
does not change the normal `7cc3904a` dependency pin or claim whole
ENT-3487 completion.
## Validation
The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`,
based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The
final product docs commit changes generated TypeDoc only.
- Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust
blobs are unchanged from their tested `788ef85b` source through
`c9bac25e`.
- Python: the PyO3 extension built successfully, and the selected native
PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The
test requests pinned version 42.
- Node: the addon and TypeScript builds passed, Biome passed, and the
selected Jest transport test passed 1/0 with 57 filtered tests. Addon
SHA-256:
`c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`.
- Python extension SHA-256:
`36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`.
- TypeDoc was generated from the native `native.d.ts` produced by that
Node build. A second generation produced no remaining diff.
The public diff contains only product, test, and generated API
documentation changes. Native runner scripts and preserved
compile-failure logs remain private validation evidence.
Agent/QN production mapping, mixed-version rollout, and the remaining
ENT-3487 server-side work are tracked separately.
## Problem
Function-backed columns need an exact `fresh`, `stale`,
`never_computed`, and
`failed` partition for the table version a caller is reading. The
existing
error listing is capped, and composing it with job and schema listings
cannot
prove which rows currently failed.
## Behavior
- Adds a strict freshness reader beside the existing permissive planner.
It
distinguishes an absent certificate from legacy evidence that omits
failures, and from a current certificate carrying failure coverage.
- Extends the immutable freshness sidecar with per-fragment known and
failed
row bitmaps. Partial or `WHERE` publications cover only observed rows;
untouched rows stay unknown. Deleted rows do not block classification.
- Adds failure-aware whole-fragment and row-repair writers. A rejected
older
attempt cannot overwrite failure evidence already recorded for a newer
input
signature.
- Adds `column_status`, `column_statuses`, and `wait_fresh` to Python
and the
corresponding Node methods. The client calls the server's exact status
endpoint, validates the four-way partition and one-version guarantee,
and
preserves the complete Function version, last-job, and automatic-refresh
metadata.
- Adds an MDX-commented draft usage section to the Enterprise
computed-column
refresh guide, covering Python, Rust, and TypeScript and remaining
hidden
until the server integration ships.
- Sends `column`, `branch`, and exact `version` selectors to the status
route.
A pinned handle rejects a nonempty response from any other table
version;
latest handles keep the existing read-freshness headers.
- Retries `wait_fresh` only for HTTP 503 with
`function_column_status_finalization_pending`. Existing failed rows
remain
visible and do not keep the wait pending; other errors propagate.
- Models the shared scheduler projection, including `up_to_date`, real
`blocked` states, their reason, failure streak, and effective cooldown.
## Validation
This is the SDK side of ENT-3563. Sophon's failure-coverage publication
and
status endpoint are a separate integration; deployed end-to-end and
whole-ticket
acceptance remain pending.
- Rust: 9 exact freshness/status tests passed, plus 1 exact
latest-handle
freshness-header test. The pinned transport test was extended with a
wrong-version response and rerun at `ebd6649a` (1 passed). Coverage
includes untouched partial rows,
selected-row failure clearing, cross-generation evidence, typed and
non-retried errors, `up_to_date`, blocked scheduler fields, and
exact-version
requests.
- Python: 2 native PyO3 remote-transport tests passed together,
including
typed retry; one local facade test reaches the existing unsupported
boundary.
- Node: the NAPI addon built successfully, TypeScript and Biome passed,
and the
selected native remote-transport test passed 1/0, including
exact-version
selection and blocked scheduler metadata.
- `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile,
Biome, and
TypeScript checks passed.
Validation is mapped by the relevant unchanged source rather than
treated as one
final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header
and Python
remote tests at `31e68b7d`, the local Python test and Node build at
`6f213584`, and
the final TypeScript/Jest delta at `4f2d5c27`. The final product tree
excludes those five private EC2 runner files and adds
six generated TypeDoc files plus one hidden draft usage section in the
computed-column refresh guide. The Node delta corrected a private
TypeScript
command and reused the unchanged native addon.
## Summary
Adds an optional `max_concurrency` setting to Function registration and
explicit refresh:
- A registered Function version can define the default maximum number of
user-code calls in flight for one logical execution.
- A refresh can override that default for that execution only.
- The value must be a positive `u32` and is independent from
`resource_budget`; it is not an account-wide quota.
- Omitting the option preserves the previous request payload. Local
embedded refreshes reject this Enterprise-only option.
Rust, Python sync/async, PyO3, and the existing Node refresh surface
forward the option. Node has no Function registration surface today, so
no registration API is added there. The generated TypeDoc pages are
included for the new public option.
## Validation
The native-tested source covers Function version decoding, registration
validation and serialization, the Flight SQL registration ticket and
returned version, remote refresh forwarding, local rejection, and Node's
numeric boundary before conversion. Exact stage results are Rust `1+3`
plus four remote cases at `1` each; Python `14/1/49/24`; Node `57`
remote and `1` local rejection. Ruff, standard repository Pyright, and
Biome passed.
The tested Python extension was
`21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the
tested Node addon and generated native declaration were
`f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and
`4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The
final 24-file diff preserves the 22 tested
product/test/fixture/source-doc blobs and adds only two generated
TypeDoc pages.
## Depends on merged #4951
Phase 1 merged as `4d22c0ce2b6825c880c818e737b079db494b4053`, so this PR
now targets `main`. It reuses the canonical
`FunctionWorkerResourceBudget` introduced there rather than copying the
type.
For Function `add_columns` declarations, Python and Rust now persist
`auto_refresh_resource_budget` in computed-column metadata and send it
as `auto_refresh_resource_budget` in the remote request. CPU
milliseconds and memory bytes are required positive integers. Omitting
`gpu_count` is accepted as zero and the canonical outgoing JSON still
contains `"gpu_count": 0`. Unknown fields, missing CPU or memory, wrong
types, zero CPU or memory, negative values, and overflow are rejected.
The same budget is applied to every public output binding of a
multi-output Function declaration. Existing metadata is retained, and
SQL-expression declarations or local execution reject the budget instead
of silently ignoring it. Node is unchanged because its current
`addColumns` surface only accepts SQL computed expressions and has no
Function declaration path.
This change does not implement fleet call concurrency limits or enforce
the budget in the runtime. The Enterprise server must support the nested
Function worker resource-budget contract before these values can act as
execution limits.
## Validation
- Python extension build passed.
- Remote Function declaration and local unsupported-path rejection each
passed.
- Canonical metadata round-trip passed, including omitted GPU decoding
to zero and explicit `gpu_count: 0` on encode.
- Function multi-output remote wire test passed with `--features
remote`.
- Repository-scoped Python type check passed with 0 errors.
- All runtime source blobs match the native-tested source. The
post-native deltas are one documentation example line and an existing
routing-test stub update for the new optional argument; the affected
parameterized test passed 3/3 against the existing extension build. No
private test runners are part of this PR.
## Problem
Function catalog lookup required an exact version string even though
callers
commonly need the latest currently visible immutable revision. On the
current
SDK, resolving that version required an out-of-band catalog query before
calling
`get_function`.
## Change
- Make Python `get_function(name)` default to latest while accepting
`int` or
`str` for exact lookup.
- Normalize integer versions to their decimal string form and reject
`bool`
explicitly. Existing string validation and errors remain unchanged.
- Thread `None` through the rebuilt PyO3 binding and route it to the new
Rust
`Connection::get_function_latest`.
- Resolve latest through the existing catalog `DESCRIBE FUNCTION` path
without
`WITH (version = ...)`. The returned value is one exact immutable
version, not
a mutable alias.
- Preserve the existing Rust exact-version API.
## Validation
On the designated EC2 host with the rebuilt extension:
- `test_local_function_catalog_operations_are_not_supported`: **1
passed**.
This crosses the Python connection, rebuilt PyO3, and embedded Rust
`NotSupported` boundary with the omitted-version call.
- Async and sync catalog-path tests: **2 passed**, 117 deselected.
- Remote SQL Function creation/description integration test: **1
passed**.
- An earlier Rust invocation with the wrong module path matched **0
tests**;
it is retained as harness evidence and is not counted as a pass.
## Scope
This is the Function lookup SDK slice. It does not include the separate
atomic
rebind/publication server work tracked under ENT-3418.
## Summary
- Let local plain queries over row-addressable legacy v1 and v2 blob
columns use `blob_mode="bytes"` without importing `pylance` through
pandas, batch, or Arrow conversion.
- Preserve aliases, filters, ordering, limits, null and empty payloads,
empty result schemas, and the caller's `_rowid` choice.
- Build the hidden row-id query separately so reusing an async builder
for lazy reads or non-blob projections cannot retain an automatic
`_rowid`.
- Keep byte reads closed for blobs inside lists, remote or namespace
pushdown, and query shapes whose result rows cannot be matched back to
the source. Blob v2 lazy auto-row-id behavior is unchanged.
Fixes#4645
## Validation
- 26 focused Python tests passed with the test subprocess blocking all
`lance` imports. The persisted fixture writes and reopens a storage-v2.1
table, then covers sync and async pandas, batches, and Arrow conversion
plus async builder reuse.
- The existing unsupported vector and hybrid reranker regressions now
assert the current eligibility diagnostic, and the legacy table-level
Arrow regression now asserts successful bytes, schema, values, and
hidden row IDs. Both focused tests passed.
- The runtime receipt records no installed `pylance` distribution and no
importable `lance` module. The Python-only change was verified with an
existing compatible 0.42.0b9 extension; the relevant query and fetch
APIs are unchanged between that source and this 0.42.0b10 base.
- Ruff format and lint pass for every changed Python file.
## Summary
- Add the canonical Rust `FunctionWorkerResourceBudget` and
`RefreshColumnOptions::resource_budget()` for explicit Enterprise
Function refreshes.
- Forward the nested budget through Python sync/async, PyO3, Rust remote
requests, Node NAPI, and TypeScript.
- Require positive CPU and memory values; normalize an omitted nested
GPU count to numeric zero; reject the nested budget when combined with
the top-level `gpu_count` shorthand.
- Reject `resource_budget` on LanceDB Embedded refresh paths instead of
silently ignoring it.
- Document the Rust, Python, and TypeScript APIs, including that SQL
servers currently require all three budget fields.
The Rust, Python, and TypeScript additions use `cpu_millis` /
`cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` /
`gpuCount`. The top-level GPU shorthand keeps its existing behavior and
remains mutually exclusive with the complete nested budget.
## Scope and compatibility
This change configures only the explicit refresh call. It does not
configure `add_columns` automatic refresh and does not introduce a
fleet-wide call-cap interface.
A LanceDB Enterprise server must expose the nested `resource_budget`
backfill contract. LanceDB Embedded continues to support its existing
expression refresh defaults and rejects the new complete budget
explicitly.
## Validation
Validated against the final signed product source:
- Rust: resource-budget remote serialization test `2 passed`; validation
test `1 passed`.
- Python: actual PyO3 build; remote-request matrix `22 passed`;
invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff
check and format clean.
- Repository-standard Pyright entry point: `0 errors`, within the
repository's configured include set.
- Node: native build and Biome lint passed; the existing remote-request
test `1 passed`, including fractional, negative, overflow, NaN, and
infinity nested GPU cases; the existing local-rejection test `1 passed`.
- TypeDoc generation and generated-document cleanliness passed.
- The native-tested product source differs from this head only in two
semantically equivalent Node range checks required by Clippy. Current CI
compiles and lints the final blobs. No dependency or lockfile changes
are included.
Unsupported remote blob materialization errors now point to row-ID
selection, fetch_blobs and fetch_blob_files. Document descriptor
structs, fetched bytes and Function BlobReader inputs, and show a
bounded subset-copy recipe that keeps the source unchanged between
selection and fetching.
Adds guidance assertions and copy coverage preserving payloads, empty
blobs and nulls.
No new blob materialization API is introduced.
Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
Printing a Job or a bound status method could dump hundreds of lines of
serialized Function initialization. Bound nested payload previews in
both sync and async handles while preserving cached identity, state,
failure and result information. Full spec/result values remain available
through their properties, with no additional backend calls.
Adds regression coverage for large payloads, bound method display,
unobserved handles and completed local jobs.
The broader method/property API design remains separate.
Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
Function source packaging rejected unsupported global values without
identifying the variable or explaining the supported captures. Include
the global name even for a nested tuple failure, describe the supported
literal values and remedy, and document the capture boundary.
Adds function/class regression cases for Arrow types, nested tuples,
dictionaries and non-finite floats, plus a supported nested literal
control.
New serialization support remains separate.
Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
Expose the durable SQL registry id separately from the connection-scoped
query UUID. Rust and Python query handles and descriptions expose
optional `job_id`; result readers preserve it in schema metadata under
`x-lancedb-sql-job-id`, without adding business columns. Clients can
reopen that id with `open_job`.
Poll success and failure metadata update the same handle. A failure
before the first successful poll retains its job id and original
structured error. The cancellation documentation distinguishes the
Flight waiter from `cancel_job(job_id)`.
Validation on the designated EC2: the two focused error interoperability
tests and Ruff passed on `a8798e81`; delivery has the identical tree.
The preceding 51 Rust SQL tests and earlier 31 native Python SQL tests
are reused for unchanged coverage. Matching-runtime independent
acceptance remains pending.
Companion server change: https://github.com/lancedb/sophon/pull/10049.
Refs ENT-3562.
Update the Rust workspace, Java lance-core, and Python test dependency
to [Lance
v15.0.0-beta.16](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.16).
Synchronize `python/uv.lock` with the published beta.16 wheels for
Linux, macOS, and Windows.
Run the native Linux x64 wheel build on an eight-core runner, matching
the existing ARM build allocation, after its optimized build repeatedly
exceeded the job deadline on the standard runner.
The dependency update passed Clippy, formatting, and CI-script checks;
the Python lockfile refresh/check passes. Fresh CI is running for the
runner adjustment. Java dependency-resolution failures caused solely by
publication lag are accepted for this upgrade.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
Python refresh APIs currently reject `publication_interval_rows`, so
callers cannot request incremental Function output publication through
the SDK. Add the optional typed parameter through the sync/async Python
APIs, PyO3, and Rust `RefreshColumnOptions`, and serialize it as a
top-level Enterprise refresh option.
An omitted value preserves the existing wire request and whole-fragment
default. Positive values set a publication threshold; zero and negative
Python values fail before a request is sent. Embedded refreshes reject
the Enterprise-only option. Documentation describes a threshold rather
than an exact number of rows per publication.
Validation on the designated development host:
- A matching baseline build reproduced the missing public keyword
argument in both sync and async submission tests.
- The updated PyO3 build passed 54 focused Python tests; Rust payload,
invalid-option, and default-wire tests each passed. Rust formatting and
Python Ruff checks passed.
- The single signed delivery commit differs from that tested
implementation only in a Rustdoc correction describing Embedded
rejection; formatting passed again. No dependency pins or lockfiles
changed.
The matching SDK is prepared for the separate server validation of
failure/cancellation retention and missing-only refill. Those end-to-end
results are still pending and are not implied by these SDK tests.
Refs [ENT-3733](https://linear.app/lancedb/issue/ENT-3733).
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.
```python
job = view.refresh_async(
cpu_millis=8000,
memory_bytes=16 * 1024**3,
gpu_count=2,
gpu_class="h100",
)
job.wait()
```
```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
gpu_count = 2, gpu_class = 'h100');
```
The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.
Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.
The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.
Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
Update Rust and Java Lance dependencies to
[v15.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.15),
pin Python tests to published pylance 15.0.0b15, and refresh both
lockfiles; no compatibility fixes were needed.
Validation passed: workspace Clippy with all features/tests and warnings
denied, cargo fmt, uv lock --check, and all 35 CI-script tests.
The [upstream wheel
workflow](https://github.com/lance-format/lance/actions/runs/42669437482)
canceled Windows ARM64 and manylinux_2_28 x86_64 builds after an hour,
so the lockfile contains the five published variants (Linux x86_64
remains covered by manylinux2014); Windows ARM64 is unavailable until
upstream publishes its wheel and the lockfile is refreshed.
Update the Rust workspace Lance crates and Python test dependency to
[Lance
v15.0.0-beta.14](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.14),
which includes JSON cache reuse across request stores and stable-row-ID
JSON index support. The Python lock contains all seven published wheels.
The independent Java module retains beta.8, as in the previous beta.13
update; its separate beta.14 publisher repair is outside this Rust
consumer upgrade.
Preserve all CI coverage while reclaiming completed workspace lancedb
artifacts before the package-level feature-test build, which otherwise
exhausted the Linux runner disk. Give only the Linux ARM64 release-wheel
job two hours after its one-hour timeout, retaining the same runner,
fat-LTO release profile and build flags.
Validation: 2,280 Rust tests passed (79 existing ignores), strict
workspace/all-feature Clippy, Rust formatting, and remote example checks
passed on unchanged Rust source and dependency lock. Python dependency
guards, lock validation, locked sync, and the published macOS wheel
import passed. The new Windows ARM64 wheel was downloaded and its
SHA-256 verified. Workflow YAML/shell checks and independent review
passed. Root Python Ruff/format checks still fail on unchanged
documentation examples; no suppressions or unrelated cleanup were added.
The CI repairs require a complete new run before readiness. This
dependency update does not establish corrected Sophon performance;
downstream native acceptance and the paired S3 replay remain separate.
Update the Rust workspace Lance dependencies to
[v15.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.13),
synchronize the Python test dependency to published pylance 15.0.0b13,
and refresh both lockfiles; no compatibility fixes were needed.
The Java SDK stays on `lance-core` 15.0.0-beta.8, as in the beta.9 and
beta.10 updates: Maven Central has no `lance-core` artifact for
15.0.0-beta.9 through beta.13.
Validation passed: workspace Clippy with all features and warnings
denied, Cargo formatting, Python lockfile checks, and all 35 CI-script
tests. Repository-wide Ruff reports 163 diagnostics, all verified
identical on unchanged HEAD; unrelated formatter changes were reverted.
---------
Co-authored-by: Vivek <vbharathan2008@gmail.com>
The server's backfill request takes a row filter, failure limits,
per-call timeouts and worker sizing, and SQL exposes all of them through
`REFRESH COLUMN ... WHERE ... WITH (...)`, but the typed SDKs could send
only batch_size, GPU placement and the retry and error policies. Anyone
needing a filtered refresh or a bounded skip policy had to drop to SQL.
This adds `where`, `max_failed_fraction`, `max_failed_rows`,
`row_timeout_seconds`, `init_timeout_seconds`, `num_workers` and
`max_executor_processes_per_worker` to Python, Rust and Node, passed
through unchanged. `RefreshColumnOptions` now serializes straight into
the request body, so the three SDKs share one producer. Node gets a
`RefreshColumnOptions` object, which also carries the options it lacked
before. A LanceDB Embedded refresh rejects every option other than
batch_size and GPU placement, naming the option.
Lance beta.11 calibrates lower `search_effort` tiers by metric and
requested `k`. With the Rust dependency and Python/Node execution tests
already on main via #4881, document the tiers across Rust, Python, Node
and the search guide, extend the remote request matrix to cover `0.1`
and `0.25`, and align the Python test package with beta.11.
`0.1` and `0.25` target empirical mean partition-routing recall of 80%
and 90% for L2, cosine and dot through `k=100000`. These are not
per-query or final Recall@k guarantees. Default Auto is unchanged;
Hamming and larger `k` remain uncalibrated, and remote queries require
corresponding server support. This PR makes no additional performance
claim. See [Lance
#10695](https://github.com/lance-format/lance/pull/10695) for
measurements and limitations.
Validation: both strict workspace Clippy configurations, Rust
formatting, 13 Rust effort tests, 208 Python query tests, 41 Node query
tests, native binding builds, TypeScript, Node lint/generated docs,
Python lint/format, full lockfile check and two dependency guards passed
on `a8b09170`. All validated Rust/Python/Node source and dependency
files are byte-identical at `2a82d07c`; the only subsequent change
restores the Java POM to main. Root Ruff has 59 existing diagnostics and
six formatting files, all byte-identical to the base; the Python package
passes.
The Java REST/Namespace client has no changed API here, and its
dependency remains identical to main. The separate [Java beta
publication failure](https://github.com/lance-format/lance/issues/10773)
is not claimed resolved or validated. Official SDK release, automatic
mirror, Sophon adoption and measured consumer acceptance remain
follow-up stages.
Downstream workspaces adopting Lance 15.0.0-beta.11 currently also
resolve beta.10 because the SDK pins every Lance crate exactly. This
produces incompatible Lance types across the SDK and the consumer.
Update the workspace pins and lockfile to the published
[v15.0.0-beta.11](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.11).
This release includes payload-free list length reads and preserves those
read modes across serialized plans (lance-format/lance#10759,
lance-format/lance#10766). Existing feature flags and unrelated locked
packages are unchanged.
The release also includes the metric/k-dependent search-effort
calibration from lance-format/lance#10695. Update the Node and Python
execution tests to its exact probe budgets (for example, effort 0.25
with L2, k=8 and Auto=6 now probes 4 partitions), and cover the 0.1
tier. Recall, caller bounds and probe-precedence assertions remain in
place.
Validation:
- `cargo fmt --all --check`
- `cargo clippy --quiet --features remote --tests --examples`
- Downstream metadata fixture with SDK source and the released Lance
beta.11 path overrides resolves exactly one Lance version.
- `cargo test --quiet --features remote --tests`: 2,271 passed, 0
failed, 79 existing ignored tests.
- Node native/TypeScript build, lint and API documentation generation
passed; 41 query tests passed.
- Python native binding build and package Ruff checks passed; 208 query
tests passed, 1 slow test deselected, 5 existing deprecation warnings.
- Rust results remain applicable: the follow-up changes only binding
tests, with product source and dependency graph unchanged from the
tested dependency update.
Repository-root Ruff currently reports 59 lint findings and 6 formatting
failures in untouched documentation snippets. Their blobs match the PR
base; the changed Python test and the entire Python package pass
lint/format checks.
Remote Python `drop_function` currently discards the dependent-view
warnings returned by SQL DROP. Both the synchronous and coroutine APIs
now emit those warnings through `warnings.warn` while preserving their
boolean result and executing DROP once. The server support is already
merged in lancedb/sophon#9820.
The Rust decoder reads warnings across every result batch, including an
empty first batch. Missing warning columns remain compatible with older
servers; malformed or null warning lists are rejected. The hidden
binding bridge carries the warnings to Python. The separate
`drop_function_async` API and its `(bool, Job)` contract are unchanged.
Python warning filters can raise after the catalog mutation has
completed.
Validation: a compiled PyO3/public-server reproduction failed before the
fix and passed afterward; the public async path also returned `true`
with the expected dependent-view warning. Final source passes two
focused Rust tests, two Python wrapper tests, strict targeted Clippy,
formatting, and Ruff. Three decoder tests cover batch/null/type and
older-server behavior. The public runs preceded a two-file
simplification of the completed Job projection; focused final-source
checks cover that delta. Current-main integration is conflict-free, and
its overlapping changes concern independent retry-policy APIs/tests.
Documentation now covers warning filters, the post-removal
warnings-as-errors behavior, and silent compatibility with older
servers. The follow-up changes only prose and a Python docstring; the
runtime validation above remains applicable.
Refs ENT-3659. Whole-issue independent acceptance remains pending.
## Summary
Bumps dependencies that Aikido flags with critical advisories, where a
patched version is available without code changes.
| Package | Where | From → To | Ships to users |
|---|---|---|---|
| jackson (BOM) | `java/pom.xml` | 2.18.11 → 2.18.12 | Yes, via the
published pom |
| zerovec / zerovec-derive | `Cargo.lock` | 0.11.6 → 0.11.8 / 0.11.3 →
0.11.7 | Yes, linked into the wheels and Node binaries |
| pillow | `python/uv.lock` | 12.3.0 → 13.0.0 | No, dev lockfile |
| pyarrow | `python/uv.lock` | 24.0.0 → 26.0.1 | No, dev lockfile |
| pyjwt, urllib3, anyio, cryptography | `python/uv.lock` | → 2.15.1,
2.8.0, 4.16.0, 50.0.2 | No, dev lockfile |
| pillow | `docs/web-tests/blob-v2/py/requirements.txt` | 12.1.1 →
13.0.0 | No |
pyarrow was held at 24.0.0 by a `pyarrow<25` pin in the `tests` extra.
It came in with #3675 without a stated reason, so this PR removes it.
pillow 13 and pyarrow 26 require Python 3.11+, so the lockfile keeps
pillow 12.3.0 and pyarrow 25.0.1 on Python 3.10, and those remain
flagged.
The sharp findings in the Node examples and docs are not addressed here.
They come from the `@huggingface/transformers@3.0.2` optional
dependency, which needs a major upgrade to 4.3+.
## Evidence
With the pyarrow pin removed and pyarrow 26.0.1 installed, the Python
suite (`-m "not slow and not s3_test"`) passed locally on Python 3.12:
3319 passed, 164 skipped. That run was on main as of #4879; this branch
is based on a slightly newer main.
## Merge Risk
**Door:** Two-way. Patch and minor version bumps that a revert undoes.
**Blast Radius:** Java consumers (jackson patch release), native
binaries (zerovec patch release), Python test environments.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Functions that import a module unavailable on the client currently
package silently even when they declare no pip/conda dependencies. Emit
an advisory warning for these unresolved imports so callers can declare
the required package or confirm that the base image supplies it.
Any pip/conda declaration suppresses this new advisory because
distribution names need not match import names. The warning does not
claim that the module is absent remotely and follows normal Python
warning filters, including warnings-as-errors.
Validation includes the original `import scipy.ndimage` with `pip=()`
through the public decorator/packager path, plus declared-package and
existing local-source controls. This does not establish remote image
availability or deployment acceptance.
Refs ENT-3448.
Xuanwo
Jack Ye
Lance Release
Yang Cen
BHAWESHBHASKAR
LanceDB RobotandJack Ye
Sarwar Bhuiyan
Bruno Ramirez
Brendan ClementandClaude
Wyatt Alt
Will JonesandClaude Opus 5.5
lancedb-gatefixer[bot]