Commit Graph
1641 Commits
Author SHA1 Message Date
Jack Ye e8a8dd1b2a feat(sql): use server-issued catalog listing tokens (#5026)
Migrates SQL-backed catalog listing APIs to `LIMIT ... FROM
'<page-token>'` and server-issued Arrow continuation metadata.
Namespace-specific APIs send an explicit `IN` schema clause, including
`IN "public"` for root, so raw SQL can default to all namespaces.

```sql
SHOW VIEWS LIMIT 100;
SHOW VIEWS IN team$ml LIMIT 100;
SHOW VIEWS IN team$ml LIMIT 100 FROM '<page-token>';
SHOW VIEWS IN other_db.team$ml LIMIT 100;
```

SDK clients forward opaque tokens unchanged, preserve empty
continuations, and decode identifier columns by name. Removes legacy
cursor reconstruction and updates the SQL reference and Python Flight
fixture. Public docs cover optional direct schema paths, all-namespace
defaults, and the shared pagination syntax across every catalog listing.
2026-11-15 08:55:42 +08:00
Xuanwo e2f836da99 feat: expose per-call merge retry controls in Python and Node (#5015)
Python and Node callers could not disable precommit retries for one
merge without changing the connection. Add `execute(..., retries=0)` for
Python sync/async and `execute(data, { retries: 0 })` for TypeScript,
forwarding into the existing Rust control without persisting the
override. Ambiguous 5xx outcomes remain non-replayable; permitted merge
retries now log status and attempt at WARN, and the API docs explain
retry budgets and timeout boundaries.

Validation at `6001300` passed 70 Python HTTP cases, 36 Node HTTP cases,
14 local merge regressions per binding, and 34 scoped Rust tests, with
rebuilt native extensions, TypeScript compilation, Clippy, scoped
formatting/lint, and generated API docs. The public API and feature diff
are unchanged after integrating main and the reviewed ingestion guide;
that binding evidence is retained. Focused merged-tree validation
additionally passed 267 Rust merge/retry/provenance cases and the CI
fixtures. Existing root Ruff issues in docs/web-tests remain; local
fixtures do not establish live proxy/server cancellation behavior.

Refs #4256 and OSS-2370.
2026-11-14 22:43:56 +08:00
Xuanwo dc9eb269b1 feat(python): support map and decimal Function signatures (#5031)
Python Function declarations reject Arrow map and decimal types before
registration. Accept `dict[str, T]`, nullable map values, explicit Arrow
maps, and `Annotated[Decimal, pa.decimal128/256(...)]` using the
existing canonical JSON representation. Preserve map key/value
nullability and decimal precision/scale; reject unsupported map keys,
sorted maps, nested Blob values, and a bare Decimal without
precision/scale.

This is the SDK map/decimal portion of ENT-3414. Server validation and
SQL support are separate changes; older servers can still reject these
signatures. Nullable scalar output support remains separate in #5030.

Validation: 129 Function signature tests passed; 30 documentation tests
passed with one existing skip. Ruff format/lint and diff checks passed.
Python tests used the existing Python 3.13 environment and prebuilt
native extension with this Python source overlaid; no Rust build or
remote-server execution was performed.
2026-11-14 22:30:47 +08:00
Xuanwo 2499342fa1 feat(function): expose resolved environments and fresh lock requests (#5034)
Function registration clients cannot request fresh dependency resolution
or inspect the measured environment returned by the server. Add
`refresh_lock=True` to Python registration and preserve the requested
runtime and measured environments in Rust/Python Function version
readback. Registration forwards both `exist_ok` and `refresh_lock`;
defaults remain unchanged.

Single-platform versions retain `resolved_environment`.
Multiarchitecture versions expose `resolved_environments`, keyed by OCI
`os/architecture`, and the Python `environment_report` alias. Platforms
are never flattened: absent entries mean unknown measurements, while
measured empty package lists remain present. Registration waits emit
scalar and per-platform server warnings once per Job; catalog reads do
not repeat them.

Requires matching server support, including Sophon #10344 for
per-platform publication. Nine focused Python source-overlay tests and
29 documentation tests passed for the additive map change; one
documentation test requiring a newer native `exist_ok` binding was
excluded. These checks reuse a prebuilt extension and do not establish
current native integration. PR CI exercises the Rust transport/readback
and Python bindings. Live multiarchitecture registration and GPU
acceptance remain separate.
2026-11-14 22:30:28 +08:00
Xuanwo ea59f8acb6 feat(python): preserve Function timestamp and duration types (#5032)
Function declarations currently reject exact timestamp and duration
types. Preserve their Arrow units and timezones in canonical signature
JSON, including fixed-offset timezones and nested fields. Bare
`datetime` and microsecond/no-timezone declarations keep their existing
representation; `timedelta` requires an explicit unit.

Compound temporal signatures require the Lance Namespace decoder in
https://github.com/lance-format/lance/pull/11059 and corresponding
server validation support. Older readers reject these signatures; this
change does not upgrade server support or dependency pins.

Validation: 128 Python signature tests and 30 documentation tests passed
(one skipped), using the changed Python source with an existing native
extension. No Rust rebuild or end-to-end Function registration/refresh
was performed.
2026-11-14 21:49:45 +08:00
Xuanwo 12b95b59ab feat(function): support literal inputs in column declarations (#5021)
Function-column declarations now preserve constants such as
`fn(video=col("mp4"), num_frames=lit(16))` without changing its Function
version. Constants travel in the application inputs; column bindings,
request input schemas, and dependency tracking contain only columns.
Changing a constant uses a new column binding rather than rebaking the
Function.

Persisted bindings with literals use metadata version 2, which older
readers reject. Column-only bindings retain version 1. Python preserves
scalar literal identity, including null and unsigned integers, and
rejects composed expressions or values outside the canonical literal
domain.

This SDK change requires matching server declaration and execution
support, after a dedicated SDK dependency update. Native validation
passed 48 Rust metadata tests, four request-transport cases and 57
rebuilt Python tests on `a42d680`; 32 pre-existing experimental tests
remained ignored. The test-assertion correction passed its exact
regression and strict Clippy on `53ff80a2`. On `993fae7d`, the three
remote metadata compatibility regressions, native Python public-export
test and strict Clippy passed. These review corrections use v3 for
unknown-version rejection, cover existing v2 literal bindings through
remote mutation guards, and export `LiteralInputBinding`. The unchanged
native implementation and verified extension are reused. Matching server
execution and original-workload acceptance are not established by this
PR.

Refs ENT-3667.
2026-11-14 20:59:52 +08:00
Xuanwo f6f8119ba4 feat: expose structured Function refresh options in SDKs (#5047)
## Problem

ENT-3267 still lacks typed refresh controls for `supervisor_resources`
and `planning_workers` in Python, Rust, and Node, and Node has no
`retry_policy` refresh option. Callers must drop to SQL for those
controls.

## Change

- Add canonical Rust `FunctionRefreshPodResources` and
`FunctionRefreshPlanningConfig`, and serialize them through
`RefreshColumnOptions`.
- Expose both options in the Python sync/async wrappers and PyO3
binding.
- Expose both options in the Node NAPI object and add a Node
`retryPolicy` object backed by the existing Rust `FunctionRetryPolicy`.
- Omitted options stay omitted; branch targeting continues through the
table handle. No server, dependency, or pin changes are included.

Client-side checks only reject non-positive structural values.
Deployment resource limits and column-kind validation remain
server-side.

## Validation

`cargo fmt --all -- --check`, `python3 -m py_compile`, and `git diff
--check` pass. No native build was run locally; CI compiles and tests
the change. Generated JS API pages are unchanged.
2026-11-14 20:59:44 +08:00
Xuanwo e5a5b67212 feat: support nullable scalar Function declarations (#5030)
Function declarations currently reject nullable scalar returns. Accept
`Optional[T]`, `T | None`, and explicit nullable Arrow fields,
preserving `nullable: true` in registration signatures, JSON round
trips, and remote `add_columns` requests.

The Rust planner accepts nullable scalar outputs, and binding readback
recognizes the server-owned assignment marker for a single nullable
scalar result. Clients do not allocate that marker. Non-nullable whole
results, whole structs, and Blob v2 retain their existing assignment
restrictions.

Validated with 127 Python tests, 30 documentation tests (1 skipped), and
seven focused Rust tests covering planning, binding readback, rejection
boundaries, and the remote request body. The final condition-collapse
change addresses strict Clippy; its delta check and CI are pending.
Python tests use a source overlay on a retained native extension, and
Rust remote coverage uses HTTP mocks.

Server support and end-to-end validation remain separate. This PR does
not claim that server publication distinguishes computed NULL values
from pending rows.
2026-11-14 20:59:20 +08:00
Xuanwo 005b33fa8b fix: preserve terminal server identity in failed job waits (#5024)
A failed remote Function job loses its persisted build identity when
converted into the exception from `job.wait()`. Append
`result.terminal_server` to that error using the existing server-build
format, preserving its message, row location, traceback and job ID. Do
not substitute the responding query node's version for the writer of the
saved terminal result; missing or malformed historical identity leaves
the error unchanged.

The reported failure was independently reproduced against a deployed
server with a real Function job. The new exact Rust regression
reproduces the missing terminal identity on the unpatched code. At
`59c1f3e`, all 10 remote Job Rust tests and five tests using the
matching native Python extension passed, including sync/async waits with
known or unknown terminal identity. These transport tests use mocked
HTTP responses; the original deployed Function job remains with
independent acceptance for recheck after delivery. This change does not
establish whole-ticket acceptance.
2026-11-14 20:59:10 +08:00
Lance Release 5de76bd8ed Bump version: 0.42.0-beta.12 → 0.42.0-beta.13 2026-11-14 12:33:27 +00:00
Yang Cen e1a10eaa66 chore(deps): upgrade Lance to v15.0.0-beta.21 (#5046)
Upgrade the Rust Lance family from beta.18 to the published
v15.0.0-beta.21 tag, and align Python test wheels and Java artifacts.
This includes the explicit-empty-segment FTS fix in
lance-format/lance#11063 needed by lancedb/sophon#10314. Enable Windows
Git long paths so Cargo can check out Lance fixtures in the Rust, NPM
and Python wheel builds.

Beta.21 also contains lance-format/lance#9858: new NGram, RTree and
MinHash LSH indexes use row addresses. Legacy row-ID segments remain
readable, but index maintenance on stable-row-ID tables can require a
rebuild. The NGram documentation now explains the consolidation
requirement; NGram/RTree tables without stable row IDs retain their
earlier format. The Python lock revision and unrelated dependency
entries are unchanged.

Validation:
- The official Lance release and internal mirror match
80bc45a195279befdec42853dac8d15a37e0504d. The seven official Fury wheels
and Maven POM/JAR were verified.
- Exact executable/dependency state 7846a15e completed the full Rust,
Python, Node, Java and CLI CI, including platform builds and package
publication dry runs. Xuanwo approved that head; Gatekeeper recommended
approve with a non-blocking qualification risk.
- Follow-up a289cb2c only adds the requested seven-line NGram upgrade
note. Static whitespace checks passed; new-head CI is running. No local
correctness tests or test builds were run by this task owner.
- Broader IVF/encoding/Blob/cache workload performance and the beta.21
Sophon composition remain separately qualified. Earlier Blob
cancellation and DynamoDB failures on #4967 are retained in their
original evidence; this PR does not count those historical failures as
fixed.

Current-head CI, full feedback and required review still gate merge. SDK
publication is a separate action. Sophon adoption follows a published
SDK release and mirror through its own dependency PR; its distributed
performance acceptance remains in #10314.
2026-11-14 20:22:13 +08:00
Xuanwo 97e62dc11b feat(remote): expose canonical job statistics reports (#5016)
Expose `Job::stats()` in Rust and synchronous/asynchronous Python jobs.
It reads `POST /v1/jobs/stats` and returns the canonical server-rendered
report verbatim, or `None` for a running job or one without a readable
durable profile. It does not wait for completion or recompute
measurements.

On `cc25005525671756b5cf5ebd96492b792e9aa1cb`, all three Rust transport
tests and six Python tests passed against the matching rebuilt native
extension; strict SDK Clippy also passed. These tests cover optional
reports, exact text preservation, fresh reads, errors, and the HTTP
boundary with a controlled server. Against the actual Phalanx HTTP route
from [Sophon #10247](https://github.com/lancedb/sophon/pull/10247)
(`5be3236`), the same native extension also completed two sync and two
async reads after durable registry archival, preserving the exact
canonical report. This is an HTTP/registry integration fixture, not a
Function workload or shared-cluster acceptance. The private runner's
JSON field-order collector failed after the successful probe; all raw
evidence was retained and hash-verified, and the original receipt was
recovered with the corrected parser without rerunning the probe.

The follow-up `ad36f66` only corrects the synchronous Python
API-reference link to the asynchronous method. Ruff checks and
formatting passed; runtime evidence above remains applicable. The actual
Phalanx/native-SDK probe covers archived reports; running-job null
availability is covered by separate backend and SDK transport tests, not
that end-to-end probe.

Part of ENT-3501.
2026-11-13 23:41:33 -08:00
Jack Ye f6311ae7e9 feat: support exist_ok for index creation (#4993) 2026-11-13 19:21:50 -08:00
Jack Ye 81d04b8c36 feat(python): support exist_ok for materialized views (#4991) 2026-11-13 19:21:29 -08:00
Jack Ye 4db0d5c7f5 feat(python): support exist_ok when cloning tables (#4990)
Adds `exist_ok=False` to Python table cloning and `.exist_ok(true)` to
the Rust clone builder. An existing target is returned with its data and
version history unchanged; strict duplicate errors remain the default.

```python
snapshot = db.clone_table("snapshot", source_uri, exist_ok=True)
```

Embedded cloning skips source reads for an existing target and handles
typed creation conflicts. Remote cloning forwards the option to the
server. Includes sync/async preservation and remote request coverage,
plus SQL syntax documentation for `CREATE TABLE IF NOT EXISTS target
CLONE source`. Requires a server supporting the option for remote
clones.

BREAKING CHANGE: Rust `CloneTableRequest` struct literals must
initialize the new `exist_ok` field; `CloneTableRequest::new` remains
compatible.

No local builds or tests were run, as requested.
2026-11-14 10:48:26 +08:00
Jack Ye a0eb6b4b71 feat(python): support exist_ok for storage integrations (#4986)
Adds `exist_ok=False` to sync and async Python storage integration
creation. `exist_ok=True` sends `CREATE STORAGE INTEGRATION IF NOT
EXISTS` and returns the server's stored configuration, preserving the
prefix, credentials and external ID.

```python
integration = db.create_storage_integration(
    "raw", storage_provider="S3", storage_prefix="s3://customer-bucket/raw/",
    aws_role_arn="arn:aws:iam::123456789012:role/access", exist_ok=True,
)
```

The non-exhaustive Rust request also supports `.exist_ok(true)` for
every provider. Requires a SQL server supporting this syntax. Includes
protocol and provider SQL coverage. No local builds or tests were run,
as requested.
2026-11-14 10:37:08 +08:00
Jack Ye 2c3c30dbdf feat(python): support exist_ok for Function creation (#4992)
Adds `exist_ok=False` to Python Function creation and job submission.
`exist_ok=True` sends `CREATE FUNCTION IF NOT EXISTS` and returns the
stored immutable version, preserving its code, credentials and execution
settings. The default retains the current version-creation behavior.

```python
version = db.create_function(analyze_caption, exist_ok=True)
```

The option also exists on the Rust registration request and is kept out
of the canonical Function definition payload. Includes
sync/async/background protocol and stored-version coverage. Requires
matching SQL server support.

BREAKING CHANGE: Rust `FunctionRegistrationRequest` struct literals must
initialize `exist_ok`; legacy serialized requests default to false.

No local builds or tests were run, as requested.
2026-11-14 10:35:54 +08:00
Jack Ye b061c3627c feat(python): support exist_ok when creating views (#4984)
Add `exist_ok=False` to synchronous and asynchronous Python view
creation. Passing `True` sends `CREATE VIEW IF NOT EXISTS` and returns
the existing view's stored query and schema unchanged; the default
continues to fail on an existing name.

```python
view = db.create_view("recent_orders", query, exist_ok=True)
view = await async_db.create_view("recent_orders", query, exist_ok=True)
```

Requires a SQL server supporting `CREATE VIEW IF NOT EXISTS`. Rust gains
an awaitable builder with `.exist_ok(true)`. Existing
`.create_view(...).await` calls continue to work; callers requiring
`Future` must convert the builder, and custom `Database` implementations
must accept the new `exist_ok` argument. Includes Python round-trip
coverage and Rust coverage for direct/deferred results and metadata
preservation.
2026-11-14 10:35:52 +08:00
Jack Ye bab22a3fbd feat(python): bundle Function package resources (#4987)
A package shipped through `code=` currently loses its non-Python files.
This breaks vendored packages that read JSON configuration, manifests,
prompts, or binary resources after deployment. Include regular package
resources in a separate base64-encoded `python_bundle.resources` map,
preserving exact bytes and relative paths. Both source and resources
contribute to the artifact digest. UTF-8 Python source bytes retain
their original line endings as well, so manifests that hash vendor code
continue to match.

For example, `@function(code=[my_package])` can use
`importlib.resources.files("my_package").joinpath("models.json").read_text()`
on the worker. Hidden files and caches are excluded; symlinks and unsafe
paths warn instead of following files outside the package. Source-only
bundles retain their existing wire form. Resource bundles require
matching server support.

Adds an isolated-import regression with a vendored subpackage, JSON
metadata, binary data, SHA256 verification, preserved CRLF bytes, and
resource-only version changes. Local builds and tests are disabled by
request. Ruff formatting and linting run locally; CI runs the tests.
2026-11-13 16:21:58 +08:00
Jack Ye 9302287bed fix: accept defining queries in Python materialized views (#4980)
Python materialized-view creation now accepts a complete defining SQL
query across sync, async, remote, and namespace connections, including
creation jobs:

```python
view = db.create_materialized_view(
    "events_by_kind",
    query="SELECT kind, count(*) AS n FROM events GROUP BY kind",
)
```

`query` is mutually exclusive with `source`, `select`, `where`, and
`limit`. Existing source-based calls remain supported.

Adds Rust `Connection::create_materialized_view_from_query`: embedded
connections plan through `MaterializedViewDefinition::from_sql`, while
remote connections submit the full query for server-side parsing.
Includes regression coverage for grouping, UNNEST, query forwarding,
creation jobs, and argument validation.
2026-11-12 14:57:00 -08:00
Xuanwo cababaecf0 feat(remote): discover server builds and retain error provenance (#4999)
Remote callers can use `db.server_info()` in Rust and Python, including
synchronous and asynchronous Python connections, to identify the
responding server without submitting a query or Job. Typed results
describe the query-node build, component versions, Function protocol and
configured images. Older servers without the endpoint return an explicit
unsupported error.

REST and Flight errors retain the reported build in diagnostic text and
Rust `RemoteErrorMetadata.server`. Typed table/job-not-found errors
preserve their HTTP source and classification, including the
schema-cache path; masked REST errors can obtain provenance from the
response header. Unknown build information stays absent.

The matching native Python SDK was exercised against a real localhost
server (`8829b074` client / `43920fa` server): sync/async discovery,
both SQL version forms and actual REST/Flight error provenance passed.
The product files match this branch; private validation drivers are
excluded. Focused coverage includes typed 404 errors, masked errors, and
the public schema-cache regression. Deployment and published-package
acceptance remain separate.

Server counterpart: https://github.com/lancedb/sophon/pull/10234.

Related: ENT-3565.
2026-11-13 06:22:35 +08:00
Xuanwo 6d24446a30 feat: add negotiated batched blob range reads across SDKs (#4994)
Existing Rust and Python blob range calls must keep working when only
the SDK is upgraded. Add negotiated batched range reads across Rust,
Python sync/async and TypeScript, preserving ordered NULLs, duplicates
and branch/version reads with splitting and reassembly at 1,024 ranges
or 64 MiB per request.

Only a successful describe response advertising `x-lancedb-blob-ranges:
1` selects the batched endpoint. Missing or unknown capability values
select the existing descriptor-query plus BlobFile Range GET route
before the operation. Endpoint failures never switch protocols. HTTP 400
preserves the server message and request ID as Rust `InvalidInput` /
Python `ValueError`; other errors retain their existing semantics.

The operation binds both numeric version and immutable snapshot identity
from its producing query or first range response. Every later partition
and oversized-range piece sends that original identity and requires an
equal response identity before decoding. Same-name table recreation
therefore fails explicitly instead of returning mixed old/new payloads;
no partial array is returned on later failure.

Targets `main`. The read-buffering prerequisite #4979, safe-write
prerequisite #4977 and deferred-query identity prerequisite #4973 are
merged into main; #5000 also supplies the merged SQL timeout-test
prerequisite. Server support is [Sophon
#10219](https://github.com/lancedb/sophon/pull/10219), including #10211.

Exact SDK `1497b508324662180086a4d1e0a6eb9e0b3970db` and server
`f39712c2c150b220eb831e00f6663c9eef32dce5` passed a real local SDK →
REST → PE experiment: one Rust harness test, 14 assertion groups, 38
REST range requests and 34 PE RPCs. All four sync/async × warm/cold
table-recreation cases failed explicitly before the second descriptor
lookup or PE dispatch, with no mixed payload or partial result. The
other groups retained 1,025-range success, 70 MiB reassembly,
branch/version reads, overwrite snapshot consistency, error behavior and
TypeScript buffer ownership. Bindings were rebuilt and their loaded
hashes verified. This is local interoperability evidence, not Cloud,
mixed-fleet or deployed header-forwarding acceptance.

The server dependency upgrade, External/cache/URI-policy acceptance and
validated rollout remain required. Enforcing PE support must precede QN
and SDK enablement; identity does not retain versions or freeze External
objects. This PR does not claim deployed support or complete ENT-3100/GA
acceptance.

Published documentation retains the Early access badge and released
Enterprise instructions, including explicit fetches for copying. The new
Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example,
future streaming restrictions and GA wording remain source-only MDX
drafts until release. The correction distinguishing released Enterprise
SQL bytes from SDK descriptors remains visible. Documentation validation
checked comment boundaries, visible instructions, Python example syntax
and whitespace; the site was not built. The main integration keeps these
release gates unchanged.

Main integration at `0491d9bdcc65ce5d52ea93e1083c89e951ea4f9d` normally
merges main `9b13e8f72530c8b1950c09685f6920711154a539`, including its
Lance beta.18 dependency. Focused current-source validation passed: 493
remote-table/query/Blob/IPC/write-progress Rust tests, 23 shared
create/write-body tests, 59 native Blob integration tests and 7 public
HTTP range tests. Rebuilt Python passed 468 selected tests (one optional
PyAV test skipped), and rebuilt TypeScript passed 194 selected tests
across four suites; lint/format, TypeScript compilation and API
documentation generation passed. Shared tests include actual HTTP
multipart failures, acknowledgment boundaries, dictionary compaction and
native Blob compaction. The 598 unrelated Jest cases excluded by the
focused shared-suite selector were not run.

The retained 14-group joint REST/PE experiment used the earlier exact
candidates stated above. Operation-identity/range/IPC code is unchanged,
but dependencies and shared write interfaces changed with main; the new
local checks validate the SDK combination and do not constitute a fresh
final-head SDK/server joint, Cloud or rollout run. Early access and
every remaining server/External/cache/policy/dependency/rollout
condition remain in force.

CI snapshot on this head (2026-11-12 18:30 UTC): `build-no-lock` failed
in the standalone `ci/tests/test_build_no_lock.py` fixture at the
retained-rlib assertion, before the actual SDK no-lock build. The
fixture, helper and Rust workflow are byte-identical to main; the
fixture does not load SDK source. The underlying CI/environment cause is
not diagnosed here. Other jobs remained pending/running at the snapshot,
so CI is not reported as passing. Failure:
https://github.com/lancedb/lancedb/actions/runs/43280277756/job/131399418284.
2026-11-13 04:03:04 +08:00
Xuanwo 9b13e8f725 feat(python): materialize remote blobs at the query snapshot (#4973)
Python remote queries materialize Blob v2 payloads with
`blob_mode="bytes"` through synchronous and asynchronous `to_arrow`,
`to_list`, `to_pandas`, and `to_batches`. Deferred reads bind to the
actual producing dataset without changing the caller's live table
handle, including concurrent writes and same-name DROP/CREATE.

Trust only the authoritative `x-lancedb-query-version` plus
`x-lancedb-query-snapshot`. A numeric version alone cannot distinguish a
recreated table; the ordinary live watermark and user-supplied Arrow
metadata cannot establish provenance. An independent immutable handle
carries the original identity through every batch, byte-limit split, and
large-Blob Range fallback, and requires a verified response echo before
accepting payloads. Aliases, NULL/empty results, rerankers, and hybrid
legs retain consistent provenance. Missing capability, mixed identities,
or a serving node ignoring the precondition fails explicitly; ordinary
descriptor queries remain available on older servers.

The source docs distinguish Enterprise SQL payload bytes from SDK
descriptors. Server dependency:
https://github.com/lancedb/sophon/pull/10211.

Validation uses retained, independently audited local evidence: 724
Python cases passed (one optional PyAV skip), 67 focused SDK Rust cases
passed, 36 server checks passed, and 15 actual REST/SDK scenarios passed
across retained runs. These cover sync/async and Arrow file/stream
recreation, concurrent overwrite, deletion of a selected version, legacy
capabilities, forged IPC metadata, WAL exclusion, and actual mid-batch
recreation. The original aggregate experiment exited 101 because its
extra harness assumed a Python batch size would split the native result;
its 36 server checks and 13 completed scenarios remain valid. The
corrected native-batch experiment passed separately for sync and async.
No aggregate all-green claim is made.

The public batched-range operation has separate cross-partition identity
integration in https://github.com/lancedb/sophon/pull/10219 and
https://github.com/lancedb/lancedb/pull/4994. This change does not claim
to fix that loop. Realtime/MemWAL remains outside Blob v2 GA, without
disabling existing writes or compaction. No dependency pins changed.

Merge/rollout hold remains: upgrade every serving node before enabling
the new SDK capability, verify proxy forwarding of both authoritative
headers and the verification echo, and preserve queried versions for
later reads. Missing or inconsistent capability fails explicitly;
identity does not retain versions or authenticate storage paths. Actual
deployed proxy/Cloud/PE behavior and production retention remain
unverified. Old SDKs do not gain incarnation protection from a
server-only upgrade.

Published documentation retains the Early access badge and released
Enterprise instructions, including explicit fetches for copying. The new
Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example,
future streaming restrictions and GA wording remain source-only MDX
drafts until release. The correction distinguishing released Enterprise
SQL bytes from SDK descriptors remains visible. Documentation validation
checked comment boundaries, visible instructions, Python example syntax
and whitespace; the site was not built. This documentation-only revision
reuses the implementation evidence above.
2026-11-13 02:05:39 +08:00
Xuanwo 725eee8a47 fix: prevent unsafe write retries and bound Blob uploads (#4977)
Ambiguous gateway failures can occur after a write commits, so replaying
the operation can append duplicate rows. Non-idempotent writes now retry
only known precommit failures. Multipart retries reuse the same upload
ID, part ID, and encoded bytes; ambiguous completion returns an
unknown-outcome error without abort or replay. A definite 409/429
completion rejection triggers best-effort cleanup using the call's retry
budget.

Replayable parts and the shared byte-based partition target default to
64 MiB. Dictionary-bearing batches are compacted before size estimation
and again when actual compressed IPC framing requires further splitting:
keys are remapped without changing their types or schema, view backing
buffers are collected, and ListView/LargeListView child ranges are
compacted as an interval union so overlapping views retain sharing.
Parts are prepared on demand; a single row and its schema can still
exceed the target. Multipart progress starts at zero and counts each
acknowledged IPC body once. Rust write builders and Python/TypeScript
`add` expose per-call retry overrides. Atomic `create_table` pre-encodes
to a temporary file and rejects compressed IPC over 256 MiB before
sending any request; the schema-first import guidance now includes
Python, TypeScript, and Rust examples. The shared partition target also
affects local automatic partitioning.

At `cb2a79473bd1b51b950c9fa3e9f247f929c8ac3b`, 70 targeted Rust cases
pass on source verified against the committed four-file patch. With a 16
KiB target and 128 unique 1 KiB dictionary values, the original ordinary
dictionary reproducer drops from 128 requests / 16,942,080 bytes to 16 /
143,488; ListView and LargeListView drop from 128 / 17,056,768 to 16 /
149,632. Encoded HTTP bodies are decoded to check values, nulls, slices,
metadata, changing dictionaries, and overlapping views; additional cases
cover framing-driven splitting, oversized rows, stable retry bytes/IDs,
acknowledged progress, and atomic create. Real HTTP fault tests commit
Blob payloads to a local Lance table and verify byte equality, no
duplicate append, and zero abort/replay after ambiguous completion.

The earlier 811 remote and 7 progress cases at
`f36d217110a25f6b501c0255df9aaa60e3c8000a`, Linux Python remote
database/options results (437 cases plus 8 fixture consumers), Node
retry-binding evidence, and Rust/TypeScript documentation example checks
remain historical validation. Python/Node extensions were not rebuilt
for the dictionary follow-up; these results do not establish current
binding integration or fresh cross-platform/minimum-dependency CI
acceptance.

A separate real Query Node lifecycle experiment used SDK
`b8efe7d2d63647d4962789ca5fc042b59efe073d` and Sophon
`58b7895c7c287810fff4e89f13226fc5fa60d9ef`. After the actual listener
forced a connection closed, a surviving QN accepted a retry with
identical upload ID, part ID, and IPC bytes. Ordinary insert and
complete response-loss scenarios sent no replay or abort; fresh reads
observed exactly one commit and exact Blob bytes in all three scenarios.
This local experiment used a 1-second drain grace, two 32 KiB payloads,
and one distinct staged part. It validates acknowledgement-loss recovery
at those revisions, not concurrent overlapping attempts, large uploads,
physical orphan cleanup, the later dictionary encoding change, or the
original sixth-insert EOF.

The part target is not a process memory limit: source/ancestor buffers
and temporary compaction allocations may coexist. Struct/FixedSizeList
parent nulls can retain hidden child keys; this is not complete
logical-reachability GC for every Arrow combination, and retained hidden
values may affect wire size. The current local Lance 15.0.0-beta.17
rejects ListView/LargeListView persistence; the tests establish remote
client IPC encoding behavior, not native or Cloud persistence support.
Live Cloud, full workspace/Clippy, the original ordinary-insert
disconnect root cause, and Sophon's asynchronous transaction cleanup
behavior are not established. Cancelling during staging retains the
existing server-session cleanup limitation. Function reader delivery is
separate.
2026-11-13 02:05:13 +08:00
Xuanwo 10f89b3c32 perf: reduce Blob read buffering and expose read window controls (#4979)
Remote Blob reads retain and concatenate intermediate IPC buffers across
batches and HTTP requests, increasing peak memory. Decode fragmented IPC
bodies using their declared length, assemble Blob values once across
batches/requests, and collect large-object Range responses into
known-size buffers. Impossible IPC/Range allocations return errors
instead of panicking. Closing a remote Blob file releases cached bytes
immediately, including while a cross-window read is waiting for the
network.

Rust and Python expose configurable Blob read buffers. Python sync/async
`fetch_blob_files` and `open_blob_file(s)` accept `buffer_size`; the
default is 512 KiB and zero disables prefetch. Explicit ranges bypass
prefetch. New PE batch-endpoint adoption is separate.

Malformed-length and retained-cache regressions fail on the previous
implementation and pass after the correction. Real HTTP tests cover
buffer propagation, close during blocked input, batch/request
aggregation, and large-object fallback. The final extension was rebuilt
before Python validation. Non-Blob Arrow arrays may still require
alignment copies; a contiguous multi-batch result still requires its
final allocation.

## Benchmark

Peak client RSS increment in KiB, using
`python/benchmarks/bench_fetch_blobs_rss.py`, independent clients and
real loopback HTTP. Both extensions were rebuilt from clean, unchanged
source with the same toolchain/configuration; loaded artifact hashes
were checked. Baseline `42f11d13a2755e44b63876b5a5edd93901291150`, after
`0a8a76de5286949d80368f95a5c529adf422c187`.

| Equivalent fetch workload | Payload | Before | After |
| --- | ---: | ---: | ---: |
| Single response, one batch | 48 MiB | 63,812 | 63,896 |
| Multiple responses, one batch per response | 144 MiB | 423,104 |
195,628 |
| Single response, 12 rows per batch | 48 MiB | 130,520 | 71,348 |
| Multiple responses, 12 rows per batch | 144 MiB | 363,144 | 172,748 |
| One oversized row, Range fallback | 70 MiB | 182,816 | 85,548 |

These are one observation per workload and build, not a statistical
distribution or memory upper bound. The single-batch workload shows no
improvement. Cloud/Sophon deployment, credentialed S3 and long-running
stress have not been exercised.
2026-11-13 02:05:09 +08:00
Xuanwo 8939ef1f01 feat(python): warn on scalar GPU Functions (#4966)
GPU Functions that use a scalar callable currently invoke the callable
once per row. The SDK documents Arrow-vectorized callables, but silently
accepts a scalar GPU callable, which makes it easy to assume that the
runtime will batch model work automatically.

Warn during Function construction when a GPU callable does not use the
documented vectorized form. A callable is recognized as vectorized when
every input is annotated as `pyarrow.Array`, or when a single input is
annotated as `pyarrow.RecordBatch`. The warning explains the resulting
per-row invocation and shows how to opt into batch execution. CPU
Functions and the vectorized GPU forms do not emit it.

The warning is advisory and does not change registration serialization,
the artifact, the runtime adapter, or execution behavior. It is one part
of the ENT-3522 documentation-and-warning alternative; it does not claim
that a single A10G meets the original batch throughput requirement.

Validation on the designated EC2 host reused the compatible existing
native extension from `ent3563-sdk-freshness-9336cfb`; this does not
claim a new native package or GPU performance run. The final head has a
clean source tree and ran:

```bash
PYTHONPATH="$sdk/python/python" /home/ec2-user/.venvs/ent3563-0a839fa/bin/python -m pytest -q python/python/tests/test_first_class_function_slice2.py -k 'gpu_marker_uses_gpu_runtime or gpu_scalar_warning_names_the_vectorized_alternative'
/home/ec2-user/.venvs/ent3563-0a839fa/bin/ruff format --check python/python/lancedb/functions.py python/python/tests/test_first_class_function_slice2.py
/home/ec2-user/.venvs/ent3563-0a839fa/bin/ruff check python/python/lancedb/functions.py python/python/tests/test_first_class_function_slice2.py
```

The actual result was `2 passed, 123 deselected`; both Ruff commands
passed. The original 2,000-row single-A10G throughput requirement
remains a separate ENT-3522 acceptance gate.
2026-11-13 02:04:32 +08:00
Jack Ye e61e21035c fix(python): explain Function namespace import restrictions (#4982)
A plain `import ctypes.util` in a Function currently reports “dynamic
namespace access” without explaining the module restriction or a way
forward. Give blocked module imports a distinct error that names the
modules, explains the reconstructed namespace constraint, and recommends
an importable helper supplied through `code=`, `pip=`, or `conda=`.
Dynamic namespace operations also get an actionable explanation.

Document the full restrictions and the helper alternative in the
decorator and Python Functions reference. Add regression coverage for
direct imports, `from` imports, global module aliases, class Functions,
the deprecated `@udf` alias, and helpers imported normally or shipped
with `code=`. Existing import restrictions remain in place.

For example, move `ctypes.util.find_library` into
`native_diagnostics/__init__.py`, then use:

```python
import native_diagnostics
from lancedb import function


@function(code=[native_diagnostics])
def probe_library(name: str) -> str:
    return native_diagnostics.find_library(name)
```

Fixes ENT-3408.
2026-11-12 09:01:39 -08:00
Jack Ye 9e03bf97d4 feat(python): support exist_ok when creating secrets (#4978)
Add `exist_ok=False` to synchronous and asynchronous Python
`create_secret`. Passing `True` uses `CREATE SECRET IF NOT EXISTS`,
preserving an existing credential. The default continues to fail on an
existing name; rotation stays explicit through `alter_secret`.

```python
db.create_secret("api-key", value, exist_ok=True)
await async_db.create_secret("api-key", value, exist_ok=True)
```

The Rust API gains an awaitable builder with `.exist_ok(true)` while
existing `create_secret(...).await` calls keep working. Includes Python
SQL round-trip coverage and Rust coverage for direct and deferred SQL
responses.
2026-11-12 08:03:11 -08:00
Jack Ye ced4799ad5 fix(python): preserve typed results for reopened refresh jobs (#4981)
Restore typed results for reopened Function and SQL-expression refresh jobs while preserving their handles and cached descriptions.

Fixes https://linear.app/lancedb/issue/ENT-3441.
2026-11-12 09:21:58 +00:00
Jack Ye 7b2dfb9e36 fix(python): align catalog and database connection inputs (#4983)
Align Python catalog connection inputs with database connections, warn when region is ignored, and preserve remote storage settings in opened databases.
2026-11-12 01:19:05 -08:00
Jack Ye 478d1d9b01 fix: expose comparable Function runtime identity (#4975)
Expose `FunctionVersion.runtime_digest` through Rust and Python, and add
`PythonRuntimeSpec.digest` so clients can compare declared environment
inputs before registering a Function. Dependency pins, environment
variables, Python version, and GPU settings affect this digest; package
ordering does not.

Clarify that matching `source_digest` alone is insufficient for reuse.
Older servers/images leave `runtime_digest` unset, which cannot
establish a match. Both digests and the other registration inputs must
match before reusing a version.

```python
request = definition.registration_request
same_source = version.source_digest == request.artifact.digest
same_runtime = version.runtime_digest == request.runtime.digest
```

Add shared runtime digest fixtures and coverage for environment-only
changes and catalog decoding. Local builds and tests were skipped as
requested.
2026-11-12 00:30:19 -08:00
Lance Release 01e9ea44aa Bump version: 0.42.0-beta.11 → 0.42.0-beta.12 2026-11-12 08:13:04 +00:00
Yang Cen 5a38066234 chore(deps): upgrade Lance to v15.0.0-beta.18 (#4965)
Upgrade the Rust Lance dependency, Java lance-core, and Python test
dependency from beta.17 to the officially published
[v15.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.18)
(`079fd77683a7acf056b0912f6ecda938e39ee120`). This provides a compatible
SDK for the newer Lance source already adopted downstream. Rust registry
versions and checksums are unchanged; the Python lock uses the six
published beta.18 wheels.

### Compatibility

The adopted release includes [Lance
#10863](https://github.com/lance-format/lance/pull/10863):
`FilteredReadOptionsProto.has_fragment_ids` distinguishes an explicit
empty fragment selection. Older executors ignore this field, so upgrade
executors before relying on the corrected empty-selection behavior. Rust
callers constructing this generated struct need the new field. [Lance
#10112](https://github.com/lance-format/lance/pull/10112) also replaces
the internal `hamming_distance_batch` API with
`hamming_distance_batch_into`.

The Linux Python matrix timeout is raised from 30 to 60 minutes: the
Python 3.13 cold native build and both test passes completed, but the
job hit its limit during cleanup. Build profiles, test commands, and
coverage are unchanged.

### Validation

- Three native FRI/ANN regressions passed: deferred-compaction refined
RQ, deferred index remapping, and approximate-query builder.
- `cargo clippy --quiet --locked --features remote --tests --examples --
-D warnings` passed with pinned Rust 1.97.0.
- Locked metadata, formatting, Python Makefile-scope Ruff checks, and
`uv lock --check --offline` passed.
- YAML, whitespace, end-of-file, and typo checks passed for the
timeout-only workflow change.

### Publication and CI

Java beta.18 is now published. After [Lance
#10968](https://github.com/lance-format/lance/pull/10968) merged, the
[normal publication
workflow](https://github.com/lance-format/lance/actions/runs/43146352505)
built all three JNI targets and published successfully; the Maven jar is
available. The upstream Python publication run was cancelled after its
Windows ARM job hit the one-hour timeout; six other wheels were
published and are the actual artifacts recorded in this lock.
Cross-platform SDK CI is still required. No performance claim is made by
this dependency update.
2026-11-12 16:09:13 +08:00
Jack Ye 1dee9b006c fix(python): reject unsupported Function callables early (#4972)
Reject synchronous and asynchronous generator Functions, including class
`__call__` methods, during decoration so invalid outputs never reach
remote execution. Give actionable errors for lambdas and decorated
wrappers such as `functools.lru_cache`. Defaulted callable inputs remain
unsupported; the error now names the parameter and explains how to use
class initialization fields.

```python
@function
def double(value: float) -> float:
    yield value * 2
# TypeError: generator Functions are not supported; return a value instead of
# yielding, or return a list for multiple values
```

The deprecated `@udf` alias shares the validation. Adds regression
coverage for both decorators, class methods, explicit schemas, and the
supported list-return alternative.

Fixes https://linear.app/lancedb/issue/ENT-3539
2026-11-11 23:32:57 -08:00
Jack Ye e622254108 fix(python): use direct lookups for table existence checks (#4974)
Table existence checks should not depend on listing every table or
searching only the first 100 names. Use a direct lookup for `name in db`
and implement `table_exists(table_id)` on synchronous local connections,
matching the remote and asynchronous APIs. Native namespace connections
use their initialized Rust connection for existence checks; custom
namespace clients retain their existing callback. Missing tables return
`False`; other lookup errors propagate.

```python
if "events" in db:
    db.drop_table("events")

exists = db.table_exists(["analytics", "events"])
all_names = list(db.list_tables())
```

Clarify that `.tables` contains one page and update deprecation guidance
with safe replacements. Add regression coverage for lookups beyond the
first page, nested namespaces, native and custom namespace clients,
missing tables, permission errors, and lookups that do not list tables.
Include an unpublished draft for the table-management guide until the
new SDK support ships.
2026-11-11 23:32:19 -08:00
Jack Ye ee231aad06 fix(python): validate function package requirements before registration (#4971)
Function dependency typos currently reach the environment build before
failing. Validate pip requirements with
`packaging.requirements.Requirement` and conda requirements with
Rattler's MatchSpec parser when constructing a function definition,
before source packaging or registration. Errors identify the original
entry and its index.

```python
@lancedb.function(pip=["numpy", "torch>=2.0,,<3"])
def score(value: float) -> float:
    return value
# ValueError: invalid pip[1] requirement 'torch>=2.0,,<3': ...
```

Preserves pip extras, markers, named references, bare archive/git URLs,
and conda channel/build specifications. The existing native extension
exposes the conda parser, so no additional Python package is required.
Adds regression coverage for invalid and supported forms, including the
deprecated `udf` alias.
2026-11-11 22:56:36 -08:00
Xuanwo 9a9b681aaf docs(python): clarify refresh result row counts (#4963)
## Summary

`RefreshColumnResult` exposed the canonical refresh counters without
documenting their unit. This change describes them as logical rows
planned for one refresh: `rows_assigned` counts successful rows written,
`rows_failed` counts rows whose computation failed, and `rows_remaining`
counts planned rows neither assigned nor terminally skipped. Rows
already covered by `rows_failed` are not counted again as remaining.
Retry attempts and user-code calls do not change that logical-row scope.
The change also documents the source and published versions and the
deprecated `rows_filled` alias.

The existing skip-policy example is now explicit: a refresh over 2,001
logical rows with 1,891 assigned, 110 failed, and 0 remaining uses 2,001
as the failure-rate denominator.

## Behavior

Documentation only. No field names, Python or native behavior, REST wire
format, dependencies, or pins change.

## Validation

- `git diff --check`
- `uvx ruff format --check`
- `uvx ruff check`
- Imported this Python source with a compatible prebuilt ABI3 extension
and asserted the public class and property documentation, the affected
source version, and the native extension identity.
2026-11-12 00:49:00 +08:00
Lance Release 08d3f448af Bump version: 0.42.0-beta.10 → 0.42.0-beta.11 2026-11-11 15:38:36 +00:00
Jack Ye ea1cf17f89 feat(python): export recorded Function job timelines (#4946)
Function jobs can now export their recorded execution timeline with
`job.export_timeline(path)` or `await job.export_timeline(path)`. The
SDK fetches a fresh Chrome Trace Event recording that opens in Perfetto,
including worker split spans and timestamped job phases. Batch counts
and latency statistics remain details rather than invented batch
intervals.

```python
job = db.open_job(job_id)
job.export_timeline("refresh.trace.json")

async_job = await async_db.open_job(job_id)
await async_job.export_timeline("refresh.trace.json")
```

The export does not wait for completion or cancel the job. Running jobs
return partial recordings. Local operations and SQL query handles remain
unsupported, and servers must support the timeline endpoint. The
destination is touched only after a successful fetch; network and
filesystem errors propagate. Rust freshness wrappers forward the
operation without changing the submitting table's read baseline.
2026-11-11 07:35:43 -08:00
Yang Cen c1fee36045 chore(deps): upgrade Lance to v15.0.0-beta.17 (#4943)
Update the Rust workspace, Java dependency, and Python test dependency
to [Lance
v15.0.0-beta.17](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.17).
This includes the FRI cache and vector-index changes required by current
Sophon and retains the RaBitQ prewarm fixes. Lockfiles use the published
source and test wheels without unrelated registry version bumps.

The dependency dry run also exposed CI resource limits. Use the existing
larger x64 runner for GNU Node and compatibility-wheel builds; allow 60
minutes for Mac Python tests, 120 minutes for the compatibility wheel,
and 150 minutes for the Mac release wheel. Disable the CLI toolchain
action's automatic target cache so only the subsequent downloads-only
cache runs. Release optimization, the compatibility wheel's manylinux
container and x86-64-v2 baseline, and test coverage stay unchanged.

Validation:

- Three native SDK regressions pass: refined RQ queries after deferred
compaction, deferred index remapping, and ANN approximation planning.
- Pinned Rust 1.97.0: `cargo clippy --quiet --locked --features remote
--tests --examples -- -D warnings` passes.
- Locked offline Rust metadata, Python lock validation, Ruff
lint/format, and diff checks pass. The Python lock contains six
published beta.17 wheels.
- Repository YAML, whitespace and typo checks pass for the workflow
changes. Product-source checks are reused for the subsequent CI-only
commits.
- At head `07033dc6`, all 65 applicable pre-review CI checks pass,
including the compatibility wheel, Mac CLI, GNU Node, Mac Python tests,
release wheels, and Java checks. Six checks are skipped by tag/main-only
conditions or draft-only app behavior; the automated review and security
checks run after the Ready transition.

The upstream Java publisher succeeded and its beta.17 Maven artifact is
available. Six upstream Python wheel builds/uploads succeeded; its
Windows ARM job was cancelled. These checks do not establish Sophon
performance or aggregate-memory acceptance.
2026-11-11 07:26:52 -08:00
Xuanwo a356950bce feat: expose queued Function refresh status (#4959)
## Problem

The existing Function `column_status` surface cannot decode the Agent
scheduler's `queued` state or the `failure_threshold_reached` and
`binding_revision_missing` blocked reasons. It also preserves only the
latest refresh job, so a later cancellation can hide an earlier actual
failure.

## Change

- Adds the `queued`, `failure_threshold_reached`, and
`binding_revision_missing` wire states in Rust, Python, and Node. The
wire change is additive, while Rust callers must update exhaustive enum
matches and public struct literals.
- Adds optional `last_failure_job` alongside `last_refresh_job`. The
latest cancellation and the most recent actual failure retain separate
meanings.
- Reuses the canonical Rust `JobFailure` and existing Python/Node
`JobFailureInfo` models for nested job failure details instead of
introducing a parallel status error model.
- Maps the existing public `sql_job_id` field through Node's
`JobFailureInfo`.
- Keeps `column_status`, `column_statuses`, and `wait_fresh` as the
existing API surfaces. There is no new endpoint, row-error list, durable
ledger, or unknown-enum fallback.
- Documents `queued` versus `running` and the separate latest-attempt
and latest-failure jobs in the existing hidden freshness release draft.

The new status values are additive but cross-service: `queued`, the two
new blocked reasons, and `last_failure_job` require matching Agent/QN
and SDK deployments before clients can rely on them. This SDK follow-up
does not change the normal `7cc3904a` dependency pin or claim whole
ENT-3487 completion.

## Validation

The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`,
based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The
final product docs commit changes generated TypeDoc only.

- Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust
blobs are unchanged from their tested `788ef85b` source through
`c9bac25e`.
- Python: the PyO3 extension built successfully, and the selected native
PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The
test requests pinned version 42.
- Node: the addon and TypeScript builds passed, Biome passed, and the
selected Jest transport test passed 1/0 with 57 filtered tests. Addon
SHA-256:
`c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`.
- Python extension SHA-256:
`36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`.
- TypeDoc was generated from the native `native.d.ts` produced by that
Node build. A second generation produced no remaining diff.

The public diff contains only product, test, and generated API
documentation changes. Native runner scripts and preserved
compile-failure logs remain private validation evidence.

Agent/QN production mapping, mixed-version rollout, and the remaining
ENT-3487 server-side work are tracked separately.
2026-11-11 21:14:30 +08:00
Xuanwo 7cc3904a7e feat: expose exact Function column freshness (#4958)
## Problem

Function-backed columns need an exact `fresh`, `stale`,
`never_computed`, and
`failed` partition for the table version a caller is reading. The
existing
error listing is capped, and composing it with job and schema listings
cannot
prove which rows currently failed.

## Behavior

- Adds a strict freshness reader beside the existing permissive planner.
It
  distinguishes an absent certificate from legacy evidence that omits
  failures, and from a current certificate carrying failure coverage.
- Extends the immutable freshness sidecar with per-fragment known and
failed
  row bitmaps. Partial or `WHERE` publications cover only observed rows;
  untouched rows stay unknown. Deleted rows do not block classification.
- Adds failure-aware whole-fragment and row-repair writers. A rejected
older
attempt cannot overwrite failure evidence already recorded for a newer
input
  signature.
- Adds `column_status`, `column_statuses`, and `wait_fresh` to Python
and the
  corresponding Node methods. The client calls the server's exact status
endpoint, validates the four-way partition and one-version guarantee,
and
preserves the complete Function version, last-job, and automatic-refresh
  metadata.
- Adds an MDX-commented draft usage section to the Enterprise
computed-column
refresh guide, covering Python, Rust, and TypeScript and remaining
hidden
  until the server integration ships.
- Sends `column`, `branch`, and exact `version` selectors to the status
route.
A pinned handle rejects a nonempty response from any other table
version;
  latest handles keep the existing read-freshness headers.
- Retries `wait_fresh` only for HTTP 503 with
`function_column_status_finalization_pending`. Existing failed rows
remain
  visible and do not keep the wait pending; other errors propagate.
- Models the shared scheduler projection, including `up_to_date`, real
`blocked` states, their reason, failure streak, and effective cooldown.

## Validation

This is the SDK side of ENT-3563. Sophon's failure-coverage publication
and
status endpoint are a separate integration; deployed end-to-end and
whole-ticket
acceptance remain pending.

- Rust: 9 exact freshness/status tests passed, plus 1 exact
latest-handle
  freshness-header test. The pinned transport test was extended with a
wrong-version response and rerun at `ebd6649a` (1 passed). Coverage
includes untouched partial rows,
  selected-row failure clearing, cross-generation evidence, typed and
non-retried errors, `up_to_date`, blocked scheduler fields, and
exact-version
  requests.
- Python: 2 native PyO3 remote-transport tests passed together,
including
typed retry; one local facade test reaches the existing unsupported
boundary.
- Node: the NAPI addon built successfully, TypeScript and Biome passed,
and the
selected native remote-transport test passed 1/0, including
exact-version
  selection and blocked scheduler metadata.
- `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile,
Biome, and
  TypeScript checks passed.

Validation is mapped by the relevant unchanged source rather than
treated as one
final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header
and Python
remote tests at `31e68b7d`, the local Python test and Node build at
`6f213584`, and
the final TypeScript/Jest delta at `4f2d5c27`. The final product tree
excludes those five private EC2 runner files and adds
six generated TypeDoc files plus one hidden draft usage section in the
computed-column refresh guide. The Node delta corrected a private
TypeScript
command and reused the unchanged native addon.
2026-11-11 19:12:06 +08:00
Xuanwo cf8e6d0fd4 feat: expose Function call concurrency limits (#4956)
## Summary

Adds an optional `max_concurrency` setting to Function registration and
explicit refresh:

- A registered Function version can define the default maximum number of
user-code calls in flight for one logical execution.
- A refresh can override that default for that execution only.
- The value must be a positive `u32` and is independent from
`resource_budget`; it is not an account-wide quota.
- Omitting the option preserves the previous request payload. Local
embedded refreshes reject this Enterprise-only option.

Rust, Python sync/async, PyO3, and the existing Node refresh surface
forward the option. Node has no Function registration surface today, so
no registration API is added there. The generated TypeDoc pages are
included for the new public option.

## Validation

The native-tested source covers Function version decoding, registration
validation and serialization, the Flight SQL registration ticket and
returned version, remote refresh forwarding, local rejection, and Node's
numeric boundary before conversion. Exact stage results are Rust `1+3`
plus four remote cases at `1` each; Python `14/1/49/24`; Node `57`
remote and `1` local rejection. Ruff, standard repository Pyright, and
Biome passed.

The tested Python extension was
`21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the
tested Node addon and generated native declaration were
`f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and
`4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The
final 24-file diff preserves the 22 tested
product/test/fixture/source-doc blobs and adds only two generated
TypeDoc pages.
2026-11-11 14:32:26 +08:00
Xuanwo 8029428c71 feat: persist automatic Function refresh resource budgets (#4953)
## Depends on merged #4951

Phase 1 merged as `4d22c0ce2b6825c880c818e737b079db494b4053`, so this PR
now targets `main`. It reuses the canonical
`FunctionWorkerResourceBudget` introduced there rather than copying the
type.

For Function `add_columns` declarations, Python and Rust now persist
`auto_refresh_resource_budget` in computed-column metadata and send it
as `auto_refresh_resource_budget` in the remote request. CPU
milliseconds and memory bytes are required positive integers. Omitting
`gpu_count` is accepted as zero and the canonical outgoing JSON still
contains `"gpu_count": 0`. Unknown fields, missing CPU or memory, wrong
types, zero CPU or memory, negative values, and overflow are rejected.

The same budget is applied to every public output binding of a
multi-output Function declaration. Existing metadata is retained, and
SQL-expression declarations or local execution reject the budget instead
of silently ignoring it. Node is unchanged because its current
`addColumns` surface only accepts SQL computed expressions and has no
Function declaration path.

This change does not implement fleet call concurrency limits or enforce
the budget in the runtime. The Enterprise server must support the nested
Function worker resource-budget contract before these values can act as
execution limits.

## Validation

- Python extension build passed.
- Remote Function declaration and local unsupported-path rejection each
passed.
- Canonical metadata round-trip passed, including omitted GPU decoding
to zero and explicit `gpu_count: 0` on encode.
- Function multi-output remote wire test passed with `--features
remote`.
- Repository-scoped Python type check passed with 0 errors.
- All runtime source blobs match the native-tested source. The
post-native deltas are one documentation example line and an existing
routing-test stub update for the new optional argument; the affected
parameterized test passed 3/3 against the existing extension build. No
private test runners are part of this PR.
2026-11-11 11:23:17 +08:00
Xuanwo c9d176ae7e feat: resolve latest Function versions (#4949)
## Problem

Function catalog lookup required an exact version string even though
callers
commonly need the latest currently visible immutable revision. On the
current
SDK, resolving that version required an out-of-band catalog query before
calling
`get_function`.

## Change

- Make Python `get_function(name)` default to latest while accepting
`int` or
  `str` for exact lookup.
- Normalize integer versions to their decimal string form and reject
`bool`
  explicitly. Existing string validation and errors remain unchanged.
- Thread `None` through the rebuilt PyO3 binding and route it to the new
Rust
  `Connection::get_function_latest`.
- Resolve latest through the existing catalog `DESCRIBE FUNCTION` path
without
`WITH (version = ...)`. The returned value is one exact immutable
version, not
  a mutable alias.
- Preserve the existing Rust exact-version API.

## Validation

On the designated EC2 host with the rebuilt extension:

- `test_local_function_catalog_operations_are_not_supported`: **1
passed**.
  This crosses the Python connection, rebuilt PyO3, and embedded Rust
  `NotSupported` boundary with the omitted-version call.
- Async and sync catalog-path tests: **2 passed**, 117 deselected.
- Remote SQL Function creation/description integration test: **1
passed**.
- An earlier Rust invocation with the wrong module path matched **0
tests**;
  it is retained as harness evidence and is not counted as a pass.

## Scope

This is the Function lookup SDK slice. It does not include the separate
atomic
rebind/publication server work tracked under ENT-3418.
2026-11-11 10:39:03 +08:00
Xuanwo 1e5b6deee3 fix(python): support legacy blob query bytes (#4947)
## Summary

- Let local plain queries over row-addressable legacy v1 and v2 blob
columns use `blob_mode="bytes"` without importing `pylance` through
pandas, batch, or Arrow conversion.
- Preserve aliases, filters, ordering, limits, null and empty payloads,
empty result schemas, and the caller's `_rowid` choice.
- Build the hidden row-id query separately so reusing an async builder
for lazy reads or non-blob projections cannot retain an automatic
`_rowid`.
- Keep byte reads closed for blobs inside lists, remote or namespace
pushdown, and query shapes whose result rows cannot be matched back to
the source. Blob v2 lazy auto-row-id behavior is unchanged.

Fixes #4645

## Validation

- 26 focused Python tests passed with the test subprocess blocking all
`lance` imports. The persisted fixture writes and reopens a storage-v2.1
table, then covers sync and async pandas, batches, and Arrow conversion
plus async builder reuse.
- The existing unsupported vector and hybrid reranker regressions now
assert the current eligibility diagnostic, and the legacy table-level
Arrow regression now asserts successful bytes, schema, values, and
hidden row IDs. Both focused tests passed.
- The runtime receipt records no installed `pylance` distribution and no
importable `lance` module. The Python-only change was verified with an
existing compatible 0.42.0b9 extension; the relevant query and fetch
APIs are unchanged between that source and this 0.42.0b10 base.
- Ruff format and lint pass for every changed Python file.
2026-11-11 10:08:13 +08:00
Xuanwo 4d22c0ce2b feat: expose Function refresh resource budgets in SDKs (#4951)
## Summary

- Add the canonical Rust `FunctionWorkerResourceBudget` and
`RefreshColumnOptions::resource_budget()` for explicit Enterprise
Function refreshes.
- Forward the nested budget through Python sync/async, PyO3, Rust remote
requests, Node NAPI, and TypeScript.
- Require positive CPU and memory values; normalize an omitted nested
GPU count to numeric zero; reject the nested budget when combined with
the top-level `gpu_count` shorthand.
- Reject `resource_budget` on LanceDB Embedded refresh paths instead of
silently ignoring it.
- Document the Rust, Python, and TypeScript APIs, including that SQL
servers currently require all three budget fields.

The Rust, Python, and TypeScript additions use `cpu_millis` /
`cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` /
`gpuCount`. The top-level GPU shorthand keeps its existing behavior and
remains mutually exclusive with the complete nested budget.

## Scope and compatibility

This change configures only the explicit refresh call. It does not
configure `add_columns` automatic refresh and does not introduce a
fleet-wide call-cap interface.

A LanceDB Enterprise server must expose the nested `resource_budget`
backfill contract. LanceDB Embedded continues to support its existing
expression refresh defaults and rejects the new complete budget
explicitly.

## Validation

Validated against the final signed product source:

- Rust: resource-budget remote serialization test `2 passed`; validation
test `1 passed`.
- Python: actual PyO3 build; remote-request matrix `22 passed`;
invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff
check and format clean.
- Repository-standard Pyright entry point: `0 errors`, within the
repository's configured include set.
- Node: native build and Biome lint passed; the existing remote-request
test `1 passed`, including fractional, negative, overflow, NaN, and
infinity nested GPU cases; the existing local-rejection test `1 passed`.
- TypeDoc generation and generated-document cleanliness passed.
- The native-tested product source differs from this head only in two
semantically equivalent Node range checks required by Clippy. Current CI
compiles and lints the final blobs. No dependency or lockfile changes
are included.
2026-11-11 10:05:52 +08:00
Jack Ye 481e82293e fix(python): explain explicit blob reads and subset copies (#4940)
Unsupported remote blob materialization errors now point to row-ID
selection, fetch_blobs and fetch_blob_files. Document descriptor
structs, fetched bytes and Function BlobReader inputs, and show a
bounded subset-copy recipe that keeps the source unchanged between
selection and fetching.

Adds guidance assertions and copy coverage preserving payloads, empty
blobs and nulls.

No new blob materialization API is introduced.

Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
2026-11-10 08:24:32 -08:00
Jack Ye 3bfa023375 fix(python): bound Job and AsyncJob payload previews (#4939)
Printing a Job or a bound status method could dump hundreds of lines of
serialized Function initialization. Bound nested payload previews in
both sync and async handles while preserving cached identity, state,
failure and result information. Full spec/result values remain available
through their properties, with no additional backend calls.

Adds regression coverage for large payloads, bound method display,
unobserved handles and completed local jobs.

The broader method/property API design remains separate.

Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
2026-11-10 08:23:13 -08:00
Jack Ye faa5987b33 fix(python): name unsupported Function globals in capture errors (#4938)
Function source packaging rejected unsupported global values without
identifying the variable or explaining the supported captures. Include
the global name even for a nested tuple failure, describe the supported
literal values and remedy, and document the capture boundary.

Adds function/class regression cases for Arrow types, nested tuples,
dictionaries and non-finite floats, plus a supported nested literal
control.

New serialization support remains separate.

Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
2026-11-10 08:22:52 -08:00