Commit Graph
865 Commits
Author SHA1 Message Date
Jack Ye f17a07a2cd feat: expose server-side job listing filters (#5041)
Expose server-side job filters and lazy Listing indexing, slicing, and length helpers. Document cache, cursor, and SDK/server compatibility behavior.
2026-11-15 00:50:58 -08:00
Lance Release 530d379fc4 Bump version: 0.42.0-beta.13 → 0.42.0-beta.14 2026-11-15 02:09:01 +00:00
Jack Ye e8a8dd1b2a feat(sql): use server-issued catalog listing tokens (#5026)
Migrates SQL-backed catalog listing APIs to `LIMIT ... FROM
'<page-token>'` and server-issued Arrow continuation metadata.
Namespace-specific APIs send an explicit `IN` schema clause, including
`IN "public"` for root, so raw SQL can default to all namespaces.

```sql
SHOW VIEWS LIMIT 100;
SHOW VIEWS IN team$ml LIMIT 100;
SHOW VIEWS IN team$ml LIMIT 100 FROM '<page-token>';
SHOW VIEWS IN other_db.team$ml LIMIT 100;
```

SDK clients forward opaque tokens unchanged, preserve empty
continuations, and decode identifier columns by name. Removes legacy
cursor reconstruction and updates the SQL reference and Python Flight
fixture. Public docs cover optional direct schema paths, all-namespace
defaults, and the shared pagination syntax across every catalog listing.
2026-11-15 08:55:42 +08:00
Xuanwo e2f836da99 feat: expose per-call merge retry controls in Python and Node (#5015)
Python and Node callers could not disable precommit retries for one
merge without changing the connection. Add `execute(..., retries=0)` for
Python sync/async and `execute(data, { retries: 0 })` for TypeScript,
forwarding into the existing Rust control without persisting the
override. Ambiguous 5xx outcomes remain non-replayable; permitted merge
retries now log status and attempt at WARN, and the API docs explain
retry budgets and timeout boundaries.

Validation at `6001300` passed 70 Python HTTP cases, 36 Node HTTP cases,
14 local merge regressions per binding, and 34 scoped Rust tests, with
rebuilt native extensions, TypeScript compilation, Clippy, scoped
formatting/lint, and generated API docs. The public API and feature diff
are unchanged after integrating main and the reviewed ingestion guide;
that binding evidence is retained. Focused merged-tree validation
additionally passed 267 Rust merge/retry/provenance cases and the CI
fixtures. Existing root Ruff issues in docs/web-tests remain; local
fixtures do not establish live proxy/server cancellation behavior.

Refs #4256 and OSS-2370.
2026-11-14 22:43:56 +08:00
Xuanwo dc9eb269b1 feat(python): support map and decimal Function signatures (#5031)
Python Function declarations reject Arrow map and decimal types before
registration. Accept `dict[str, T]`, nullable map values, explicit Arrow
maps, and `Annotated[Decimal, pa.decimal128/256(...)]` using the
existing canonical JSON representation. Preserve map key/value
nullability and decimal precision/scale; reject unsupported map keys,
sorted maps, nested Blob values, and a bare Decimal without
precision/scale.

This is the SDK map/decimal portion of ENT-3414. Server validation and
SQL support are separate changes; older servers can still reject these
signatures. Nullable scalar output support remains separate in #5030.

Validation: 129 Function signature tests passed; 30 documentation tests
passed with one existing skip. Ruff format/lint and diff checks passed.
Python tests used the existing Python 3.13 environment and prebuilt
native extension with this Python source overlaid; no Rust build or
remote-server execution was performed.
2026-11-14 22:30:47 +08:00
Xuanwo 2499342fa1 feat(function): expose resolved environments and fresh lock requests (#5034)
Function registration clients cannot request fresh dependency resolution
or inspect the measured environment returned by the server. Add
`refresh_lock=True` to Python registration and preserve the requested
runtime and measured environments in Rust/Python Function version
readback. Registration forwards both `exist_ok` and `refresh_lock`;
defaults remain unchanged.

Single-platform versions retain `resolved_environment`.
Multiarchitecture versions expose `resolved_environments`, keyed by OCI
`os/architecture`, and the Python `environment_report` alias. Platforms
are never flattened: absent entries mean unknown measurements, while
measured empty package lists remain present. Registration waits emit
scalar and per-platform server warnings once per Job; catalog reads do
not repeat them.

Requires matching server support, including Sophon #10344 for
per-platform publication. Nine focused Python source-overlay tests and
29 documentation tests passed for the additive map change; one
documentation test requiring a newer native `exist_ok` binding was
excluded. These checks reuse a prebuilt extension and do not establish
current native integration. PR CI exercises the Rust transport/readback
and Python bindings. Live multiarchitecture registration and GPU
acceptance remain separate.
2026-11-14 22:30:28 +08:00
Xuanwo ea59f8acb6 feat(python): preserve Function timestamp and duration types (#5032)
Function declarations currently reject exact timestamp and duration
types. Preserve their Arrow units and timezones in canonical signature
JSON, including fixed-offset timezones and nested fields. Bare
`datetime` and microsecond/no-timezone declarations keep their existing
representation; `timedelta` requires an explicit unit.

Compound temporal signatures require the Lance Namespace decoder in
https://github.com/lance-format/lance/pull/11059 and corresponding
server validation support. Older readers reject these signatures; this
change does not upgrade server support or dependency pins.

Validation: 128 Python signature tests and 30 documentation tests passed
(one skipped), using the changed Python source with an existing native
extension. No Rust rebuild or end-to-end Function registration/refresh
was performed.
2026-11-14 21:49:45 +08:00
Xuanwo 12b95b59ab feat(function): support literal inputs in column declarations (#5021)
Function-column declarations now preserve constants such as
`fn(video=col("mp4"), num_frames=lit(16))` without changing its Function
version. Constants travel in the application inputs; column bindings,
request input schemas, and dependency tracking contain only columns.
Changing a constant uses a new column binding rather than rebaking the
Function.

Persisted bindings with literals use metadata version 2, which older
readers reject. Column-only bindings retain version 1. Python preserves
scalar literal identity, including null and unsigned integers, and
rejects composed expressions or values outside the canonical literal
domain.

This SDK change requires matching server declaration and execution
support, after a dedicated SDK dependency update. Native validation
passed 48 Rust metadata tests, four request-transport cases and 57
rebuilt Python tests on `a42d680`; 32 pre-existing experimental tests
remained ignored. The test-assertion correction passed its exact
regression and strict Clippy on `53ff80a2`. On `993fae7d`, the three
remote metadata compatibility regressions, native Python public-export
test and strict Clippy passed. These review corrections use v3 for
unknown-version rejection, cover existing v2 literal bindings through
remote mutation guards, and export `LiteralInputBinding`. The unchanged
native implementation and verified extension are reused. Matching server
execution and original-workload acceptance are not established by this
PR.

Refs ENT-3667.
2026-11-14 20:59:52 +08:00
Xuanwo f6f8119ba4 feat: expose structured Function refresh options in SDKs (#5047)
## Problem

ENT-3267 still lacks typed refresh controls for `supervisor_resources`
and `planning_workers` in Python, Rust, and Node, and Node has no
`retry_policy` refresh option. Callers must drop to SQL for those
controls.

## Change

- Add canonical Rust `FunctionRefreshPodResources` and
`FunctionRefreshPlanningConfig`, and serialize them through
`RefreshColumnOptions`.
- Expose both options in the Python sync/async wrappers and PyO3
binding.
- Expose both options in the Node NAPI object and add a Node
`retryPolicy` object backed by the existing Rust `FunctionRetryPolicy`.
- Omitted options stay omitted; branch targeting continues through the
table handle. No server, dependency, or pin changes are included.

Client-side checks only reject non-positive structural values.
Deployment resource limits and column-kind validation remain
server-side.

## Validation

`cargo fmt --all -- --check`, `python3 -m py_compile`, and `git diff
--check` pass. No native build was run locally; CI compiles and tests
the change. Generated JS API pages are unchanged.
2026-11-14 20:59:44 +08:00
Xuanwo e5a5b67212 feat: support nullable scalar Function declarations (#5030)
Function declarations currently reject nullable scalar returns. Accept
`Optional[T]`, `T | None`, and explicit nullable Arrow fields,
preserving `nullable: true` in registration signatures, JSON round
trips, and remote `add_columns` requests.

The Rust planner accepts nullable scalar outputs, and binding readback
recognizes the server-owned assignment marker for a single nullable
scalar result. Clients do not allocate that marker. Non-nullable whole
results, whole structs, and Blob v2 retain their existing assignment
restrictions.

Validated with 127 Python tests, 30 documentation tests (1 skipped), and
seven focused Rust tests covering planning, binding readback, rejection
boundaries, and the remote request body. The final condition-collapse
change addresses strict Clippy; its delta check and CI are pending.
Python tests use a source overlay on a retained native extension, and
Rust remote coverage uses HTTP mocks.

Server support and end-to-end validation remain separate. This PR does
not claim that server publication distinguishes computed NULL values
from pending rows.
2026-11-14 20:59:20 +08:00
Xuanwo beda423318 fix(node): bind blob byte queries to their response snapshot (#5019)
TypeScript remote `toArrow` and `toArray` could combine scalar values
from query snapshot A with Blob payloads from a later snapshot B. Cached
table schemas and native descriptor layouts could also leave descriptors
in a `blobMode: "bytes"` result.

Discover Blob fields and snapshot provenance from the executed query
stream, recognizing canonical native v2 descriptors as well as
extension-marked fields. Fetch payloads through an independent Rust
snapshot handle bound to the query version and immutable dataset
identity, without changing the live table. Preserve aliases, nested
fields, NULL/empty values, explicit row IDs, and scalar-only results.
Hybrid byte queries require matching provenance from both actual
responses. Missing or mismatched server provenance fails explicitly;
logical-input validation and local materialization remain unchanged.

Manual integration validation at `4daf354df`: all five real SDK/NAPI →
Sophon REST scenarios passed. Both `toArrow` and `toArray` retain A
bytes across overwrite and reject same-name/same-URI recreation even
when version and row IDs are reused. A previously cached handle
discovers a subsequently added Blob field. Ten actual payload fetches
include the two expected identity rejections; later queries on the same
live handle read B. The canonical-descriptor and Blob regression suites
pass 119 tests, with independent source and wire-evidence review.
Earlier broader SDK and hybrid evidence is retained within its
unchanged-source scope.

This is local REST interoperability evidence, not deployed
Cloud/header-forwarding acceptance. Remote bytes require the
authoritative server version/identity and fetch-echo contract. CI and
formal review remain merge requirements; this follow-up does not close
ENT-3556 or Blob V2 GA.
2026-11-14 20:52:50 +08:00
Xuanwo 2f496322ef fix: report multipart rows only after part acknowledgement (#5020)
Remote multipart progress currently counts every scanned row before the
corresponding IPC part is acknowledged. A rejected first part can
therefore report uploaded rows, even though its acknowledged byte count
remains zero.

Count both rows and IPC bytes only after each part succeeds, including
parts assembled from multiple batches or split from one batch. Retried
parts count once. Native and single-request writes retain their existing
accounting; acknowledged multipart progress describes staged data, not a
successful commit.

Validation follows the actual TCP upload path: one scanned batch becomes
six decoded IPC parts, the sixth response is lost, and each
request/retry observes only previously acknowledged rows and bytes.
Rejection and complete-outcome cases preserve the same accounting
boundary. This validation covers local HTTP behavior; deployed service
throughput and the original disconnect investigation remain outside its
scope.


Frozen candidate validation: 839 remote tests and 7 progress tests
passed. Scoped Clippy (`lancedb`, remote feature, lib/tests/examples,
warnings denied) passed with unchanged source. Python/Node bindings were
unchanged and were not rebuilt for this patch.
2026-11-14 20:51:25 +08:00
Lance Release 5de76bd8ed Bump version: 0.42.0-beta.12 → 0.42.0-beta.13 2026-11-14 12:33:27 +00:00
Yang Cen e1a10eaa66 chore(deps): upgrade Lance to v15.0.0-beta.21 (#5046)
Upgrade the Rust Lance family from beta.18 to the published
v15.0.0-beta.21 tag, and align Python test wheels and Java artifacts.
This includes the explicit-empty-segment FTS fix in
lance-format/lance#11063 needed by lancedb/sophon#10314. Enable Windows
Git long paths so Cargo can check out Lance fixtures in the Rust, NPM
and Python wheel builds.

Beta.21 also contains lance-format/lance#9858: new NGram, RTree and
MinHash LSH indexes use row addresses. Legacy row-ID segments remain
readable, but index maintenance on stable-row-ID tables can require a
rebuild. The NGram documentation now explains the consolidation
requirement; NGram/RTree tables without stable row IDs retain their
earlier format. The Python lock revision and unrelated dependency
entries are unchanged.

Validation:
- The official Lance release and internal mirror match
80bc45a195279befdec42853dac8d15a37e0504d. The seven official Fury wheels
and Maven POM/JAR were verified.
- Exact executable/dependency state 7846a15e completed the full Rust,
Python, Node, Java and CLI CI, including platform builds and package
publication dry runs. Xuanwo approved that head; Gatekeeper recommended
approve with a non-blocking qualification risk.
- Follow-up a289cb2c only adds the requested seven-line NGram upgrade
note. Static whitespace checks passed; new-head CI is running. No local
correctness tests or test builds were run by this task owner.
- Broader IVF/encoding/Blob/cache workload performance and the beta.21
Sophon composition remain separately qualified. Earlier Blob
cancellation and DynamoDB failures on #4967 are retained in their
original evidence; this PR does not count those historical failures as
fixed.

Current-head CI, full feedback and required review still gate merge. SDK
publication is a separate action. Sophon adoption follows a published
SDK release and mirror through its own dependency PR; its distributed
performance acceptance remains in #10314.
2026-11-14 20:22:13 +08:00
Jack Ye dd449978d9 docs: clarify Function workflows and search contracts
Clarify Function signatures, refresh and recovery workflows, job snapshots, FTS operators, and IVF deduplication limits.
2026-11-14 01:09:40 -08:00
d67ec47921 docs: make the landing page concise (#4898)
Shortens the docs landing page and organizes its overview around
database features, with the multimodal data model, ML workflow, SDKs,
and PyTorch loader explained in prose. Keeps versioning visible,
branching in draft, and the existing `{#architecture}` anchor.

Validation: added documentation links and anchors verified; `git diff
--check` passes. `mint validate` reports the same four pre-existing
missing snippet-export warnings in the reranking and vector-search
pages, also reproduced on the unchanged revision.

Closes
https://linear.app/lancedb/issue/DOC-80/strengthen-what-is-lancedb

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
2026-11-13 20:23:41 -08:00
Jack Ye f6311ae7e9 feat: support exist_ok for index creation (#4993) 2026-11-13 19:21:50 -08:00
Jack Ye 81d04b8c36 feat(python): support exist_ok for materialized views (#4991) 2026-11-13 19:21:29 -08:00
Jack Ye 4db0d5c7f5 feat(python): support exist_ok when cloning tables (#4990)
Adds `exist_ok=False` to Python table cloning and `.exist_ok(true)` to
the Rust clone builder. An existing target is returned with its data and
version history unchanged; strict duplicate errors remain the default.

```python
snapshot = db.clone_table("snapshot", source_uri, exist_ok=True)
```

Embedded cloning skips source reads for an existing target and handles
typed creation conflicts. Remote cloning forwards the option to the
server. Includes sync/async preservation and remote request coverage,
plus SQL syntax documentation for `CREATE TABLE IF NOT EXISTS target
CLONE source`. Requires a server supporting the option for remote
clones.

BREAKING CHANGE: Rust `CloneTableRequest` struct literals must
initialize the new `exist_ok` field; `CloneTableRequest::new` remains
compatible.

No local builds or tests were run, as requested.
2026-11-14 10:48:26 +08:00
Jack Ye a0eb6b4b71 feat(python): support exist_ok for storage integrations (#4986)
Adds `exist_ok=False` to sync and async Python storage integration
creation. `exist_ok=True` sends `CREATE STORAGE INTEGRATION IF NOT
EXISTS` and returns the server's stored configuration, preserving the
prefix, credentials and external ID.

```python
integration = db.create_storage_integration(
    "raw", storage_provider="S3", storage_prefix="s3://customer-bucket/raw/",
    aws_role_arn="arn:aws:iam::123456789012:role/access", exist_ok=True,
)
```

The non-exhaustive Rust request also supports `.exist_ok(true)` for
every provider. Requires a SQL server supporting this syntax. Includes
protocol and provider SQL coverage. No local builds or tests were run,
as requested.
2026-11-14 10:37:08 +08:00
Jack Ye 2c3c30dbdf feat(python): support exist_ok for Function creation (#4992)
Adds `exist_ok=False` to Python Function creation and job submission.
`exist_ok=True` sends `CREATE FUNCTION IF NOT EXISTS` and returns the
stored immutable version, preserving its code, credentials and execution
settings. The default retains the current version-creation behavior.

```python
version = db.create_function(analyze_caption, exist_ok=True)
```

The option also exists on the Rust registration request and is kept out
of the canonical Function definition payload. Includes
sync/async/background protocol and stored-version coverage. Requires
matching SQL server support.

BREAKING CHANGE: Rust `FunctionRegistrationRequest` struct literals must
initialize `exist_ok`; legacy serialized requests default to false.

No local builds or tests were run, as requested.
2026-11-14 10:35:54 +08:00
Jack Ye b061c3627c feat(python): support exist_ok when creating views (#4984)
Add `exist_ok=False` to synchronous and asynchronous Python view
creation. Passing `True` sends `CREATE VIEW IF NOT EXISTS` and returns
the existing view's stored query and schema unchanged; the default
continues to fail on an existing name.

```python
view = db.create_view("recent_orders", query, exist_ok=True)
view = await async_db.create_view("recent_orders", query, exist_ok=True)
```

Requires a SQL server supporting `CREATE VIEW IF NOT EXISTS`. Rust gains
an awaitable builder with `.exist_ok(true)`. Existing
`.create_view(...).await` calls continue to work; callers requiring
`Future` must convert the builder, and custom `Database` implementations
must accept the new `exist_ok` argument. Includes Python round-trip
coverage and Rust coverage for direct/deferred results and metadata
preservation.
2026-11-14 10:35:52 +08:00
Will JonesandClaude Opus 5.5 7d50d2d28f chore(deps): bump Rust deps flagged by security scans (#5035)
## Summary

Bumps the Rust dependencies flagged by Aikido that have a
semver-compatible fix available. Supersedes #5009 (rustls in the docs
lockfile is included here).

Root `Cargo.lock`:

| Crate | From | To |
| --- | --- | --- |
| mio | 1.2.0 | 1.2.5 |
| smallvec | 1.15.1 | 1.16.5 |
| thread_local | 1.1.9 | 1.1.10 |
| tinyvec | 1.11.0 | 1.13.3 |
| zerocopy / zerocopy-derive | 0.8.48 | 0.8.63 |
| event-listener | 5.4.1 | 5.4.2 |
| arrayvec | 0.7.6 | 0.7.8 |
| wasm-bindgen (+ js-sys, web-sys) | 0.2.121 | 0.2.130 |
| getrandom 0.4 | 0.4.2 | 0.4.3 (drops wasip3 → wasmparser 0.244) |
| brotli | 3.5.0 | 9.1.1 (via `random_word` dev-dep 0.4.3 → 1.0.0) |

`docs/web-tests/rs/Cargo.lock` gets the same crates plus rustls, hyper,
h2, moka, bytes, crossbeam-epoch and anyhow. That crate is not a
workspace member, so cargo refused to resolve it; it now has an empty
`[workspace]` table.

Still open, blocked upstream: quick-xml 0.41 (lance-io), rand 0.9
(datafusion/lance/candle), heapless 0.8 (rstar via geo), paste,
unsafe-libyaml (rattler_conda_types, lindera), fast-float/smartstring
(polars 0.39), number_prefix (needs hf-hub 1.0 migration), zstd 0.13
(arrow-ipc).

## Merge Risk

**Door:** two-way. Lockfile patch/minor bumps and a dev-only dependency.
**Blast Radius:** none expected; no source changes. The docs crate is
not built in CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:44:40 -08:00
Taylor 3a980054a1 docs: standardize Realtime Engine naming (#4969)
The Enterprise feature name was inconsistently written as "Real Time
Engine" and "real time engine". Standardize it to **Realtime Engine** in
eight website guides, including page and sidebar titles, descriptions,
prose, link labels, cards, and search keywords.

The remaining MemWAL references describe Lance's open source format or
literal API/job identifiers, including `memwal_compaction` and
`memwal_catchup_<type>`, and are preserved. Only `docs/web/*.mdx` pages
change; SDK source, generated API reference, code examples, URL
destinations, and section anchors are unchanged.

Validation:
- `git diff --check` passes.
- Reviewed all MemWAL and Realtime Engine references under `docs/`.
- Verified each changed file differs only by the intended naming
substitutions and preserves code blocks, inline code, links, anchors,
and MemWAL identifiers.
- Confirmed no split or hyphenated "real time engine" names remain in
website guides.
- No snippets needed regeneration because code examples are unchanged.
Vale and Mintlify are not installed locally, so their checks were not
run locally.
2026-11-13 11:24:55 -08:00
Jack Ye bab22a3fbd feat(python): bundle Function package resources (#4987)
A package shipped through `code=` currently loses its non-Python files.
This breaks vendored packages that read JSON configuration, manifests,
prompts, or binary resources after deployment. Include regular package
resources in a separate base64-encoded `python_bundle.resources` map,
preserving exact bytes and relative paths. Both source and resources
contribute to the artifact digest. UTF-8 Python source bytes retain
their original line endings as well, so manifests that hash vendor code
continue to match.

For example, `@function(code=[my_package])` can use
`importlib.resources.files("my_package").joinpath("models.json").read_text()`
on the worker. Hidden files and caches are excluded; symlinks and unsafe
paths warn instead of following files outside the package. Source-only
bundles retain their existing wire form. Resource bundles require
matching server support.

Adds an isolated-import regression with a vendored subpackage, JSON
metadata, binary data, SHA256 verification, preserved CRLF bytes, and
resource-only version changes. Local builds and tests are disabled by
request. Ruff formatting and linting run locally; CI runs the tests.
2026-11-13 16:21:58 +08:00
Jack Ye 9302287bed fix: accept defining queries in Python materialized views (#4980)
Python materialized-view creation now accepts a complete defining SQL
query across sync, async, remote, and namespace connections, including
creation jobs:

```python
view = db.create_materialized_view(
    "events_by_kind",
    query="SELECT kind, count(*) AS n FROM events GROUP BY kind",
)
```

`query` is mutually exclusive with `source`, `select`, `where`, and
`limit`. Existing source-based calls remain supported.

Adds Rust `Connection::create_materialized_view_from_query`: embedded
connections plan through `MaterializedViewDefinition::from_sql`, while
remote connections submit the full query for server-side parsing.
Includes regression coverage for grouping, UNNEST, query forwarding,
creation jobs, and argument validation.
2026-11-12 14:57:00 -08:00
Xuanwo cababaecf0 feat(remote): discover server builds and retain error provenance (#4999)
Remote callers can use `db.server_info()` in Rust and Python, including
synchronous and asynchronous Python connections, to identify the
responding server without submitting a query or Job. Typed results
describe the query-node build, component versions, Function protocol and
configured images. Older servers without the endpoint return an explicit
unsupported error.

REST and Flight errors retain the reported build in diagnostic text and
Rust `RemoteErrorMetadata.server`. Typed table/job-not-found errors
preserve their HTTP source and classification, including the
schema-cache path; masked REST errors can obtain provenance from the
response header. Unknown build information stays absent.

The matching native Python SDK was exercised against a real localhost
server (`8829b074` client / `43920fa` server): sync/async discovery,
both SQL version forms and actual REST/Flight error provenance passed.
The product files match this branch; private validation drivers are
excluded. Focused coverage includes typed 404 errors, masked errors, and
the public schema-cache regression. Deployment and published-package
acceptance remain separate.

Server counterpart: https://github.com/lancedb/sophon/pull/10234.

Related: ENT-3565.
2026-11-13 06:22:35 +08:00
Xuanwo 6d24446a30 feat: add negotiated batched blob range reads across SDKs (#4994)
Existing Rust and Python blob range calls must keep working when only
the SDK is upgraded. Add negotiated batched range reads across Rust,
Python sync/async and TypeScript, preserving ordered NULLs, duplicates
and branch/version reads with splitting and reassembly at 1,024 ranges
or 64 MiB per request.

Only a successful describe response advertising `x-lancedb-blob-ranges:
1` selects the batched endpoint. Missing or unknown capability values
select the existing descriptor-query plus BlobFile Range GET route
before the operation. Endpoint failures never switch protocols. HTTP 400
preserves the server message and request ID as Rust `InvalidInput` /
Python `ValueError`; other errors retain their existing semantics.

The operation binds both numeric version and immutable snapshot identity
from its producing query or first range response. Every later partition
and oversized-range piece sends that original identity and requires an
equal response identity before decoding. Same-name table recreation
therefore fails explicitly instead of returning mixed old/new payloads;
no partial array is returned on later failure.

Targets `main`. The read-buffering prerequisite #4979, safe-write
prerequisite #4977 and deferred-query identity prerequisite #4973 are
merged into main; #5000 also supplies the merged SQL timeout-test
prerequisite. Server support is [Sophon
#10219](https://github.com/lancedb/sophon/pull/10219), including #10211.

Exact SDK `1497b508324662180086a4d1e0a6eb9e0b3970db` and server
`f39712c2c150b220eb831e00f6663c9eef32dce5` passed a real local SDK →
REST → PE experiment: one Rust harness test, 14 assertion groups, 38
REST range requests and 34 PE RPCs. All four sync/async × warm/cold
table-recreation cases failed explicitly before the second descriptor
lookup or PE dispatch, with no mixed payload or partial result. The
other groups retained 1,025-range success, 70 MiB reassembly,
branch/version reads, overwrite snapshot consistency, error behavior and
TypeScript buffer ownership. Bindings were rebuilt and their loaded
hashes verified. This is local interoperability evidence, not Cloud,
mixed-fleet or deployed header-forwarding acceptance.

The server dependency upgrade, External/cache/URI-policy acceptance and
validated rollout remain required. Enforcing PE support must precede QN
and SDK enablement; identity does not retain versions or freeze External
objects. This PR does not claim deployed support or complete ENT-3100/GA
acceptance.

Published documentation retains the Early access badge and released
Enterprise instructions, including explicit fetches for copying. The new
Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example,
future streaming restrictions and GA wording remain source-only MDX
drafts until release. The correction distinguishing released Enterprise
SQL bytes from SDK descriptors remains visible. Documentation validation
checked comment boundaries, visible instructions, Python example syntax
and whitespace; the site was not built. The main integration keeps these
release gates unchanged.

Main integration at `0491d9bdcc65ce5d52ea93e1083c89e951ea4f9d` normally
merges main `9b13e8f72530c8b1950c09685f6920711154a539`, including its
Lance beta.18 dependency. Focused current-source validation passed: 493
remote-table/query/Blob/IPC/write-progress Rust tests, 23 shared
create/write-body tests, 59 native Blob integration tests and 7 public
HTTP range tests. Rebuilt Python passed 468 selected tests (one optional
PyAV test skipped), and rebuilt TypeScript passed 194 selected tests
across four suites; lint/format, TypeScript compilation and API
documentation generation passed. Shared tests include actual HTTP
multipart failures, acknowledgment boundaries, dictionary compaction and
native Blob compaction. The 598 unrelated Jest cases excluded by the
focused shared-suite selector were not run.

The retained 14-group joint REST/PE experiment used the earlier exact
candidates stated above. Operation-identity/range/IPC code is unchanged,
but dependencies and shared write interfaces changed with main; the new
local checks validate the SDK combination and do not constitute a fresh
final-head SDK/server joint, Cloud or rollout run. Early access and
every remaining server/External/cache/policy/dependency/rollout
condition remain in force.

CI snapshot on this head (2026-11-12 18:30 UTC): `build-no-lock` failed
in the standalone `ci/tests/test_build_no_lock.py` fixture at the
retained-rlib assertion, before the actual SDK no-lock build. The
fixture, helper and Rust workflow are byte-identical to main; the
fixture does not load SDK source. The underlying CI/environment cause is
not diagnosed here. Other jobs remained pending/running at the snapshot,
so CI is not reported as passing. Failure:
https://github.com/lancedb/lancedb/actions/runs/43280277756/job/131399418284.
2026-11-13 04:03:04 +08:00
Xuanwo 9b13e8f725 feat(python): materialize remote blobs at the query snapshot (#4973)
Python remote queries materialize Blob v2 payloads with
`blob_mode="bytes"` through synchronous and asynchronous `to_arrow`,
`to_list`, `to_pandas`, and `to_batches`. Deferred reads bind to the
actual producing dataset without changing the caller's live table
handle, including concurrent writes and same-name DROP/CREATE.

Trust only the authoritative `x-lancedb-query-version` plus
`x-lancedb-query-snapshot`. A numeric version alone cannot distinguish a
recreated table; the ordinary live watermark and user-supplied Arrow
metadata cannot establish provenance. An independent immutable handle
carries the original identity through every batch, byte-limit split, and
large-Blob Range fallback, and requires a verified response echo before
accepting payloads. Aliases, NULL/empty results, rerankers, and hybrid
legs retain consistent provenance. Missing capability, mixed identities,
or a serving node ignoring the precondition fails explicitly; ordinary
descriptor queries remain available on older servers.

The source docs distinguish Enterprise SQL payload bytes from SDK
descriptors. Server dependency:
https://github.com/lancedb/sophon/pull/10211.

Validation uses retained, independently audited local evidence: 724
Python cases passed (one optional PyAV skip), 67 focused SDK Rust cases
passed, 36 server checks passed, and 15 actual REST/SDK scenarios passed
across retained runs. These cover sync/async and Arrow file/stream
recreation, concurrent overwrite, deletion of a selected version, legacy
capabilities, forged IPC metadata, WAL exclusion, and actual mid-batch
recreation. The original aggregate experiment exited 101 because its
extra harness assumed a Python batch size would split the native result;
its 36 server checks and 13 completed scenarios remain valid. The
corrected native-batch experiment passed separately for sync and async.
No aggregate all-green claim is made.

The public batched-range operation has separate cross-partition identity
integration in https://github.com/lancedb/sophon/pull/10219 and
https://github.com/lancedb/lancedb/pull/4994. This change does not claim
to fix that loop. Realtime/MemWAL remains outside Blob v2 GA, without
disabling existing writes or compaction. No dependency pins changed.

Merge/rollout hold remains: upgrade every serving node before enabling
the new SDK capability, verify proxy forwarding of both authoritative
headers and the verification echo, and preserve queried versions for
later reads. Missing or inconsistent capability fails explicitly;
identity does not retain versions or authenticate storage paths. Actual
deployed proxy/Cloud/PE behavior and production retention remain
unverified. Old SDKs do not gain incarnation protection from a
server-only upgrade.

Published documentation retains the Early access badge and released
Enterprise instructions, including explicit fetches for copying. The new
Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example,
future streaming restrictions and GA wording remain source-only MDX
drafts until release. The correction distinguishing released Enterprise
SQL bytes from SDK descriptors remains visible. Documentation validation
checked comment boundaries, visible instructions, Python example syntax
and whitespace; the site was not built. This documentation-only revision
reuses the implementation evidence above.
2026-11-13 02:05:39 +08:00
Xuanwo 725eee8a47 fix: prevent unsafe write retries and bound Blob uploads (#4977)
Ambiguous gateway failures can occur after a write commits, so replaying
the operation can append duplicate rows. Non-idempotent writes now retry
only known precommit failures. Multipart retries reuse the same upload
ID, part ID, and encoded bytes; ambiguous completion returns an
unknown-outcome error without abort or replay. A definite 409/429
completion rejection triggers best-effort cleanup using the call's retry
budget.

Replayable parts and the shared byte-based partition target default to
64 MiB. Dictionary-bearing batches are compacted before size estimation
and again when actual compressed IPC framing requires further splitting:
keys are remapped without changing their types or schema, view backing
buffers are collected, and ListView/LargeListView child ranges are
compacted as an interval union so overlapping views retain sharing.
Parts are prepared on demand; a single row and its schema can still
exceed the target. Multipart progress starts at zero and counts each
acknowledged IPC body once. Rust write builders and Python/TypeScript
`add` expose per-call retry overrides. Atomic `create_table` pre-encodes
to a temporary file and rejects compressed IPC over 256 MiB before
sending any request; the schema-first import guidance now includes
Python, TypeScript, and Rust examples. The shared partition target also
affects local automatic partitioning.

At `cb2a79473bd1b51b950c9fa3e9f247f929c8ac3b`, 70 targeted Rust cases
pass on source verified against the committed four-file patch. With a 16
KiB target and 128 unique 1 KiB dictionary values, the original ordinary
dictionary reproducer drops from 128 requests / 16,942,080 bytes to 16 /
143,488; ListView and LargeListView drop from 128 / 17,056,768 to 16 /
149,632. Encoded HTTP bodies are decoded to check values, nulls, slices,
metadata, changing dictionaries, and overlapping views; additional cases
cover framing-driven splitting, oversized rows, stable retry bytes/IDs,
acknowledged progress, and atomic create. Real HTTP fault tests commit
Blob payloads to a local Lance table and verify byte equality, no
duplicate append, and zero abort/replay after ambiguous completion.

The earlier 811 remote and 7 progress cases at
`f36d217110a25f6b501c0255df9aaa60e3c8000a`, Linux Python remote
database/options results (437 cases plus 8 fixture consumers), Node
retry-binding evidence, and Rust/TypeScript documentation example checks
remain historical validation. Python/Node extensions were not rebuilt
for the dictionary follow-up; these results do not establish current
binding integration or fresh cross-platform/minimum-dependency CI
acceptance.

A separate real Query Node lifecycle experiment used SDK
`b8efe7d2d63647d4962789ca5fc042b59efe073d` and Sophon
`58b7895c7c287810fff4e89f13226fc5fa60d9ef`. After the actual listener
forced a connection closed, a surviving QN accepted a retry with
identical upload ID, part ID, and IPC bytes. Ordinary insert and
complete response-loss scenarios sent no replay or abort; fresh reads
observed exactly one commit and exact Blob bytes in all three scenarios.
This local experiment used a 1-second drain grace, two 32 KiB payloads,
and one distinct staged part. It validates acknowledgement-loss recovery
at those revisions, not concurrent overlapping attempts, large uploads,
physical orphan cleanup, the later dictionary encoding change, or the
original sixth-insert EOF.

The part target is not a process memory limit: source/ancestor buffers
and temporary compaction allocations may coexist. Struct/FixedSizeList
parent nulls can retain hidden child keys; this is not complete
logical-reachability GC for every Arrow combination, and retained hidden
values may affect wire size. The current local Lance 15.0.0-beta.17
rejects ListView/LargeListView persistence; the tests establish remote
client IPC encoding behavior, not native or Cloud persistence support.
Live Cloud, full workspace/Clippy, the original ordinary-insert
disconnect root cause, and Sophon's asynchronous transaction cleanup
behavior are not established. Cancelling during staging retains the
existing server-session cleanup limitation. Function reader delivery is
separate.
2026-11-13 02:05:13 +08:00
Xuanwo 10f89b3c32 perf: reduce Blob read buffering and expose read window controls (#4979)
Remote Blob reads retain and concatenate intermediate IPC buffers across
batches and HTTP requests, increasing peak memory. Decode fragmented IPC
bodies using their declared length, assemble Blob values once across
batches/requests, and collect large-object Range responses into
known-size buffers. Impossible IPC/Range allocations return errors
instead of panicking. Closing a remote Blob file releases cached bytes
immediately, including while a cross-window read is waiting for the
network.

Rust and Python expose configurable Blob read buffers. Python sync/async
`fetch_blob_files` and `open_blob_file(s)` accept `buffer_size`; the
default is 512 KiB and zero disables prefetch. Explicit ranges bypass
prefetch. New PE batch-endpoint adoption is separate.

Malformed-length and retained-cache regressions fail on the previous
implementation and pass after the correction. Real HTTP tests cover
buffer propagation, close during blocked input, batch/request
aggregation, and large-object fallback. The final extension was rebuilt
before Python validation. Non-Blob Arrow arrays may still require
alignment copies; a contiguous multi-batch result still requires its
final allocation.

## Benchmark

Peak client RSS increment in KiB, using
`python/benchmarks/bench_fetch_blobs_rss.py`, independent clients and
real loopback HTTP. Both extensions were rebuilt from clean, unchanged
source with the same toolchain/configuration; loaded artifact hashes
were checked. Baseline `42f11d13a2755e44b63876b5a5edd93901291150`, after
`0a8a76de5286949d80368f95a5c529adf422c187`.

| Equivalent fetch workload | Payload | Before | After |
| --- | ---: | ---: | ---: |
| Single response, one batch | 48 MiB | 63,812 | 63,896 |
| Multiple responses, one batch per response | 144 MiB | 423,104 |
195,628 |
| Single response, 12 rows per batch | 48 MiB | 130,520 | 71,348 |
| Multiple responses, 12 rows per batch | 144 MiB | 363,144 | 172,748 |
| One oversized row, Range fallback | 70 MiB | 182,816 | 85,548 |

These are one observation per workload and build, not a statistical
distribution or memory upper bound. The single-batch workload shows no
improvement. Cloud/Sophon deployment, credentialed S3 and long-running
stress have not been exercised.
2026-11-13 02:05:09 +08:00
Jack Ye e61e21035c fix(python): explain Function namespace import restrictions (#4982)
A plain `import ctypes.util` in a Function currently reports “dynamic
namespace access” without explaining the module restriction or a way
forward. Give blocked module imports a distinct error that names the
modules, explains the reconstructed namespace constraint, and recommends
an importable helper supplied through `code=`, `pip=`, or `conda=`.
Dynamic namespace operations also get an actionable explanation.

Document the full restrictions and the helper alternative in the
decorator and Python Functions reference. Add regression coverage for
direct imports, `from` imports, global module aliases, class Functions,
the deprecated `@udf` alias, and helpers imported normally or shipped
with `code=`. Existing import restrictions remain in place.

For example, move `ctypes.util.find_library` into
`native_diagnostics/__init__.py`, then use:

```python
import native_diagnostics
from lancedb import function


@function(code=[native_diagnostics])
def probe_library(name: str) -> str:
    return native_diagnostics.find_library(name)
```

Fixes ENT-3408.
2026-11-12 09:01:39 -08:00
Jack Ye 7b2dfb9e36 fix(python): align catalog and database connection inputs (#4983)
Align Python catalog connection inputs with database connections, warn when region is ignored, and preserve remote storage settings in opened databases.
2026-11-12 01:19:05 -08:00
Lance Release 01e9ea44aa Bump version: 0.42.0-beta.11 → 0.42.0-beta.12 2026-11-12 08:13:04 +00:00
Jack Ye e622254108 fix(python): use direct lookups for table existence checks (#4974)
Table existence checks should not depend on listing every table or
searching only the first 100 names. Use a direct lookup for `name in db`
and implement `table_exists(table_id)` on synchronous local connections,
matching the remote and asynchronous APIs. Native namespace connections
use their initialized Rust connection for existence checks; custom
namespace clients retain their existing callback. Missing tables return
`False`; other lookup errors propagate.

```python
if "events" in db:
    db.drop_table("events")

exists = db.table_exists(["analytics", "events"])
all_names = list(db.list_tables())
```

Clarify that `.tables` contains one page and update deprecation guidance
with safe replacements. Add regression coverage for lookups beyond the
first page, nested namespaces, native and custom namespace clients,
missing tables, permission errors, and lookups that do not list tables.
Include an unpublished draft for the table-management guide until the
new SDK support ships.
2026-11-11 23:32:19 -08:00
Brendan Clement 42f11d13a2 docs: clarify Python job states and waiting behavior (#4954)
We weren’t explicit about the job states returned by the Python SDK.
This documents those values and clarifies the difference between
checking the current status and waiting for completion.
2026-11-11 12:50:07 -08:00
Bruno Ramirez 28bbc81c3e fix(remote): explain unsupported optimize syntax (#4944)
This PR addresses [OSS-2913](https://linear.app/lancedb/issue/OSS-2913)
([#4845](https://github.com/lancedb/lancedb/issues/4845)) and
[ENT-2909](https://linear.app/lancedb/issue/ENT-2909):
`table.optimize()` on an Enterprise table exposes SQL parser errors when
the deployment does not support its generated SQL. The SDK generates the
statement itself, but previously propagated unsupported grammar as a
generic SQL failure. Parser errors and unknown-option errors reported as
`InvalidArgument` now become `NotSupported` (`NotImplementedError` in
Python), retaining the original diagnostic and request ID.
2026-11-11 12:06:00 -08:00
Lance Release 08d3f448af Bump version: 0.42.0-beta.10 → 0.42.0-beta.11 2026-11-11 15:38:36 +00:00
Jack Ye e9774ed168 feat(nodejs): expose Function refresh publication intervals (#4957)
Expose `publicationIntervalRows` on TypeScript refresh options and
forward it as `publication_interval_rows`. Accept positive safe integers
and preserve whole-fragment publication when omitted.

```typescript
const job = await table.refreshColumnAsync("embedding", {
  publicationIntervalRows: 10000,
});
await job.wait();
```

Document the threshold semantics and add coverage for serialization,
omission, invalid values, safe-integer boundaries, and Embedded
rejection.

Closes ENT-3755.
2026-11-11 07:34:54 -08:00
Xuanwo a356950bce feat: expose queued Function refresh status (#4959)
## Problem

The existing Function `column_status` surface cannot decode the Agent
scheduler's `queued` state or the `failure_threshold_reached` and
`binding_revision_missing` blocked reasons. It also preserves only the
latest refresh job, so a later cancellation can hide an earlier actual
failure.

## Change

- Adds the `queued`, `failure_threshold_reached`, and
`binding_revision_missing` wire states in Rust, Python, and Node. The
wire change is additive, while Rust callers must update exhaustive enum
matches and public struct literals.
- Adds optional `last_failure_job` alongside `last_refresh_job`. The
latest cancellation and the most recent actual failure retain separate
meanings.
- Reuses the canonical Rust `JobFailure` and existing Python/Node
`JobFailureInfo` models for nested job failure details instead of
introducing a parallel status error model.
- Maps the existing public `sql_job_id` field through Node's
`JobFailureInfo`.
- Keeps `column_status`, `column_statuses`, and `wait_fresh` as the
existing API surfaces. There is no new endpoint, row-error list, durable
ledger, or unknown-enum fallback.
- Documents `queued` versus `running` and the separate latest-attempt
and latest-failure jobs in the existing hidden freshness release draft.

The new status values are additive but cross-service: `queued`, the two
new blocked reasons, and `last_failure_job` require matching Agent/QN
and SDK deployments before clients can rely on them. This SDK follow-up
does not change the normal `7cc3904a` dependency pin or claim whole
ENT-3487 completion.

## Validation

The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`,
based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The
final product docs commit changes generated TypeDoc only.

- Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust
blobs are unchanged from their tested `788ef85b` source through
`c9bac25e`.
- Python: the PyO3 extension built successfully, and the selected native
PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The
test requests pinned version 42.
- Node: the addon and TypeScript builds passed, Biome passed, and the
selected Jest transport test passed 1/0 with 57 filtered tests. Addon
SHA-256:
`c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`.
- Python extension SHA-256:
`36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`.
- TypeDoc was generated from the native `native.d.ts` produced by that
Node build. A second generation produced no remaining diff.

The public diff contains only product, test, and generated API
documentation changes. Native runner scripts and preserved
compile-failure logs remain private validation evidence.

Agent/QN production mapping, mixed-version rollout, and the remaining
ENT-3487 server-side work are tracked separately.
2026-11-11 21:14:30 +08:00
Xuanwo 7cc3904a7e feat: expose exact Function column freshness (#4958)
## Problem

Function-backed columns need an exact `fresh`, `stale`,
`never_computed`, and
`failed` partition for the table version a caller is reading. The
existing
error listing is capped, and composing it with job and schema listings
cannot
prove which rows currently failed.

## Behavior

- Adds a strict freshness reader beside the existing permissive planner.
It
  distinguishes an absent certificate from legacy evidence that omits
  failures, and from a current certificate carrying failure coverage.
- Extends the immutable freshness sidecar with per-fragment known and
failed
  row bitmaps. Partial or `WHERE` publications cover only observed rows;
  untouched rows stay unknown. Deleted rows do not block classification.
- Adds failure-aware whole-fragment and row-repair writers. A rejected
older
attempt cannot overwrite failure evidence already recorded for a newer
input
  signature.
- Adds `column_status`, `column_statuses`, and `wait_fresh` to Python
and the
  corresponding Node methods. The client calls the server's exact status
endpoint, validates the four-way partition and one-version guarantee,
and
preserves the complete Function version, last-job, and automatic-refresh
  metadata.
- Adds an MDX-commented draft usage section to the Enterprise
computed-column
refresh guide, covering Python, Rust, and TypeScript and remaining
hidden
  until the server integration ships.
- Sends `column`, `branch`, and exact `version` selectors to the status
route.
A pinned handle rejects a nonempty response from any other table
version;
  latest handles keep the existing read-freshness headers.
- Retries `wait_fresh` only for HTTP 503 with
`function_column_status_finalization_pending`. Existing failed rows
remain
  visible and do not keep the wait pending; other errors propagate.
- Models the shared scheduler projection, including `up_to_date`, real
`blocked` states, their reason, failure streak, and effective cooldown.

## Validation

This is the SDK side of ENT-3563. Sophon's failure-coverage publication
and
status endpoint are a separate integration; deployed end-to-end and
whole-ticket
acceptance remain pending.

- Rust: 9 exact freshness/status tests passed, plus 1 exact
latest-handle
  freshness-header test. The pinned transport test was extended with a
wrong-version response and rerun at `ebd6649a` (1 passed). Coverage
includes untouched partial rows,
  selected-row failure clearing, cross-generation evidence, typed and
non-retried errors, `up_to_date`, blocked scheduler fields, and
exact-version
  requests.
- Python: 2 native PyO3 remote-transport tests passed together,
including
typed retry; one local facade test reaches the existing unsupported
boundary.
- Node: the NAPI addon built successfully, TypeScript and Biome passed,
and the
selected native remote-transport test passed 1/0, including
exact-version
  selection and blocked scheduler metadata.
- `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile,
Biome, and
  TypeScript checks passed.

Validation is mapped by the relevant unchanged source rather than
treated as one
final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header
and Python
remote tests at `31e68b7d`, the local Python test and Node build at
`6f213584`, and
the final TypeScript/Jest delta at `4f2d5c27`. The final product tree
excludes those five private EC2 runner files and adds
six generated TypeDoc files plus one hidden draft usage section in the
computed-column refresh guide. The Node delta corrected a private
TypeScript
command and reused the unchanged native addon.
2026-11-11 19:12:06 +08:00
Xuanwo cf8e6d0fd4 feat: expose Function call concurrency limits (#4956)
## Summary

Adds an optional `max_concurrency` setting to Function registration and
explicit refresh:

- A registered Function version can define the default maximum number of
user-code calls in flight for one logical execution.
- A refresh can override that default for that execution only.
- The value must be a positive `u32` and is independent from
`resource_budget`; it is not an account-wide quota.
- Omitting the option preserves the previous request payload. Local
embedded refreshes reject this Enterprise-only option.

Rust, Python sync/async, PyO3, and the existing Node refresh surface
forward the option. Node has no Function registration surface today, so
no registration API is added there. The generated TypeDoc pages are
included for the new public option.

## Validation

The native-tested source covers Function version decoding, registration
validation and serialization, the Flight SQL registration ticket and
returned version, remote refresh forwarding, local rejection, and Node's
numeric boundary before conversion. Exact stage results are Rust `1+3`
plus four remote cases at `1` each; Python `14/1/49/24`; Node `57`
remote and `1` local rejection. Ruff, standard repository Pyright, and
Biome passed.

The tested Python extension was
`21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the
tested Node addon and generated native declaration were
`f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and
`4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The
final 24-file diff preserves the 22 tested
product/test/fixture/source-doc blobs and adds only two generated
TypeDoc pages.
2026-11-11 14:32:26 +08:00
Xuanwo 8029428c71 feat: persist automatic Function refresh resource budgets (#4953)
## Depends on merged #4951

Phase 1 merged as `4d22c0ce2b6825c880c818e737b079db494b4053`, so this PR
now targets `main`. It reuses the canonical
`FunctionWorkerResourceBudget` introduced there rather than copying the
type.

For Function `add_columns` declarations, Python and Rust now persist
`auto_refresh_resource_budget` in computed-column metadata and send it
as `auto_refresh_resource_budget` in the remote request. CPU
milliseconds and memory bytes are required positive integers. Omitting
`gpu_count` is accepted as zero and the canonical outgoing JSON still
contains `"gpu_count": 0`. Unknown fields, missing CPU or memory, wrong
types, zero CPU or memory, negative values, and overflow are rejected.

The same budget is applied to every public output binding of a
multi-output Function declaration. Existing metadata is retained, and
SQL-expression declarations or local execution reject the budget instead
of silently ignoring it. Node is unchanged because its current
`addColumns` surface only accepts SQL computed expressions and has no
Function declaration path.

This change does not implement fleet call concurrency limits or enforce
the budget in the runtime. The Enterprise server must support the nested
Function worker resource-budget contract before these values can act as
execution limits.

## Validation

- Python extension build passed.
- Remote Function declaration and local unsupported-path rejection each
passed.
- Canonical metadata round-trip passed, including omitted GPU decoding
to zero and explicit `gpu_count: 0` on encode.
- Function multi-output remote wire test passed with `--features
remote`.
- Repository-scoped Python type check passed with 0 errors.
- All runtime source blobs match the native-tested source. The
post-native deltas are one documentation example line and an existing
routing-test stub update for the new optional argument; the affected
parameterized test passed 3/3 against the existing extension build. No
private test runners are part of this PR.
2026-11-11 11:23:17 +08:00
Xuanwo c9d176ae7e feat: resolve latest Function versions (#4949)
## Problem

Function catalog lookup required an exact version string even though
callers
commonly need the latest currently visible immutable revision. On the
current
SDK, resolving that version required an out-of-band catalog query before
calling
`get_function`.

## Change

- Make Python `get_function(name)` default to latest while accepting
`int` or
  `str` for exact lookup.
- Normalize integer versions to their decimal string form and reject
`bool`
  explicitly. Existing string validation and errors remain unchanged.
- Thread `None` through the rebuilt PyO3 binding and route it to the new
Rust
  `Connection::get_function_latest`.
- Resolve latest through the existing catalog `DESCRIBE FUNCTION` path
without
`WITH (version = ...)`. The returned value is one exact immutable
version, not
  a mutable alias.
- Preserve the existing Rust exact-version API.

## Validation

On the designated EC2 host with the rebuilt extension:

- `test_local_function_catalog_operations_are_not_supported`: **1
passed**.
  This crosses the Python connection, rebuilt PyO3, and embedded Rust
  `NotSupported` boundary with the omitted-version call.
- Async and sync catalog-path tests: **2 passed**, 117 deselected.
- Remote SQL Function creation/description integration test: **1
passed**.
- An earlier Rust invocation with the wrong module path matched **0
tests**;
  it is retained as harness evidence and is not counted as a pass.

## Scope

This is the Function lookup SDK slice. It does not include the separate
atomic
rebind/publication server work tracked under ENT-3418.
2026-11-11 10:39:03 +08:00
Xuanwo 1e5b6deee3 fix(python): support legacy blob query bytes (#4947)
## Summary

- Let local plain queries over row-addressable legacy v1 and v2 blob
columns use `blob_mode="bytes"` without importing `pylance` through
pandas, batch, or Arrow conversion.
- Preserve aliases, filters, ordering, limits, null and empty payloads,
empty result schemas, and the caller's `_rowid` choice.
- Build the hidden row-id query separately so reusing an async builder
for lazy reads or non-blob projections cannot retain an automatic
`_rowid`.
- Keep byte reads closed for blobs inside lists, remote or namespace
pushdown, and query shapes whose result rows cannot be matched back to
the source. Blob v2 lazy auto-row-id behavior is unchanged.

Fixes #4645

## Validation

- 26 focused Python tests passed with the test subprocess blocking all
`lance` imports. The persisted fixture writes and reopens a storage-v2.1
table, then covers sync and async pandas, batches, and Arrow conversion
plus async builder reuse.
- The existing unsupported vector and hybrid reranker regressions now
assert the current eligibility diagnostic, and the legacy table-level
Arrow regression now asserts successful bytes, schema, values, and
hidden row IDs. Both focused tests passed.
- The runtime receipt records no installed `pylance` distribution and no
importable `lance` module. The Python-only change was verified with an
existing compatible 0.42.0b9 extension; the relevant query and fetch
APIs are unchanged between that source and this 0.42.0b10 base.
- Ruff format and lint pass for every changed Python file.
2026-11-11 10:08:13 +08:00
Xuanwo 4d22c0ce2b feat: expose Function refresh resource budgets in SDKs (#4951)
## Summary

- Add the canonical Rust `FunctionWorkerResourceBudget` and
`RefreshColumnOptions::resource_budget()` for explicit Enterprise
Function refreshes.
- Forward the nested budget through Python sync/async, PyO3, Rust remote
requests, Node NAPI, and TypeScript.
- Require positive CPU and memory values; normalize an omitted nested
GPU count to numeric zero; reject the nested budget when combined with
the top-level `gpu_count` shorthand.
- Reject `resource_budget` on LanceDB Embedded refresh paths instead of
silently ignoring it.
- Document the Rust, Python, and TypeScript APIs, including that SQL
servers currently require all three budget fields.

The Rust, Python, and TypeScript additions use `cpu_millis` /
`cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` /
`gpuCount`. The top-level GPU shorthand keeps its existing behavior and
remains mutually exclusive with the complete nested budget.

## Scope and compatibility

This change configures only the explicit refresh call. It does not
configure `add_columns` automatic refresh and does not introduce a
fleet-wide call-cap interface.

A LanceDB Enterprise server must expose the nested `resource_budget`
backfill contract. LanceDB Embedded continues to support its existing
expression refresh defaults and rejects the new complete budget
explicitly.

## Validation

Validated against the final signed product source:

- Rust: resource-budget remote serialization test `2 passed`; validation
test `1 passed`.
- Python: actual PyO3 build; remote-request matrix `22 passed`;
invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff
check and format clean.
- Repository-standard Pyright entry point: `0 errors`, within the
repository's configured include set.
- Node: native build and Biome lint passed; the existing remote-request
test `1 passed`, including fractional, negative, overflow, NaN, and
infinity nested GPU cases; the existing local-rejection test `1 passed`.
- TypeDoc generation and generated-document cleanliness passed.
- The native-tested product source differs from this head only in two
semantically equivalent Node range checks required by Clippy. Current CI
compiles and lints the final blobs. No dependency or lockfile changes
are included.
2026-11-11 10:05:52 +08:00
Lucia Cerchie d44f768920 docs: add generated agent prompt catalog section (UAX-192) (#4945)
Add an automated prompt catalog section to the agentic skills page. The
combined docs build fills its insertion marker from the evaluation
source, avoiding a manually maintained prompt list. Keep the evaluation
context and fixture guidance in this page so docs reviewers can edit it
independently of the generator. Existing workflow examples remain in
place.

Part of UAX-192.

Depends on https://github.com/lancedb/docs/pull/434; merge that docs
pipeline change first. Validation: combined assembly and Mintlify
validation pass.
2026-11-10 15:07:59 -08:00
Jack Ye 481e82293e fix(python): explain explicit blob reads and subset copies (#4940)
Unsupported remote blob materialization errors now point to row-ID
selection, fetch_blobs and fetch_blob_files. Document descriptor
structs, fetched bytes and Function BlobReader inputs, and show a
bounded subset-copy recipe that keeps the source unchanged between
selection and fetching.

Adds guidance assertions and copy coverage preserving payloads, empty
blobs and nulls.

No new blob materialization API is introduced.

Formatting applied; no local builds or tests were run. CI will execute
the regression coverage.
2026-11-10 08:24:32 -08:00
Lance Release 3851938b6e Bump version: 0.42.0-beta.9 → 0.42.0-beta.10 2026-11-10 08:27:20 +00:00
Xuanwo 13e4cf7e58 feat: expose Function publication intervals in refresh options (#4928)
Python refresh APIs currently reject `publication_interval_rows`, so
callers cannot request incremental Function output publication through
the SDK. Add the optional typed parameter through the sync/async Python
APIs, PyO3, and Rust `RefreshColumnOptions`, and serialize it as a
top-level Enterprise refresh option.

An omitted value preserves the existing wire request and whole-fragment
default. Positive values set a publication threshold; zero and negative
Python values fail before a request is sent. Embedded refreshes reject
the Enterprise-only option. Documentation describes a threshold rather
than an exact number of rows per publication.

Validation on the designated development host:
- A matching baseline build reproduced the missing public keyword
argument in both sync and async submission tests.
- The updated PyO3 build passed 54 focused Python tests; Rust payload,
invalid-option, and default-wire tests each passed. Rust formatting and
Python Ruff checks passed.
- The single signed delivery commit differs from that tested
implementation only in a Rustdoc correction describing Embedded
rejection; formatting passed again. No dependency pins or lockfiles
changed.

The matching SDK is prepared for the separate server validation of
failure/cancellation retention and missing-only refill. Those end-to-end
results are still pending and are not implied by these SDK tests.

Refs [ENT-3733](https://linear.app/lancedb/issue/ENT-3733).
2026-11-10 15:22:56 +08:00