Commit Graph
3586 Commits
Author SHA1 Message Date
Xuanwo 2499342fa1 feat(function): expose resolved environments and fresh lock requests (#5034)
Function registration clients cannot request fresh dependency resolution
or inspect the measured environment returned by the server. Add
`refresh_lock=True` to Python registration and preserve the requested
runtime and measured environments in Rust/Python Function version
readback. Registration forwards both `exist_ok` and `refresh_lock`;
defaults remain unchanged.

Single-platform versions retain `resolved_environment`.
Multiarchitecture versions expose `resolved_environments`, keyed by OCI
`os/architecture`, and the Python `environment_report` alias. Platforms
are never flattened: absent entries mean unknown measurements, while
measured empty package lists remain present. Registration waits emit
scalar and per-platform server warnings once per Job; catalog reads do
not repeat them.

Requires matching server support, including Sophon #10344 for
per-platform publication. Nine focused Python source-overlay tests and
29 documentation tests passed for the additive map change; one
documentation test requiring a newer native `exist_ok` binding was
excluded. These checks reuse a prebuilt extension and do not establish
current native integration. PR CI exercises the Rust transport/readback
and Python bindings. Live multiarchitecture registration and GPU
acceptance remain separate.
2026-11-14 22:30:28 +08:00
Xuanwo ea59f8acb6 feat(python): preserve Function timestamp and duration types (#5032)
Function declarations currently reject exact timestamp and duration
types. Preserve their Arrow units and timezones in canonical signature
JSON, including fixed-offset timezones and nested fields. Bare
`datetime` and microsecond/no-timezone declarations keep their existing
representation; `timedelta` requires an explicit unit.

Compound temporal signatures require the Lance Namespace decoder in
https://github.com/lance-format/lance/pull/11059 and corresponding
server validation support. Older readers reject these signatures; this
change does not upgrade server support or dependency pins.

Validation: 128 Python signature tests and 30 documentation tests passed
(one skipped), using the changed Python source with an existing native
extension. No Rust rebuild or end-to-end Function registration/refresh
was performed.
2026-11-14 21:49:45 +08:00
Xuanwo 12b95b59ab feat(function): support literal inputs in column declarations (#5021)
Function-column declarations now preserve constants such as
`fn(video=col("mp4"), num_frames=lit(16))` without changing its Function
version. Constants travel in the application inputs; column bindings,
request input schemas, and dependency tracking contain only columns.
Changing a constant uses a new column binding rather than rebaking the
Function.

Persisted bindings with literals use metadata version 2, which older
readers reject. Column-only bindings retain version 1. Python preserves
scalar literal identity, including null and unsigned integers, and
rejects composed expressions or values outside the canonical literal
domain.

This SDK change requires matching server declaration and execution
support, after a dedicated SDK dependency update. Native validation
passed 48 Rust metadata tests, four request-transport cases and 57
rebuilt Python tests on `a42d680`; 32 pre-existing experimental tests
remained ignored. The test-assertion correction passed its exact
regression and strict Clippy on `53ff80a2`. On `993fae7d`, the three
remote metadata compatibility regressions, native Python public-export
test and strict Clippy passed. These review corrections use v3 for
unknown-version rejection, cover existing v2 literal bindings through
remote mutation guards, and export `LiteralInputBinding`. The unchanged
native implementation and verified extension are reused. Matching server
execution and original-workload acceptance are not established by this
PR.

Refs ENT-3667.
2026-11-14 20:59:52 +08:00
Xuanwo f6f8119ba4 feat: expose structured Function refresh options in SDKs (#5047)
## Problem

ENT-3267 still lacks typed refresh controls for `supervisor_resources`
and `planning_workers` in Python, Rust, and Node, and Node has no
`retry_policy` refresh option. Callers must drop to SQL for those
controls.

## Change

- Add canonical Rust `FunctionRefreshPodResources` and
`FunctionRefreshPlanningConfig`, and serialize them through
`RefreshColumnOptions`.
- Expose both options in the Python sync/async wrappers and PyO3
binding.
- Expose both options in the Node NAPI object and add a Node
`retryPolicy` object backed by the existing Rust `FunctionRetryPolicy`.
- Omitted options stay omitted; branch targeting continues through the
table handle. No server, dependency, or pin changes are included.

Client-side checks only reject non-positive structural values.
Deployment resource limits and column-kind validation remain
server-side.

## Validation

`cargo fmt --all -- --check`, `python3 -m py_compile`, and `git diff
--check` pass. No native build was run locally; CI compiles and tests
the change. Generated JS API pages are unchanged.
2026-11-14 20:59:44 +08:00
Xuanwo 4843ac4c7e fix(remote): observe new publications while waiting for Function freshness (#5033)
Remote `wait_fresh()` could keep observing a cached table version after
automatic refresh published new outputs. Each poll on a latest handle
now requests a current snapshot, so the wait can finish without an
external `checkout_latest()` call. Explicit version, tag, and
query-snapshot selections remain pinned.

The wait preserves existing finalization-pending and error handling and
continues to perform read-only status requests.
2026-11-14 20:59:41 +08:00
Xuanwo e5a5b67212 feat: support nullable scalar Function declarations (#5030)
Function declarations currently reject nullable scalar returns. Accept
`Optional[T]`, `T | None`, and explicit nullable Arrow fields,
preserving `nullable: true` in registration signatures, JSON round
trips, and remote `add_columns` requests.

The Rust planner accepts nullable scalar outputs, and binding readback
recognizes the server-owned assignment marker for a single nullable
scalar result. Clients do not allocate that marker. Non-nullable whole
results, whole structs, and Blob v2 retain their existing assignment
restrictions.

Validated with 127 Python tests, 30 documentation tests (1 skipped), and
seven focused Rust tests covering planning, binding readback, rejection
boundaries, and the remote request body. The final condition-collapse
change addresses strict Clippy; its delta check and CI are pending.
Python tests use a source overlay on a retained native extension, and
Rust remote coverage uses HTTP mocks.

Server support and end-to-end validation remain separate. This PR does
not claim that server publication distinguishes computed NULL values
from pending rows.
2026-11-14 20:59:20 +08:00
Xuanwo 005b33fa8b fix: preserve terminal server identity in failed job waits (#5024)
A failed remote Function job loses its persisted build identity when
converted into the exception from `job.wait()`. Append
`result.terminal_server` to that error using the existing server-build
format, preserving its message, row location, traceback and job ID. Do
not substitute the responding query node's version for the writer of the
saved terminal result; missing or malformed historical identity leaves
the error unchanged.

The reported failure was independently reproduced against a deployed
server with a real Function job. The new exact Rust regression
reproduces the missing terminal identity on the unpatched code. At
`59c1f3e`, all 10 remote Job Rust tests and five tests using the
matching native Python extension passed, including sync/async waits with
known or unknown terminal identity. These transport tests use mocked
HTTP responses; the original deployed Function job remains with
independent acceptance for recheck after delivery. This change does not
establish whole-ticket acceptance.
2026-11-14 20:59:10 +08:00
Xuanwo 0d55457d9e test(node): make blob read cancellation deterministic (#5029)
The Node blob cancellation test assumes a 32 MiB local read is still in
flight when JavaScript immediately aborts. The native future can finish
first, in which case returning its completed buffer is valid.

Replace that timing assumption with the existing HTTP fixture: send one
byte of a five-byte range response, hold back the remaining body, then
abort. Verify `AbortError`, response disconnection, listener cleanup,
and a subsequent read on the same handle with the single request permit.
Existing local pre-abort, successful-read, cursor, and close coverage
remains in place; production cancellation semantics are unchanged.

Validated the 97 scoped Blob tests on Linux x64 / Node 22.23.4 with both
a locally built beta18 binding and the [beta19 native binding built for
#4967](https://github.com/lancedb/lancedb/actions/runs/43381450237/job/131739908253).
The beta19 run used the published repair tests at `8096f848`, passed all
97 cases with exit 0, and verified the downloaded artifact digest,
loaded binding hash, and native build checkout against #4967's source
tree. The pre-existing `CustomGC` import diagnostic remains; the process
exited normally.

The [hosted macOS ARM64
job](https://github.com/lancedb/lancedb/actions/runs/43395339041/job/131784902005)
also passed the cancellation regression and the complete applicable Jest
suite: 1,695 tests passed and 22 were skipped on Node 24.22.2 / beta18.
Its actual merge checkout `e906a2da` has matching test, native Blob,
dependency, and workflow sources to PR head `8096f848`.

A separate [macOS ARM64 / Node 22.23.4 / beta19 composition
run](https://github.com/lancedb/lancedb/actions/runs/43559120836/job/132353156252)
now passed 1,694 tests (23 skipped), including both in-flight
cancellation and later-response-body cancellation/permit release. It
tested the normal merge of #4967 `6b175764` and this repair `8096f848`,
at validation head `144e5294` whose only subsequent change narrows the
workflow and disables publication. The loaded native binding hash and
downloaded artifact digest were captured. The original failed #4967 job
is preserved; this separate validation does not replace normal
prerequisite integration or current-branch CI on #4967.
2026-11-14 20:57:49 +08:00
Xuanwo 56c4b3a6f2 ci: stabilize macOS Node toolchain and build budget (#5017)
The macOS Node job installed floating stable before restoring the Rust
cache, although native builds use the repository-pinned compiler. Use
the existing file-aware Rust setup action with its implicit cache
disabled so setup, cache selection and compilation use the same pin and
one cache owner. Preserve the existing native build flags.

Allow 35 minutes for the complete macOS job. In the [pinned-toolchain
cold
run](https://github.com/lancedb/lancedb/actions/runs/43291714333/job/131439350321),
native compilation finished in 28m02s, but setup and packaging exhausted
the former 30-minute job limit before TypeScript checking and Jest.
Retained full Jest runs took 38–59 seconds; the new bound covers the
measured work with a small margin. The runner, build profile, features,
targets and complete test commands are preserved.

Hosted acceptance at `ff718c03`: the [complete macOS
job](https://github.com/lancedb/lancedb/actions/runs/43300424344/job/131469805138)
succeeded within the 35-minute limit (20:27:07–20:56:58 UTC). Native
compilation took 27m38s; the complete applicable Jest suite passed 1,695
tests, with 22 skipped, in 46.129s. This single successful run does not
guarantee every cold-build duration, validate the separate beta19
composition, or establish the cause of the DynamoDB timeout.
2026-11-14 20:55:23 +08:00
Xuanwo 193f203b72 ci: run remote Functions gate only in the full test suite (#5022)
The remote job runs the default local Functions gate twice: once before
building the remaining test targets, and again in the full remote suite.
Run it only through the existing full suite, removing the serial Cargo
invocation barrier while retaining the same features, authentication
wrapper, targets, and 30-minute deadline.

The [hosted remote
job](https://github.com/lancedb/lancedb/actions/runs/43302507952/job/131477104217)
at `7886d79e` completed successfully in 25m36s within the unchanged
30-minute limit. Its full suite passed 2,164 unit, 114 integration, and
120 doctests, including the three default gate tests. This single run
establishes completion for that workload, not a general timing
improvement or cold-build guarantee. The branch now includes main's
Cargo artifact fixture repair (#5010). The workflow, full-suite command,
authentication wrapper, three default Functions gate tests, and
30-minute deadline are unchanged from that hosted run. The 2,398-test
result remains evidence for `7886d79e`. On the updated head `810d6558`,
the [hosted remote
job](https://github.com/lancedb/lancedb/actions/runs/43382023177/job/131741745645)
completed in 17m58s with 2,171 unit tests, 119 integration tests, and
122 doctests passing (82 ignored), including all three default Functions
gate tests. This is current-head execution evidence, not a controlled
speedup comparison.
2026-11-14 20:54:19 +08:00
Xuanwo beda423318 fix(node): bind blob byte queries to their response snapshot (#5019)
TypeScript remote `toArrow` and `toArray` could combine scalar values
from query snapshot A with Blob payloads from a later snapshot B. Cached
table schemas and native descriptor layouts could also leave descriptors
in a `blobMode: "bytes"` result.

Discover Blob fields and snapshot provenance from the executed query
stream, recognizing canonical native v2 descriptors as well as
extension-marked fields. Fetch payloads through an independent Rust
snapshot handle bound to the query version and immutable dataset
identity, without changing the live table. Preserve aliases, nested
fields, NULL/empty values, explicit row IDs, and scalar-only results.
Hybrid byte queries require matching provenance from both actual
responses. Missing or mismatched server provenance fails explicitly;
logical-input validation and local materialization remain unchanged.

Manual integration validation at `4daf354df`: all five real SDK/NAPI →
Sophon REST scenarios passed. Both `toArrow` and `toArray` retain A
bytes across overwrite and reject same-name/same-URI recreation even
when version and row IDs are reused. A previously cached handle
discovers a subsequently added Blob field. Ten actual payload fetches
include the two expected identity rejections; later queries on the same
live handle read B. The canonical-descriptor and Blob regression suites
pass 119 tests, with independent source and wire-evidence review.
Earlier broader SDK and hybrid evidence is retained within its
unchanged-source scope.

This is local REST interoperability evidence, not deployed
Cloud/header-forwarding acceptance. Remote bytes require the
authoritative server version/identity and fetch-echo contract. CI and
formal review remain merge requirements; this follow-up does not close
ENT-3556 or Blob V2 GA.
2026-11-14 20:52:50 +08:00
Xuanwo 2f496322ef fix: report multipart rows only after part acknowledgement (#5020)
Remote multipart progress currently counts every scanned row before the
corresponding IPC part is acknowledged. A rejected first part can
therefore report uploaded rows, even though its acknowledged byte count
remains zero.

Count both rows and IPC bytes only after each part succeeds, including
parts assembled from multiple batches or split from one batch. Retried
parts count once. Native and single-request writes retain their existing
accounting; acknowledged multipart progress describes staged data, not a
successful commit.

Validation follows the actual TCP upload path: one scanned batch becomes
six decoded IPC parts, the sixth response is lost, and each
request/retry observes only previously acknowledged rows and bytes.
Rejection and complete-outcome cases preserve the same accounting
boundary. This validation covers local HTTP behavior; deployed service
throughput and the original disconnect investigation remain outside its
scope.


Frozen candidate validation: 839 remote tests and 7 progress tests
passed. Scoped Clippy (`lancedb`, remote feature, lib/tests/examples,
warnings denied) passed with unchanged source. Python/Node bindings were
unchanged and were not rebuilt for this patch.
2026-11-14 20:51:25 +08:00
Lance Release 5de76bd8ed Bump version: 0.42.0-beta.12 → 0.42.0-beta.13 v0.42.0-beta.13 2026-11-14 12:33:27 +00:00
Yang Cen e1a10eaa66 chore(deps): upgrade Lance to v15.0.0-beta.21 (#5046)
Upgrade the Rust Lance family from beta.18 to the published
v15.0.0-beta.21 tag, and align Python test wheels and Java artifacts.
This includes the explicit-empty-segment FTS fix in
lance-format/lance#11063 needed by lancedb/sophon#10314. Enable Windows
Git long paths so Cargo can check out Lance fixtures in the Rust, NPM
and Python wheel builds.

Beta.21 also contains lance-format/lance#9858: new NGram, RTree and
MinHash LSH indexes use row addresses. Legacy row-ID segments remain
readable, but index maintenance on stable-row-ID tables can require a
rebuild. The NGram documentation now explains the consolidation
requirement; NGram/RTree tables without stable row IDs retain their
earlier format. The Python lock revision and unrelated dependency
entries are unchanged.

Validation:
- The official Lance release and internal mirror match
80bc45a195279befdec42853dac8d15a37e0504d. The seven official Fury wheels
and Maven POM/JAR were verified.
- Exact executable/dependency state 7846a15e completed the full Rust,
Python, Node, Java and CLI CI, including platform builds and package
publication dry runs. Xuanwo approved that head; Gatekeeper recommended
approve with a non-blocking qualification risk.
- Follow-up a289cb2c only adds the requested seven-line NGram upgrade
note. Static whitespace checks passed; new-head CI is running. No local
correctness tests or test builds were run by this task owner.
- Broader IVF/encoding/Blob/cache workload performance and the beta.21
Sophon composition remain separately qualified. Earlier Blob
cancellation and DynamoDB failures on #4967 are retained in their
original evidence; this PR does not count those historical failures as
fixed.

Current-head CI, full feedback and required review still gate merge. SDK
publication is a separate action. Sophon adoption follows a published
SDK release and mirror through its own dependency PR; its distributed
performance acceptance remains in #10314.
2026-11-14 20:22:13 +08:00
Jack Ye dd449978d9 docs: clarify Function workflows and search contracts
Clarify Function signatures, refresh and recovery workflows, job snapshots, FTS operators, and IVF deduplication limits.
2026-11-14 01:09:40 -08:00
Xuanwo 97e62dc11b feat(remote): expose canonical job statistics reports (#5016)
Expose `Job::stats()` in Rust and synchronous/asynchronous Python jobs.
It reads `POST /v1/jobs/stats` and returns the canonical server-rendered
report verbatim, or `None` for a running job or one without a readable
durable profile. It does not wait for completion or recompute
measurements.

On `cc25005525671756b5cf5ebd96492b792e9aa1cb`, all three Rust transport
tests and six Python tests passed against the matching rebuilt native
extension; strict SDK Clippy also passed. These tests cover optional
reports, exact text preservation, fresh reads, errors, and the HTTP
boundary with a controlled server. Against the actual Phalanx HTTP route
from [Sophon #10247](https://github.com/lancedb/sophon/pull/10247)
(`5be3236`), the same native extension also completed two sync and two
async reads after durable registry archival, preserving the exact
canonical report. This is an HTTP/registry integration fixture, not a
Function workload or shared-cluster acceptance. The private runner's
JSON field-order collector failed after the successful probe; all raw
evidence was retained and hash-verified, and the original receipt was
recovered with the corrected parser without rerunning the probe.

The follow-up `ad36f66` only corrects the synchronous Python
API-reference link to the asynchronous method. Ruff checks and
formatting passed; runtime evidence above remains applicable. The actual
Phalanx/native-SDK probe covers archived reports; running-job null
availability is covered by separate backend and SDK transport tests, not
that end-to-end probe.

Part of ENT-3501.
2026-11-13 23:41:33 -08:00
d67ec47921 docs: make the landing page concise (#4898)
Shortens the docs landing page and organizes its overview around
database features, with the multimodal data model, ML workflow, SDKs,
and PyTorch loader explained in prose. Keeps versioning visible,
branching in draft, and the existing `{#architecture}` anchor.

Validation: added documentation links and anchors verified; `git diff
--check` passes. `mint validate` reports the same four pre-existing
missing snippet-export warnings in the reranking and vector-search
pages, also reproduced on the unchanged revision.

Closes
https://linear.app/lancedb/issue/DOC-80/strengthen-what-is-lancedb

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
2026-11-13 20:23:41 -08:00
Jack Ye f6311ae7e9 feat: support exist_ok for index creation (#4993) 2026-11-13 19:21:50 -08:00
Jack Ye 81d04b8c36 feat(python): support exist_ok for materialized views (#4991) 2026-11-13 19:21:29 -08:00
Jack Ye 4db0d5c7f5 feat(python): support exist_ok when cloning tables (#4990)
Adds `exist_ok=False` to Python table cloning and `.exist_ok(true)` to
the Rust clone builder. An existing target is returned with its data and
version history unchanged; strict duplicate errors remain the default.

```python
snapshot = db.clone_table("snapshot", source_uri, exist_ok=True)
```

Embedded cloning skips source reads for an existing target and handles
typed creation conflicts. Remote cloning forwards the option to the
server. Includes sync/async preservation and remote request coverage,
plus SQL syntax documentation for `CREATE TABLE IF NOT EXISTS target
CLONE source`. Requires a server supporting the option for remote
clones.

BREAKING CHANGE: Rust `CloneTableRequest` struct literals must
initialize the new `exist_ok` field; `CloneTableRequest::new` remains
compatible.

No local builds or tests were run, as requested.
2026-11-14 10:48:26 +08:00
Jack Ye a0eb6b4b71 feat(python): support exist_ok for storage integrations (#4986)
Adds `exist_ok=False` to sync and async Python storage integration
creation. `exist_ok=True` sends `CREATE STORAGE INTEGRATION IF NOT
EXISTS` and returns the server's stored configuration, preserving the
prefix, credentials and external ID.

```python
integration = db.create_storage_integration(
    "raw", storage_provider="S3", storage_prefix="s3://customer-bucket/raw/",
    aws_role_arn="arn:aws:iam::123456789012:role/access", exist_ok=True,
)
```

The non-exhaustive Rust request also supports `.exist_ok(true)` for
every provider. Requires a SQL server supporting this syntax. Includes
protocol and provider SQL coverage. No local builds or tests were run,
as requested.
2026-11-14 10:37:08 +08:00
Jack Ye 2c3c30dbdf feat(python): support exist_ok for Function creation (#4992)
Adds `exist_ok=False` to Python Function creation and job submission.
`exist_ok=True` sends `CREATE FUNCTION IF NOT EXISTS` and returns the
stored immutable version, preserving its code, credentials and execution
settings. The default retains the current version-creation behavior.

```python
version = db.create_function(analyze_caption, exist_ok=True)
```

The option also exists on the Rust registration request and is kept out
of the canonical Function definition payload. Includes
sync/async/background protocol and stored-version coverage. Requires
matching SQL server support.

BREAKING CHANGE: Rust `FunctionRegistrationRequest` struct literals must
initialize `exist_ok`; legacy serialized requests default to false.

No local builds or tests were run, as requested.
2026-11-14 10:35:54 +08:00
Jack Ye b061c3627c feat(python): support exist_ok when creating views (#4984)
Add `exist_ok=False` to synchronous and asynchronous Python view
creation. Passing `True` sends `CREATE VIEW IF NOT EXISTS` and returns
the existing view's stored query and schema unchanged; the default
continues to fail on an existing name.

```python
view = db.create_view("recent_orders", query, exist_ok=True)
view = await async_db.create_view("recent_orders", query, exist_ok=True)
```

Requires a SQL server supporting `CREATE VIEW IF NOT EXISTS`. Rust gains
an awaitable builder with `.exist_ok(true)`. Existing
`.create_view(...).await` calls continue to work; callers requiring
`Future` must convert the builder, and custom `Database` implementations
must accept the new `exist_ok` argument. Includes Python round-trip
coverage and Rust coverage for direct/deferred results and metadata
preservation.
2026-11-14 10:35:52 +08:00
Will JonesandClaude Opus 5.5 7d50d2d28f chore(deps): bump Rust deps flagged by security scans (#5035)
## Summary

Bumps the Rust dependencies flagged by Aikido that have a
semver-compatible fix available. Supersedes #5009 (rustls in the docs
lockfile is included here).

Root `Cargo.lock`:

| Crate | From | To |
| --- | --- | --- |
| mio | 1.2.0 | 1.2.5 |
| smallvec | 1.15.1 | 1.16.5 |
| thread_local | 1.1.9 | 1.1.10 |
| tinyvec | 1.11.0 | 1.13.3 |
| zerocopy / zerocopy-derive | 0.8.48 | 0.8.63 |
| event-listener | 5.4.1 | 5.4.2 |
| arrayvec | 0.7.6 | 0.7.8 |
| wasm-bindgen (+ js-sys, web-sys) | 0.2.121 | 0.2.130 |
| getrandom 0.4 | 0.4.2 | 0.4.3 (drops wasip3 → wasmparser 0.244) |
| brotli | 3.5.0 | 9.1.1 (via `random_word` dev-dep 0.4.3 → 1.0.0) |

`docs/web-tests/rs/Cargo.lock` gets the same crates plus rustls, hyper,
h2, moka, bytes, crossbeam-epoch and anyhow. That crate is not a
workspace member, so cargo refused to resolve it; it now has an empty
`[workspace]` table.

Still open, blocked upstream: quick-xml 0.41 (lance-io), rand 0.9
(datafusion/lance/candle), heapless 0.8 (rstar via geo), paste,
unsafe-libyaml (rattler_conda_types, lindera), fast-float/smartstring
(polars 0.39), number_prefix (needs hf-hub 1.0 migration), zstd 0.13
(arrow-ipc).

## Merge Risk

**Door:** two-way. Lockfile patch/minor bumps and a dev-only dependency.
**Blast Radius:** none expected; no source changes. The docs crate is
not built in CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:44:40 -08:00
Taylor 3a980054a1 docs: standardize Realtime Engine naming (#4969)
The Enterprise feature name was inconsistently written as "Real Time
Engine" and "real time engine". Standardize it to **Realtime Engine** in
eight website guides, including page and sidebar titles, descriptions,
prose, link labels, cards, and search keywords.

The remaining MemWAL references describe Lance's open source format or
literal API/job identifiers, including `memwal_compaction` and
`memwal_catchup_<type>`, and are preserved. Only `docs/web/*.mdx` pages
change; SDK source, generated API reference, code examples, URL
destinations, and section anchors are unchanged.

Validation:
- `git diff --check` passes.
- Reviewed all MemWAL and Realtime Engine references under `docs/`.
- Verified each changed file differs only by the intended naming
substitutions and preserves code blocks, inline code, links, anchors,
and MemWAL identifiers.
- Confirmed no split or hyphenated "real time engine" names remain in
website guides.
- No snippets needed regeneration because code examples are unchanged.
Vale and Mintlify are not installed locally, so their checks were not
run locally.
2026-11-13 11:24:55 -08:00
Jack Ye 1765747f91 fix: allow updates to Function input columns (#4985)
Updating an input column on a table with a Function binding currently
fails before the write, although the write path already preserves
freshness for bindings whose inputs are unchanged. Allow input updates
through the normal update builder while retaining supported
binding-metadata validation and the guard against writing computed or
internal assignment outputs directly.

Lance rewrites only the matched rows. Their affected Function outputs
keep their previous values and their changed input signatures make them
eligible for refresh; unchanged fragments retain their freshness.
Rewrites affecting only unrelated inputs keep the existing
freshness-preservation path.

For a table whose Function reads `title`, this update now succeeds while
retaining the previous computed values until refresh:

```python
table.update(where="id = 7", values={"title": "revised"})
```

Server consumers need this crate update before remote input updates gain
the behavior.

Adds regressions for a one-row input update, retained output values,
freshness scope, independent binding preservation, and atomic rejection
of mixed input/output writes. Local builds and tests are disabled by
request; CI runs the regressions.
2026-11-14 03:02:07 +08:00
Bruno Ramirez 927b243948 fix(remote): preserve branches in cache prewarm requests (#5012)
Calling `prewarm_data` or `prewarm_index` on a checked-out remote branch
currently targets main. Both methods omit the branch context, which can
warm the wrong snapshot, miss a branch-only index, or fail when the
branch's minimum read version is ahead of main. This PR carries the
table handle's branch through the existing request helpers. Main
requests retain their existing format, with no public API changes.
2026-11-13 09:38:50 -08:00
Jack Ye bab22a3fbd feat(python): bundle Function package resources (#4987)
A package shipped through `code=` currently loses its non-Python files.
This breaks vendored packages that read JSON configuration, manifests,
prompts, or binary resources after deployment. Include regular package
resources in a separate base64-encoded `python_bundle.resources` map,
preserving exact bytes and relative paths. Both source and resources
contribute to the artifact digest. UTF-8 Python source bytes retain
their original line endings as well, so manifests that hash vendor code
continue to match.

For example, `@function(code=[my_package])` can use
`importlib.resources.files("my_package").joinpath("models.json").read_text()`
on the worker. Hidden files and caches are excluded; symlinks and unsafe
paths warn instead of following files outside the package. Source-only
bundles retain their existing wire form. Resource bundles require
matching server support.

Adds an isolated-import regression with a vendored subpackage, JSON
metadata, binary data, SHA256 verification, preserved CRLF bytes, and
resource-only version changes. Local builds and tests are disabled by
request. Ruff formatting and linting run locally; CI runs the tests.
2026-11-13 16:21:58 +08:00
Xuanwo db098ba812 fix(ci): support Cargo 1.100 artifact layout in cleanup fixture (#5010)
Cargo 1.100 moves intermediate compiler artifacts into per-unit build
directories and emits `build_script_build` instead of
`build-script-build`. The unlocked-build fixture currently fails on its
old rlib path before the SDK build starts, even though the library
survives cleanup.

Discover compiler inputs recursively and accept both build-script names
while preserving the library, generated-input, ordinary-executable,
discarded-test, linker-failure and outside-target assertions. The
standalone fixture reproduces the failure before this change and passes
afterward on Rust/Cargo 1.100.0; it also passes on 1.97.0. Fault
injection confirms missing libraries and other retained inputs still
fail their assertions. Production cleanup and normal workflow coverage
are unchanged.
2026-11-12 20:34:31 -08:00
Jack Ye 9302287bed fix: accept defining queries in Python materialized views (#4980)
Python materialized-view creation now accepts a complete defining SQL
query across sync, async, remote, and namespace connections, including
creation jobs:

```python
view = db.create_materialized_view(
    "events_by_kind",
    query="SELECT kind, count(*) AS n FROM events GROUP BY kind",
)
```

`query` is mutually exclusive with `source`, `select`, `where`, and
`limit`. Existing source-based calls remain supported.

Adds Rust `Connection::create_materialized_view_from_query`: embedded
connections plan through `MaterializedViewDefinition::from_sql`, while
remote connections submit the full query for server-side parsing.
Includes regression coverage for grouping, UNNEST, query forwarding,
creation jobs, and argument validation.
2026-11-12 14:57:00 -08:00
Xuanwo f863ea5e08 feat(rust): support cross-table blob copies in DataFusion (#4976)
`INSERT INTO target SELECT ... FROM source` currently cannot copy Blob
v2 descriptors into an independent table. This enables copies from a
single native source snapshot through projections, filters, sorting, and
limits, while preserving descriptor queries such as `SELECT
payload.size`. Source-owned payloads become target-owned data, so
subsequent source mutation, compaction, or deletion does not invalidate
the copy; nested fields retain their named identity across metadata
differences, child reordering, scalar casts, and nullable parents.

External blobs remain URI/range references under the destination's
existing authorization policy: a destination that independently
registers the matching external base accepts the copy, while an
unregistered destination rejects it without committing. Source base
registrations are not transferred, and
`allow_external_blob_outside_bases` is not relaxed. JOIN, UNION,
aggregation, and computed descriptors are rejected during planning. This
addresses the DataFusion cross-table copy portion of OSS-696;
remote/cloud execution and Realtime/MemWAL remain outside this support
boundary.

Local acceptance covers 15 passing DataFusion Blob integration cases,
including byte reads after physical source-directory deletion,
authorized and unauthorized External ranges, and a source-read failure
after an earlier partition has written an uncommitted data file, with
both the live destination and an independently reopened manifest
unchanged. Existing regression evidence also includes 195 DataFusion
unit tests and 59 Blob integration tests. Rust formatting and whitespace
checks passed. Remote-feature builds, cloud/S3 execution, clippy, and
the full workspace suite were not run.
2026-11-13 06:34:38 +08:00
Xuanwo cababaecf0 feat(remote): discover server builds and retain error provenance (#4999)
Remote callers can use `db.server_info()` in Rust and Python, including
synchronous and asynchronous Python connections, to identify the
responding server without submitting a query or Job. Typed results
describe the query-node build, component versions, Function protocol and
configured images. Older servers without the endpoint return an explicit
unsupported error.

REST and Flight errors retain the reported build in diagnostic text and
Rust `RemoteErrorMetadata.server`. Typed table/job-not-found errors
preserve their HTTP source and classification, including the
schema-cache path; masked REST errors can obtain provenance from the
response header. Unknown build information stays absent.

The matching native Python SDK was exercised against a real localhost
server (`8829b074` client / `43920fa` server): sync/async discovery,
both SQL version forms and actual REST/Flight error provenance passed.
The product files match this branch; private validation drivers are
excluded. Focused coverage includes typed 404 errors, masked errors, and
the public schema-cache regression. Deployment and published-package
acceptance remain separate.

Server counterpart: https://github.com/lancedb/sophon/pull/10234.

Related: ENT-3565.
2026-11-13 06:22:35 +08:00
Xuanwo 6d24446a30 feat: add negotiated batched blob range reads across SDKs (#4994)
Existing Rust and Python blob range calls must keep working when only
the SDK is upgraded. Add negotiated batched range reads across Rust,
Python sync/async and TypeScript, preserving ordered NULLs, duplicates
and branch/version reads with splitting and reassembly at 1,024 ranges
or 64 MiB per request.

Only a successful describe response advertising `x-lancedb-blob-ranges:
1` selects the batched endpoint. Missing or unknown capability values
select the existing descriptor-query plus BlobFile Range GET route
before the operation. Endpoint failures never switch protocols. HTTP 400
preserves the server message and request ID as Rust `InvalidInput` /
Python `ValueError`; other errors retain their existing semantics.

The operation binds both numeric version and immutable snapshot identity
from its producing query or first range response. Every later partition
and oversized-range piece sends that original identity and requires an
equal response identity before decoding. Same-name table recreation
therefore fails explicitly instead of returning mixed old/new payloads;
no partial array is returned on later failure.

Targets `main`. The read-buffering prerequisite #4979, safe-write
prerequisite #4977 and deferred-query identity prerequisite #4973 are
merged into main; #5000 also supplies the merged SQL timeout-test
prerequisite. Server support is [Sophon
#10219](https://github.com/lancedb/sophon/pull/10219), including #10211.

Exact SDK `1497b508324662180086a4d1e0a6eb9e0b3970db` and server
`f39712c2c150b220eb831e00f6663c9eef32dce5` passed a real local SDK →
REST → PE experiment: one Rust harness test, 14 assertion groups, 38
REST range requests and 34 PE RPCs. All four sync/async × warm/cold
table-recreation cases failed explicitly before the second descriptor
lookup or PE dispatch, with no mixed payload or partial result. The
other groups retained 1,025-range success, 70 MiB reassembly,
branch/version reads, overwrite snapshot consistency, error behavior and
TypeScript buffer ownership. Bindings were rebuilt and their loaded
hashes verified. This is local interoperability evidence, not Cloud,
mixed-fleet or deployed header-forwarding acceptance.

The server dependency upgrade, External/cache/URI-policy acceptance and
validated rollout remain required. Enforcing PE support must precede QN
and SDK enablement; identity does not retain versions or freeze External
objects. This PR does not claim deployed support or complete ENT-3100/GA
acceptance.

Published documentation retains the Early access badge and released
Enterprise instructions, including explicit fetches for copying. The new
Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example,
future streaming restrictions and GA wording remain source-only MDX
drafts until release. The correction distinguishing released Enterprise
SQL bytes from SDK descriptors remains visible. Documentation validation
checked comment boundaries, visible instructions, Python example syntax
and whitespace; the site was not built. The main integration keeps these
release gates unchanged.

Main integration at `0491d9bdcc65ce5d52ea93e1083c89e951ea4f9d` normally
merges main `9b13e8f72530c8b1950c09685f6920711154a539`, including its
Lance beta.18 dependency. Focused current-source validation passed: 493
remote-table/query/Blob/IPC/write-progress Rust tests, 23 shared
create/write-body tests, 59 native Blob integration tests and 7 public
HTTP range tests. Rebuilt Python passed 468 selected tests (one optional
PyAV test skipped), and rebuilt TypeScript passed 194 selected tests
across four suites; lint/format, TypeScript compilation and API
documentation generation passed. Shared tests include actual HTTP
multipart failures, acknowledgment boundaries, dictionary compaction and
native Blob compaction. The 598 unrelated Jest cases excluded by the
focused shared-suite selector were not run.

The retained 14-group joint REST/PE experiment used the earlier exact
candidates stated above. Operation-identity/range/IPC code is unchanged,
but dependencies and shared write interfaces changed with main; the new
local checks validate the SDK combination and do not constitute a fresh
final-head SDK/server joint, Cloud or rollout run. Early access and
every remaining server/External/cache/policy/dependency/rollout
condition remain in force.

CI snapshot on this head (2026-11-12 18:30 UTC): `build-no-lock` failed
in the standalone `ci/tests/test_build_no_lock.py` fixture at the
retained-rlib assertion, before the actual SDK no-lock build. The
fixture, helper and Rust workflow are byte-identical to main; the
fixture does not load SDK source. The underlying CI/environment cause is
not diagnosed here. Other jobs remained pending/running at the snapshot,
so CI is not reported as passing. Failure:
https://github.com/lancedb/lancedb/actions/runs/43280277756/job/131399418284.
2026-11-13 04:03:04 +08:00
Will JonesandClaude Opus 5.5 1806dc2323 ci(java): publish beta releases to Fury instead of Maven Central (#5011)
## Summary

Java publishing has failed since `v0.42.0-beta.10` because the `lancedb`
Sonatype org hit Maven Central's monthly release-count limit:

```text
Publishing blocked: your organization has reached its monthly release count limit. Grace period ended 2026-11-08. Usage resets 2026-12-01.
```

This sends beta tags to `https://maven.fury.io/lancedb/` instead,
matching how Python publishes betas to `pypi.fury.io/lancedb`. Stable
releases still go to Maven Central.

```text
tag contains "beta" -> setup-java server "fury"  -> mvn deploy -P deploy-to-fury
otherwise           -> setup-java server "ossrh" -> mvn deploy -P deploy-to-ossrh
```

- `java/pom.xml`: new `deploy-to-fury` profile that points
`distributionManagement` at Fury.
- `java/README.md`: explains how to add the Fury and `maven.lance.org`
repositories to use a beta.

Requires a `FURY_TOKEN` secret with push access to the `lancedb` Fury
account. The token is passed as the Maven username with a placeholder
password. That follows Gemfury's convention but hasn't been tested
against Fury yet.

## Evidence

Running the beta path locally (`deploy -pl lancedb-core -am -P
deploy-to-fury`) against a file repository deployed `lancedb-parent`
(POM) and `lancedb-core` (JAR, POM, sources, javadoc).
`help:effective-pom -P deploy-to-fury` resolves the release repository
to `https://maven.fury.io/lancedb/`.

## Merge Risk

**Door:** Two-way. This only changes where future betas are published;
reverting the workflow restores Central publishing.
**Blast Radius:** `release-ci`. Beta consumers need the extra
repository; stable consumers are unaffected.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:34:14 -08:00
Xuanwo 9755c57d41 fix: prune Function outcomes for replaced fragments (#5005)
An input-preserving merge can fail freshness preservation after a
successful data write: the replaced fragment loses its signature, but
its known/failed-row records remain and fail sidecar validation. This
affects tables with persisted Function outcome metadata.

Prune both outcome maps with the same live-fragment set used for
signatures and row state. Outcomes on untouched fragments remain intact,
and inserted rows remain pending fill. The regression performs a real
mixed update/insert and reopens the resulting snapshot to check both
output columns.

Native validation on Linux: the exact regression fails before the fix
with `failure evidence names a fragment without a signature`. At
`20b10d80408cb455d9e5827f5d7d36a117622dea`, all 24 discovered
`table::freshness::tests::function_write_` cases pass, including the new
real merge regression. Test lists, counts and logs were retained and
SHA-256 verified. [Current-head CI passed both workspace Clippy
configurations](https://github.com/lancedb/lancedb/actions/runs/43274052568/job/131377627811).
A separate package-only, default-feature Clippy command reported
dead-code errors in seven files unchanged from the base; that local
failure is retained rather than represented as a pass. Other hosted
gates are pending. This does not claim downstream Sophon integration or
deployed acceptance.
2026-11-13 03:20:20 +08:00
Xuanwo 9b13e8f725 feat(python): materialize remote blobs at the query snapshot (#4973)
Python remote queries materialize Blob v2 payloads with
`blob_mode="bytes"` through synchronous and asynchronous `to_arrow`,
`to_list`, `to_pandas`, and `to_batches`. Deferred reads bind to the
actual producing dataset without changing the caller's live table
handle, including concurrent writes and same-name DROP/CREATE.

Trust only the authoritative `x-lancedb-query-version` plus
`x-lancedb-query-snapshot`. A numeric version alone cannot distinguish a
recreated table; the ordinary live watermark and user-supplied Arrow
metadata cannot establish provenance. An independent immutable handle
carries the original identity through every batch, byte-limit split, and
large-Blob Range fallback, and requires a verified response echo before
accepting payloads. Aliases, NULL/empty results, rerankers, and hybrid
legs retain consistent provenance. Missing capability, mixed identities,
or a serving node ignoring the precondition fails explicitly; ordinary
descriptor queries remain available on older servers.

The source docs distinguish Enterprise SQL payload bytes from SDK
descriptors. Server dependency:
https://github.com/lancedb/sophon/pull/10211.

Validation uses retained, independently audited local evidence: 724
Python cases passed (one optional PyAV skip), 67 focused SDK Rust cases
passed, 36 server checks passed, and 15 actual REST/SDK scenarios passed
across retained runs. These cover sync/async and Arrow file/stream
recreation, concurrent overwrite, deletion of a selected version, legacy
capabilities, forged IPC metadata, WAL exclusion, and actual mid-batch
recreation. The original aggregate experiment exited 101 because its
extra harness assumed a Python batch size would split the native result;
its 36 server checks and 13 completed scenarios remain valid. The
corrected native-batch experiment passed separately for sync and async.
No aggregate all-green claim is made.

The public batched-range operation has separate cross-partition identity
integration in https://github.com/lancedb/sophon/pull/10219 and
https://github.com/lancedb/lancedb/pull/4994. This change does not claim
to fix that loop. Realtime/MemWAL remains outside Blob v2 GA, without
disabling existing writes or compaction. No dependency pins changed.

Merge/rollout hold remains: upgrade every serving node before enabling
the new SDK capability, verify proxy forwarding of both authoritative
headers and the verification echo, and preserve queried versions for
later reads. Missing or inconsistent capability fails explicitly;
identity does not retain versions or authenticate storage paths. Actual
deployed proxy/Cloud/PE behavior and production retention remain
unverified. Old SDKs do not gain incarnation protection from a
server-only upgrade.

Published documentation retains the Early access badge and released
Enterprise instructions, including explicit fetches for copying. The new
Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example,
future streaming restrictions and GA wording remain source-only MDX
drafts until release. The correction distinguishing released Enterprise
SQL bytes from SDK descriptors remains visible. Documentation validation
checked comment boundaries, visible instructions, Python example syntax
and whitespace; the site was not built. This documentation-only revision
reuses the implementation evidence above.
2026-11-13 02:05:39 +08:00
Xuanwo da40aa6c44 test: isolate SQL reader timeout from submission latency (#5000)
The reader timeout regression can fail on a busy runner while submitting
its real loopback Flight exchange, before it ever calls `reader()`.
Freeze Tokio time and inhibit automatic clock advancement during
submission, then let the existing 200ms result timeout fire. Preserve
the exact result-stage error through both describe APIs and a second
reader, and verify that the timeout closes the server response stream.

With a diagnostic-only 300ms wall-clock delay in request-header
preparation, main fails with `SQL query submission timed out`; the
repaired fixture passes the same scenario through the real Flight/reader
path. The focused regression and all 54 adjacent SQL tests pass locally.
Hosted Linux and macOS CLI tests and the complete PR CI pass on
`fb96931be004166a479cc031aca4b1db05f4b78d`.
2026-11-13 02:05:23 +08:00
Xuanwo 725eee8a47 fix: prevent unsafe write retries and bound Blob uploads (#4977)
Ambiguous gateway failures can occur after a write commits, so replaying
the operation can append duplicate rows. Non-idempotent writes now retry
only known precommit failures. Multipart retries reuse the same upload
ID, part ID, and encoded bytes; ambiguous completion returns an
unknown-outcome error without abort or replay. A definite 409/429
completion rejection triggers best-effort cleanup using the call's retry
budget.

Replayable parts and the shared byte-based partition target default to
64 MiB. Dictionary-bearing batches are compacted before size estimation
and again when actual compressed IPC framing requires further splitting:
keys are remapped without changing their types or schema, view backing
buffers are collected, and ListView/LargeListView child ranges are
compacted as an interval union so overlapping views retain sharing.
Parts are prepared on demand; a single row and its schema can still
exceed the target. Multipart progress starts at zero and counts each
acknowledged IPC body once. Rust write builders and Python/TypeScript
`add` expose per-call retry overrides. Atomic `create_table` pre-encodes
to a temporary file and rejects compressed IPC over 256 MiB before
sending any request; the schema-first import guidance now includes
Python, TypeScript, and Rust examples. The shared partition target also
affects local automatic partitioning.

At `cb2a79473bd1b51b950c9fa3e9f247f929c8ac3b`, 70 targeted Rust cases
pass on source verified against the committed four-file patch. With a 16
KiB target and 128 unique 1 KiB dictionary values, the original ordinary
dictionary reproducer drops from 128 requests / 16,942,080 bytes to 16 /
143,488; ListView and LargeListView drop from 128 / 17,056,768 to 16 /
149,632. Encoded HTTP bodies are decoded to check values, nulls, slices,
metadata, changing dictionaries, and overlapping views; additional cases
cover framing-driven splitting, oversized rows, stable retry bytes/IDs,
acknowledged progress, and atomic create. Real HTTP fault tests commit
Blob payloads to a local Lance table and verify byte equality, no
duplicate append, and zero abort/replay after ambiguous completion.

The earlier 811 remote and 7 progress cases at
`f36d217110a25f6b501c0255df9aaa60e3c8000a`, Linux Python remote
database/options results (437 cases plus 8 fixture consumers), Node
retry-binding evidence, and Rust/TypeScript documentation example checks
remain historical validation. Python/Node extensions were not rebuilt
for the dictionary follow-up; these results do not establish current
binding integration or fresh cross-platform/minimum-dependency CI
acceptance.

A separate real Query Node lifecycle experiment used SDK
`b8efe7d2d63647d4962789ca5fc042b59efe073d` and Sophon
`58b7895c7c287810fff4e89f13226fc5fa60d9ef`. After the actual listener
forced a connection closed, a surviving QN accepted a retry with
identical upload ID, part ID, and IPC bytes. Ordinary insert and
complete response-loss scenarios sent no replay or abort; fresh reads
observed exactly one commit and exact Blob bytes in all three scenarios.
This local experiment used a 1-second drain grace, two 32 KiB payloads,
and one distinct staged part. It validates acknowledgement-loss recovery
at those revisions, not concurrent overlapping attempts, large uploads,
physical orphan cleanup, the later dictionary encoding change, or the
original sixth-insert EOF.

The part target is not a process memory limit: source/ancestor buffers
and temporary compaction allocations may coexist. Struct/FixedSizeList
parent nulls can retain hidden child keys; this is not complete
logical-reachability GC for every Arrow combination, and retained hidden
values may affect wire size. The current local Lance 15.0.0-beta.17
rejects ListView/LargeListView persistence; the tests establish remote
client IPC encoding behavior, not native or Cloud persistence support.
Live Cloud, full workspace/Clippy, the original ordinary-insert
disconnect root cause, and Sophon's asynchronous transaction cleanup
behavior are not established. Cancelling during staging retains the
existing server-session cleanup limitation. Function reader delivery is
separate.
2026-11-13 02:05:13 +08:00
Xuanwo 10f89b3c32 perf: reduce Blob read buffering and expose read window controls (#4979)
Remote Blob reads retain and concatenate intermediate IPC buffers across
batches and HTTP requests, increasing peak memory. Decode fragmented IPC
bodies using their declared length, assemble Blob values once across
batches/requests, and collect large-object Range responses into
known-size buffers. Impossible IPC/Range allocations return errors
instead of panicking. Closing a remote Blob file releases cached bytes
immediately, including while a cross-window read is waiting for the
network.

Rust and Python expose configurable Blob read buffers. Python sync/async
`fetch_blob_files` and `open_blob_file(s)` accept `buffer_size`; the
default is 512 KiB and zero disables prefetch. Explicit ranges bypass
prefetch. New PE batch-endpoint adoption is separate.

Malformed-length and retained-cache regressions fail on the previous
implementation and pass after the correction. Real HTTP tests cover
buffer propagation, close during blocked input, batch/request
aggregation, and large-object fallback. The final extension was rebuilt
before Python validation. Non-Blob Arrow arrays may still require
alignment copies; a contiguous multi-batch result still requires its
final allocation.

## Benchmark

Peak client RSS increment in KiB, using
`python/benchmarks/bench_fetch_blobs_rss.py`, independent clients and
real loopback HTTP. Both extensions were rebuilt from clean, unchanged
source with the same toolchain/configuration; loaded artifact hashes
were checked. Baseline `42f11d13a2755e44b63876b5a5edd93901291150`, after
`0a8a76de5286949d80368f95a5c529adf422c187`.

| Equivalent fetch workload | Payload | Before | After |
| --- | ---: | ---: | ---: |
| Single response, one batch | 48 MiB | 63,812 | 63,896 |
| Multiple responses, one batch per response | 144 MiB | 423,104 |
195,628 |
| Single response, 12 rows per batch | 48 MiB | 130,520 | 71,348 |
| Multiple responses, 12 rows per batch | 144 MiB | 363,144 | 172,748 |
| One oversized row, Range fallback | 70 MiB | 182,816 | 85,548 |

These are one observation per workload and build, not a statistical
distribution or memory upper bound. The single-batch workload shows no
improvement. Cloud/Sophon deployment, credentialed S3 and long-running
stress have not been exercised.
2026-11-13 02:05:09 +08:00
Xuanwo 8939ef1f01 feat(python): warn on scalar GPU Functions (#4966)
GPU Functions that use a scalar callable currently invoke the callable
once per row. The SDK documents Arrow-vectorized callables, but silently
accepts a scalar GPU callable, which makes it easy to assume that the
runtime will batch model work automatically.

Warn during Function construction when a GPU callable does not use the
documented vectorized form. A callable is recognized as vectorized when
every input is annotated as `pyarrow.Array`, or when a single input is
annotated as `pyarrow.RecordBatch`. The warning explains the resulting
per-row invocation and shows how to opt into batch execution. CPU
Functions and the vectorized GPU forms do not emit it.

The warning is advisory and does not change registration serialization,
the artifact, the runtime adapter, or execution behavior. It is one part
of the ENT-3522 documentation-and-warning alternative; it does not claim
that a single A10G meets the original batch throughput requirement.

Validation on the designated EC2 host reused the compatible existing
native extension from `ent3563-sdk-freshness-9336cfb`; this does not
claim a new native package or GPU performance run. The final head has a
clean source tree and ran:

```bash
PYTHONPATH="$sdk/python/python" /home/ec2-user/.venvs/ent3563-0a839fa/bin/python -m pytest -q python/python/tests/test_first_class_function_slice2.py -k 'gpu_marker_uses_gpu_runtime or gpu_scalar_warning_names_the_vectorized_alternative'
/home/ec2-user/.venvs/ent3563-0a839fa/bin/ruff format --check python/python/lancedb/functions.py python/python/tests/test_first_class_function_slice2.py
/home/ec2-user/.venvs/ent3563-0a839fa/bin/ruff check python/python/lancedb/functions.py python/python/tests/test_first_class_function_slice2.py
```

The actual result was `2 passed, 123 deselected`; both Ruff commands
passed. The original 2,000-row single-A10G throughput requirement
remains a separate ENT-3522 acceptance gate.
2026-11-13 02:04:32 +08:00
Jack Ye e61e21035c fix(python): explain Function namespace import restrictions (#4982)
A plain `import ctypes.util` in a Function currently reports “dynamic
namespace access” without explaining the module restriction or a way
forward. Give blocked module imports a distinct error that names the
modules, explains the reconstructed namespace constraint, and recommends
an importable helper supplied through `code=`, `pip=`, or `conda=`.
Dynamic namespace operations also get an actionable explanation.

Document the full restrictions and the helper alternative in the
decorator and Python Functions reference. Add regression coverage for
direct imports, `from` imports, global module aliases, class Functions,
the deprecated `@udf` alias, and helpers imported normally or shipped
with `code=`. Existing import restrictions remain in place.

For example, move `ctypes.util.find_library` into
`native_diagnostics/__init__.py`, then use:

```python
import native_diagnostics
from lancedb import function


@function(code=[native_diagnostics])
def probe_library(name: str) -> str:
    return native_diagnostics.find_library(name)
```

Fixes ENT-3408.
2026-11-12 09:01:39 -08:00
Xuanwo b4d1c55f62 test: isolate DynamoDB integration test databases (#4998)
The concurrent-append and drop-all DynamoDB tests shared a database
prefix. A timeout in the first could leave its table visible to the
second. Give each test its own prefix while retaining five independent
concurrent appends, the six-row assertion, and all drop/recreate checks.

The final two-prefix diff passed both tests in hosted LocalStack across
Node 22, 24, 26, and Node 24 experimental, without timing
instrumentation ([run
43245851764](https://github.com/lancedb/lancedb/actions/runs/43245851764),
head `5896c419`). Concurrency and the 5000 ms budget are unchanged. This
fixes cascading fixture contamination; the original timeout remains
unexplained, and these passing runs do not prove its root cause was
fixed.
2026-11-13 00:22:35 +08:00
Xuanwo b5623f8881 ci: separate macOS Rust test phases (#4995)
macOS doctests and library/integration tests currently share a 60-minute
job budget. In #4967, compilation and doctests left only 3–4 minutes for
an integration group that takes 6–7 minutes on the base revision, so the
job was cancelled before it could finish.

Run doctests and library/integration tests in separate matrix jobs on
both macOS 15 and 26, each retaining the existing 60-minute limit.
Preserve the library/integration check names, feature selection, locked
dependencies and all existing coverage. This adds runner/build cost but
removes the cumulative deadline coupling without reducing coverage.

YAML parsing, shell syntax and matrix coverage checks pass. Hosted macOS
execution and subsequent beta19 integration still need verification;
this does not address the separate unlocked-build timeout.
2026-11-13 00:22:21 +08:00
Xuanwo ec5e2cd621 ci: use two Cargo jobs for unlocked builds (#4996)
Fresh unlocked workspace builds can exhaust the 30-minute CI job while
Cargo is restricted to one compiler/linker task. Allow two concurrent
tasks while retaining fresh dependency resolution, all workspace
features and targets, the existing profile and timeout, and immediate
cleanup of completed test executables. The single-cache prerequisite
#4997 is included by a normal merge; merge it first under normal review
gates.

The final combined candidate passed hosted build-no-lock on head
`8a3cda494f79a414101db392986cd18a64e5d80d` (tested merge `8d56029`, run
[43236975778](https://github.com/lancedb/lancedb/actions/runs/43236975778/job/131247077833)).
The build step took 18m42s and the full job 19m58s, within the unchanged
30-minute budget. Exactly one 1,624,797,998-byte cache restore
completed; free disk samples before and after the build were 85 GiB and
79 GiB. These are endpoint samples, not peak RAM/disk measurements or a
controlled speedup comparison. This job compiles and links targets but
does not execute their tests.

Temporary diagnostics are removed. The three existing driver tests pass,
and the combined workflow and driver match their independently reviewed
parents.
2026-11-13 00:21:31 +08:00
Xuanwo a2a0d7f632 fix(ci): restore the unlocked Rust build cache once (#4997)
`build-no-lock` restores Rust build outputs twice: toolchain setup
enables its own cache by default before Cargo.lock removal, then the
workflow restores an explicit cache afterward. Both restore into the
same Cargo and target paths; the implicit action also allows PR cache
saves.

Disable only toolchain setup's implicit cache. The explicit restore
remains after lock removal and retains its main-only save condition.

This configuration is inherited unchanged by #4996. Its [failed
job](https://github.com/lancedb/lancedb/actions/runs/43229196192/job/131220088196)
reported runner ENOSPC while the second cache action was active, before
dependency installation or Cargo. Earlier hosted logs confirm two
restores into overlapping paths, but the failed job's full logs are
unavailable, so the exact disk peak and ENOSPC cause remain unproven.

A bounded probe using the resolved composite-action definitions verifies
two restores before this change versus one afterward, preserves ordering
and main-only saves, and confirms all other workflow configuration is
unchanged. The [hosted
run](https://github.com/lancedb/lancedb/actions/runs/43230840086/job/131225407007)
confirms one successful 1.62 GB cache restore after lock removal and no
PR cache save. It reached Cargo and linking without ENOSPC, but hit the
inherited 30-minute deadline with the unchanged one-job driver. Free
disk was 85 GiB before the build and 79 GiB at cancellation; these
endpoint observations do not establish peak disk use or a general ENOSPC
guarantee. The complementary two-job repair in #4996 now includes this
prerequisite for combined hosted acceptance; this PR remains limited to
cache ownership.
2026-11-13 00:20:53 +08:00
Jack Ye 9e03bf97d4 feat(python): support exist_ok when creating secrets (#4978)
Add `exist_ok=False` to synchronous and asynchronous Python
`create_secret`. Passing `True` uses `CREATE SECRET IF NOT EXISTS`,
preserving an existing credential. The default continues to fail on an
existing name; rotation stays explicit through `alter_secret`.

```python
db.create_secret("api-key", value, exist_ok=True)
await async_db.create_secret("api-key", value, exist_ok=True)
```

The Rust API gains an awaitable builder with `.exist_ok(true)` while
existing `create_secret(...).await` calls keep working. Includes Python
SQL round-trip coverage and Rust coverage for direct and deferred SQL
responses.
2026-11-12 08:03:11 -08:00
Jack Ye ced4799ad5 fix(python): preserve typed results for reopened refresh jobs (#4981)
Restore typed results for reopened Function and SQL-expression refresh jobs while preserving their handles and cached descriptions.

Fixes https://linear.app/lancedb/issue/ENT-3441.
2026-11-12 09:21:58 +00:00
Jack Ye 7b2dfb9e36 fix(python): align catalog and database connection inputs (#4983)
Align Python catalog connection inputs with database connections, warn when region is ignored, and preserve remote storage settings in opened databases.
2026-11-12 01:19:05 -08:00
Jack Ye 478d1d9b01 fix: expose comparable Function runtime identity (#4975)
Expose `FunctionVersion.runtime_digest` through Rust and Python, and add
`PythonRuntimeSpec.digest` so clients can compare declared environment
inputs before registering a Function. Dependency pins, environment
variables, Python version, and GPU settings affect this digest; package
ordering does not.

Clarify that matching `source_digest` alone is insufficient for reuse.
Older servers/images leave `runtime_digest` unset, which cannot
establish a match. Both digests and the other registration inputs must
match before reusing a version.

```python
request = definition.registration_request
same_source = version.source_digest == request.artifact.digest
same_runtime = version.runtime_digest == request.runtime.digest
```

Add shared runtime digest fixtures and coverage for environment-only
changes and catalog decoding. Local builds and tests were skipped as
requested.
2026-11-12 00:30:19 -08:00
Lance Release 01e9ea44aa Bump version: 0.42.0-beta.11 → 0.42.0-beta.12 v0.42.0-beta.12 2026-11-12 08:13:04 +00:00