mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
bab22a3fbd0f120b6d9e23cd4f3ee37621bb9bb5
3559
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jack Ye
|
bab22a3fbd |
feat(python): bundle Function package resources (#4987)
A package shipped through `code=` currently loses its non-Python files.
This breaks vendored packages that read JSON configuration, manifests,
prompts, or binary resources after deployment. Include regular package
resources in a separate base64-encoded `python_bundle.resources` map,
preserving exact bytes and relative paths. Both source and resources
contribute to the artifact digest. UTF-8 Python source bytes retain
their original line endings as well, so manifests that hash vendor code
continue to match.
For example, `@function(code=[my_package])` can use
`importlib.resources.files("my_package").joinpath("models.json").read_text()`
on the worker. Hidden files and caches are excluded; symlinks and unsafe
paths warn instead of following files outside the package. Source-only
bundles retain their existing wire form. Resource bundles require
matching server support.
Adds an isolated-import regression with a vendored subpackage, JSON
metadata, binary data, SHA256 verification, preserved CRLF bytes, and
resource-only version changes. Local builds and tests are disabled by
request. Ruff formatting and linting run locally; CI runs the tests.
|
||
 Xuanwo
|
db098ba812 |
fix(ci): support Cargo 1.100 artifact layout in cleanup fixture (#5010)
Cargo 1.100 moves intermediate compiler artifacts into per-unit build directories and emits `build_script_build` instead of `build-script-build`. The unlocked-build fixture currently fails on its old rlib path before the SDK build starts, even though the library survives cleanup. Discover compiler inputs recursively and accept both build-script names while preserving the library, generated-input, ordinary-executable, discarded-test, linker-failure and outside-target assertions. The standalone fixture reproduces the failure before this change and passes afterward on Rust/Cargo 1.100.0; it also passes on 1.97.0. Fault injection confirms missing libraries and other retained inputs still fail their assertions. Production cleanup and normal workflow coverage are unchanged. |
||
|
Jack Ye
|
9302287bed |
fix: accept defining queries in Python materialized views (#4980)
Python materialized-view creation now accepts a complete defining SQL
query across sync, async, remote, and namespace connections, including
creation jobs:
```python
view = db.create_materialized_view(
"events_by_kind",
query="SELECT kind, count(*) AS n FROM events GROUP BY kind",
)
```
`query` is mutually exclusive with `source`, `select`, `where`, and
`limit`. Existing source-based calls remain supported.
Adds Rust `Connection::create_materialized_view_from_query`: embedded
connections plan through `MaterializedViewDefinition::from_sql`, while
remote connections submit the full query for server-side parsing.
Includes regression coverage for grouping, UNNEST, query forwarding,
creation jobs, and argument validation.
|
||
|
Xuanwo
|
f863ea5e08 |
feat(rust): support cross-table blob copies in DataFusion (#4976)
`INSERT INTO target SELECT ... FROM source` currently cannot copy Blob v2 descriptors into an independent table. This enables copies from a single native source snapshot through projections, filters, sorting, and limits, while preserving descriptor queries such as `SELECT payload.size`. Source-owned payloads become target-owned data, so subsequent source mutation, compaction, or deletion does not invalidate the copy; nested fields retain their named identity across metadata differences, child reordering, scalar casts, and nullable parents. External blobs remain URI/range references under the destination's existing authorization policy: a destination that independently registers the matching external base accepts the copy, while an unregistered destination rejects it without committing. Source base registrations are not transferred, and `allow_external_blob_outside_bases` is not relaxed. JOIN, UNION, aggregation, and computed descriptors are rejected during planning. This addresses the DataFusion cross-table copy portion of OSS-696; remote/cloud execution and Realtime/MemWAL remain outside this support boundary. Local acceptance covers 15 passing DataFusion Blob integration cases, including byte reads after physical source-directory deletion, authorized and unauthorized External ranges, and a source-read failure after an earlier partition has written an uncommitted data file, with both the live destination and an independently reopened manifest unchanged. Existing regression evidence also includes 195 DataFusion unit tests and 59 Blob integration tests. Rust formatting and whitespace checks passed. Remote-feature builds, cloud/S3 execution, clippy, and the full workspace suite were not run. |
||
|
Xuanwo
|
cababaecf0 |
feat(remote): discover server builds and retain error provenance (#4999)
Remote callers can use `db.server_info()` in Rust and Python, including synchronous and asynchronous Python connections, to identify the responding server without submitting a query or Job. Typed results describe the query-node build, component versions, Function protocol and configured images. Older servers without the endpoint return an explicit unsupported error. REST and Flight errors retain the reported build in diagnostic text and Rust `RemoteErrorMetadata.server`. Typed table/job-not-found errors preserve their HTTP source and classification, including the schema-cache path; masked REST errors can obtain provenance from the response header. Unknown build information stays absent. The matching native Python SDK was exercised against a real localhost server (`8829b074` client / `43920fa` server): sync/async discovery, both SQL version forms and actual REST/Flight error provenance passed. The product files match this branch; private validation drivers are excluded. Focused coverage includes typed 404 errors, masked errors, and the public schema-cache regression. Deployment and published-package acceptance remain separate. Server counterpart: https://github.com/lancedb/sophon/pull/10234. Related: ENT-3565. |
||
|
Xuanwo
|
6d24446a30 |
feat: add negotiated batched blob range reads across SDKs (#4994)
Existing Rust and Python blob range calls must keep working when only the SDK is upgraded. Add negotiated batched range reads across Rust, Python sync/async and TypeScript, preserving ordered NULLs, duplicates and branch/version reads with splitting and reassembly at 1,024 ranges or 64 MiB per request. Only a successful describe response advertising `x-lancedb-blob-ranges: 1` selects the batched endpoint. Missing or unknown capability values select the existing descriptor-query plus BlobFile Range GET route before the operation. Endpoint failures never switch protocols. HTTP 400 preserves the server message and request ID as Rust `InvalidInput` / Python `ValueError`; other errors retain their existing semantics. The operation binds both numeric version and immutable snapshot identity from its producing query or first range response. Every later partition and oversized-range piece sends that original identity and requires an equal response identity before decoding. Same-name table recreation therefore fails explicitly instead of returning mixed old/new payloads; no partial array is returned on later failure. Targets `main`. The read-buffering prerequisite #4979, safe-write prerequisite #4977 and deferred-query identity prerequisite #4973 are merged into main; #5000 also supplies the merged SQL timeout-test prerequisite. Server support is [Sophon #10219](https://github.com/lancedb/sophon/pull/10219), including #10211. Exact SDK `1497b508324662180086a4d1e0a6eb9e0b3970db` and server `f39712c2c150b220eb831e00f6663c9eef32dce5` passed a real local SDK → REST → PE experiment: one Rust harness test, 14 assertion groups, 38 REST range requests and 34 PE RPCs. All four sync/async × warm/cold table-recreation cases failed explicitly before the second descriptor lookup or PE dispatch, with no mixed payload or partial result. The other groups retained 1,025-range success, 70 MiB reassembly, branch/version reads, overwrite snapshot consistency, error behavior and TypeScript buffer ownership. Bindings were rebuilt and their loaded hashes verified. This is local interoperability evidence, not Cloud, mixed-fleet or deployed header-forwarding acceptance. The server dependency upgrade, External/cache/URI-policy acceptance and validated rollout remain required. Enforcing PE support must precede QN and SDK enablement; identity does not retain versions or freeze External objects. This PR does not claim deployed support or complete ENT-3100/GA acceptance. Published documentation retains the Early access badge and released Enterprise instructions, including explicit fetches for copying. The new Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example, future streaming restrictions and GA wording remain source-only MDX drafts until release. The correction distinguishing released Enterprise SQL bytes from SDK descriptors remains visible. Documentation validation checked comment boundaries, visible instructions, Python example syntax and whitespace; the site was not built. The main integration keeps these release gates unchanged. Main integration at `0491d9bdcc65ce5d52ea93e1083c89e951ea4f9d` normally merges main `9b13e8f72530c8b1950c09685f6920711154a539`, including its Lance beta.18 dependency. Focused current-source validation passed: 493 remote-table/query/Blob/IPC/write-progress Rust tests, 23 shared create/write-body tests, 59 native Blob integration tests and 7 public HTTP range tests. Rebuilt Python passed 468 selected tests (one optional PyAV test skipped), and rebuilt TypeScript passed 194 selected tests across four suites; lint/format, TypeScript compilation and API documentation generation passed. Shared tests include actual HTTP multipart failures, acknowledgment boundaries, dictionary compaction and native Blob compaction. The 598 unrelated Jest cases excluded by the focused shared-suite selector were not run. The retained 14-group joint REST/PE experiment used the earlier exact candidates stated above. Operation-identity/range/IPC code is unchanged, but dependencies and shared write interfaces changed with main; the new local checks validate the SDK combination and do not constitute a fresh final-head SDK/server joint, Cloud or rollout run. Early access and every remaining server/External/cache/policy/dependency/rollout condition remain in force. CI snapshot on this head (2026-11-12 18:30 UTC): `build-no-lock` failed in the standalone `ci/tests/test_build_no_lock.py` fixture at the retained-rlib assertion, before the actual SDK no-lock build. The fixture, helper and Rust workflow are byte-identical to main; the fixture does not load SDK source. The underlying CI/environment cause is not diagnosed here. Other jobs remained pending/running at the snapshot, so CI is not reported as passing. Failure: https://github.com/lancedb/lancedb/actions/runs/43280277756/job/131399418284. |
||
  Will JonesandClaude Opus 5.5
|
1806dc2323 |
ci(java): publish beta releases to Fury instead of Maven Central (#5011)
## Summary Java publishing has failed since `v0.42.0-beta.10` because the `lancedb` Sonatype org hit Maven Central's monthly release-count limit: ```text Publishing blocked: your organization has reached its monthly release count limit. Grace period ended 2026-11-08. Usage resets 2026-12-01. ``` This sends beta tags to `https://maven.fury.io/lancedb/` instead, matching how Python publishes betas to `pypi.fury.io/lancedb`. Stable releases still go to Maven Central. ```text tag contains "beta" -> setup-java server "fury" -> mvn deploy -P deploy-to-fury otherwise -> setup-java server "ossrh" -> mvn deploy -P deploy-to-ossrh ``` - `java/pom.xml`: new `deploy-to-fury` profile that points `distributionManagement` at Fury. - `java/README.md`: explains how to add the Fury and `maven.lance.org` repositories to use a beta. Requires a `FURY_TOKEN` secret with push access to the `lancedb` Fury account. The token is passed as the Maven username with a placeholder password. That follows Gemfury's convention but hasn't been tested against Fury yet. ## Evidence Running the beta path locally (`deploy -pl lancedb-core -am -P deploy-to-fury`) against a file repository deployed `lancedb-parent` (POM) and `lancedb-core` (JAR, POM, sources, javadoc). `help:effective-pom -P deploy-to-fury` resolves the release repository to `https://maven.fury.io/lancedb/`. ## Merge Risk **Door:** Two-way. This only changes where future betas are published; reverting the workflow restores Central publishing. **Blast Radius:** `release-ci`. Beta consumers need the extra repository; stable consumers are unaffected. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Xuanwo
|
9755c57d41 |
fix: prune Function outcomes for replaced fragments (#5005)
An input-preserving merge can fail freshness preservation after a successful data write: the replaced fragment loses its signature, but its known/failed-row records remain and fail sidecar validation. This affects tables with persisted Function outcome metadata. Prune both outcome maps with the same live-fragment set used for signatures and row state. Outcomes on untouched fragments remain intact, and inserted rows remain pending fill. The regression performs a real mixed update/insert and reopens the resulting snapshot to check both output columns. Native validation on Linux: the exact regression fails before the fix with `failure evidence names a fragment without a signature`. At `20b10d80408cb455d9e5827f5d7d36a117622dea`, all 24 discovered `table::freshness::tests::function_write_` cases pass, including the new real merge regression. Test lists, counts and logs were retained and SHA-256 verified. [Current-head CI passed both workspace Clippy configurations](https://github.com/lancedb/lancedb/actions/runs/43274052568/job/131377627811). A separate package-only, default-feature Clippy command reported dead-code errors in seven files unchanged from the base; that local failure is retained rather than represented as a pass. Other hosted gates are pending. This does not claim downstream Sophon integration or deployed acceptance. |
||
|
Xuanwo
|
9b13e8f725 |
feat(python): materialize remote blobs at the query snapshot (#4973)
Python remote queries materialize Blob v2 payloads with `blob_mode="bytes"` through synchronous and asynchronous `to_arrow`, `to_list`, `to_pandas`, and `to_batches`. Deferred reads bind to the actual producing dataset without changing the caller's live table handle, including concurrent writes and same-name DROP/CREATE. Trust only the authoritative `x-lancedb-query-version` plus `x-lancedb-query-snapshot`. A numeric version alone cannot distinguish a recreated table; the ordinary live watermark and user-supplied Arrow metadata cannot establish provenance. An independent immutable handle carries the original identity through every batch, byte-limit split, and large-Blob Range fallback, and requires a verified response echo before accepting payloads. Aliases, NULL/empty results, rerankers, and hybrid legs retain consistent provenance. Missing capability, mixed identities, or a serving node ignoring the precondition fails explicitly; ordinary descriptor queries remain available on older servers. The source docs distinguish Enterprise SQL payload bytes from SDK descriptors. Server dependency: https://github.com/lancedb/sophon/pull/10211. Validation uses retained, independently audited local evidence: 724 Python cases passed (one optional PyAV skip), 67 focused SDK Rust cases passed, 36 server checks passed, and 15 actual REST/SDK scenarios passed across retained runs. These cover sync/async and Arrow file/stream recreation, concurrent overwrite, deletion of a selected version, legacy capabilities, forged IPC metadata, WAL exclusion, and actual mid-batch recreation. The original aggregate experiment exited 101 because its extra harness assumed a Python batch size would split the native result; its 36 server checks and 13 completed scenarios remain valid. The corrected native-batch experiment passed separately for sync and async. No aggregate all-green claim is made. The public batched-range operation has separate cross-partition identity integration in https://github.com/lancedb/sophon/pull/10219 and https://github.com/lancedb/lancedb/pull/4994. This change does not claim to fix that loop. Realtime/MemWAL remains outside Blob v2 GA, without disabling existing writes or compaction. No dependency pins changed. Merge/rollout hold remains: upgrade every serving node before enabling the new SDK capability, verify proxy forwarding of both authoritative headers and the verification echo, and preserve queried versions for later reads. Missing or inconsistent capability fails explicitly; identity does not retain versions or authenticate storage paths. Actual deployed proxy/Cloud/PE behavior and production retention remain unverified. Old SDKs do not gain incarnation protection from a server-only upgrade. Published documentation retains the Early access badge and released Enterprise instructions, including explicit fetches for copying. The new Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example, future streaming restrictions and GA wording remain source-only MDX drafts until release. The correction distinguishing released Enterprise SQL bytes from SDK descriptors remains visible. Documentation validation checked comment boundaries, visible instructions, Python example syntax and whitespace; the site was not built. This documentation-only revision reuses the implementation evidence above. |
||
|
Xuanwo
|
da40aa6c44 |
test: isolate SQL reader timeout from submission latency (#5000)
The reader timeout regression can fail on a busy runner while submitting its real loopback Flight exchange, before it ever calls `reader()`. Freeze Tokio time and inhibit automatic clock advancement during submission, then let the existing 200ms result timeout fire. Preserve the exact result-stage error through both describe APIs and a second reader, and verify that the timeout closes the server response stream. With a diagnostic-only 300ms wall-clock delay in request-header preparation, main fails with `SQL query submission timed out`; the repaired fixture passes the same scenario through the real Flight/reader path. The focused regression and all 54 adjacent SQL tests pass locally. Hosted Linux and macOS CLI tests and the complete PR CI pass on `fb96931be004166a479cc031aca4b1db05f4b78d`. |
||
|
Xuanwo
|
725eee8a47 |
fix: prevent unsafe write retries and bound Blob uploads (#4977)
Ambiguous gateway failures can occur after a write commits, so replaying the operation can append duplicate rows. Non-idempotent writes now retry only known precommit failures. Multipart retries reuse the same upload ID, part ID, and encoded bytes; ambiguous completion returns an unknown-outcome error without abort or replay. A definite 409/429 completion rejection triggers best-effort cleanup using the call's retry budget. Replayable parts and the shared byte-based partition target default to 64 MiB. Dictionary-bearing batches are compacted before size estimation and again when actual compressed IPC framing requires further splitting: keys are remapped without changing their types or schema, view backing buffers are collected, and ListView/LargeListView child ranges are compacted as an interval union so overlapping views retain sharing. Parts are prepared on demand; a single row and its schema can still exceed the target. Multipart progress starts at zero and counts each acknowledged IPC body once. Rust write builders and Python/TypeScript `add` expose per-call retry overrides. Atomic `create_table` pre-encodes to a temporary file and rejects compressed IPC over 256 MiB before sending any request; the schema-first import guidance now includes Python, TypeScript, and Rust examples. The shared partition target also affects local automatic partitioning. At `cb2a79473bd1b51b950c9fa3e9f247f929c8ac3b`, 70 targeted Rust cases pass on source verified against the committed four-file patch. With a 16 KiB target and 128 unique 1 KiB dictionary values, the original ordinary dictionary reproducer drops from 128 requests / 16,942,080 bytes to 16 / 143,488; ListView and LargeListView drop from 128 / 17,056,768 to 16 / 149,632. Encoded HTTP bodies are decoded to check values, nulls, slices, metadata, changing dictionaries, and overlapping views; additional cases cover framing-driven splitting, oversized rows, stable retry bytes/IDs, acknowledged progress, and atomic create. Real HTTP fault tests commit Blob payloads to a local Lance table and verify byte equality, no duplicate append, and zero abort/replay after ambiguous completion. The earlier 811 remote and 7 progress cases at `f36d217110a25f6b501c0255df9aaa60e3c8000a`, Linux Python remote database/options results (437 cases plus 8 fixture consumers), Node retry-binding evidence, and Rust/TypeScript documentation example checks remain historical validation. Python/Node extensions were not rebuilt for the dictionary follow-up; these results do not establish current binding integration or fresh cross-platform/minimum-dependency CI acceptance. A separate real Query Node lifecycle experiment used SDK `b8efe7d2d63647d4962789ca5fc042b59efe073d` and Sophon `58b7895c7c287810fff4e89f13226fc5fa60d9ef`. After the actual listener forced a connection closed, a surviving QN accepted a retry with identical upload ID, part ID, and IPC bytes. Ordinary insert and complete response-loss scenarios sent no replay or abort; fresh reads observed exactly one commit and exact Blob bytes in all three scenarios. This local experiment used a 1-second drain grace, two 32 KiB payloads, and one distinct staged part. It validates acknowledgement-loss recovery at those revisions, not concurrent overlapping attempts, large uploads, physical orphan cleanup, the later dictionary encoding change, or the original sixth-insert EOF. The part target is not a process memory limit: source/ancestor buffers and temporary compaction allocations may coexist. Struct/FixedSizeList parent nulls can retain hidden child keys; this is not complete logical-reachability GC for every Arrow combination, and retained hidden values may affect wire size. The current local Lance 15.0.0-beta.17 rejects ListView/LargeListView persistence; the tests establish remote client IPC encoding behavior, not native or Cloud persistence support. Live Cloud, full workspace/Clippy, the original ordinary-insert disconnect root cause, and Sophon's asynchronous transaction cleanup behavior are not established. Cancelling during staging retains the existing server-session cleanup limitation. Function reader delivery is separate. |
||
|
Xuanwo
|
10f89b3c32 |
perf: reduce Blob read buffering and expose read window controls (#4979)
Remote Blob reads retain and concatenate intermediate IPC buffers across batches and HTTP requests, increasing peak memory. Decode fragmented IPC bodies using their declared length, assemble Blob values once across batches/requests, and collect large-object Range responses into known-size buffers. Impossible IPC/Range allocations return errors instead of panicking. Closing a remote Blob file releases cached bytes immediately, including while a cross-window read is waiting for the network. Rust and Python expose configurable Blob read buffers. Python sync/async `fetch_blob_files` and `open_blob_file(s)` accept `buffer_size`; the default is 512 KiB and zero disables prefetch. Explicit ranges bypass prefetch. New PE batch-endpoint adoption is separate. Malformed-length and retained-cache regressions fail on the previous implementation and pass after the correction. Real HTTP tests cover buffer propagation, close during blocked input, batch/request aggregation, and large-object fallback. The final extension was rebuilt before Python validation. Non-Blob Arrow arrays may still require alignment copies; a contiguous multi-batch result still requires its final allocation. ## Benchmark Peak client RSS increment in KiB, using `python/benchmarks/bench_fetch_blobs_rss.py`, independent clients and real loopback HTTP. Both extensions were rebuilt from clean, unchanged source with the same toolchain/configuration; loaded artifact hashes were checked. Baseline `42f11d13a2755e44b63876b5a5edd93901291150`, after `0a8a76de5286949d80368f95a5c529adf422c187`. | Equivalent fetch workload | Payload | Before | After | | --- | ---: | ---: | ---: | | Single response, one batch | 48 MiB | 63,812 | 63,896 | | Multiple responses, one batch per response | 144 MiB | 423,104 | 195,628 | | Single response, 12 rows per batch | 48 MiB | 130,520 | 71,348 | | Multiple responses, 12 rows per batch | 144 MiB | 363,144 | 172,748 | | One oversized row, Range fallback | 70 MiB | 182,816 | 85,548 | These are one observation per workload and build, not a statistical distribution or memory upper bound. The single-batch workload shows no improvement. Cloud/Sophon deployment, credentialed S3 and long-running stress have not been exercised. |
||
|
Xuanwo
|
8939ef1f01 |
feat(python): warn on scalar GPU Functions (#4966)
GPU Functions that use a scalar callable currently invoke the callable once per row. The SDK documents Arrow-vectorized callables, but silently accepts a scalar GPU callable, which makes it easy to assume that the runtime will batch model work automatically. Warn during Function construction when a GPU callable does not use the documented vectorized form. A callable is recognized as vectorized when every input is annotated as `pyarrow.Array`, or when a single input is annotated as `pyarrow.RecordBatch`. The warning explains the resulting per-row invocation and shows how to opt into batch execution. CPU Functions and the vectorized GPU forms do not emit it. The warning is advisory and does not change registration serialization, the artifact, the runtime adapter, or execution behavior. It is one part of the ENT-3522 documentation-and-warning alternative; it does not claim that a single A10G meets the original batch throughput requirement. Validation on the designated EC2 host reused the compatible existing native extension from `ent3563-sdk-freshness-9336cfb`; this does not claim a new native package or GPU performance run. The final head has a clean source tree and ran: ```bash PYTHONPATH="$sdk/python/python" /home/ec2-user/.venvs/ent3563-0a839fa/bin/python -m pytest -q python/python/tests/test_first_class_function_slice2.py -k 'gpu_marker_uses_gpu_runtime or gpu_scalar_warning_names_the_vectorized_alternative' /home/ec2-user/.venvs/ent3563-0a839fa/bin/ruff format --check python/python/lancedb/functions.py python/python/tests/test_first_class_function_slice2.py /home/ec2-user/.venvs/ent3563-0a839fa/bin/ruff check python/python/lancedb/functions.py python/python/tests/test_first_class_function_slice2.py ``` The actual result was `2 passed, 123 deselected`; both Ruff commands passed. The original 2,000-row single-A10G throughput requirement remains a separate ENT-3522 acceptance gate. |
||
|
Jack Ye
|
e61e21035c |
fix(python): explain Function namespace import restrictions (#4982)
A plain `import ctypes.util` in a Function currently reports “dynamic
namespace access” without explaining the module restriction or a way
forward. Give blocked module imports a distinct error that names the
modules, explains the reconstructed namespace constraint, and recommends
an importable helper supplied through `code=`, `pip=`, or `conda=`.
Dynamic namespace operations also get an actionable explanation.
Document the full restrictions and the helper alternative in the
decorator and Python Functions reference. Add regression coverage for
direct imports, `from` imports, global module aliases, class Functions,
the deprecated `@udf` alias, and helpers imported normally or shipped
with `code=`. Existing import restrictions remain in place.
For example, move `ctypes.util.find_library` into
`native_diagnostics/__init__.py`, then use:
```python
import native_diagnostics
from lancedb import function
@function(code=[native_diagnostics])
def probe_library(name: str) -> str:
return native_diagnostics.find_library(name)
```
Fixes ENT-3408.
|
||
|
Xuanwo
|
b4d1c55f62 |
test: isolate DynamoDB integration test databases (#4998)
The concurrent-append and drop-all DynamoDB tests shared a database prefix. A timeout in the first could leave its table visible to the second. Give each test its own prefix while retaining five independent concurrent appends, the six-row assertion, and all drop/recreate checks. The final two-prefix diff passed both tests in hosted LocalStack across Node 22, 24, 26, and Node 24 experimental, without timing instrumentation ([run 43245851764](https://github.com/lancedb/lancedb/actions/runs/43245851764), head `5896c419`). Concurrency and the 5000 ms budget are unchanged. This fixes cascading fixture contamination; the original timeout remains unexplained, and these passing runs do not prove its root cause was fixed. |
||
|
Xuanwo
|
b5623f8881 |
ci: separate macOS Rust test phases (#4995)
macOS doctests and library/integration tests currently share a 60-minute job budget. In #4967, compilation and doctests left only 3–4 minutes for an integration group that takes 6–7 minutes on the base revision, so the job was cancelled before it could finish. Run doctests and library/integration tests in separate matrix jobs on both macOS 15 and 26, each retaining the existing 60-minute limit. Preserve the library/integration check names, feature selection, locked dependencies and all existing coverage. This adds runner/build cost but removes the cumulative deadline coupling without reducing coverage. YAML parsing, shell syntax and matrix coverage checks pass. Hosted macOS execution and subsequent beta19 integration still need verification; this does not address the separate unlocked-build timeout. |
||
|
Xuanwo
|
ec5e2cd621 |
ci: use two Cargo jobs for unlocked builds (#4996)
Fresh unlocked workspace builds can exhaust the 30-minute CI job while Cargo is restricted to one compiler/linker task. Allow two concurrent tasks while retaining fresh dependency resolution, all workspace features and targets, the existing profile and timeout, and immediate cleanup of completed test executables. The single-cache prerequisite #4997 is included by a normal merge; merge it first under normal review gates. The final combined candidate passed hosted build-no-lock on head `8a3cda494f79a414101db392986cd18a64e5d80d` (tested merge `8d56029`, run [43236975778](https://github.com/lancedb/lancedb/actions/runs/43236975778/job/131247077833)). The build step took 18m42s and the full job 19m58s, within the unchanged 30-minute budget. Exactly one 1,624,797,998-byte cache restore completed; free disk samples before and after the build were 85 GiB and 79 GiB. These are endpoint samples, not peak RAM/disk measurements or a controlled speedup comparison. This job compiles and links targets but does not execute their tests. Temporary diagnostics are removed. The three existing driver tests pass, and the combined workflow and driver match their independently reviewed parents. |
||
|
Xuanwo
|
a2a0d7f632 |
fix(ci): restore the unlocked Rust build cache once (#4997)
`build-no-lock` restores Rust build outputs twice: toolchain setup enables its own cache by default before Cargo.lock removal, then the workflow restores an explicit cache afterward. Both restore into the same Cargo and target paths; the implicit action also allows PR cache saves. Disable only toolchain setup's implicit cache. The explicit restore remains after lock removal and retains its main-only save condition. This configuration is inherited unchanged by #4996. Its [failed job](https://github.com/lancedb/lancedb/actions/runs/43229196192/job/131220088196) reported runner ENOSPC while the second cache action was active, before dependency installation or Cargo. Earlier hosted logs confirm two restores into overlapping paths, but the failed job's full logs are unavailable, so the exact disk peak and ENOSPC cause remain unproven. A bounded probe using the resolved composite-action definitions verifies two restores before this change versus one afterward, preserves ordering and main-only saves, and confirms all other workflow configuration is unchanged. The [hosted run](https://github.com/lancedb/lancedb/actions/runs/43230840086/job/131225407007) confirms one successful 1.62 GB cache restore after lock removal and no PR cache save. It reached Cargo and linking without ENOSPC, but hit the inherited 30-minute deadline with the unchanged one-job driver. Free disk was 85 GiB before the build and 79 GiB at cancellation; these endpoint observations do not establish peak disk use or a general ENOSPC guarantee. The complementary two-job repair in #4996 now includes this prerequisite for combined hosted acceptance; this PR remains limited to cache ownership. |
||
|
Jack Ye
|
9e03bf97d4 |
feat(python): support exist_ok when creating secrets (#4978)
Add `exist_ok=False` to synchronous and asynchronous Python
`create_secret`. Passing `True` uses `CREATE SECRET IF NOT EXISTS`,
preserving an existing credential. The default continues to fail on an
existing name; rotation stays explicit through `alter_secret`.
```python
db.create_secret("api-key", value, exist_ok=True)
await async_db.create_secret("api-key", value, exist_ok=True)
```
The Rust API gains an awaitable builder with `.exist_ok(true)` while
existing `create_secret(...).await` calls keep working. Includes Python
SQL round-trip coverage and Rust coverage for direct and deferred SQL
responses.
|
||
|
Jack Ye
|
ced4799ad5 |
fix(python): preserve typed results for reopened refresh jobs (#4981)
Restore typed results for reopened Function and SQL-expression refresh jobs while preserving their handles and cached descriptions. Fixes https://linear.app/lancedb/issue/ENT-3441. |
||
|
Jack Ye
|
7b2dfb9e36 |
fix(python): align catalog and database connection inputs (#4983)
Align Python catalog connection inputs with database connections, warn when region is ignored, and preserve remote storage settings in opened databases. |
||
|
Jack Ye
|
478d1d9b01 |
fix: expose comparable Function runtime identity (#4975)
Expose `FunctionVersion.runtime_digest` through Rust and Python, and add `PythonRuntimeSpec.digest` so clients can compare declared environment inputs before registering a Function. Dependency pins, environment variables, Python version, and GPU settings affect this digest; package ordering does not. Clarify that matching `source_digest` alone is insufficient for reuse. Older servers/images leave `runtime_digest` unset, which cannot establish a match. Both digests and the other registration inputs must match before reusing a version. ```python request = definition.registration_request same_source = version.source_digest == request.artifact.digest same_runtime = version.runtime_digest == request.runtime.digest ``` Add shared runtime digest fixtures and coverage for environment-only changes and catalog decoding. Local builds and tests were skipped as requested. |
||
 Lance Release
|
01e9ea44aa | Bump version: 0.42.0-beta.11 → 0.42.0-beta.12 v0.42.0-beta.12 | ||
 Yang Cen
|
5a38066234 |
chore(deps): upgrade Lance to v15.0.0-beta.18 (#4965)
Upgrade the Rust Lance dependency, Java lance-core, and Python test dependency from beta.17 to the officially published [v15.0.0-beta.18](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.18) (`079fd77683a7acf056b0912f6ecda938e39ee120`). This provides a compatible SDK for the newer Lance source already adopted downstream. Rust registry versions and checksums are unchanged; the Python lock uses the six published beta.18 wheels. ### Compatibility The adopted release includes [Lance #10863](https://github.com/lance-format/lance/pull/10863): `FilteredReadOptionsProto.has_fragment_ids` distinguishes an explicit empty fragment selection. Older executors ignore this field, so upgrade executors before relying on the corrected empty-selection behavior. Rust callers constructing this generated struct need the new field. [Lance #10112](https://github.com/lance-format/lance/pull/10112) also replaces the internal `hamming_distance_batch` API with `hamming_distance_batch_into`. The Linux Python matrix timeout is raised from 30 to 60 minutes: the Python 3.13 cold native build and both test passes completed, but the job hit its limit during cleanup. Build profiles, test commands, and coverage are unchanged. ### Validation - Three native FRI/ANN regressions passed: deferred-compaction refined RQ, deferred index remapping, and approximate-query builder. - `cargo clippy --quiet --locked --features remote --tests --examples -- -D warnings` passed with pinned Rust 1.97.0. - Locked metadata, formatting, Python Makefile-scope Ruff checks, and `uv lock --check --offline` passed. - YAML, whitespace, end-of-file, and typo checks passed for the timeout-only workflow change. ### Publication and CI Java beta.18 is now published. After [Lance #10968](https://github.com/lance-format/lance/pull/10968) merged, the [normal publication workflow](https://github.com/lance-format/lance/actions/runs/43146352505) built all three JNI targets and published successfully; the Maven jar is available. The upstream Python publication run was cancelled after its Windows ARM job hit the one-hour timeout; six other wheels were published and are the actual artifacts recorded in this lock. Cross-platform SDK CI is still required. No performance claim is made by this dependency update. |
||
|
Jack Ye
|
1dee9b006c |
fix(python): reject unsupported Function callables early (#4972)
Reject synchronous and asynchronous generator Functions, including class
`__call__` methods, during decoration so invalid outputs never reach
remote execution. Give actionable errors for lambdas and decorated
wrappers such as `functools.lru_cache`. Defaulted callable inputs remain
unsupported; the error now names the parameter and explains how to use
class initialization fields.
```python
@function
def double(value: float) -> float:
yield value * 2
# TypeError: generator Functions are not supported; return a value instead of
# yielding, or return a list for multiple values
```
The deprecated `@udf` alias shares the validation. Adds regression
coverage for both decorators, class methods, explicit schemas, and the
supported list-return alternative.
Fixes https://linear.app/lancedb/issue/ENT-3539
|
||
|
Jack Ye
|
e622254108 |
fix(python): use direct lookups for table existence checks (#4974)
Table existence checks should not depend on listing every table or
searching only the first 100 names. Use a direct lookup for `name in db`
and implement `table_exists(table_id)` on synchronous local connections,
matching the remote and asynchronous APIs. Native namespace connections
use their initialized Rust connection for existence checks; custom
namespace clients retain their existing callback. Missing tables return
`False`; other lookup errors propagate.
```python
if "events" in db:
db.drop_table("events")
exists = db.table_exists(["analytics", "events"])
all_names = list(db.list_tables())
```
Clarify that `.tables` contains one page and update deprecation guidance
with safe replacements. Add regression coverage for lookups beyond the
first page, nested namespaces, native and custom namespace clients,
missing tables, permission errors, and lookups that do not list tables.
Include an unpublished draft for the table-management guide until the
new SDK support ships.
|
||
|
Jack Ye
|
ee231aad06 |
fix(python): validate function package requirements before registration (#4971)
Function dependency typos currently reach the environment build before
failing. Validate pip requirements with
`packaging.requirements.Requirement` and conda requirements with
Rattler's MatchSpec parser when constructing a function definition,
before source packaging or registration. Errors identify the original
entry and its index.
```python
@lancedb.function(pip=["numpy", "torch>=2.0,,<3"])
def score(value: float) -> float:
return value
# ValueError: invalid pip[1] requirement 'torch>=2.0,,<3': ...
```
Preserves pip extras, markers, named references, bare archive/git URLs,
and conda channel/build specifications. The existing native extension
exposes the conda parser, so no additional Python package is required.
Adds regression coverage for invalid and supported forms, including the
deprecated `udf` alias.
|
||
 Brendan Clement
|
42f11d13a2 |
docs: clarify Python job states and waiting behavior (#4954)
We weren’t explicit about the job states returned by the Python SDK. This documents those values and clarifies the difference between checking the current status and waiting for completion. |
||
 Bruno Ramirez
|
28bbc81c3e |
fix(remote): explain unsupported optimize syntax (#4944)
This PR addresses [OSS-2913](https://linear.app/lancedb/issue/OSS-2913) ([#4845](https://github.com/lancedb/lancedb/issues/4845)) and [ENT-2909](https://linear.app/lancedb/issue/ENT-2909): `table.optimize()` on an Enterprise table exposes SQL parser errors when the deployment does not support its generated SQL. The SDK generates the statement itself, but previously propagated unsupported grammar as a generic SQL failure. Parser errors and unknown-option errors reported as `InvalidArgument` now become `NotSupported` (`NotImplementedError` in Python), retaining the original diagnostic and request ID. |
||
|
Xuanwo
|
9a9b681aaf |
docs(python): clarify refresh result row counts (#4963)
## Summary `RefreshColumnResult` exposed the canonical refresh counters without documenting their unit. This change describes them as logical rows planned for one refresh: `rows_assigned` counts successful rows written, `rows_failed` counts rows whose computation failed, and `rows_remaining` counts planned rows neither assigned nor terminally skipped. Rows already covered by `rows_failed` are not counted again as remaining. Retry attempts and user-code calls do not change that logical-row scope. The change also documents the source and published versions and the deprecated `rows_filled` alias. The existing skip-policy example is now explicit: a refresh over 2,001 logical rows with 1,891 assigned, 110 failed, and 0 remaining uses 2,001 as the failure-rate denominator. ## Behavior Documentation only. No field names, Python or native behavior, REST wire format, dependencies, or pins change. ## Validation - `git diff --check` - `uvx ruff format --check` - `uvx ruff check` - Imported this Python source with a compatible prebuilt ABI3 extension and asserted the public class and property documentation, the affected source version, and the native extension identity. |
||
|
Lance Release
|
08d3f448af | Bump version: 0.42.0-beta.10 → 0.42.0-beta.11 v0.42.0-beta.11 | ||
|
Jack Ye
|
ea1cf17f89 |
feat(python): export recorded Function job timelines (#4946)
Function jobs can now export their recorded execution timeline with
`job.export_timeline(path)` or `await job.export_timeline(path)`. The
SDK fetches a fresh Chrome Trace Event recording that opens in Perfetto,
including worker split spans and timestamped job phases. Batch counts
and latency statistics remain details rather than invented batch
intervals.
```python
job = db.open_job(job_id)
job.export_timeline("refresh.trace.json")
async_job = await async_db.open_job(job_id)
await async_job.export_timeline("refresh.trace.json")
```
The export does not wait for completion or cancel the job. Running jobs
return partial recordings. Local operations and SQL query handles remain
unsupported, and servers must support the timeline endpoint. The
destination is touched only after a successful fetch; network and
filesystem errors propagate. Rust freshness wrappers forward the
operation without changing the submitting table's read baseline.
|
||
|
Jack Ye
|
e9774ed168 |
feat(nodejs): expose Function refresh publication intervals (#4957)
Expose `publicationIntervalRows` on TypeScript refresh options and
forward it as `publication_interval_rows`. Accept positive safe integers
and preserve whole-fragment publication when omitted.
```typescript
const job = await table.refreshColumnAsync("embedding", {
publicationIntervalRows: 10000,
});
await job.wait();
```
Document the threshold semantics and add coverage for serialization,
omission, invalid values, safe-integer boundaries, and Embedded
rejection.
Closes ENT-3755.
|
||
|
Yang Cen
|
c1fee36045 |
chore(deps): upgrade Lance to v15.0.0-beta.17 (#4943)
Update the Rust workspace, Java dependency, and Python test dependency to [Lance v15.0.0-beta.17](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.17). This includes the FRI cache and vector-index changes required by current Sophon and retains the RaBitQ prewarm fixes. Lockfiles use the published source and test wheels without unrelated registry version bumps. The dependency dry run also exposed CI resource limits. Use the existing larger x64 runner for GNU Node and compatibility-wheel builds; allow 60 minutes for Mac Python tests, 120 minutes for the compatibility wheel, and 150 minutes for the Mac release wheel. Disable the CLI toolchain action's automatic target cache so only the subsequent downloads-only cache runs. Release optimization, the compatibility wheel's manylinux container and x86-64-v2 baseline, and test coverage stay unchanged. Validation: - Three native SDK regressions pass: refined RQ queries after deferred compaction, deferred index remapping, and ANN approximation planning. - Pinned Rust 1.97.0: `cargo clippy --quiet --locked --features remote --tests --examples -- -D warnings` passes. - Locked offline Rust metadata, Python lock validation, Ruff lint/format, and diff checks pass. The Python lock contains six published beta.17 wheels. - Repository YAML, whitespace and typo checks pass for the workflow changes. Product-source checks are reused for the subsequent CI-only commits. - At head `07033dc6`, all 65 applicable pre-review CI checks pass, including the compatibility wheel, Mac CLI, GNU Node, Mac Python tests, release wheels, and Java checks. Six checks are skipped by tag/main-only conditions or draft-only app behavior; the automated review and security checks run after the Ready transition. The upstream Java publisher succeeded and its beta.17 Maven artifact is available. Six upstream Python wheel builds/uploads succeeded; its Windows ARM job was cancelled. These checks do not establish Sophon performance or aggregate-memory acceptance. |
||
|
Xuanwo
|
a356950bce |
feat: expose queued Function refresh status (#4959)
## Problem The existing Function `column_status` surface cannot decode the Agent scheduler's `queued` state or the `failure_threshold_reached` and `binding_revision_missing` blocked reasons. It also preserves only the latest refresh job, so a later cancellation can hide an earlier actual failure. ## Change - Adds the `queued`, `failure_threshold_reached`, and `binding_revision_missing` wire states in Rust, Python, and Node. The wire change is additive, while Rust callers must update exhaustive enum matches and public struct literals. - Adds optional `last_failure_job` alongside `last_refresh_job`. The latest cancellation and the most recent actual failure retain separate meanings. - Reuses the canonical Rust `JobFailure` and existing Python/Node `JobFailureInfo` models for nested job failure details instead of introducing a parallel status error model. - Maps the existing public `sql_job_id` field through Node's `JobFailureInfo`. - Keeps `column_status`, `column_statuses`, and `wait_fresh` as the existing API surfaces. There is no new endpoint, row-error list, durable ledger, or unknown-enum fallback. - Documents `queued` versus `running` and the separate latest-attempt and latest-failure jobs in the existing hidden freshness release draft. The new status values are additive but cross-service: `queued`, the two new blocked reasons, and `last_failure_job` require matching Agent/QN and SDK deployments before clients can rely on them. This SDK follow-up does not change the normal `7cc3904a` dependency pin or claim whole ENT-3487 completion. ## Validation The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`, based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The final product docs commit changes generated TypeDoc only. - Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust blobs are unchanged from their tested `788ef85b` source through `c9bac25e`. - Python: the PyO3 extension built successfully, and the selected native PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The test requests pinned version 42. - Node: the addon and TypeScript builds passed, Biome passed, and the selected Jest transport test passed 1/0 with 57 filtered tests. Addon SHA-256: `c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`. - Python extension SHA-256: `36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`. - TypeDoc was generated from the native `native.d.ts` produced by that Node build. A second generation produced no remaining diff. The public diff contains only product, test, and generated API documentation changes. Native runner scripts and preserved compile-failure logs remain private validation evidence. Agent/QN production mapping, mixed-version rollout, and the remaining ENT-3487 server-side work are tracked separately. |
||
|
Xuanwo
|
7cc3904a7e |
feat: expose exact Function column freshness (#4958)
## Problem Function-backed columns need an exact `fresh`, `stale`, `never_computed`, and `failed` partition for the table version a caller is reading. The existing error listing is capped, and composing it with job and schema listings cannot prove which rows currently failed. ## Behavior - Adds a strict freshness reader beside the existing permissive planner. It distinguishes an absent certificate from legacy evidence that omits failures, and from a current certificate carrying failure coverage. - Extends the immutable freshness sidecar with per-fragment known and failed row bitmaps. Partial or `WHERE` publications cover only observed rows; untouched rows stay unknown. Deleted rows do not block classification. - Adds failure-aware whole-fragment and row-repair writers. A rejected older attempt cannot overwrite failure evidence already recorded for a newer input signature. - Adds `column_status`, `column_statuses`, and `wait_fresh` to Python and the corresponding Node methods. The client calls the server's exact status endpoint, validates the four-way partition and one-version guarantee, and preserves the complete Function version, last-job, and automatic-refresh metadata. - Adds an MDX-commented draft usage section to the Enterprise computed-column refresh guide, covering Python, Rust, and TypeScript and remaining hidden until the server integration ships. - Sends `column`, `branch`, and exact `version` selectors to the status route. A pinned handle rejects a nonempty response from any other table version; latest handles keep the existing read-freshness headers. - Retries `wait_fresh` only for HTTP 503 with `function_column_status_finalization_pending`. Existing failed rows remain visible and do not keep the wait pending; other errors propagate. - Models the shared scheduler projection, including `up_to_date`, real `blocked` states, their reason, failure streak, and effective cooldown. ## Validation This is the SDK side of ENT-3563. Sophon's failure-coverage publication and status endpoint are a separate integration; deployed end-to-end and whole-ticket acceptance remain pending. - Rust: 9 exact freshness/status tests passed, plus 1 exact latest-handle freshness-header test. The pinned transport test was extended with a wrong-version response and rerun at `ebd6649a` (1 passed). Coverage includes untouched partial rows, selected-row failure clearing, cross-generation evidence, typed and non-retried errors, `up_to_date`, blocked scheduler fields, and exact-version requests. - Python: 2 native PyO3 remote-transport tests passed together, including typed retry; one local facade test reaches the existing unsupported boundary. - Node: the NAPI addon built successfully, TypeScript and Biome passed, and the selected native remote-transport test passed 1/0, including exact-version selection and blocked scheduler metadata. - `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile, Biome, and TypeScript checks passed. Validation is mapped by the relevant unchanged source rather than treated as one final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header and Python remote tests at `31e68b7d`, the local Python test and Node build at `6f213584`, and the final TypeScript/Jest delta at `4f2d5c27`. The final product tree excludes those five private EC2 runner files and adds six generated TypeDoc files plus one hidden draft usage section in the computed-column refresh guide. The Node delta corrected a private TypeScript command and reused the unchanged native addon. |
||
|
Yang Cen
|
34e8b8087b |
fix: preserve function freshness across input-preserving writes (#4948)
Updating an unrelated column or merging rows with unchanged Function inputs rewrites fragment files, causing already computed outputs to be treated as stale. Preserve their freshness from the successful write transaction when the inputs and outputs are provably unchanged; keep changed or uncertain inputs eligible for recomputation. Mixed update/insert writes keep a separate pending-fill proof so retained outputs can be reused while inserted NULL outputs still need computation. Partial refreshes under a new Function definition discard the previous definition’s completed and pending proofs. Preservation uses a version-fenced metadata commit, and an optional preservation failure does not turn a successful data write into an error. Validation: 67 freshness tests pass, including the new partial-definition regression (confirmed failing before its fix), unrelated updates, identical/mixed merges, NULL/nested/blob inputs, dirty-state preservation, concurrency fences and sidecar retention. `cargo fmt --all` and `cargo clippy --quiet --locked --features remote --tests --examples` pass. Compatibility: `StalenessPlan` adds a public `pending_fill` field, so direct Rust struct literals must be updated. Refresh consumers must handle pending-fill admission and publication before adopting this behavior; a dependency bump alone does not complete consumer integration. End-to-end consumer validation and deployment remain separate. No latency claim is made for this SDK-only change. |
||
|
Xuanwo
|
cf8e6d0fd4 |
feat: expose Function call concurrency limits (#4956)
## Summary Adds an optional `max_concurrency` setting to Function registration and explicit refresh: - A registered Function version can define the default maximum number of user-code calls in flight for one logical execution. - A refresh can override that default for that execution only. - The value must be a positive `u32` and is independent from `resource_budget`; it is not an account-wide quota. - Omitting the option preserves the previous request payload. Local embedded refreshes reject this Enterprise-only option. Rust, Python sync/async, PyO3, and the existing Node refresh surface forward the option. Node has no Function registration surface today, so no registration API is added there. The generated TypeDoc pages are included for the new public option. ## Validation The native-tested source covers Function version decoding, registration validation and serialization, the Flight SQL registration ticket and returned version, remote refresh forwarding, local rejection, and Node's numeric boundary before conversion. Exact stage results are Rust `1+3` plus four remote cases at `1` each; Python `14/1/49/24`; Node `57` remote and `1` local rejection. Ruff, standard repository Pyright, and Biome passed. The tested Python extension was `21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the tested Node addon and generated native declaration were `f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and `4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The final 24-file diff preserves the 22 tested product/test/fixture/source-doc blobs and adds only two generated TypeDoc pages. |
||
|
Xuanwo
|
8029428c71 |
feat: persist automatic Function refresh resource budgets (#4953)
## Depends on merged #4951 Phase 1 merged as `4d22c0ce2b6825c880c818e737b079db494b4053`, so this PR now targets `main`. It reuses the canonical `FunctionWorkerResourceBudget` introduced there rather than copying the type. For Function `add_columns` declarations, Python and Rust now persist `auto_refresh_resource_budget` in computed-column metadata and send it as `auto_refresh_resource_budget` in the remote request. CPU milliseconds and memory bytes are required positive integers. Omitting `gpu_count` is accepted as zero and the canonical outgoing JSON still contains `"gpu_count": 0`. Unknown fields, missing CPU or memory, wrong types, zero CPU or memory, negative values, and overflow are rejected. The same budget is applied to every public output binding of a multi-output Function declaration. Existing metadata is retained, and SQL-expression declarations or local execution reject the budget instead of silently ignoring it. Node is unchanged because its current `addColumns` surface only accepts SQL computed expressions and has no Function declaration path. This change does not implement fleet call concurrency limits or enforce the budget in the runtime. The Enterprise server must support the nested Function worker resource-budget contract before these values can act as execution limits. ## Validation - Python extension build passed. - Remote Function declaration and local unsupported-path rejection each passed. - Canonical metadata round-trip passed, including omitted GPU decoding to zero and explicit `gpu_count: 0` on encode. - Function multi-output remote wire test passed with `--features remote`. - Repository-scoped Python type check passed with 0 errors. - All runtime source blobs match the native-tested source. The post-native deltas are one documentation example line and an existing routing-test stub update for the new optional argument; the affected parameterized test passed 3/3 against the existing extension build. No private test runners are part of this PR. |
||
|
Xuanwo
|
c9d176ae7e |
feat: resolve latest Function versions (#4949)
## Problem Function catalog lookup required an exact version string even though callers commonly need the latest currently visible immutable revision. On the current SDK, resolving that version required an out-of-band catalog query before calling `get_function`. ## Change - Make Python `get_function(name)` default to latest while accepting `int` or `str` for exact lookup. - Normalize integer versions to their decimal string form and reject `bool` explicitly. Existing string validation and errors remain unchanged. - Thread `None` through the rebuilt PyO3 binding and route it to the new Rust `Connection::get_function_latest`. - Resolve latest through the existing catalog `DESCRIBE FUNCTION` path without `WITH (version = ...)`. The returned value is one exact immutable version, not a mutable alias. - Preserve the existing Rust exact-version API. ## Validation On the designated EC2 host with the rebuilt extension: - `test_local_function_catalog_operations_are_not_supported`: **1 passed**. This crosses the Python connection, rebuilt PyO3, and embedded Rust `NotSupported` boundary with the omitted-version call. - Async and sync catalog-path tests: **2 passed**, 117 deselected. - Remote SQL Function creation/description integration test: **1 passed**. - An earlier Rust invocation with the wrong module path matched **0 tests**; it is retained as harness evidence and is not counted as a pass. ## Scope This is the Function lookup SDK slice. It does not include the separate atomic rebind/publication server work tracked under ENT-3418. |
||
|
Xuanwo
|
1e5b6deee3 |
fix(python): support legacy blob query bytes (#4947)
## Summary - Let local plain queries over row-addressable legacy v1 and v2 blob columns use `blob_mode="bytes"` without importing `pylance` through pandas, batch, or Arrow conversion. - Preserve aliases, filters, ordering, limits, null and empty payloads, empty result schemas, and the caller's `_rowid` choice. - Build the hidden row-id query separately so reusing an async builder for lazy reads or non-blob projections cannot retain an automatic `_rowid`. - Keep byte reads closed for blobs inside lists, remote or namespace pushdown, and query shapes whose result rows cannot be matched back to the source. Blob v2 lazy auto-row-id behavior is unchanged. Fixes #4645 ## Validation - 26 focused Python tests passed with the test subprocess blocking all `lance` imports. The persisted fixture writes and reopens a storage-v2.1 table, then covers sync and async pandas, batches, and Arrow conversion plus async builder reuse. - The existing unsupported vector and hybrid reranker regressions now assert the current eligibility diagnostic, and the legacy table-level Arrow regression now asserts successful bytes, schema, values, and hidden row IDs. Both focused tests passed. - The runtime receipt records no installed `pylance` distribution and no importable `lance` module. The Python-only change was verified with an existing compatible 0.42.0b9 extension; the relevant query and fetch APIs are unchanged between that source and this 0.42.0b10 base. - Ruff format and lint pass for every changed Python file. |
||
|
Xuanwo
|
4d22c0ce2b |
feat: expose Function refresh resource budgets in SDKs (#4951)
## Summary - Add the canonical Rust `FunctionWorkerResourceBudget` and `RefreshColumnOptions::resource_budget()` for explicit Enterprise Function refreshes. - Forward the nested budget through Python sync/async, PyO3, Rust remote requests, Node NAPI, and TypeScript. - Require positive CPU and memory values; normalize an omitted nested GPU count to numeric zero; reject the nested budget when combined with the top-level `gpu_count` shorthand. - Reject `resource_budget` on LanceDB Embedded refresh paths instead of silently ignoring it. - Document the Rust, Python, and TypeScript APIs, including that SQL servers currently require all three budget fields. The Rust, Python, and TypeScript additions use `cpu_millis` / `cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` / `gpuCount`. The top-level GPU shorthand keeps its existing behavior and remains mutually exclusive with the complete nested budget. ## Scope and compatibility This change configures only the explicit refresh call. It does not configure `add_columns` automatic refresh and does not introduce a fleet-wide call-cap interface. A LanceDB Enterprise server must expose the nested `resource_budget` backfill contract. LanceDB Embedded continues to support its existing expression refresh defaults and rejects the new complete budget explicitly. ## Validation Validated against the final signed product source: - Rust: resource-budget remote serialization test `2 passed`; validation test `1 passed`. - Python: actual PyO3 build; remote-request matrix `22 passed`; invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff check and format clean. - Repository-standard Pyright entry point: `0 errors`, within the repository's configured include set. - Node: native build and Biome lint passed; the existing remote-request test `1 passed`, including fractional, negative, overflow, NaN, and infinity nested GPU cases; the existing local-rejection test `1 passed`. - TypeDoc generation and generated-document cleanliness passed. - The native-tested product source differs from this head only in two semantically equivalent Node range checks required by Clippy. Current CI compiles and lints the final blobs. No dependency or lockfile changes are included. |
||
 Lucia Cerchie
|
d44f768920 |
docs: add generated agent prompt catalog section (UAX-192) (#4945)
Add an automated prompt catalog section to the agentic skills page. The combined docs build fills its insertion marker from the evaluation source, avoiding a manually maintained prompt list. Keep the evaluation context and fixture guidance in this page so docs reviewers can edit it independently of the generator. Existing workflow examples remain in place. Part of UAX-192. Depends on https://github.com/lancedb/docs/pull/434; merge that docs pipeline change first. Validation: combined assembly and Mintlify validation pass. |
||
|
Jack Ye
|
481e82293e |
fix(python): explain explicit blob reads and subset copies (#4940)
Unsupported remote blob materialization errors now point to row-ID selection, fetch_blobs and fetch_blob_files. Document descriptor structs, fetched bytes and Function BlobReader inputs, and show a bounded subset-copy recipe that keeps the source unchanged between selection and fetching. Adds guidance assertions and copy coverage preserving payloads, empty blobs and nulls. No new blob materialization API is introduced. Formatting applied; no local builds or tests were run. CI will execute the regression coverage. |
||
|
Jack Ye
|
3bfa023375 |
fix(python): bound Job and AsyncJob payload previews (#4939)
Printing a Job or a bound status method could dump hundreds of lines of serialized Function initialization. Bound nested payload previews in both sync and async handles while preserving cached identity, state, failure and result information. Full spec/result values remain available through their properties, with no additional backend calls. Adds regression coverage for large payloads, bound method display, unobserved handles and completed local jobs. The broader method/property API design remains separate. Formatting applied; no local builds or tests were run. CI will execute the regression coverage. |
||
|
Jack Ye
|
faa5987b33 |
fix(python): name unsupported Function globals in capture errors (#4938)
Function source packaging rejected unsupported global values without identifying the variable or explaining the supported captures. Include the global name even for a nested tuple failure, describe the supported literal values and remedy, and document the capture boundary. Adds function/class regression cases for Arrow types, nested tuples, dictionaries and non-finite floats, plus a supported nested literal control. New serialization support remains separate. Formatting applied; no local builds or tests were run. CI will execute the regression coverage. |
||
 BHAWESHBHASKAR
|
f7648d4a22 | feat(python): expose the ZoneMap scalar index (#4816) | ||
|
Xuanwo
|
c0bd83ef29 |
feat(sql): expose durable job ids on query handles (#4941)
Expose the durable SQL registry id separately from the connection-scoped query UUID. Rust and Python query handles and descriptions expose optional `job_id`; result readers preserve it in schema metadata under `x-lancedb-sql-job-id`, without adding business columns. Clients can reopen that id with `open_job`. Poll success and failure metadata update the same handle. A failure before the first successful poll retains its job id and original structured error. The cancellation documentation distinguishes the Flight waiter from `cancel_job(job_id)`. Validation on the designated EC2: the two focused error interoperability tests and Ruff passed on `a8798e81`; delivery has the identical tree. The preceding 51 Rust SQL tests and earlier 31 native Python SQL tests are reused for unchanged coverage. Matching-runtime independent acceptance remains pending. Companion server change: https://github.com/lancedb/sophon/pull/10049. Refs ENT-3562. |
||
|
Lance Release
|
3851938b6e | Bump version: 0.42.0-beta.9 → 0.42.0-beta.10 v0.42.0-beta.10 | ||
 LanceDB RobotandJack Ye
|
e928e374d0 |
chore: update lance dependency to v15.0.0-beta.16 (#4935)
Update the Rust workspace, Java lance-core, and Python test dependency to [Lance v15.0.0-beta.16](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.16). Synchronize `python/uv.lock` with the published beta.16 wheels for Linux, macOS, and Windows. Run the native Linux x64 wheel build on an eight-core runner, matching the existing ARM build allocation, after its optimized build repeatedly exceeded the job deadline on the standard runner. The dependency update passed Clippy, formatting, and CI-script checks; the Python lockfile refresh/check passes. Fresh CI is running for the runner adjustment. Java dependency-resolution failures caused solely by publication lag are accepted for this upgrade. --------- Co-authored-by: Jack Ye <yezhaoqin@gmail.com> |