mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
28bbc81c3e0488c9eca9dab738f6db713f24d12c
313
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jack Ye
|
ea1cf17f89 |
feat(python): export recorded Function job timelines (#4946)
Function jobs can now export their recorded execution timeline with
`job.export_timeline(path)` or `await job.export_timeline(path)`. The
SDK fetches a fresh Chrome Trace Event recording that opens in Perfetto,
including worker split spans and timestamped job phases. Batch counts
and latency statistics remain details rather than invented batch
intervals.
```python
job = db.open_job(job_id)
job.export_timeline("refresh.trace.json")
async_job = await async_db.open_job(job_id)
await async_job.export_timeline("refresh.trace.json")
```
The export does not wait for completion or cancel the job. Running jobs
return partial recordings. Local operations and SQL query handles remain
unsupported, and servers must support the timeline endpoint. The
destination is touched only after a successful fetch; network and
filesystem errors propagate. Rust freshness wrappers forward the
operation without changing the submitting table's read baseline.
|
||
 Xuanwo
|
a356950bce |
feat: expose queued Function refresh status (#4959)
## Problem The existing Function `column_status` surface cannot decode the Agent scheduler's `queued` state or the `failure_threshold_reached` and `binding_revision_missing` blocked reasons. It also preserves only the latest refresh job, so a later cancellation can hide an earlier actual failure. ## Change - Adds the `queued`, `failure_threshold_reached`, and `binding_revision_missing` wire states in Rust, Python, and Node. The wire change is additive, while Rust callers must update exhaustive enum matches and public struct literals. - Adds optional `last_failure_job` alongside `last_refresh_job`. The latest cancellation and the most recent actual failure retain separate meanings. - Reuses the canonical Rust `JobFailure` and existing Python/Node `JobFailureInfo` models for nested job failure details instead of introducing a parallel status error model. - Maps the existing public `sql_job_id` field through Node's `JobFailureInfo`. - Keeps `column_status`, `column_statuses`, and `wait_fresh` as the existing API surfaces. There is no new endpoint, row-error list, durable ledger, or unknown-enum fallback. - Documents `queued` versus `running` and the separate latest-attempt and latest-failure jobs in the existing hidden freshness release draft. The new status values are additive but cross-service: `queued`, the two new blocked reasons, and `last_failure_job` require matching Agent/QN and SDK deployments before clients can rely on them. This SDK follow-up does not change the normal `7cc3904a` dependency pin or claim whole ENT-3487 completion. ## Validation The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`, based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The final product docs commit changes generated TypeDoc only. - Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust blobs are unchanged from their tested `788ef85b` source through `c9bac25e`. - Python: the PyO3 extension built successfully, and the selected native PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The test requests pinned version 42. - Node: the addon and TypeScript builds passed, Biome passed, and the selected Jest transport test passed 1/0 with 57 filtered tests. Addon SHA-256: `c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`. - Python extension SHA-256: `36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`. - TypeDoc was generated from the native `native.d.ts` produced by that Node build. A second generation produced no remaining diff. The public diff contains only product, test, and generated API documentation changes. Native runner scripts and preserved compile-failure logs remain private validation evidence. Agent/QN production mapping, mixed-version rollout, and the remaining ENT-3487 server-side work are tracked separately. |
||
|
Xuanwo
|
7cc3904a7e |
feat: expose exact Function column freshness (#4958)
## Problem Function-backed columns need an exact `fresh`, `stale`, `never_computed`, and `failed` partition for the table version a caller is reading. The existing error listing is capped, and composing it with job and schema listings cannot prove which rows currently failed. ## Behavior - Adds a strict freshness reader beside the existing permissive planner. It distinguishes an absent certificate from legacy evidence that omits failures, and from a current certificate carrying failure coverage. - Extends the immutable freshness sidecar with per-fragment known and failed row bitmaps. Partial or `WHERE` publications cover only observed rows; untouched rows stay unknown. Deleted rows do not block classification. - Adds failure-aware whole-fragment and row-repair writers. A rejected older attempt cannot overwrite failure evidence already recorded for a newer input signature. - Adds `column_status`, `column_statuses`, and `wait_fresh` to Python and the corresponding Node methods. The client calls the server's exact status endpoint, validates the four-way partition and one-version guarantee, and preserves the complete Function version, last-job, and automatic-refresh metadata. - Adds an MDX-commented draft usage section to the Enterprise computed-column refresh guide, covering Python, Rust, and TypeScript and remaining hidden until the server integration ships. - Sends `column`, `branch`, and exact `version` selectors to the status route. A pinned handle rejects a nonempty response from any other table version; latest handles keep the existing read-freshness headers. - Retries `wait_fresh` only for HTTP 503 with `function_column_status_finalization_pending`. Existing failed rows remain visible and do not keep the wait pending; other errors propagate. - Models the shared scheduler projection, including `up_to_date`, real `blocked` states, their reason, failure streak, and effective cooldown. ## Validation This is the SDK side of ENT-3563. Sophon's failure-coverage publication and status endpoint are a separate integration; deployed end-to-end and whole-ticket acceptance remain pending. - Rust: 9 exact freshness/status tests passed, plus 1 exact latest-handle freshness-header test. The pinned transport test was extended with a wrong-version response and rerun at `ebd6649a` (1 passed). Coverage includes untouched partial rows, selected-row failure clearing, cross-generation evidence, typed and non-retried errors, `up_to_date`, blocked scheduler fields, and exact-version requests. - Python: 2 native PyO3 remote-transport tests passed together, including typed retry; one local facade test reaches the existing unsupported boundary. - Node: the NAPI addon built successfully, TypeScript and Biome passed, and the selected native remote-transport test passed 1/0, including exact-version selection and blocked scheduler metadata. - `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile, Biome, and TypeScript checks passed. Validation is mapped by the relevant unchanged source rather than treated as one final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header and Python remote tests at `31e68b7d`, the local Python test and Node build at `6f213584`, and the final TypeScript/Jest delta at `4f2d5c27`. The final product tree excludes those five private EC2 runner files and adds six generated TypeDoc files plus one hidden draft usage section in the computed-column refresh guide. The Node delta corrected a private TypeScript command and reused the unchanged native addon. |
||
|
Xuanwo
|
cf8e6d0fd4 |
feat: expose Function call concurrency limits (#4956)
## Summary Adds an optional `max_concurrency` setting to Function registration and explicit refresh: - A registered Function version can define the default maximum number of user-code calls in flight for one logical execution. - A refresh can override that default for that execution only. - The value must be a positive `u32` and is independent from `resource_budget`; it is not an account-wide quota. - Omitting the option preserves the previous request payload. Local embedded refreshes reject this Enterprise-only option. Rust, Python sync/async, PyO3, and the existing Node refresh surface forward the option. Node has no Function registration surface today, so no registration API is added there. The generated TypeDoc pages are included for the new public option. ## Validation The native-tested source covers Function version decoding, registration validation and serialization, the Flight SQL registration ticket and returned version, remote refresh forwarding, local rejection, and Node's numeric boundary before conversion. Exact stage results are Rust `1+3` plus four remote cases at `1` each; Python `14/1/49/24`; Node `57` remote and `1` local rejection. Ruff, standard repository Pyright, and Biome passed. The tested Python extension was `21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the tested Node addon and generated native declaration were `f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and `4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The final 24-file diff preserves the 22 tested product/test/fixture/source-doc blobs and adds only two generated TypeDoc pages. |
||
|
Xuanwo
|
8029428c71 |
feat: persist automatic Function refresh resource budgets (#4953)
## Depends on merged #4951 Phase 1 merged as `4d22c0ce2b6825c880c818e737b079db494b4053`, so this PR now targets `main`. It reuses the canonical `FunctionWorkerResourceBudget` introduced there rather than copying the type. For Function `add_columns` declarations, Python and Rust now persist `auto_refresh_resource_budget` in computed-column metadata and send it as `auto_refresh_resource_budget` in the remote request. CPU milliseconds and memory bytes are required positive integers. Omitting `gpu_count` is accepted as zero and the canonical outgoing JSON still contains `"gpu_count": 0`. Unknown fields, missing CPU or memory, wrong types, zero CPU or memory, negative values, and overflow are rejected. The same budget is applied to every public output binding of a multi-output Function declaration. Existing metadata is retained, and SQL-expression declarations or local execution reject the budget instead of silently ignoring it. Node is unchanged because its current `addColumns` surface only accepts SQL computed expressions and has no Function declaration path. This change does not implement fleet call concurrency limits or enforce the budget in the runtime. The Enterprise server must support the nested Function worker resource-budget contract before these values can act as execution limits. ## Validation - Python extension build passed. - Remote Function declaration and local unsupported-path rejection each passed. - Canonical metadata round-trip passed, including omitted GPU decoding to zero and explicit `gpu_count: 0` on encode. - Function multi-output remote wire test passed with `--features remote`. - Repository-scoped Python type check passed with 0 errors. - All runtime source blobs match the native-tested source. The post-native deltas are one documentation example line and an existing routing-test stub update for the new optional argument; the affected parameterized test passed 3/3 against the existing extension build. No private test runners are part of this PR. |
||
|
Xuanwo
|
c9d176ae7e |
feat: resolve latest Function versions (#4949)
## Problem Function catalog lookup required an exact version string even though callers commonly need the latest currently visible immutable revision. On the current SDK, resolving that version required an out-of-band catalog query before calling `get_function`. ## Change - Make Python `get_function(name)` default to latest while accepting `int` or `str` for exact lookup. - Normalize integer versions to their decimal string form and reject `bool` explicitly. Existing string validation and errors remain unchanged. - Thread `None` through the rebuilt PyO3 binding and route it to the new Rust `Connection::get_function_latest`. - Resolve latest through the existing catalog `DESCRIBE FUNCTION` path without `WITH (version = ...)`. The returned value is one exact immutable version, not a mutable alias. - Preserve the existing Rust exact-version API. ## Validation On the designated EC2 host with the rebuilt extension: - `test_local_function_catalog_operations_are_not_supported`: **1 passed**. This crosses the Python connection, rebuilt PyO3, and embedded Rust `NotSupported` boundary with the omitted-version call. - Async and sync catalog-path tests: **2 passed**, 117 deselected. - Remote SQL Function creation/description integration test: **1 passed**. - An earlier Rust invocation with the wrong module path matched **0 tests**; it is retained as harness evidence and is not counted as a pass. ## Scope This is the Function lookup SDK slice. It does not include the separate atomic rebind/publication server work tracked under ENT-3418. |
||
|
Xuanwo
|
4d22c0ce2b |
feat: expose Function refresh resource budgets in SDKs (#4951)
## Summary - Add the canonical Rust `FunctionWorkerResourceBudget` and `RefreshColumnOptions::resource_budget()` for explicit Enterprise Function refreshes. - Forward the nested budget through Python sync/async, PyO3, Rust remote requests, Node NAPI, and TypeScript. - Require positive CPU and memory values; normalize an omitted nested GPU count to numeric zero; reject the nested budget when combined with the top-level `gpu_count` shorthand. - Reject `resource_budget` on LanceDB Embedded refresh paths instead of silently ignoring it. - Document the Rust, Python, and TypeScript APIs, including that SQL servers currently require all three budget fields. The Rust, Python, and TypeScript additions use `cpu_millis` / `cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` / `gpuCount`. The top-level GPU shorthand keeps its existing behavior and remains mutually exclusive with the complete nested budget. ## Scope and compatibility This change configures only the explicit refresh call. It does not configure `add_columns` automatic refresh and does not introduce a fleet-wide call-cap interface. A LanceDB Enterprise server must expose the nested `resource_budget` backfill contract. LanceDB Embedded continues to support its existing expression refresh defaults and rejects the new complete budget explicitly. ## Validation Validated against the final signed product source: - Rust: resource-budget remote serialization test `2 passed`; validation test `1 passed`. - Python: actual PyO3 build; remote-request matrix `22 passed`; invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff check and format clean. - Repository-standard Pyright entry point: `0 errors`, within the repository's configured include set. - Node: native build and Biome lint passed; the existing remote-request test `1 passed`, including fractional, negative, overflow, NaN, and infinity nested GPU cases; the existing local-rejection test `1 passed`. - TypeDoc generation and generated-document cleanliness passed. - The native-tested product source differs from this head only in two semantically equivalent Node range checks required by Clippy. Current CI compiles and lints the final blobs. No dependency or lockfile changes are included. |
||
 BHAWESHBHASKAR
|
f7648d4a22 | feat(python): expose the ZoneMap scalar index (#4816) | ||
|
Xuanwo
|
c0bd83ef29 |
feat(sql): expose durable job ids on query handles (#4941)
Expose the durable SQL registry id separately from the connection-scoped query UUID. Rust and Python query handles and descriptions expose optional `job_id`; result readers preserve it in schema metadata under `x-lancedb-sql-job-id`, without adding business columns. Clients can reopen that id with `open_job`. Poll success and failure metadata update the same handle. A failure before the first successful poll retains its job id and original structured error. The cancellation documentation distinguishes the Flight waiter from `cancel_job(job_id)`. Validation on the designated EC2: the two focused error interoperability tests and Ruff passed on `a8798e81`; delivery has the identical tree. The preceding 51 Rust SQL tests and earlier 31 native Python SQL tests are reused for unchanged coverage. Matching-runtime independent acceptance remains pending. Companion server change: https://github.com/lancedb/sophon/pull/10049. Refs ENT-3562. |
||
|
Xuanwo
|
13e4cf7e58 |
feat: expose Function publication intervals in refresh options (#4928)
Python refresh APIs currently reject `publication_interval_rows`, so callers cannot request incremental Function output publication through the SDK. Add the optional typed parameter through the sync/async Python APIs, PyO3, and Rust `RefreshColumnOptions`, and serialize it as a top-level Enterprise refresh option. An omitted value preserves the existing wire request and whole-fragment default. Positive values set a publication threshold; zero and negative Python values fail before a request is sent. Embedded refreshes reject the Enterprise-only option. Documentation describes a threshold rather than an exact number of rows per publication. Validation on the designated development host: - A matching baseline build reproduced the missing public keyword argument in both sync and async submission tests. - The updated PyO3 build passed 54 focused Python tests; Rust payload, invalid-option, and default-wire tests each passed. Rust formatting and Python Ruff checks passed. - The single signed delivery commit differs from that tested implementation only in a Rustdoc correction describing Embedded rejection; formatting passed again. No dependency pins or lockfiles changed. The matching SDK is prepared for the separate server validation of failure/cancellation retention and missing-only refill. Those end-to-end results are still pending and are not implied by these SDK tests. Refs [ENT-3733](https://linear.app/lancedb/issue/ENT-3733). |
||
 Sarwar Bhuiyan
|
d8957238bf | fix(remote): stream query responses instead of buffering them (#4929) | ||
|
Jack Ye
|
794fa023cd |
feat: add materialized view refresh resource budgets (#4902)
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.
```python
job = view.refresh_async(
cpu_millis=8000,
memory_bytes=16 * 1024**3,
gpu_count=2,
gpu_class="h100",
)
job.wait()
```
```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
gpu_count = 2, gpu_class = 'h100');
```
The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.
Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.
The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.
Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
|
||
 Bruno Ramirez
|
546a28ad2c | feat: support remote compaction tuning and table inheritance (#4886) | ||
 Wyatt Alt
|
2af85bea89 |
feat: expose the remaining refresh options on the SDKs (#4890)
The server's backfill request takes a row filter, failure limits, per-call timeouts and worker sizing, and SQL exposes all of them through `REFRESH COLUMN ... WHERE ... WITH (...)`, but the typed SDKs could send only batch_size, GPU placement and the retry and error policies. Anyone needing a filtered refresh or a bounded skip policy had to drop to SQL. This adds `where`, `max_failed_fraction`, `max_failed_rows`, `row_timeout_seconds`, `init_timeout_seconds`, `num_workers` and `max_executor_processes_per_worker` to Python, Rust and Node, passed through unchanged. `RefreshColumnOptions` now serializes straight into the request body, so the three SDKs share one producer. Node gets a `RefreshColumnOptions` object, which also carries the options it lacked before. A LanceDB Embedded refresh rejects every option other than batch_size and GPU placement, naming the option. |
||
|
Xuanwo
|
1b40e479d3 |
fix(python): surface Function drop warnings (#4880)
Remote Python `drop_function` currently discards the dependent-view warnings returned by SQL DROP. Both the synchronous and coroutine APIs now emit those warnings through `warnings.warn` while preserving their boolean result and executing DROP once. The server support is already merged in lancedb/sophon#9820. The Rust decoder reads warnings across every result batch, including an empty first batch. Missing warning columns remain compatible with older servers; malformed or null warning lists are rejected. The hidden binding bridge carries the warnings to Python. The separate `drop_function_async` API and its `(bool, Job)` contract are unchanged. Python warning filters can raise after the catalog mutation has completed. Validation: a compiled PyO3/public-server reproduction failed before the fix and passed afterward; the public async path also returned `true` with the expected dependent-view warning. Final source passes two focused Rust tests, two Python wrapper tests, strict targeted Clippy, formatting, and Ruff. Three decoder tests cover batch/null/type and older-server behavior. The public runs preceded a two-file simplification of the completed Job projection; focused final-source checks cover that delta. Current-main integration is conflict-free, and its overlapping changes concern independent retry-policy APIs/tests. Documentation now covers warning filters, the post-removal warnings-as-errors behavior, and silent compatibility with older servers. The follow-up changes only prose and a Python docstring; the runtime validation above remains applicable. Refs ENT-3659. Whole-issue independent acceptance remains pending. |
||
 Yang Cen
|
6687db979d |
feat(functions): expose retry policies in Python and Rust (#4873)
Expose `FunctionRetryPolicy` on Function declarations and refresh overrides, together with `error_policy` and recorded attempt counts. Python uses the Rust validation rules before sending a request; local refreshes reject these Enterprise options. The policy bounds calls and backoff, supports class/message/provider-code matching, and can disable retries with `max_attempts=1`. Batch retry and skip operate on the whole failed call. Requires Enterprise server support for the new policy. Validation: rebuilt the Python extension; 47 targeted Python tests and 10 Rust tests passed. Python `make check`, Rust formatting, and `cargo clippy --features remote --tests --examples -- -D warnings` passed. The additional repository-root Ruff sweep reports existing errors under `docs/web-tests`; the Python workspace checks pass. Draft user guidance covers the supporting server requirement, safe repetition of partial work and whole-batch exhaustion. The guide sections stay hidden until release. Vale 3.24.0 passed for the added draft prose; no paired deployed SDK/server retry acceptance was run. Companion server: https://github.com/lancedb/sophon/pull/9812. BREAKING CHANGE: Rust callers constructing `FunctionRegistrationRequest` or `FunctionErrorRecord` with struct literals must supply the new `retry_policy` or `attempts` field, respectively. |
||
|
Xuanwo
|
57cd31e1aa |
test(python): load checked-in error classes for Rust tests (#4876)
Bare Rust test runners do not install the `lancedb` Python package, so the typed-error tests fail with `ModuleNotFoundError`. Load the checked-in exception modules once for the embedded interpreter, preserving exception-class identity across parallel tests and the existing type/cause assertions. Rust CI now watches the binding sources and the Python exception modules embedded by these tests, so those changes exercise the macOS Rust test jobs. |
||
|
Xuanwo
|
3c4aaa8e62 |
fix: preserve typed catalog errors through Flight and Python (#4867)
Catalog lookup failures lose their structured resource identity at the Rust/Python SDK boundaries, leaving callers with generic errors or empty correlation fields. Decode the existing Flight SQL error metadata and preserve resource kind/name, namespace, error code, and request/job IDs in the existing error types. Direct `get_function` calls unwrap named Function or namespace failures. Generic SQL job failures remain `JobFailedError` with the typed resource error as their cause; native deferred commands correctly leave durable job IDs null. Existing unstructured SQL classifications, including `FailedPrecondition`, remain supported. Validation: focused Rust/Python regressions and a native-extension run against a real Sophon process passed all five Function/Secret resource-error cases. The final one-commit candidate passes `cargo check -p lancedb-python` and formatting; earlier runtime evidence applies after removal of an unused tuple member. Independent deployed acceptance remains pending for ENT-3373. Server companion: https://github.com/lancedb/sophon/pull/9804. Integration validation: all 14 DocLinks pass on the merge candidate with #4869. The subsequent main documentation routing change was inspected; it does not alter the typed-error paths. The inherited pin-utils cargo-deny finding is tracked in a separate dependency repair. |
||
|
Yang Cen
|
69ac2158a1 |
fix: preserve take_row_ids request order (#4826)
`take_row_ids` used an `IN` filter, so the backend could reorder rows and collapse duplicate IDs. That can silently misalign training rows with labels or sample weights. Restore the requested order and duplicate occurrences in the shared Rust take path for native and remote tables, and retain that information through Python query serialization. Missing or deleted IDs are omitted. Explicit Rust `order_by` overrides request order, and pagination applies after restoring occurrences. Empty projections and `_rowid` projection aliases retain their output schema. RTE tables reject unsupported takes with the existing base-table guidance before checking the internal row-ID projection used for ordering. An explicit `use_rte(False)` reads the base table with the same requested order and occurrences. A pre-existing fixture defect also blocked CI: `test_table_add_in_threadpool` responded without reading uploads, allowing a connection reset during response decoding. This PR includes a fix confined to that test's handler: consume each request body before responding and verify one create row plus ten inserted rows. This was not a regression in the take implementation. Production write retries, concurrency and HTTP client code are unchanged. The selective sampling guide scopes the row-ID guarantees to SDK releases containing this fix, preserves the distinct offset-take semantics, and explains that ordered takes collect and combine matching rows before output batching. It recommends smaller ID requests and narrow projections because `max_batch_length` does not cap working memory. This uses the existing REST query protocol and requires a new SDK release; no deployed service was changed. Validation: - Docs follow-up: the changed page compiles with the Mint 4.2.888 MDX compiler; Typos 1.26.0 passes and Vale 3.24.0 has no findings on changed lines. Full-site Mint validation reports four existing snippet-import warnings on the reranking/vector-search pages, identical with the original page restored in a baseline comparison. - The applied threadpool test passed locally after the fixture change. The identical candidate previously passed 10 trials with the ordinary reader and 10 with an unbuffered reader; each trial verifies all uploaded rows. Those checks used macOS arm64/Python 3.14/default features, not the full failing CI environment. The normal new-head macOS Python 3.13/experimental-functions job remains the required platform check. - Python package Ruff format/check passed. Repository-root Ruff still reports 59 existing errors and six formatting differences in unchanged `docs/web-tests` files; none are in changed files. - At `a569d3f0`, the RTE diagnostic regression was reproduced and fixed; 5 targeted cases and strict workspace Clippy passed. The fixture-only follow-up changes no production Rust/Python code, so that evidence is reused. - Earlier validation at `6bcc0878`: 439 Rust query/remote-table tests and 24 Python take/getitems tests passed; the focused 49-test suite also covered offset dataloaders. A local 9,216-row, four-fragment reproduction returned 200 seeded random batches of 64 rows with zero ordering or label-alignment mismatches. The ordering and projection implementation is unchanged by the follow-ups. - No live Enterprise run was performed. Fixes [ENT-3469](https://linear.app/lancedb/issue/ENT-3469). |
||
|
Xuanwo
|
edc49e60e2 |
feat: expose Function failure tracebacks (#4861)
Function refresh failures and skipped-row diagnostics lose the traceback when decoded by the SDK. Expose the server-provided optional `traceback` on job failure details and Function error records in Rust, Python, and Node.js, while keeping existing compact error messages. Responses from older servers remain supported when the field is absent. Tracebacks may be truncated by the server. The Rust `Error::JobFailed` variant now stores `Box<JobFailure>` so the added diagnostics do not enlarge every `Result` error payload. |
||
|
Xuanwo
|
deb8834b5a |
fix: expose stable row IDs in Function errors (#4841)
Function error records expose physical row locations, which can change when a table is compacted. Expose the server-provided optional stable `row_id` through Rust and Python so callers can locate the same row after compaction. Document stable-ID retries and retain physical addresses for tables without stable row IDs. Older server responses continue to deserialize with `row_id=None`. The Python remote response test and Rust remote deserialization test passed on verification SDK `4e1725fa352035e2079dfd235b8271cd05adf433`, including decoding a legacy response without the new field. This branch applies the same SDK patch to current main; intervening main changes leave the affected error types and assertions unchanged. Formatting and lint checks passed after correcting the retry examples to use the `table.name` property. In a separate Linux server experiment, a real skipped Function row retained `_rowid=506` after compaction changed its physical address. The SDK field requires a server that emits stable row IDs; this PR does not change server dependencies. |
||
|
Yang Cen
|
2dedde9b6c |
feat(python)!: expose typed query errors and request metadata (#4832)
Remote SQL failures currently become generic runtime errors, and HTTP table lookup can discard server error metadata. Preserve transport status, current request ID, SQL job ID and worker request ID across Rust and Python. Select typed column, table, reference and schema exceptions only when an explicit server code matches the transport status. Remote `HttpError` and `RuntimeError` handlers remain compatible. Failed SQL jobs retain `JobFailedError` with a typed SQL cause when available. Missing metadata stays absent; message text never determines the exception class. Local missing-column errors retain their type and qualified-field suggestions, and `JobFailure.source` documents structured remote Flight SQL failures. This branch integrates main at `a9ad4997`, including its released Lance 15.0.0-beta.9 dependency update and remote optimize support. The only textual conflict was two independent additions to `test_remote_db.py`; both test groups and all original assertions are retained. The feature has no dependency or lockfile delta against current main. SDK runtime validation on `03ee633a` (runtime source unchanged in `f3f2d0b8`), macOS ARM64, pinned Rust 1.97.0 and the frozen uv Python 3.14 environment: - Rebuilt the native extension from the integrated source. All 22 targeted Python tests passed: native local errors, HTTP/Flight metadata, legacy sync/async table-not-found behavior, and the three main remote-optimize regressions. - All 52 Rust SQL, metadata/source and nested-field tests passed, including the new main optimize cases. - Root and Python-binding Rust Clippy with `-D warnings`, Rust formatting, Python package Ruff and conflict-file formatting passed. - Repository-wide Ruff reports 59 existing documentation findings, and its format check identifies six documentation files. Every affected file is byte-identical to current main; these checks are not reported as passing. - Direct integration review and AST comparison confirm both conflicting test groups were preserved. Earlier exception-constructor/pickling evidence remains applicable to the unchanged Python error modules. The unlocked build exhausted disk again on a runner with the same label but only 9.4 GiB free on a 72 GiB root filesystem. Keep the existing runner and dependency caches, serialize Cargo compilation, and remove only completed test/bench executables reported by Cargo. This job never runs those binaries; normal binaries, libraries and build-script outputs remain available. Fresh resolution, the ci profile, all features, and test/bench compilation and linking are retained. Three probes using Cargo and target-path validation pass with `-D warnings`: all fixture targets link while ordinary outputs remain usable; an undefined symbol still produces Cargo exit 101 and its linker diagnostic; files outside the target directory are never removed. The [current-head build-no-lock job](https://github.com/lancedb/lancedb/actions/runs/41911613851/job/126860364065) passed on `f3f2d0b8`: all compilation/linking completed, 20 completed test executables were removed, and `target` retained about 7.3 GiB. This successful run had a 145 GiB root with about 78 GiB free afterward; it was not an identical rerun of the failed 72 GiB image. The driver tests also passed in this job. All previous-head jobs ended naturally before the push; new-head CI and review gates still apply. Typed remote input exceptions require the server metadata contract. Older servers retain generic transport errors and existing missing-table compatibility. The optimize tests use mock servers and do not establish downstream server acceptance. BREAKING CHANGE: Rust callers with exhaustive matches on `lancedb::Error` must handle the new `FieldNotFound` and `Sql` variants. Local `Error::ColumnNotFound` failures now raise `lancedb.exceptions.ColumnNotFoundError`, a `ValueError` subclass. Python callers that previously caught only `RuntimeError` for this local failure must catch `ColumnNotFoundError` or `ValueError`. |
||
 Bruno RamirezandClaude Sonnet 5
|
a9ad499781 |
feat: support remote table compaction via SQL OPTIMIZE TABLE statement (#4802)
`Table::optimize()` previously returned `NotSupported` for remote tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL. Only `OptimizeAction::Compact` without a custom `remap_options` or non-default `CompactionOptions` is supported remotely. `OptimizeAction::All`, `::Prune`, and `::Index` all return a clear `NotSupported` error instead of silently doing the wrong thing. A branch-scoped table handle submits a `WITH (branch = '<branch>')` clause; the statement doesn't validate the branch exists itself, so an unknown branch surfaces as an async job failure rather than an error from this call. The server's SQL job adapter doesn't expose compaction metrics, so `OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a placeholder rather than real numbers — it must still be `Some`, since callers (including the Python/Node bindings) unwrap it unconditionally on success. This mirrors how `refresh_materialized_view_async` already handles the same limitation for its own metrics. The Python and Node `optimize()` convenience methods call a new shared `Table::optimize_full`, which runs `Compact` and then, on a remote table, skips `Prune` and `Index` instead of attempting and failing on them — returning zeroed prune stats rather than submitting a compaction job and then raising. Python's `RemoteTable.optimize()` also now forwards `compaction_options` to the Rust layer, so a non-default value is rejected before submission instead of being silently dropped. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Xuanwo
|
b09735529b |
feat(sql): return durable materialized view job handles (#4824)
Materialized-view creation and refresh can outlive the submitting connection, but their SQL query IDs cannot be reopened through the Job Registry. On servers that negotiate early handles, return the durable registry job ID and use the existing job API to describe, reopen, wait for, or cancel it. Older servers retain the connection-scoped query handle. Registry refresh handles forward running and terminal progress observations and establish the committed-version read-freshness fence before wait returns. Preserve a nested SQL failure ID separately from the outer job ID through Rust and the Python native bindings, so Function failures remain traceable. Rust callers constructing `JobFailure` directly must supply the new optional `sql_job_id` field or use `..Default::default()`. Validation: seven SDK regression cases and the docs-link test pass, including plain wait and running-to-terminal progress observations; the observer regression fails before the fix. Formatting and strict Clippy pass. Earlier accepted coverage includes a real Flight test service exercising materialized-view creation followed by registry describe/reopen/cancel, and two Python tests after rebuilding the native extension. The published product tree matches the validated candidate. Refs ENT-3562. The matching Sophon server change is being validated separately before publication. |
||
|
Xuanwo
|
1544ad65a8 |
feat: expose structured job progress to SDK callers (#4808)
Job callers currently lose the server's structured progress when describing, listing, or waiting for a job. Expose Rust and Python progress snapshots and optional wait observers using the existing backend poller. Python supports callbacks, a caller-owned tqdm-like bar, or `progress=True`; `status()` still returns a string and the default wait behavior is unchanged. Preserve unknown progress fields and phases from newer servers, and report omitted counters as unknown. Each wait calibrates a caller-owned bar from its first complete snapshot, then uses a high-water mark within the same execution epoch; a new epoch or smaller plan resets the bar. ETA is best-effort and omitted for missing, stale, paused, or incompatible samples. Newly submitted index and refresh handles forward observers through their freshness wrapper while retaining successful-wait read consistency. The Jobs guide includes draft Python/Rust snapshot and observer examples for release; TypeScript progress support remains tracked under ENT-3500. Validation on the development EC2 host: the initial 11 Python progress tests, REST fixture caller, and two Rust progress regressions passed. After fixing freshness-wrapper forwarding, rebuilt the Python extension and ran two submitted-handle Python tests plus the new Rust refresh and index regressions (one test each); all passed. Strict Clippy, rustfmt, and targeted Ruff passed. The final commit only wraps the new guide section in an MDX draft and clarifies language support. Repository-wide Ruff reports 48 existing findings in six unchanged docs/web-tests files. This publishes the SDK portion of ENT-3500. Server-side live-split and terminal-progress changes and the combined server/SDK end-to-end scenario remain in progress. Refs ENT-3500. BREAKING CHANGE: Rust struct literals for `JobInfo` must add `progress: None`, `phase: None`, and `paused: false`. Literals for `JobDescription` must add `progress: None`, `phase: None`, `queue_reason: None`, `waiting_for_job_id: None`, and `paused_at_millis: None` when that metadata is unavailable. Existing Python calls remain compatible. |
||
|
Xuanwo
|
2d70bec53a |
feat: add UTM-tagged docs links to SDK warnings and enterprise-only errors (#4797)
Add a central documentation link registry (`lancedb::docs::DocLink`) in the Rust core that builds `docs.lancedb.com` URLs with UTM parameters (`utm_source=lancedb_sdk`, `utm_medium=warning|error`, `utm_campaign=<id>`, `utm_content=<sdk>-<version>`) placed before any `#anchor` fragment and without user-identifying data. - Initialize `utm_content` per language binding (`python`, `nodejs`, `cli`, defaulting to `rust`) and expose the link builder to Python so `DBConnection` and `Job` stubs reuse the core definitions. - Append published documentation links to Enterprise-only `NotSupported` / `NotImplementedError` errors across database, job, catalog, and table operations. - Emit rate-limited maintenance warnings (once per table per process when `warn` logging is enabled) when opening a native table with at least 100 small fragments (`< 100k` physical rows each) or at least 1,000,000 unindexed rows on an optimizable user index, reading only manifest metadata without opening data or `_indices/` files, and document the thresholds in the compaction and reindexing guides. - Verify published/in-repo `docs/web/` targets and `#anchor` fragments in unit tests and include extracted SDK links in the scheduled `docs-link-check` workflow. Linear: OSS-2772 |
||
  LeoandLeo
|
f2bcb156ad |
feat: expose approx_mode on vector queries
`ApproxMode` landed in the Rust core in #3549, but the Python and Node.js bindings were never plumbed, so `approx_mode` is unreachable from either SDK. Both were built mirroring the existing `ef` plumbing. Co-authored-by: Leo <leo@insight.co> |
||
 Srijan Keshri
|
90b2b264d9 |
feat(python): accept DataFrames and readers in add_columns
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader. Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them. SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits. Fixes #1959. |
||
 lancedb-gatefixer[bot]
|
7466dc9d1d |
fix(python): reject embedding source updates without vectors
Fixes #4761 `Table.update` does not compute embeddings, so changing an embedding source column could leave the previous embedding in place. Source-column updates now raise `ValueError` unless every associated vector column is supplied in the same call. The guard reads Python embedding metadata and native `lancedb::column_definitions` without loading embedding models. It uses Lance's field-path parser for quoted targets and covers synchronous and asynchronous local and remote tables. Supplying every associated vector column, vector-only updates, and unrelated updates remain supported. This is a breaking change for source-only updates. The API documentation and Update rows guide explain the requirement and the `merge_insert` alternative for regenerating embeddings. Validation at the reviewed head covered 44 targeted public update cases and the full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable exact-head checks and workflows completed successfully. |
||
|
lancedb-gatefixer[bot]
|
ea9194a7f3 |
fix(python): retry transient job polling errors
Fixes #4773. Python job polling now retries transient connection, send, and read failures with capped exponential backoff under the caller's original timeout. If the deadline expires while disconnected, `wait()` raises `ConnectionError` with the job ID, reconnect guidance, and the original cause; after recovery, ordinary timeouts remain `TimeoutError`. Terminal job errors, cancellation, and malformed responses continue to surface immediately. Validation: exact-head CI passed across Rust all-features, Python on Linux, macOS, and Windows, NodeJS, deny, and packaging checks. The hosted macOS wheel suite passed 3049 tests (162 skipped, 70 deselected), including sync and async recovery, timeout reuse, cancellation, terminal errors, and error preservation. |
||
  TaylorandBrendan Clement
|
9e65bd2c27 |
docs: use Enterprise terminology across website and SDK documentation (#4787)
Use “LanceDB Enterprise” throughout the staged website guides, SDK reference sources, generated API pages, and user-facing SDK messages. A separate branded “LanceDB Distributed” name adds terminology without clarifying behavior; describe where operations run directly. - Update `docs/web`, including the architecture diagram title and alt text. Use Enterprise in headings, table labels, and prose; simplify redundant product/execution-model wording and name the product only once in the diagram alt text. - Update Python docstrings and the Python/Java reference pages, TypeScript comments and native declaration comments, and Rust API documentation. Regenerate the TypeScript reference, including 24 replacements across six pages. - Update SDK warnings/errors and existing test expectations to match, along with contributor guidance and Vale rules. Remove the obsolete rule distinguishing Enterprise from Distributed. - Keep lowercase “distributed” for technical descriptions. API identifiers, anchors, asset paths, and execution behavior are unchanged. No branded-name exceptions are proposed. Validation: - Python and Node native builds succeeded; focused Python tests passed (36 passed, 4 skipped), and focused Node tests passed (14). - TypeDoc generation and the complete MkDocs reference build passed. MkDocs retains an existing unresolved `lancedb.functions.SecretReference` cross-reference. - Rust formatting, Node lint/format, changed-file Python lint/format, 32 Vale tests, and whitespace checks passed. Repository-wide Python lint has 48 existing findings in six untouched files. - Combined documentation assembly passed. Mintlify validation and link/anchor/redirect checks previously passed for both site builds; no code snippets changed. Companion PRs: [saved editions, terminology guide, and shared rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview and writing rules](https://github.com/lancedb/sophon/pull/9511). --------- Co-authored-by: Brendan Clement <brendan.clement@lancedb.com> |
||
 SHAIK SHIVAJI
|
45edaa350e |
feat(python): expose code analyzer options on FTS and tokenize
Part of #4364 (Python side; the TypeScript `FtsOptions` change will follow in a separate PR). Lance already supports the code analyzer, but `FTS` and `lancedb.tokenize()` had no way to set it, so `FTS({'analyzer': 'code'})` and `FTS(split_identifiers=True)` raised `TypeError`. ## Changes - `FTS` and `lancedb.tokenize()` gain `analyzer` (`"text"` or `"code"`), `split_identifiers`, `split_on_numerics`, `preserve_original` and `index_operators`. They default to `None`, which keeps the tokenizer's own defaults, so existing indexes and calls are unchanged. - Validation follows pylance: `analyzer="code"` with a `base_tokenizer` other than `"code"` or the default `"simple"` is rejected, `analyzer="text"` with `base_tokenizer="code"` is rejected, and turning on a code flag without the code analyzer is rejected. Passing a flag as `False` is always allowed. - `analyzer="code"` with the default `base_tokenizer="simple"` selects the code tokenizer. `FTS` can't tell an explicit `"simple"` from the default, so that case is accepted too. - The deprecated `create_fts_index` is left alone; it points users at `create_index(config=FTS(...))`. - Rust `FtsIndexBuilder` is Lance's `InvertedIndexParams`, so the Rust SDK already had these options. The remote client serializes the same struct, so the options reach LanceDB Distributed in the `create_index` request body. A new case in the Rust `test_create_index` checks that body. ## Tests - `python/tests/test_fts.py`: tokenizer output for each flag, validation errors for `tokenize` and `FTS`, `False` flags on a text index, and local sync and async indexes that match `http`, `request` and `parseHTTPRequest` with splitting on and not match subwords with it off. - `cargo test --features remote remote::table::tests::test_create_index` |
||
|
Xuanwo
|
63aa06ef7d |
feat(python): expose Function GPU classes and placement (#4791)
## Summary - Add sync/async `DBConnection.list_gpu_classes()` returning typed `GpuClass` records for `GET /v1/gpu/classes`. Records include the class name, Kubernetes resource, GPU model and memory, per-worker GPU maximum, and worker CPU/memory/ephemeral-storage limits. - Add `udf(gpu_class=...)` for immutable Function-level GPU placement. It requires `gpu=True`; Function versions expose their registered `gpu_class`. - Add `gpu_class` and `gpu_count` to `refresh_column` / `refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as top-level `backfill_column` fields and validated before any request. `gpu_count` is the aggregate GPU budget across Function workers. - Validate GPU class names with the server lowercase 63-character DNS-label rule. - Document the discovery-to-registration flow, configured hardware versus free capacity, System Usage permission, and per-refresh GPU placement in the API and web guides. Older servers without `/v1/gpu/classes` produce a clear `NotImplementedError` instead of a bare 404. Part of ENT-3470. The client follows the in-flight Sophon contract on `xuanwo/ent-3470-fix` for GPU class discovery and backfill placement options. ## Breaking change `PythonRuntimeSpec::PythonV2` adds `gpu_class`, and `RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers constructing either value with all fields explicitly must supply the new fields. ## Validation At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python extension build, pyright, ruff, and Python Function/runtime, discovery, refresh placement, and refresh regressions (147 passed, 6 skipped). At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright, Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed). The final commit only updates documentation and Rust doc comments. |
||
|
Xuanwo
|
6a255926d7 |
fix(python): improve function failure diagnostics (#4790)
## Summary - Make `AsyncJob.wait(timeout)` and `Job.wait(timeout)` raise a descriptive `TimeoutError` containing the job ID and elapsed timeout. Registry jobs explain that work continues on the server and include the `db.open_job(id).wait()` reattach call. SQL query handles instead say to keep the original handle because query IDs cannot be reopened through the registry jobs API. - Render `JobFailureInfo` and `FunctionErrorRecord` with Python values instead of Rust `Option` wrappers. - Decode and expose the server's `row_address` on `FunctionErrorRecord`, and document how to retry recorded rows with `REFRESH COLUMN ... WHERE _rowaddr IN (...)` before compaction or rewrite. - Warn when an import inside a packaged Function body resolves to undeclared local source. The `udf` documentation explains why arbitrary base-image coverage and distribution-to-import-name mapping cannot be validated reliably by the SDK. `row_address` is optional, so records from older servers still decode with `None`. Rust code that constructs `FunctionErrorRecord` literals must now supply this new field. Part of ENT-3448 and ENT-3442. This consumes the server-side diagnostics from lancedb/sophon#9509 and logical row addresses from lancedb/sophon#9512. ## Validation - Rust `function_errors` remote decoding and legacy-missing-`row_address` tests. - Rust/PyO3 clippy for `lancedb` and `lancedb-python`. - Python extension build, pyright, and ruff. - Targeted Python job, Function packaging, Function error, and job detail tests: 62 passed, plus timeout-handle tests: 3 passed. |
||
|
Xuanwo
|
9786f2b63b |
fix(python): surface typed remote resource errors (#4789)
## Summary - Parse structured `error_code` and `resource` fields from remote HTTP error bodies and carry them through Rust/PyO3. - Add `SecretNotFoundError`, `FunctionNotFoundError`, `NamespaceNotFoundError`, and `AlreadyExistsError` as `HttpError` subclasses. - Expose `error_code`, `resource_kind`, `resource_name`, and `namespace_path` on HTTP exceptions while preserving `status_code`, `request_id`, and existing `except HttpError` handlers. - Do not retry structured domain errors that would otherwise become `RetryError` (notably `already_exists` on 409). Older servers that do not provide the structured fields continue to raise a plain `HttpError`. ## Compatibility This consumes the stable resource error contract added in lancedb/sophon#9516 (merge commit `35858561b3`). The existing `TableNotFoundError` behavior is unchanged. Part of ENT-3373. |
||
|
BHAWESHBHASKAR
|
a9146d536a |
feat(python): expose the NGram scalar index
## What kind of change does this PR introduce?
Feature. Exposes the NGram scalar index in the Python SDK.
## What is the current behavior?
The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.
## What is the new behavior?
Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:
```python
from lancedb.index import NGram
await table.create_index("text", config=NGram())
```
The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.
Changes:
- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.
This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
|
||
|
lancedb-gatefixer[bot]
|
71b23ed9fb |
fix(python): preserve request metadata for missing remote tables
Fixes #4750 Remote missing-table errors retained HTTP metadata in Rust but the Python binding converted them to a plain `ValueError`, discarding the request ID and status. This raises a `TableNotFoundError` that preserves the original message, request ID, status code, and chained HTTP cause while remaining compatible with existing `ValueError` and `lance_namespace.errors.TableNotFoundError` handlers. Local and non-HTTP errors retain their previous behavior. Validation covers sync and async opens across cached/uncached and root/nested paths, exception pickling, local compatibility, the remote and error test suites, Clippy, formatting, Ruff, and all applicable PR checks including CodeQL. |
||
 Yui Numazawa
|
faa8953d81 |
fix(python): allow attribute access on namespace request dicts
Allows Python namespace implementations to read request dictionary entries as attributes while preserving mapping and `model_dump()` behavior. Missing dunder attributes continue to raise `AttributeError`. Adds a real namespace connection regression covering list/open operations and omitted optional fields. |
||
|
lancedb-gatefixer[bot]
|
9804f26bc3 |
fix: support batch size options for column refreshes (#4777)
Python column refresh methods rejected `batch_size`, and the Rust remote backend never included it in the backfill request. Callers could not reduce the rows passed to a slow Function to stay within the executor's per-call deadline. Add optional, keyword-only `batch_size` arguments throughout the synchronous and asynchronous Python APIs and bindings. Add compatible Rust `RefreshColumnOptions` APIs while preserving existing Rust method signatures. Reject non-positive sizes before submitting a job, send positive sizes in `backfill_column`, and omit the field when unset while retaining branch scoping. Batch size controls remote Function calls. Embedded expression refreshes accept and validate the option without changing execution, and blocking remote refreshes retain their existing unsupported behavior; remote callers use `refresh_column_async` and wait on the returned job. Validation: - Reproduced the original Python `TypeError` from the committed method. - Five focused Rust tests passed, covering request contents, omission, branch scoping, and rejection before any request. - `cargo check --quiet --features remote --tests --examples` passed. - `cargo clippy --quiet --features remote --tests --examples -- -D warnings` passed. - Nineteen targeted Python tests passed against the rebuilt extension, including synchronous and asynchronous remote requests, invalid sizes, and default-build behavior. Eight embedded refresh cases require the experimental-functions build and were skipped. - Rust formatting and Python formatting/lint checks passed, including CI's pinned Ruff version. Repository-wide Ruff reports 48 existing findings in untouched documentation files; none are in changed files. Fixes #4776 <!-- lance-gatekeeper-fix:v1 agent=4906ff3f5f7797a1d49a13803302021c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
Bruno Ramirez
|
5f7ec4e0a2 |
feat(sdk): expose table overrides for remote tables
Expose table-level maintenance override read, update, and reset APIs for remote
Rust and Python tables. Local tables continue to return `NotSupported`.
Updates recursively merge the caller-provided JSON object into the current
override document so unrelated and unknown service fields are preserved. Reset
uses the explicit reset endpoint.
The integrated change keeps current main's remote database identity, namespace,
branch, SQL, WAL, and threaded mock-server behavior.
Validated with focused Rust and Python override tests and the complete applicable
CI run for
|
||
|
Jack Ye
|
29691719fa |
feat: manage property graphs defined over tables
Replaces #4322, keeping only its property graph defined over tables. A property graph maps node labels and relationship types to existing tables of its namespace, in the shape of SQL/PGQ's `CREATE PROPERTY GRAPH`. It stores only that definition: a query reads the tables as nodes and edges through SQL `GRAPH_TABLE`, as they are when it runs. - **Rust:** `Connection::create_property_graph` takes a `PropertyGraphDefinition` of `NodeTable`s and `EdgeTable`s. `describe_property_graph`, `drop_property_graph` and `list_property_graphs` manage it. - **Python:** `lancedb.graph` has `NodeTable`, `EdgeTable` and `Endpoint`. Sync and async connections gain `create_property_graph`, `open_property_graph`, `drop_property_graph` and `list_property_graphs`, and a `PropertyGraph` handle describes its graph. - **Support:** remote connections only. Local connections raise `NotImplementedError` in Python and return `NotSupported` in Rust. ```python import lancedb from lancedb.graph import EdgeTable, Endpoint, NodeTable db = lancedb.connect("db://demo", api_key="...") db.create_property_graph( "social", nodes=[NodeTable("person", key="person_id", label="Person")], edges=[ EdgeTable( "knows", label="KNOWS", source=Endpoint("src_id", references=("person", "person_id")), destination=Endpoint("dst_id", references=("person", "person_id")), ) ], ) db.execute_query(""" SELECT * FROM GRAPH_TABLE (social MATCH (p:Person)-[:KNOWS]->(f:Person) WHERE p.age > 30 COLUMNS (p.name AS person, f.name AS friend)) """).read_all() ``` |
||
  ryan-haynesandRyan Haynes
|
d316cbbed4 |
fix!: support opt-in PKCE for device-code login (#4747)
## Problem and change
Some Keycloak client policies require S256 PKCE during device-code
login. Sending those parameters unconditionally breaks interoperability
with providers that expect standard RFC 8628 requests.
Configure device login with `OAuthFlow::DeviceCode(DeviceCodeOptions)`,
following the existing authorization-code options pattern.
`DeviceCodeOptions::new()` keeps PKCE disabled; `.use_pkce(true)` sends
an S256 challenge during device authorization and its matching verifier
during token polling. There is no automatic downgrade, and enabling PKCE
preserves the existing token-cache identity. Python, Node, and CLI
conversions retain ordinary device login without PKCE.
OIDC discovery and token requests now include a versioned User-Agent.
`env!("CARGO_PKG_VERSION")` is expanded at compile time and embeds the
version in the binary; no Cargo environment variable is needed when
running an installed binary or native binding.
The supported-flows guide documents the Rust-only device PKCE option and
its default.
## Rust API migration
BREAKING CHANGE: `OAuthFlow::DeviceCode` now carries
`DeviceCodeOptions`. Replace construction with
`OAuthFlow::DeviceCode(DeviceCodeOptions::new())` or
`OAuthFlow::DeviceCode(Default::default())`, and match with
`OAuthFlow::DeviceCode(_)`. Import `DeviceCodeOptions` from
`lancedb::remote` to opt into PKCE. Python and TypeScript public APIs
are unchanged.
## Validation
- `cargo +1.97.1 test --locked -p lancedb --features remote --lib
remote::`: 679 passed, including PKCE-on/off polling, option defaults,
User-Agent headers, and cache identity.
- `cargo +1.97.1 test --locked -p lancedb-python -p lancedb-nodejs --lib
test_device_code_conversion`: both native-binding conversion tests
passed.
- `cargo +1.97.1 test --locked -p lancedb --features remote --doc
remote::`: 6 passed, including the new options example and updated
session examples.
- `cargo +1.97.1 test --locked -p lancedb-cli --lib auth::tests::`: 11
passed.
- `cargo +1.97.1 fmt --all -- --check` and `git diff --check`: passed.
- A compiled version-string check ran successfully with
`CARGO_PKG_VERSION` unset at runtime.
---------
Co-authored-by: Ryan Haynes <ryan.haynes@lancedb.com>
|
||
|
Jack Ye
|
6d126a3601 |
feat: add database storage integration APIs (#4618)
Adds database-owned S3 storage integrations with Secret-style namespace
paths in a separate keyspace. Use a regular database under the cluster
root for bootstrap integrations and Secrets. `default` is an example
name, with no enforced or automatically provisioned database; `system`
remains read-only. Authentication uses either an IAM role or two
existing Secret references, including cross-database references;
configuration metadata never contains credential values.
### Python experience
```python
catalog = lancedb.connect_catalog("https://my-server.example", api_key="secret")
db = catalog.connect_database("default")
info = db.create_storage_integration(
"raw", "s3://customer-bucket/raw/",
aws_role_arn="arn:aws:iam::123456789012:role/lancedb-access",
aws_region="us-east-1",
)
print(info.configuration_instructions)
info = db.describe_storage_integration("raw")
print(list(db.list_storage_integrations()))
db.drop_storage_integration("raw")
```
For static credentials, replace `aws_role_arn` with
`aws_access_key_id_secret=SecretReference("s3_key_id",
database="default")` and
`aws_secret_access_key_secret=SecretReference("s3_secret_key",
database="default")`, imported from `lancedb.storage_integration`. Pass
`namespace_path` to address nested namespaces. Async database
connections expose the same APIs; listing is lazy. All four client
operations execute SQL through the existing SQL transport, with typed
Arrow result decoding and name-based listing cursors.
### SQL experience
```sql
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/lancedb-access'
STORAGE_AWS_REGION = 'us-east-1';
DESCRIBE STORAGE INTEGRATION default.public.raw;
SHOW STORAGE INTEGRATIONS IN default.public;
DROP STORAGE INTEGRATION IF EXISTS default.public.raw;
CREATE STORAGE INTEGRATION default.public.raw
STORAGE_PROVIDER = 'S3'
STORAGE_PREFIX = 's3://customer-bucket/raw/'
STORAGE_AWS_ACCESS_KEY_ID_SECRET = default.public.s3_key_id
STORAGE_AWS_SECRET_ACCESS_KEY_SECRET = default.public.s3_secret_key;
```
CREATE and DESCRIBE return configuration and a single sentence of setup
guidance. IAM mode also returns the deployment role principal, stable
external ID, customer trust policy, and prefix-scoped permissions
policy. Registration does not verify access. Creation requires namespace
CREATE_STORAGE_INTEGRATION plus parent USAGE; Secret references require
source database, namespace, and Secret USAGE. Describe requires object
USAGE; drop requires OWNERSHIP. SQL supports GRANT/REVOKE for these
objects and their creation privilege.
Includes client documentation and Flight SQL regression coverage; no
dedicated storage-integration REST endpoints are used. Local
builds/tests are excluded by request.
|
||
|
BHAWESHBHASKAR
|
891774343e |
feat(python): add CombinedFieldsQuery for BM25F full text search
Add a typed Python `CombinedFieldsQuery` for BM25F search across multiple columns. The query mirrors Lance's combined-fields semantics with ordered per-column boosts and AND/OR term matching. Convert it in both directions at the PyO3 boundary, including nested Boolean queries, and document the Python typed convenience entry. Closes #4652. |
||
|
lancedb-gatefixer[bot]
|
741cb9ad4e |
fix(python): implement remote namespace and table existence checks (#4725)
Remote Python connections inherited the unsupported-operation stubs for `namespace_exists` and `table_exists`, so both raised `NotImplementedError` even when namespace and table operations worked. Add the missing remote overrides, async methods, and PyO3 bindings using the existing namespace describe and table open paths. Only an explicit not-found result returns `False`; permission, transport, and server errors propagate. This preserves authentication, dynamic headers, retries, and nested namespace handling, and rechecks cached tables before reporting that they exist. The async namespace helper also recognizes native `NamespaceNotFound` errors so missing local namespaces return `False`. The table and namespace guides document Python remote support and distinguish it from the unsupported `exists` routes used by direct REST namespace clients. Validation: - All 40 focused regression cases pass, covering successful and missing lookups, local and remote namespaces, dynamic headers, permission/server errors, invalid table identifiers, and cached tables that disappear. - The original inherited stubs reproduce the reported `NotImplementedError`; the local namespace regression reproduced the native missing-namespace `RuntimeError` before its fix. - Full remote connection suite: 255 passed. Local namespace tests: 14 passed. - `cargo clippy --quiet -p lancedb-python --features remote --lib`, `cargo fmt --all`, Python Ruff lint, and Python formatting checks pass. - Root `ruff check .` reports 48 existing errors in documentation examples under `docs/web-tests`; the Python subtree passes and those unrelated files are unchanged. Fixes #4722 <!-- lance-gatekeeper-fix:v1 agent=4d369a0681756fdb5a8b35853d7f5607 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Will JonesandClaude Opus 5.5
|
4d7a64712f |
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Wyatt Alt
|
bb6aa1a147 |
feat: add a cascade option to drop_database (#4697)
The remote catalog could only drop an empty database: it always asked
the server for a restricted drop.
`DropDatabaseRequest::cascade`, `drop_database(..., cascade=True)` in
Python and `dropDatabase(name, { cascade: true })` in TypeScript ask the
server to drop everything in the database first. The name stays taken
until the server finishes cleaning up the dropped objects' storage. A
restricted drop remains the default.
|
||
|
lancedb-gatefixer[bot]andXuanwo
|
2cd635d41c |
fix: correct remote materialized view refresh results and options (#4672)
Remote materialized-view refreshes drained the SQL result and then fabricated `mode=rebuild`, `rows_written=<total view rows>`, and `source_version=0`. This made incremental refreshes report incorrect statistics. Make these three result fields optional across Rust, Python, and TypeScript, and leave them unset for remote SQL refreshes. Remove the post-refresh row count request. Local refreshes continue to populate all metrics, and `version` remains the view version observed after completion. Remote per-operation statistics remain unavailable until the SQL job adapter can retrieve them. Expose the forced-rebuild option through the Rust builder, synchronous/asynchronous Python refresh and job methods, and TypeScript refresh. Remote `full=true` sends `WITH (full = true)`; the default lets the server choose its refresh mode. Local source-version pinning and remote job cancellation are preserved. Local refresh regression tests respect the `experimental-functions` build gate, while remote refresh coverage also runs in default builds. Validation: - Rust: 20 materialized-view tests and 21 remote SQL tests passed with `remote,experimental-functions`; `cargo fmt --all` passed. - Python: rebuilt the extension in the uv-managed environment. The default build passed 23 tests with 27 feature-dependent skips. The opt-in build passed 61 tests with 3 expected default-build-only skips, covering the materialized-view suite and 14 async query-serialization cases from current main. Python Ruff lint and formatting checks passed. - TypeScript: default and opt-in native builds and TypeScript compilation passed. The default build passed 5 tests with 10 expected feature-dependent skips; the opt-in build passed all 14 materialized-view tests with the default-build-only test skipped. Lint/format checks and documentation generation passed. - Repository-wide `ruff check .` reports 48 pre-existing violations in unchanged `docs/web-tests` files; `ruff check python` passes. Fixes #4670 <!-- lance-gatekeeper-fix:v1 agent=ece43ec4b88a48d1789f60005fa3ecfc generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
7fca4d666a |
fix(python): preserve async query search options during serialization (#4659)
Async `to_query_object()` dropped enabled `fast_search` on every query type and `postfilter` on plain and full-text queries. Re-executing a serialized fast search could therefore return unindexed rows. Copy enabled `fast_search` into the Python `Query` and serialize enabled plain/FTS postfilter from the Rust request's prefilter setting. Preserve the existing unset defaults: `None` for these options, with the vector query's existing `False` postfilter default. Validation: - Added 12 serialization cases across plain, FTS, and vector queries, covering both enabled and unset options, plus two FTS execution round-trip cases. Before the fix, nine cases failed, including returning an unindexed row after serialization. - Rebuilt the Python extension using the uv-managed environment. - Query tests and existing FTS fast-search tests: 192 passed, 1 slow test deselected. - Rust formatting and Python package lint/format checks pass. Repository-wide `ruff check .` reports 49 pre-existing errors in unchanged `docs/web-tests` files. Fixes #4658 <!-- lance-gatekeeper-fix:v1 agent=f828b24871a2dc928668be2de4f85078 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
1edfa6077b |
fix(python): allow phrase queries without a column (#4653)
Quoted async full-text queries without `columns` carry an unset phrase column, but converting them to Python unwrapped it and raised `PanicException`. Python also required `PhraseQuery.column`, preventing callers from constructing the equivalent query object. Make the Python column optional and preserve `Option<String>` in both Rust conversion directions, leaving column selection to query planning. Regression tests cover quoted strings and structured queries with omitted, `None`, and explicit columns, query reuse, JSON serialization, slop, and document granularity. Validation: - Reproduced both missing-column failures before the fix; explicit-column controls passed. - Rebuilt the extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`. - `cd python && uv run --no-sync --extra tests pytest python/tests/test_fts.py python/tests/test_query.py::test_query_serialization_async -q`: 85 passed. - `cargo fmt --all -- --check` and `ruff check python` pass; touched Python files pass formatting checks. - `cargo clippy --quiet -p lancedb-python --lib --features remote` passes. - Root-wide `ruff check .` reports 49 existing findings in unchanged documentation examples. Fixes #4651 <!-- lance-gatekeeper-fix:v1 agent=62d36fad70c8717236283b8e97dd6b30 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |