mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-16 07:36:50 +00:00
5de76bd8edebb1aae915591683bdcfa642807032
140
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f6311ae7e9 | feat: support exist_ok for index creation (#4993) | ||
|
|
9b13e8f725 |
feat(python): materialize remote blobs at the query snapshot (#4973)
Python remote queries materialize Blob v2 payloads with `blob_mode="bytes"` through synchronous and asynchronous `to_arrow`, `to_list`, `to_pandas`, and `to_batches`. Deferred reads bind to the actual producing dataset without changing the caller's live table handle, including concurrent writes and same-name DROP/CREATE. Trust only the authoritative `x-lancedb-query-version` plus `x-lancedb-query-snapshot`. A numeric version alone cannot distinguish a recreated table; the ordinary live watermark and user-supplied Arrow metadata cannot establish provenance. An independent immutable handle carries the original identity through every batch, byte-limit split, and large-Blob Range fallback, and requires a verified response echo before accepting payloads. Aliases, NULL/empty results, rerankers, and hybrid legs retain consistent provenance. Missing capability, mixed identities, or a serving node ignoring the precondition fails explicitly; ordinary descriptor queries remain available on older servers. The source docs distinguish Enterprise SQL payload bytes from SDK descriptors. Server dependency: https://github.com/lancedb/sophon/pull/10211. Validation uses retained, independently audited local evidence: 724 Python cases passed (one optional PyAV skip), 67 focused SDK Rust cases passed, 36 server checks passed, and 15 actual REST/SDK scenarios passed across retained runs. These cover sync/async and Arrow file/stream recreation, concurrent overwrite, deletion of a selected version, legacy capabilities, forged IPC metadata, WAL exclusion, and actual mid-batch recreation. The original aggregate experiment exited 101 because its extra harness assumed a Python batch size would split the native result; its 36 server checks and 13 completed scenarios remain valid. The corrected native-batch experiment passed separately for sync and async. No aggregate all-green claim is made. The public batched-range operation has separate cross-partition identity integration in https://github.com/lancedb/sophon/pull/10219 and https://github.com/lancedb/lancedb/pull/4994. This change does not claim to fix that loop. Realtime/MemWAL remains outside Blob v2 GA, without disabling existing writes or compaction. No dependency pins changed. Merge/rollout hold remains: upgrade every serving node before enabling the new SDK capability, verify proxy forwarding of both authoritative headers and the verification echo, and preserve queried versions for later reads. Missing or inconsistent capability fails explicitly; identity does not retain versions or authenticate storage paths. Actual deployed proxy/Cloud/PE behavior and production retention remain unverified. Old SDKs do not gain incarnation protection from a server-only upgrade. Published documentation retains the Early access badge and released Enterprise instructions, including explicit fetches for copying. The new Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example, future streaming restrictions and GA wording remain source-only MDX drafts until release. The correction distinguishing released Enterprise SQL bytes from SDK descriptors remains visible. Documentation validation checked comment boundaries, visible instructions, Python example syntax and whitespace; the site was not built. This documentation-only revision reuses the implementation evidence above. |
||
|
|
725eee8a47 |
fix: prevent unsafe write retries and bound Blob uploads (#4977)
Ambiguous gateway failures can occur after a write commits, so replaying the operation can append duplicate rows. Non-idempotent writes now retry only known precommit failures. Multipart retries reuse the same upload ID, part ID, and encoded bytes; ambiguous completion returns an unknown-outcome error without abort or replay. A definite 409/429 completion rejection triggers best-effort cleanup using the call's retry budget. Replayable parts and the shared byte-based partition target default to 64 MiB. Dictionary-bearing batches are compacted before size estimation and again when actual compressed IPC framing requires further splitting: keys are remapped without changing their types or schema, view backing buffers are collected, and ListView/LargeListView child ranges are compacted as an interval union so overlapping views retain sharing. Parts are prepared on demand; a single row and its schema can still exceed the target. Multipart progress starts at zero and counts each acknowledged IPC body once. Rust write builders and Python/TypeScript `add` expose per-call retry overrides. Atomic `create_table` pre-encodes to a temporary file and rejects compressed IPC over 256 MiB before sending any request; the schema-first import guidance now includes Python, TypeScript, and Rust examples. The shared partition target also affects local automatic partitioning. At `cb2a79473bd1b51b950c9fa3e9f247f929c8ac3b`, 70 targeted Rust cases pass on source verified against the committed four-file patch. With a 16 KiB target and 128 unique 1 KiB dictionary values, the original ordinary dictionary reproducer drops from 128 requests / 16,942,080 bytes to 16 / 143,488; ListView and LargeListView drop from 128 / 17,056,768 to 16 / 149,632. Encoded HTTP bodies are decoded to check values, nulls, slices, metadata, changing dictionaries, and overlapping views; additional cases cover framing-driven splitting, oversized rows, stable retry bytes/IDs, acknowledged progress, and atomic create. Real HTTP fault tests commit Blob payloads to a local Lance table and verify byte equality, no duplicate append, and zero abort/replay after ambiguous completion. The earlier 811 remote and 7 progress cases at `f36d217110a25f6b501c0255df9aaa60e3c8000a`, Linux Python remote database/options results (437 cases plus 8 fixture consumers), Node retry-binding evidence, and Rust/TypeScript documentation example checks remain historical validation. Python/Node extensions were not rebuilt for the dictionary follow-up; these results do not establish current binding integration or fresh cross-platform/minimum-dependency CI acceptance. A separate real Query Node lifecycle experiment used SDK `b8efe7d2d63647d4962789ca5fc042b59efe073d` and Sophon `58b7895c7c287810fff4e89f13226fc5fa60d9ef`. After the actual listener forced a connection closed, a surviving QN accepted a retry with identical upload ID, part ID, and IPC bytes. Ordinary insert and complete response-loss scenarios sent no replay or abort; fresh reads observed exactly one commit and exact Blob bytes in all three scenarios. This local experiment used a 1-second drain grace, two 32 KiB payloads, and one distinct staged part. It validates acknowledgement-loss recovery at those revisions, not concurrent overlapping attempts, large uploads, physical orphan cleanup, the later dictionary encoding change, or the original sixth-insert EOF. The part target is not a process memory limit: source/ancestor buffers and temporary compaction allocations may coexist. Struct/FixedSizeList parent nulls can retain hidden child keys; this is not complete logical-reachability GC for every Arrow combination, and retained hidden values may affect wire size. The current local Lance 15.0.0-beta.17 rejects ListView/LargeListView persistence; the tests establish remote client IPC encoding behavior, not native or Cloud persistence support. Live Cloud, full workspace/Clippy, the original ordinary-insert disconnect root cause, and Sophon's asynchronous transaction cleanup behavior are not established. Cancelling during staging retains the existing server-session cleanup limitation. Function reader delivery is separate. |
||
|
|
10f89b3c32 |
perf: reduce Blob read buffering and expose read window controls (#4979)
Remote Blob reads retain and concatenate intermediate IPC buffers across batches and HTTP requests, increasing peak memory. Decode fragmented IPC bodies using their declared length, assemble Blob values once across batches/requests, and collect large-object Range responses into known-size buffers. Impossible IPC/Range allocations return errors instead of panicking. Closing a remote Blob file releases cached bytes immediately, including while a cross-window read is waiting for the network. Rust and Python expose configurable Blob read buffers. Python sync/async `fetch_blob_files` and `open_blob_file(s)` accept `buffer_size`; the default is 512 KiB and zero disables prefetch. Explicit ranges bypass prefetch. New PE batch-endpoint adoption is separate. Malformed-length and retained-cache regressions fail on the previous implementation and pass after the correction. Real HTTP tests cover buffer propagation, close during blocked input, batch/request aggregation, and large-object fallback. The final extension was rebuilt before Python validation. Non-Blob Arrow arrays may still require alignment copies; a contiguous multi-batch result still requires its final allocation. ## Benchmark Peak client RSS increment in KiB, using `python/benchmarks/bench_fetch_blobs_rss.py`, independent clients and real loopback HTTP. Both extensions were rebuilt from clean, unchanged source with the same toolchain/configuration; loaded artifact hashes were checked. Baseline `42f11d13a2755e44b63876b5a5edd93901291150`, after `0a8a76de5286949d80368f95a5c529adf422c187`. | Equivalent fetch workload | Payload | Before | After | | --- | ---: | ---: | ---: | | Single response, one batch | 48 MiB | 63,812 | 63,896 | | Multiple responses, one batch per response | 144 MiB | 423,104 | 195,628 | | Single response, 12 rows per batch | 48 MiB | 130,520 | 71,348 | | Multiple responses, 12 rows per batch | 144 MiB | 363,144 | 172,748 | | One oversized row, Range fallback | 70 MiB | 182,816 | 85,548 | These are one observation per workload and build, not a statistical distribution or memory upper bound. The single-batch workload shows no improvement. Cloud/Sophon deployment, credentialed S3 and long-running stress have not been exercised. |
||
|
|
a356950bce |
feat: expose queued Function refresh status (#4959)
## Problem The existing Function `column_status` surface cannot decode the Agent scheduler's `queued` state or the `failure_threshold_reached` and `binding_revision_missing` blocked reasons. It also preserves only the latest refresh job, so a later cancellation can hide an earlier actual failure. ## Change - Adds the `queued`, `failure_threshold_reached`, and `binding_revision_missing` wire states in Rust, Python, and Node. The wire change is additive, while Rust callers must update exhaustive enum matches and public struct literals. - Adds optional `last_failure_job` alongside `last_refresh_job`. The latest cancellation and the most recent actual failure retain separate meanings. - Reuses the canonical Rust `JobFailure` and existing Python/Node `JobFailureInfo` models for nested job failure details instead of introducing a parallel status error model. - Maps the existing public `sql_job_id` field through Node's `JobFailureInfo`. - Keeps `column_status`, `column_statuses`, and `wait_fresh` as the existing API surfaces. There is no new endpoint, row-error list, durable ledger, or unknown-enum fallback. - Documents `queued` versus `running` and the separate latest-attempt and latest-failure jobs in the existing hidden freshness release draft. The new status values are additive but cross-service: `queued`, the two new blocked reasons, and `last_failure_job` require matching Agent/QN and SDK deployments before clients can rely on them. This SDK follow-up does not change the normal `7cc3904a` dependency pin or claim whole ENT-3487 completion. ## Validation The tested private source is `b19c8adbdc936c5976def6c17be47f6d5e3faf3e`, based on product commit `c9bac25ecb8b73b1bd0256bbea65c2621bcf6a94`. The final product docs commit changes generated TypeDoc only. - Rust: 2 selected tests passed, 0 failed, exit 0. The same two Rust blobs are unchanged from their tested `788ef85b` source through `c9bac25e`. - Python: the PyO3 extension built successfully, and the selected native PyO3 SDK test against a mock HTTP REST transport passed 1/0, exit 0. The test requests pinned version 42. - Node: the addon and TypeScript builds passed, Biome passed, and the selected Jest transport test passed 1/0 with 57 filtered tests. Addon SHA-256: `c888e260b61d41e79689ed2f20dd0dc98586a2377ce34f21610b06111734d426`. - Python extension SHA-256: `36c961d8a0cbd2e102a7765f2e47efb4c5af6310baa6262c8e3120407463da2b`. - TypeDoc was generated from the native `native.d.ts` produced by that Node build. A second generation produced no remaining diff. The public diff contains only product, test, and generated API documentation changes. Native runner scripts and preserved compile-failure logs remain private validation evidence. Agent/QN production mapping, mixed-version rollout, and the remaining ENT-3487 server-side work are tracked separately. |
||
|
|
7cc3904a7e |
feat: expose exact Function column freshness (#4958)
## Problem Function-backed columns need an exact `fresh`, `stale`, `never_computed`, and `failed` partition for the table version a caller is reading. The existing error listing is capped, and composing it with job and schema listings cannot prove which rows currently failed. ## Behavior - Adds a strict freshness reader beside the existing permissive planner. It distinguishes an absent certificate from legacy evidence that omits failures, and from a current certificate carrying failure coverage. - Extends the immutable freshness sidecar with per-fragment known and failed row bitmaps. Partial or `WHERE` publications cover only observed rows; untouched rows stay unknown. Deleted rows do not block classification. - Adds failure-aware whole-fragment and row-repair writers. A rejected older attempt cannot overwrite failure evidence already recorded for a newer input signature. - Adds `column_status`, `column_statuses`, and `wait_fresh` to Python and the corresponding Node methods. The client calls the server's exact status endpoint, validates the four-way partition and one-version guarantee, and preserves the complete Function version, last-job, and automatic-refresh metadata. - Adds an MDX-commented draft usage section to the Enterprise computed-column refresh guide, covering Python, Rust, and TypeScript and remaining hidden until the server integration ships. - Sends `column`, `branch`, and exact `version` selectors to the status route. A pinned handle rejects a nonempty response from any other table version; latest handles keep the existing read-freshness headers. - Retries `wait_fresh` only for HTTP 503 with `function_column_status_finalization_pending`. Existing failed rows remain visible and do not keep the wait pending; other errors propagate. - Models the shared scheduler projection, including `up_to_date`, real `blocked` states, their reason, failure streak, and effective cooldown. ## Validation This is the SDK side of ENT-3563. Sophon's failure-coverage publication and status endpoint are a separate integration; deployed end-to-end and whole-ticket acceptance remain pending. - Rust: 9 exact freshness/status tests passed, plus 1 exact latest-handle freshness-header test. The pinned transport test was extended with a wrong-version response and rerun at `ebd6649a` (1 passed). Coverage includes untouched partial rows, selected-row failure clearing, cross-generation evidence, typed and non-retried errors, `up_to_date`, blocked scheduler fields, and exact-version requests. - Python: 2 native PyO3 remote-transport tests passed together, including typed retry; one local facade test reaches the existing unsupported boundary. - Node: the NAPI addon built successfully, TypeScript and Biome passed, and the selected native remote-transport test passed 1/0, including exact-version selection and blocked scheduler metadata. - `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile, Biome, and TypeScript checks passed. Validation is mapped by the relevant unchanged source rather than treated as one final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header and Python remote tests at `31e68b7d`, the local Python test and Node build at `6f213584`, and the final TypeScript/Jest delta at `4f2d5c27`. The final product tree excludes those five private EC2 runner files and adds six generated TypeDoc files plus one hidden draft usage section in the computed-column refresh guide. The Node delta corrected a private TypeScript command and reused the unchanged native addon. |
||
|
|
cf8e6d0fd4 |
feat: expose Function call concurrency limits (#4956)
## Summary Adds an optional `max_concurrency` setting to Function registration and explicit refresh: - A registered Function version can define the default maximum number of user-code calls in flight for one logical execution. - A refresh can override that default for that execution only. - The value must be a positive `u32` and is independent from `resource_budget`; it is not an account-wide quota. - Omitting the option preserves the previous request payload. Local embedded refreshes reject this Enterprise-only option. Rust, Python sync/async, PyO3, and the existing Node refresh surface forward the option. Node has no Function registration surface today, so no registration API is added there. The generated TypeDoc pages are included for the new public option. ## Validation The native-tested source covers Function version decoding, registration validation and serialization, the Flight SQL registration ticket and returned version, remote refresh forwarding, local rejection, and Node's numeric boundary before conversion. Exact stage results are Rust `1+3` plus four remote cases at `1` each; Python `14/1/49/24`; Node `57` remote and `1` local rejection. Ruff, standard repository Pyright, and Biome passed. The tested Python extension was `21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the tested Node addon and generated native declaration were `f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and `4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The final 24-file diff preserves the 22 tested product/test/fixture/source-doc blobs and adds only two generated TypeDoc pages. |
||
|
|
8029428c71 |
feat: persist automatic Function refresh resource budgets (#4953)
## Depends on merged #4951 Phase 1 merged as `4d22c0ce2b6825c880c818e737b079db494b4053`, so this PR now targets `main`. It reuses the canonical `FunctionWorkerResourceBudget` introduced there rather than copying the type. For Function `add_columns` declarations, Python and Rust now persist `auto_refresh_resource_budget` in computed-column metadata and send it as `auto_refresh_resource_budget` in the remote request. CPU milliseconds and memory bytes are required positive integers. Omitting `gpu_count` is accepted as zero and the canonical outgoing JSON still contains `"gpu_count": 0`. Unknown fields, missing CPU or memory, wrong types, zero CPU or memory, negative values, and overflow are rejected. The same budget is applied to every public output binding of a multi-output Function declaration. Existing metadata is retained, and SQL-expression declarations or local execution reject the budget instead of silently ignoring it. Node is unchanged because its current `addColumns` surface only accepts SQL computed expressions and has no Function declaration path. This change does not implement fleet call concurrency limits or enforce the budget in the runtime. The Enterprise server must support the nested Function worker resource-budget contract before these values can act as execution limits. ## Validation - Python extension build passed. - Remote Function declaration and local unsupported-path rejection each passed. - Canonical metadata round-trip passed, including omitted GPU decoding to zero and explicit `gpu_count: 0` on encode. - Function multi-output remote wire test passed with `--features remote`. - Repository-scoped Python type check passed with 0 errors. - All runtime source blobs match the native-tested source. The post-native deltas are one documentation example line and an existing routing-test stub update for the new optional argument; the affected parameterized test passed 3/3 against the existing extension build. No private test runners are part of this PR. |
||
|
|
4d22c0ce2b |
feat: expose Function refresh resource budgets in SDKs (#4951)
## Summary - Add the canonical Rust `FunctionWorkerResourceBudget` and `RefreshColumnOptions::resource_budget()` for explicit Enterprise Function refreshes. - Forward the nested budget through Python sync/async, PyO3, Rust remote requests, Node NAPI, and TypeScript. - Require positive CPU and memory values; normalize an omitted nested GPU count to numeric zero; reject the nested budget when combined with the top-level `gpu_count` shorthand. - Reject `resource_budget` on LanceDB Embedded refresh paths instead of silently ignoring it. - Document the Rust, Python, and TypeScript APIs, including that SQL servers currently require all three budget fields. The Rust, Python, and TypeScript additions use `cpu_millis` / `cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` / `gpuCount`. The top-level GPU shorthand keeps its existing behavior and remains mutually exclusive with the complete nested budget. ## Scope and compatibility This change configures only the explicit refresh call. It does not configure `add_columns` automatic refresh and does not introduce a fleet-wide call-cap interface. A LanceDB Enterprise server must expose the nested `resource_budget` backfill contract. LanceDB Embedded continues to support its existing expression refresh defaults and rejects the new complete budget explicitly. ## Validation Validated against the final signed product source: - Rust: resource-budget remote serialization test `2 passed`; validation test `1 passed`. - Python: actual PyO3 build; remote-request matrix `22 passed`; invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff check and format clean. - Repository-standard Pyright entry point: `0 errors`, within the repository's configured include set. - Node: native build and Biome lint passed; the existing remote-request test `1 passed`, including fractional, negative, overflow, NaN, and infinity nested GPU cases; the existing local-rejection test `1 passed`. - TypeDoc generation and generated-document cleanliness passed. - The native-tested product source differs from this head only in two semantically equivalent Node range checks required by Clippy. Current CI compiles and lints the final blobs. No dependency or lockfile changes are included. |
||
|
|
13e4cf7e58 |
feat: expose Function publication intervals in refresh options (#4928)
Python refresh APIs currently reject `publication_interval_rows`, so callers cannot request incremental Function output publication through the SDK. Add the optional typed parameter through the sync/async Python APIs, PyO3, and Rust `RefreshColumnOptions`, and serialize it as a top-level Enterprise refresh option. An omitted value preserves the existing wire request and whole-fragment default. Positive values set a publication threshold; zero and negative Python values fail before a request is sent. Embedded refreshes reject the Enterprise-only option. Documentation describes a threshold rather than an exact number of rows per publication. Validation on the designated development host: - A matching baseline build reproduced the missing public keyword argument in both sync and async submission tests. - The updated PyO3 build passed 54 focused Python tests; Rust payload, invalid-option, and default-wire tests each passed. Rust formatting and Python Ruff checks passed. - The single signed delivery commit differs from that tested implementation only in a Rustdoc correction describing Embedded rejection; formatting passed again. No dependency pins or lockfiles changed. The matching SDK is prepared for the separate server validation of failure/cancellation retention and missing-only refill. Those end-to-end results are still pending and are not implied by these SDK tests. Refs [ENT-3733](https://linear.app/lancedb/issue/ENT-3733). |
||
|
|
794fa023cd |
feat: add materialized view refresh resource budgets (#4902)
Expose aggregate CPU, memory, and GPU budgets plus GPU-class placement
on materialized-view refreshes in Python and Rust. Forward options
through SQL and reject invalid or unsupported Embedded requests instead
of silently dropping constraints.
```python
job = view.refresh_async(
cpu_millis=8000,
memory_bytes=16 * 1024**3,
gpu_count=2,
gpu_class="h100",
)
job.wait()
```
```sql
REFRESH MATERIALIZED VIEW curated_training_set
WITH (cpu_millis = 8000, memory_bytes = 17179869184,
gpu_count = 2, gpu_class = 'h100');
```
The same options apply to blocking refreshes, async Python connections,
and the Rust refresh builder. CPU/memory budgets require all three
resource fields; GPU-only caps and class-only placement are also
supported. These are execution-wide caps and require a supporting
Enterprise server with an isolated execution backend. A cap does not
increase refresh parallelism; current MV refreshes use at most one GPU.
Updates API documentation and SQL reference, with Python sync/async and
Rust remote-submission regression coverage. Formatting and Python lint
pass; local builds and tests are intentionally skipped.
The Linux CI build and example steps explicitly select the LanceDB
package, matching the feature-test step. This avoids rebuilding and
retaining a second dependency graph after a workspace-wide build, which
exhausted runner disk space on both this branch and main.
Related: [ENT-3386](https://linear.app/lancedb/issue/ENT-3386)
|
||
|
|
546a28ad2c | feat: support remote compaction tuning and table inheritance (#4886) | ||
|
|
2af85bea89 |
feat: expose the remaining refresh options on the SDKs (#4890)
The server's backfill request takes a row filter, failure limits, per-call timeouts and worker sizing, and SQL exposes all of them through `REFRESH COLUMN ... WHERE ... WITH (...)`, but the typed SDKs could send only batch_size, GPU placement and the retry and error policies. Anyone needing a filtered refresh or a bounded skip policy had to drop to SQL. This adds `where`, `max_failed_fraction`, `max_failed_rows`, `row_timeout_seconds`, `init_timeout_seconds`, `num_workers` and `max_executor_processes_per_worker` to Python, Rust and Node, passed through unchanged. `RefreshColumnOptions` now serializes straight into the request body, so the three SDKs share one producer. Node gets a `RefreshColumnOptions` object, which also carries the options it lacked before. A LanceDB Embedded refresh rejects every option other than batch_size and GPU placement, naming the option. |
||
|
|
6687db979d |
feat(functions): expose retry policies in Python and Rust (#4873)
Expose `FunctionRetryPolicy` on Function declarations and refresh overrides, together with `error_policy` and recorded attempt counts. Python uses the Rust validation rules before sending a request; local refreshes reject these Enterprise options. The policy bounds calls and backoff, supports class/message/provider-code matching, and can disable retries with `max_attempts=1`. Batch retry and skip operate on the whole failed call. Requires Enterprise server support for the new policy. Validation: rebuilt the Python extension; 47 targeted Python tests and 10 Rust tests passed. Python `make check`, Rust formatting, and `cargo clippy --features remote --tests --examples -- -D warnings` passed. The additional repository-root Ruff sweep reports existing errors under `docs/web-tests`; the Python workspace checks pass. Draft user guidance covers the supporting server requirement, safe repetition of partial work and whole-batch exhaustion. The guide sections stay hidden until release. Vale 3.24.0 passed for the added draft prose; no paired deployed SDK/server retry acceptance was run. Companion server: https://github.com/lancedb/sophon/pull/9812. BREAKING CHANGE: Rust callers constructing `FunctionRegistrationRequest` or `FunctionErrorRecord` with struct literals must supply the new `retry_policy` or `attempts` field, respectively. |
||
|
|
edc49e60e2 |
feat: expose Function failure tracebacks (#4861)
Function refresh failures and skipped-row diagnostics lose the traceback when decoded by the SDK. Expose the server-provided optional `traceback` on job failure details and Function error records in Rust, Python, and Node.js, while keeping existing compact error messages. Responses from older servers remain supported when the field is absent. Tracebacks may be truncated by the server. The Rust `Error::JobFailed` variant now stores `Box<JobFailure>` so the added diagnostics do not enlarge every `Result` error payload. |
||
|
|
deb8834b5a |
fix: expose stable row IDs in Function errors (#4841)
Function error records expose physical row locations, which can change when a table is compacted. Expose the server-provided optional stable `row_id` through Rust and Python so callers can locate the same row after compaction. Document stable-ID retries and retain physical addresses for tables without stable row IDs. Older server responses continue to deserialize with `row_id=None`. The Python remote response test and Rust remote deserialization test passed on verification SDK `4e1725fa352035e2079dfd235b8271cd05adf433`, including decoding a legacy response without the new field. This branch applies the same SDK patch to current main; intervening main changes leave the affected error types and assertions unchanged. Formatting and lint checks passed after correcting the retry examples to use the `table.name` property. In a separate Linux server experiment, a real skipped Function row retained `_rowid=506` after compaction changed its physical address. The SDK field requires a server that emits stable row IDs; this PR does not change server dependencies. |
||
|
|
a9ad499781 |
feat: support remote table compaction via SQL OPTIMIZE TABLE statement (#4802)
`Table::optimize()` previously returned `NotSupported` for remote tables. This PR wires it to the LanceDB Enterprise server's `OPTIMIZE TABLE ... REWRITE DATA` SQL statement, submitted over Flight SQL. Only `OptimizeAction::Compact` without a custom `remap_options` or non-default `CompactionOptions` is supported remotely. `OptimizeAction::All`, `::Prune`, and `::Index` all return a clear `NotSupported` error instead of silently doing the wrong thing. A branch-scoped table handle submits a `WITH (branch = '<branch>')` clause; the statement doesn't validate the branch exists itself, so an unknown branch surfaces as an async job failure rather than an error from this call. The server's SQL job adapter doesn't expose compaction metrics, so `OptimizeStats::compaction` is `Some(CompactionMetrics::default())`, a placeholder rather than real numbers — it must still be `Some`, since callers (including the Python/Node bindings) unwrap it unconditionally on success. This mirrors how `refresh_materialized_view_async` already handles the same limitation for its own metrics. The Python and Node `optimize()` convenience methods call a new shared `Table::optimize_full`, which runs `Compact` and then, on a remote table, skips `Prune` and `Index` instead of attempting and failing on them — returning zeroed prune stats rather than submitting a compaction job and then raising. Python's `RemoteTable.optimize()` also now forwards `compaction_options` to the Rust layer, so a non-default value is rejected before submission instead of being silently dropped. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
90b2b264d9 |
feat(python): accept DataFrames and readers in add_columns
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader. Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them. SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits. Fixes #1959. |
||
|
|
9e65bd2c27 |
docs: use Enterprise terminology across website and SDK documentation (#4787)
Use “LanceDB Enterprise” throughout the staged website guides, SDK reference sources, generated API pages, and user-facing SDK messages. A separate branded “LanceDB Distributed” name adds terminology without clarifying behavior; describe where operations run directly. - Update `docs/web`, including the architecture diagram title and alt text. Use Enterprise in headings, table labels, and prose; simplify redundant product/execution-model wording and name the product only once in the diagram alt text. - Update Python docstrings and the Python/Java reference pages, TypeScript comments and native declaration comments, and Rust API documentation. Regenerate the TypeScript reference, including 24 replacements across six pages. - Update SDK warnings/errors and existing test expectations to match, along with contributor guidance and Vale rules. Remove the obsolete rule distinguishing Enterprise from Distributed. - Keep lowercase “distributed” for technical descriptions. API identifiers, anchors, asset paths, and execution behavior are unchanged. No branded-name exceptions are proposed. Validation: - Python and Node native builds succeeded; focused Python tests passed (36 passed, 4 skipped), and focused Node tests passed (14). - TypeDoc generation and the complete MkDocs reference build passed. MkDocs retains an existing unresolved `lancedb.functions.SecretReference` cross-reference. - Rust formatting, Node lint/format, changed-file Python lint/format, 32 Vale tests, and whitespace checks passed. Repository-wide Python lint has 48 existing findings in six untouched files. - Combined documentation assembly passed. Mintlify validation and link/anchor/redirect checks previously passed for both site builds; no code snippets changed. Companion PRs: [saved editions, terminology guide, and shared rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview and writing rules](https://github.com/lancedb/sophon/pull/9511). --------- Co-authored-by: Brendan Clement <brendan.clement@lancedb.com> |
||
|
|
45edaa350e |
feat(python): expose code analyzer options on FTS and tokenize
Part of #4364 (Python side; the TypeScript `FtsOptions` change will follow in a separate PR). Lance already supports the code analyzer, but `FTS` and `lancedb.tokenize()` had no way to set it, so `FTS({'analyzer': 'code'})` and `FTS(split_identifiers=True)` raised `TypeError`. ## Changes - `FTS` and `lancedb.tokenize()` gain `analyzer` (`"text"` or `"code"`), `split_identifiers`, `split_on_numerics`, `preserve_original` and `index_operators`. They default to `None`, which keeps the tokenizer's own defaults, so existing indexes and calls are unchanged. - Validation follows pylance: `analyzer="code"` with a `base_tokenizer` other than `"code"` or the default `"simple"` is rejected, `analyzer="text"` with `base_tokenizer="code"` is rejected, and turning on a code flag without the code analyzer is rejected. Passing a flag as `False` is always allowed. - `analyzer="code"` with the default `base_tokenizer="simple"` selects the code tokenizer. `FTS` can't tell an explicit `"simple"` from the default, so that case is accepted too. - The deprecated `create_fts_index` is left alone; it points users at `create_index(config=FTS(...))`. - Rust `FtsIndexBuilder` is Lance's `InvertedIndexParams`, so the Rust SDK already had these options. The remote client serializes the same struct, so the options reach LanceDB Distributed in the `create_index` request body. A new case in the Rust `test_create_index` checks that body. ## Tests - `python/tests/test_fts.py`: tokenizer output for each flag, validation errors for `tokenize` and `FTS`, `False` flags on a text index, and local sync and async indexes that match `http`, `request` and `parseHTTPRequest` with splitting on and not match subwords with it off. - `cargo test --features remote remote::table::tests::test_create_index` |
||
|
|
63aa06ef7d |
feat(python): expose Function GPU classes and placement (#4791)
## Summary - Add sync/async `DBConnection.list_gpu_classes()` returning typed `GpuClass` records for `GET /v1/gpu/classes`. Records include the class name, Kubernetes resource, GPU model and memory, per-worker GPU maximum, and worker CPU/memory/ephemeral-storage limits. - Add `udf(gpu_class=...)` for immutable Function-level GPU placement. It requires `gpu=True`; Function versions expose their registered `gpu_class`. - Add `gpu_class` and `gpu_count` to `refresh_column` / `refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as top-level `backfill_column` fields and validated before any request. `gpu_count` is the aggregate GPU budget across Function workers. - Validate GPU class names with the server lowercase 63-character DNS-label rule. - Document the discovery-to-registration flow, configured hardware versus free capacity, System Usage permission, and per-refresh GPU placement in the API and web guides. Older servers without `/v1/gpu/classes` produce a clear `NotImplementedError` instead of a bare 404. Part of ENT-3470. The client follows the in-flight Sophon contract on `xuanwo/ent-3470-fix` for GPU class discovery and backfill placement options. ## Breaking change `PythonRuntimeSpec::PythonV2` adds `gpu_class`, and `RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers constructing either value with all fields explicitly must supply the new fields. ## Validation At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python extension build, pyright, ruff, and Python Function/runtime, discovery, refresh placement, and refresh regressions (147 passed, 6 skipped). At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright, Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed). The final commit only updates documentation and Rust doc comments. |
||
|
|
6a255926d7 |
fix(python): improve function failure diagnostics (#4790)
## Summary - Make `AsyncJob.wait(timeout)` and `Job.wait(timeout)` raise a descriptive `TimeoutError` containing the job ID and elapsed timeout. Registry jobs explain that work continues on the server and include the `db.open_job(id).wait()` reattach call. SQL query handles instead say to keep the original handle because query IDs cannot be reopened through the registry jobs API. - Render `JobFailureInfo` and `FunctionErrorRecord` with Python values instead of Rust `Option` wrappers. - Decode and expose the server's `row_address` on `FunctionErrorRecord`, and document how to retry recorded rows with `REFRESH COLUMN ... WHERE _rowaddr IN (...)` before compaction or rewrite. - Warn when an import inside a packaged Function body resolves to undeclared local source. The `udf` documentation explains why arbitrary base-image coverage and distribution-to-import-name mapping cannot be validated reliably by the SDK. `row_address` is optional, so records from older servers still decode with `None`. Rust code that constructs `FunctionErrorRecord` literals must now supply this new field. Part of ENT-3448 and ENT-3442. This consumes the server-side diagnostics from lancedb/sophon#9509 and logical row addresses from lancedb/sophon#9512. ## Validation - Rust `function_errors` remote decoding and legacy-missing-`row_address` tests. - Rust/PyO3 clippy for `lancedb` and `lancedb-python`. - Python extension build, pyright, and ruff. - Targeted Python job, Function packaging, Function error, and job detail tests: 62 passed, plus timeout-handle tests: 3 passed. |
||
|
|
9804f26bc3 |
fix: support batch size options for column refreshes (#4777)
Python column refresh methods rejected `batch_size`, and the Rust remote backend never included it in the backfill request. Callers could not reduce the rows passed to a slow Function to stay within the executor's per-call deadline. Add optional, keyword-only `batch_size` arguments throughout the synchronous and asynchronous Python APIs and bindings. Add compatible Rust `RefreshColumnOptions` APIs while preserving existing Rust method signatures. Reject non-positive sizes before submitting a job, send positive sizes in `backfill_column`, and omit the field when unset while retaining branch scoping. Batch size controls remote Function calls. Embedded expression refreshes accept and validate the option without changing execution, and blocking remote refreshes retain their existing unsupported behavior; remote callers use `refresh_column_async` and wait on the returned job. Validation: - Reproduced the original Python `TypeError` from the committed method. - Five focused Rust tests passed, covering request contents, omission, branch scoping, and rejection before any request. - `cargo check --quiet --features remote --tests --examples` passed. - `cargo clippy --quiet --features remote --tests --examples -- -D warnings` passed. - Nineteen targeted Python tests passed against the rebuilt extension, including synchronous and asynchronous remote requests, invalid sizes, and default-build behavior. Eight embedded refresh cases require the experimental-functions build and were skipped. - Rust formatting and Python formatting/lint checks passed, including CI's pinned Ruff version. Repository-wide Ruff reports 48 existing findings in untouched documentation files; none are in changed files. Fixes #4776 <!-- lance-gatekeeper-fix:v1 agent=4906ff3f5f7797a1d49a13803302021c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
5f7ec4e0a2 |
feat(sdk): expose table overrides for remote tables
Expose table-level maintenance override read, update, and reset APIs for remote
Rust and Python tables. Local tables continue to return `NotSupported`.
Updates recursively merge the caller-provided JSON object into the current
override document so unrelated and unknown service fields are preserved. Reset
uses the explicit reset endpoint.
The integrated change keeps current main's remote database identity, namespace,
branch, SQL, WAL, and threaded mock-server behavior.
Validated with focused Rust and Python override tests and the complete applicable
CI run for
|
||
|
|
4d7a64712f |
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2cd635d41c |
fix: correct remote materialized view refresh results and options (#4672)
Remote materialized-view refreshes drained the SQL result and then fabricated `mode=rebuild`, `rows_written=<total view rows>`, and `source_version=0`. This made incremental refreshes report incorrect statistics. Make these three result fields optional across Rust, Python, and TypeScript, and leave them unset for remote SQL refreshes. Remove the post-refresh row count request. Local refreshes continue to populate all metrics, and `version` remains the view version observed after completion. Remote per-operation statistics remain unavailable until the SQL job adapter can retrieve them. Expose the forced-rebuild option through the Rust builder, synchronous/asynchronous Python refresh and job methods, and TypeScript refresh. Remote `full=true` sends `WITH (full = true)`; the default lets the server choose its refresh mode. Local source-version pinning and remote job cancellation are preserved. Local refresh regression tests respect the `experimental-functions` build gate, while remote refresh coverage also runs in default builds. Validation: - Rust: 20 materialized-view tests and 21 remote SQL tests passed with `remote,experimental-functions`; `cargo fmt --all` passed. - Python: rebuilt the extension in the uv-managed environment. The default build passed 23 tests with 27 feature-dependent skips. The opt-in build passed 61 tests with 3 expected default-build-only skips, covering the materialized-view suite and 14 async query-serialization cases from current main. Python Ruff lint and formatting checks passed. - TypeScript: default and opt-in native builds and TypeScript compilation passed. The default build passed 5 tests with 10 expected feature-dependent skips; the opt-in build passed all 14 materialized-view tests with the default-build-only test skipped. Lint/format checks and documentation generation passed. - Repository-wide `ruff check .` reports 48 pre-existing violations in unchanged `docs/web-tests` files; `ruff check python` passes. Fixes #4670 <!-- lance-gatekeeper-fix:v1 agent=ece43ec4b88a48d1789f60005fa3ecfc generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
3352bf49df |
fix(python): expose optimize compaction options (#3870)
## Summary - expose typed compaction options on synchronous and asynchronous table optimization - translate validated Python options into native Lance compaction settings - document the row-based steady-state limit and first-pass/source-budget caveats - carry the Lance beta.15 fragment-exclusion option through the typed Python API ## Root cause The native Python optimize binding always constructed default compaction settings, and the public AsyncTable API offered no way to lower the fragment row target. Wide-row fragments therefore remained compaction candidates on every later optimize call, repeatedly rewriting the accumulated table. ## Limitations This is an explicit row-based mitigation, not byte-native task sizing or a hard memory cap. Callers must derive target_rows_per_fragment from expected row size. Existing oversized fragments are rewritten whole on the first pass; max_bytes_per_file limits output file size rather than compaction memory; and source budgets admit whole planned tasks, so a budget below the first task performs no compaction. Splitting tasks by a byte budget or returning a planner error requires a Lance-side change. ## Validation - cargo check --release -p lancedb-python - cargo clippy --quiet -p lancedb-python - cargo fmt --all --check - ruff format --check python tests - ruff check python tests - uv run --no-sync pytest python/tests/test_table.py -k optimize_compaction -q (16 passed) Fixes #2325 <!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
dcfef7c5ab |
feat(python): open lazy blob files from SQL descriptors (#4668)
SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.
### Python experience
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.
The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.
For an async table:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
### Execution
The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.
Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.
Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.
Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
|
||
|
|
3d883af674 |
fix(python): propagate invalid index wait_timeout errors (#4495)
Invalid `wait_timeout` values currently reach `Duration::extract(...).unwrap()` in both native index-creation bindings. A normal Python argument error becomes `pyo3_runtime.PanicException`, which is not caught by `except Exception`. Propagate the extraction error with `map(...).transpose()?` before constructing the index operation. Valid `None` and `timedelta` behavior is unchanged. Tests cover both `create_index` and `create_index_async`, three invalid types, valid `None`/`timedelta`, and no index created after rejection. Validation against main `0aa842b`, Python 3.12, uv-locked dependencies, freshly compiled PyO3 extension for each arm: - Unmodified source: 6 new regressions fail with `PanicException`; 6 controls pass. - Fixed source: 12/12 pass; broader index-creation regression selection: 18/18 pass. - `cargo fmt --all --check`, Ruff check/format, and diff check pass. The broader selection includes the 12 cases above. No cloud, full-suite, or performance claim. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
|
a12aef2a8c |
fix: return a Python error when querying a closed table (#4455)
`AsyncTable.query()` unwraps the native closed-table error, so querying a closed table raises `PanicException` instead of the documented `RuntimeError`. `AsyncTable.head()` reaches the same path. Propagate `inner_ref()`'s error through `PyResult` instead. Add regressions for closed `query()` and `head()`, with `count_rows()`, repeated close, and normal open-table queries as controls. Validation against separately built native bindings: - Before: 2 regression failures (`PanicException`), 2 controls passed. - After: all 4 regressions passed. - Existing query tests plus the basic database test: 102 passed, 1 skipped on each arm (Torch is not installed). - `cargo fmt --all -- --check`, Ruff, and `git diff --check` passed. The native build used the locked Cargo dependencies with `remote,lancedb/metrics-otel,pyo3/extension-module`. This is source-tree native testing, not a complete wheel-packaging or cloud-service test. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
|
a4544c5472 |
feat: configure automatic refresh when declaring computed columns (#4607)
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
|
||
|
|
92825f6f13 |
refactor!: rename MemWAL APIs to RTE (realtime engine) (#4598)
## What Renames the user-facing MemWAL surface to **RTE** (realtime engine) across Rust, Python, TypeScript, and Java, including the REST wire format. This follows the LSM → MemWAL rename in #4075. | Before | After | |---|---| | `MemWalWriteSpec` / `MemWalStats` | `RTEWriteSpec` / `RTEStats` | | `enable_memwal` / `disable_memwal` / `get_memwal_spec` | `enable_rte` / `disable_rte` / `get_rte_spec` | | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | `flush_rte` / `compact_rte` / `checkpoint_rte` | | `get_memwal_stats` / `close_memwal_writers` | `get_rte_stats` / `close_rte_writers` | | `use_memwal` (query, merge insert) | `use_rte` | | TS/Java `enableMemWal`, `useMemWal`, ... | `enableRTE`, `useRTE`, ... (napi `js_name` set explicitly) | | Java `LanceDbTableMemWal` | `LanceDbTableRTE` | | `table/memwal_stats.rs` | `table/rte_stats.rs` | Docs and comments now say "realtime engine (RTE)". Text about Lance's underlying mechanism keeps "MemWAL" ("Lance's MemWAL", "MemWAL index"). **Left alone:** Lance-owned names (`MemWalIndexDetails`, `DatasetMemWalExt`, `InitializeMemWal*`, `mem_wal` paths), internal `Lsm*` names and modules, and the `does not support with_row_id` refusal text that the hybrid fallback matches on. ## Compatibility Breaking, for an experimental API. The wire format changes, so this must roll out together with the matching sophon change (lancedb/sophon#8870): - Routes: `/enable_memwal/` → `/enable_rte/`, `/disable_memwal/` → `/disable_rte/`, `/get_memwal_spec/` → `/get_rte_spec/`, `/get_memwal_stats/` → `/get_rte_stats/`, `/flush_memwal/` → `/flush_rte/`, `/compact_memwal/` → `/compact_rte/` - Response keys: `memwal_spec` → `rte_spec`, `memwal_stats` → `rte_stats` - Request field: `use_memwal` → `use_rte` on query and merge-insert bodies The SQL keyword becomes `ALTER TABLE ... SET RTE` server-side. The guides under `docs/web` will be updated separately. ## Validation - Rust: `cargo clippy --features remote --tests --examples` clean; `cargo test --features remote -p lancedb --lib`: 1533 passed - Java: `mvn test`: 37 passed (Spotless applied with JDK 11) - Python: `ruff format` / `ruff check` clean; targeted tests (`test_rte_spec`, `test_merge_insert_lsm`, `test_remote_db`, `test_wal_hybrid`, `test_query`, `test_permutation`, `test_table`): 708 passed - TypeScript: `pnpm build`; `table`, `query` tests: 606 passed; `biome check` clean; `docs/src/js` regenerated with `pnpm run docs` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
fe34eb3549 |
refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a1e4dc9f91 | fix(python): align BlobFile seek and read behavior (#4290) | ||
|
|
0212cfd3d6 | fix(python): normalize blob row ID inputs and errors (#4293) | ||
|
|
0acb61471c |
fix(python): route async remote blob reads through server (#4306)
## Cause Async remote pandas conversion could call `AsyncTable.to_lance()`, which opened the server's table storage URI in the client process. This bypassed server query and authorization APIs. The first repair routed blob reads through the server, but a query and subsequent blob fetch could resolve different table snapshots after replacement or compaction. ## Fix - Reject `AsyncTable.to_lance()` for remote tables before requesting a table URI. - Convert remote nonblob and blob description results through the server Arrow path. - Reject automatic remote byte and lazy blob materialization until the server can bind the query and fetches to one verified table identity and version. The default lazy mode now raises `NotImplementedError` when a blob column is projected; callers can request `blob_mode="descriptions"`. ## Validation - Mocked remote tests cover direct storage rejection, description results, byte and lazy rejection before any blob fetch, and nonblob projections. - Focused local and remote blob tests passed (19 tests). `ruff check .`, `ruff format .`, and pyright passed. Fixes #4272 <!-- lance-gatekeeper-fix:v1 agent=35937e6e1d968f62eea59adcc56c84e9 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
bcf5ee263f |
fix(python): preserve code tokenizer filter defaults (#4370)
Python's FTS bindings always forwarded `stem=True` and `remove_stop_words=True`, overriding Lance's code tokenizer defaults. As a result, code indexes stemmed identifiers and discarded stop words, and `lancedb.tokenize` reproduced the same behavior. Use `None` for omitted filter options and only apply explicit booleans in the Rust bindings. This preserves tokenizer defaults across local and remote index creation, the deprecated `create_fts_index` helpers, and standalone tokenization. Update the signatures and documentation to describe the inherited defaults. Regression tests cover stored local settings, token output, stop-word searches, remote request payloads, explicit overrides, and the existing simple tokenizer behavior. The regression suite produced 16 failures before the fix and passes all 28 cases after it. Validation: - Rebuilt the Python extension using the uv-managed development environment. - Full Python FTS and index test files plus relevant remote index tests: 105 passed. - `cargo clippy --quiet --features remote --tests --examples -p lancedb-python` - `cargo fmt --all`, Ruff format/lint with CI's pinned version 0.9.9, and Pyright (zero errors). Fixes #4365 <!-- lance-gatekeeper-fix:v1 agent=9c1db2d3fc25c49c5b340f8debf43a0f generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
1e4662ac23 |
fix(remote): unify unsupported operation errors and warnings (#4414)
Remote SDKs report Cloud-specific errors against self-hosted servers, and some Python wrappers raise different exception types for the same unsupported operation. The sync index wrapper also warns that `replace` is ignored even though Rust forwards it to the server. Use backend-neutral table/server diagnostics and the existing `NotSupported` error conversions for unsupported manifest operations, blob ranges, primary keys, optimization, and HNSW-PQ indexes. Sync Python delegates to the async paths, so manifest operations and `optimize()` consistently raise `NotImplementedError`. Legacy and config-based HNSW-PQ creation share the same error and alternative-index advice. Remove the false replacement warning and simplify the ignored GPU-accelerator warning. Keep the deprecated cleanup/compaction no-ops, with backend-neutral warnings that do not assume automatic server maintenance. Embedding configuration forwarding and stale warning removal are now supplied by #4408 in `main`. Retain the regression that verifies full embedding metadata and warning-free remote creation alongside the diagnostics tests. Validation against the merged `main` revision `bbc4aebe` and Lance 14.0.0-beta.1: - Rebuilt the Python extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`; all 122 remote Python tests passed. Root Ruff checks and formatting passed using the uv-managed environment. - `pnpm build`, `pnpm lint`, formatting, and `pnpm run docs` passed; all 47 remote Node tests and the focused local vector-index regression passed. - All 583 Rust remote tests and 54 query unit tests passed, along with `cargo check --quiet --features remote --tests --examples`, `cargo clippy --quiet --features remote --tests --examples`, and `cargo fmt --all`. Fixes #4393 <!-- lance-gatekeeper-fix:v1 agent=10bafd337999f4910855c3a9b71e7dcf generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
069b2f7d31 |
feat: add delete rows to table stats (#3783)
add to table stats the number of rows that has been marked for deletion, but not yet deleted. note that when a table fragment is removed due to deletion of all of its rows, these rows are not counted. since the main use of this feature is to determine when table compaction is needed, the fact that these rows are not reported should not impact that decision (deleted table fragments don't need compaction). Fixes: https://github.com/lancedb/lancedb/issues/3761 Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com> |
||
|
|
2bf70758a7 |
fix: read back the maintained-index selection a table was given (#4447)
`maintained_indexes` on a MemWAL table has three values: a named list, `[]` for none, and `null` for every index the table has, including ones created later. This PR makes every client read all three back as they were set. - **Rust remote client** (also used by the Python and Node SDKs) reads `null` as `None`. - **Java client** reads `null` as `null`, not as an empty list. - **Docs** for the setter, the field and the getter describe the three values the same way in Rust, Python, Node and Java. The generated JS docs are regenerated to match. ## Testing - `rust/lancedb`: 73 LSM tests pass, including one case per value. With the decoder typed as a list, the `null` case fails. - Java: two cases for `null` and `[]` on the read path, run by the Java CI job. - `cargo fmt`, `cargo clippy -D warnings` and `ruff format` are clean. |
||
|
|
7e8f0d4982 |
refactor: always fully refresh materialized views (#4373)
Simplify experimental materialized views to one output-only storage
layout and one full-rebuild refresh implementation.
## Current layout
Local materialized views currently maintain incremental-refresh state
alongside the materialized values:
- an internal `__source_row_id` column for source provenance;
- `mv.definition` plus `mv.incarnation`;
- source version/timestamp, view version, and last-refresh watermarks;
- optional `mv.staging` state for table-function sources; and
- stable-row-ID requirements on the source table.
Refresh classifies source changes, appends or evicts rows when possible,
and falls back to a rebuild for unsupported deltas. Remote
create/refresh also use separate REST mutation paths.
## New layout
An MV is an ordinary Lance backing table containing only:
- the defining query's output columns;
- its current data fragments; and
- `mv.definition`, encoded as a versioned SQL envelope such as
`{"kind":"query","format":1,"query":"SELECT ..."}`.
There is no provenance column, incarnation, source/view watermark,
staging metadata, or stable-row-ID requirement. Local refresh may still
pin a source version for reproducibility, but it always recomputes the
complete result.
Full refresh streams the new result into uncommitted fragments and then
publishes a single transaction that removes every old fragment and
installs the new set. A shared conflict token prevents concurrent
replacements from both committing. This fragment-replacement primitive
is also used by Sophon's SQL worker.
## New remote create workflow
```text
Connection::create_materialized_view(...)
-> builder renders the defining SELECT
-> remote Database receives a CreateMaterializedViewRequest
-> submit CREATE MATERIALIZED VIEW ... AS ... through Flight SQL
-> retain the SQL Query handle while the Job is running
-> consume the durable SQL-job result
-> open the created table and read mv.definition
-> return MaterializedView
```
Remote refresh similarly submits `REFRESH MATERIALIZED VIEW` through
Flight SQL. Cancelling either wrapper forwards cancellation to the
retained SQL query before aborting the local task, so the server
operation is not silently left running.
## API changes
- remove incremental/no-op modes and the `full` switch;
- remove distributed grouped/partitioned refresh APIs and native
vector-source MV APIs;
- keep full-refresh results across Rust, Python, and TypeScript;
- preserve list/describe/drop as object-management operations while
routing create/refresh mutations through SQL.
The storage contract is experimental and has no compatibility shim;
views using the removed layout must be recreated.
|
||
|
|
de19ecbf63 |
fix: reject empty column alterations and unknown Python keys (#4417)
Fixes #4390 Lance accepts column alterations with every change field unset, so a path-only Python call could commit a new local version without changing the schema. The Python binding also silently discarded unknown keys such as `nulable`. Validate every alteration in the shared Rust `Table` API before dispatching to either backend, and reject unknown dictionary keys in the Python binding while preserving the legacy `name` alias. Invalid batches leave local schema and version unchanged and send no remote request. Validation: - Reproduced both reported calls on the original extension: versions advanced from 1 to 2 to 3 with an unchanged schema; all 13 new invalid-input Python cases failed before the fix. - Rebuilt the extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`; all 18 focused Python cases pass across sync/async and local/mocked remote tables. - `cargo test --quiet --features remote -p lancedb --lib alter_column`: all eight tests pass, including both new regressions. - `cargo clippy --quiet --features remote --tests --examples` passes across the workspace. - `cargo fmt --all`, `ruff check .`, and `ruff format .` pass. - `typos` 1.26.0 passes across the repository; intentional misspellings in regression fixtures use the existing line-specific spelling exemptions. <!-- lance-gatekeeper-fix:v1 agent=36a547cd11108898893456042e38628b generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
2216586023 |
feat(query): dedup hybrid search by primary key on MemWAL tables (#4268)
Implementing a PK-based dedup to support WAL in hybrid search. Basically, the existing code just calls a vector query + FTS query and dedups locally using the row ID. Since row ID is unsupported in WAL (currently), we update the algorithm to use the tables PK (which is required in WAL, and uniquely linked to a PK on updates). To enable PK-dedup branch, we've chosen to fallback on error rather than fire an exploratory query. Specifically, we issue the regular hybrid query, using row IDs, and when that fails with "WAL does not support row IDs" we fallback to the PK-based variant, and then cache the fallback for future calls from the same client. The alternative is that we chec if WAL is enabled before issuing the query. This would require a `get_lsm_write_spec` call, slowing down all the existing non-WAL paths unnecessarily. The ultimate goal is to get row ID supported in WAL, but that is pending stable row ID GA and subsequent integration into the enterprise stack. Once that lands, we will rip out the PK-based dedup that is only for WAL-enabled tables and use the existing row ID code. This is the main reason for building this adaptively. I had spec-ed out a refactor so that rather than duplicating code between python and rust we build this in rust and have python call that through PyO3. However, that made the reverse path (ie. ) more difficult. So I think the tradeoff of maintaining duplicate code in the short term for the ease of reverting this in the medium term (hopefully 2-3 months). --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
75e71176cb |
fix(python): prevent BlobFile finalizer panics on Tokio workers (#4291)
## Cause `io.RawIOBase.__del__` checks `closed` and calls `close()` during GC. Both PyO3 methods entered `block_on`, which panics when GC runs on a LanceDB Tokio worker. ## Fix Track Python blob handle closure atomically so `closed` is synchronous. `close()` marks the handle closed immediately and, when called on a Tokio thread, schedules the underlying async cleanup without blocking that thread. Reads and seeks after close fail immediately; ordinary callers still wait for cleanup. ## Validation - Rebuilt the Python extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`. - Ran the blob test file and remote blob handle test: 69 passed, 1 skipped. - Ran `cargo fmt --all`, `ruff format .`, and `ruff check .`. Fixes #4283 <!-- lance-gatekeeper-fix:v1 agent=15a7bd545e63dea4b69faa0429f0a684 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
|
01ee01dbc8 |
feat: define a materialized view by its query, with Functions in FROM position (#4190)
A view definition was a structured record under a `kind` tag, one kind per query shape, and a Function returning `list<struct>` was about to add a third. That names shapes instead of describing a relation. A materialized view is now a relation defined by a query, stored as one canonical SQL string under a format number: ```sql SELECT columns FROM [ns.]table [, function(args) AS alias | , UNNEST(column) AS alias] [WHERE predicate] [LIMIT n] ``` Any other clause is refused at parse time. Older readers report the view as unrefreshable, the pre-format layouts still read, and a legacy view is rewritten on its next refresh that commits, rebuilt only where its raw text meant something else under lance's parser. A Function in FROM position yields one row per element it returns, as a table function does in any dialect. The server stages its list output in a hidden table and records that binding beside the query, which stays as the user wrote it; refresh scans the staging and unnests the column, the same operator as UNNEST over a list column the table already holds. Row ids repeat per element, so eviction and incremental append are unchanged. A local database refuses a Function in FROM position, since it has no executor. |
||
|
|
5231d37f8d |
feat: list a table's per-row Function errors from the client (#4208)
A refresh running under a skip policy records each row it skipped, with the failing input and the error, but the client could not read that store: the server exposes it over SQL and, since recently, a REST route. A user who hit per-row failures still had to open a SQL session. `Table::function_errors` calls the route. The listing is table-addressed with optional job and column filters, the same addressing the SQL surface uses, so the two cannot disagree about what a table's errors are. The two non-record signals come back as their own fields rather than as rows: capped-fragment summaries, and whether the listing stopped at its limit. Local tables refuse rather than answer with an empty list. Python and Node expose the same call, with the same optional filters. |
||
|
|
f12996557f |
feat(remote): support materialized view APIs (#4180)
## Summary
Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.
Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.
## Route comparison
| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |
Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.
## Python API changes
| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |
The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
|
||
|
|
a614400755 |
feat: accept blob URI writes (#3954)
#3528 added blob declarations and binary coercion. String values were still rejected. They now coerce to the blob `uri` child. ```python table.add([{"id": 1, "image": "s3://bucket/media/cat.jpg"}]) payload = table.fetch_blobs("image", table.search().to_arrow()) ``` A URI under a registered base writes with no extra options. An unregistered URI fails. `allow_external_blob_outside_bases` is a local escape hatch that stores an absolute URI. It does not register a base. Remote `add` rejects that flag before making a request. String input still coerces and is sent as a `uri` struct. `add_bases` is a follow-up. `merge_insert` does not coerce string blob input. ### Testing - `cargo test -p lancedb --test blob_integration` - `cargo test -p lancedb blob_coerce` - `cargo test -p lancedb --features remote --lib add_rejects_external_blob_flag add_string_blob_becomes_uri_struct` - `cd python && uv run --extra tests pytest python/tests/test_blob.py -k uri -q` Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
|
b0dae5eb0b |
feat: return typed refresh job results (#4013)
## Problem `refresh_column_async` returned a unit-result job even though durable refresh jobs carry a canonical terminal result. Python callers could not obtain row counts or source and published versions through the public `Job` API, and local and remote refresh jobs exposed different result semantics. ## Behavior `refresh_column_async` now returns `Job[RefreshColumnResult]` for local and remote tables. The general typed-job bridge binds each endpoint to its public result model while preserving unit-result jobs and existing status, wait, cancel, and timeout behavior. A local no-op refresh reports no published version. The Node.js API continues to resolve `wait()` as `void`; its binding erases the Rust result type internally to preserve the existing public contract. ## Ownership and integration boundary LanceDB owns the language-neutral `Job<T>` contract and language-binding decode. Sophon owns production and durable persistence of terminal payloads. Sophon #7348 and #7378 now publish the canonical refresh result for Function-backed and expression-backed refresh jobs, respectively. The remote client fixture matches the merged server schema; live deployment and end-to-end demo acceptance remain separate rollout checks. |
||
|
|
e98d8ac685 |
feat!: rename branch merge to cherry_pick (#3986)
This PR is a **breaking** rename of #3686. merge reads like git merge w/ three-way, replay history, combine two lines of work. That is not this API. This call takes one additive change on a branch and lands it on main. New column, including a blob column. Main's existing columns are not rewritten. If it cannot land, you get `status="failed"` and `diff.errors`, not a merge conflict to resolve. Cherry-pick is terminology that aligns more with that. ```python table = db.open_table("images") table.branches.create("exp") exp = table.branches.checkout("exp") exp.add_columns({"tag": "cast('draft' as string)"}) diff = table.branches.diff("exp") preview = table.branches.cherry_pick("exp", dry_run=True) result = table.branches.cherry_pick("exp") if result["status"] == "cherryPicked": print("landed at", result["mainVersionAfter"]) elif result["status"] == "failed": print(result["diff"]["errors"]) ``` ### Behavior - Remote / Enterprise only. Local still NotSupported. - HTTP 409 is not an exception. It is Ok with status="failed" and diff.errors (CherryPickError). - Unknown error / status codes still parse as Unknown. - Requests are not retried. 409 is final and carries the body. - Endpoint is POST /v1/table/{id}/branches/cherry_pick/. - merge_insert and Table.merge are unchanged. ### Testing - `cargo test -p lancedb --features remote diff_branch` - `cargo test -p lancedb --features remote cherry_pick` - `pytest python/python/tests/test_remote_db.py -k cherry_pick` - node `remote.test.ts` diffs / cherry-picks path |