mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 14:16:54 +00:00
90b2b264d99dd9474f86f258b1336cc9a15c57cf
123
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Srijan Keshri
|
90b2b264d9 |
feat(python): accept DataFrames and readers in add_columns
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader. Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them. SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits. Fixes #1959. |
||
  TaylorandBrendan Clement
|
9e65bd2c27 |
docs: use Enterprise terminology across website and SDK documentation (#4787)
Use “LanceDB Enterprise” throughout the staged website guides, SDK reference sources, generated API pages, and user-facing SDK messages. A separate branded “LanceDB Distributed” name adds terminology without clarifying behavior; describe where operations run directly. - Update `docs/web`, including the architecture diagram title and alt text. Use Enterprise in headings, table labels, and prose; simplify redundant product/execution-model wording and name the product only once in the diagram alt text. - Update Python docstrings and the Python/Java reference pages, TypeScript comments and native declaration comments, and Rust API documentation. Regenerate the TypeScript reference, including 24 replacements across six pages. - Update SDK warnings/errors and existing test expectations to match, along with contributor guidance and Vale rules. Remove the obsolete rule distinguishing Enterprise from Distributed. - Keep lowercase “distributed” for technical descriptions. API identifiers, anchors, asset paths, and execution behavior are unchanged. No branded-name exceptions are proposed. Validation: - Python and Node native builds succeeded; focused Python tests passed (36 passed, 4 skipped), and focused Node tests passed (14). - TypeDoc generation and the complete MkDocs reference build passed. MkDocs retains an existing unresolved `lancedb.functions.SecretReference` cross-reference. - Rust formatting, Node lint/format, changed-file Python lint/format, 32 Vale tests, and whitespace checks passed. Repository-wide Python lint has 48 existing findings in six untouched files. - Combined documentation assembly passed. Mintlify validation and link/anchor/redirect checks previously passed for both site builds; no code snippets changed. Companion PRs: [saved editions, terminology guide, and shared rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview and writing rules](https://github.com/lancedb/sophon/pull/9511). --------- Co-authored-by: Brendan Clement <brendan.clement@lancedb.com> |
||
 SHAIK SHIVAJI
|
45edaa350e |
feat(python): expose code analyzer options on FTS and tokenize
Part of #4364 (Python side; the TypeScript `FtsOptions` change will follow in a separate PR). Lance already supports the code analyzer, but `FTS` and `lancedb.tokenize()` had no way to set it, so `FTS({'analyzer': 'code'})` and `FTS(split_identifiers=True)` raised `TypeError`. ## Changes - `FTS` and `lancedb.tokenize()` gain `analyzer` (`"text"` or `"code"`), `split_identifiers`, `split_on_numerics`, `preserve_original` and `index_operators`. They default to `None`, which keeps the tokenizer's own defaults, so existing indexes and calls are unchanged. - Validation follows pylance: `analyzer="code"` with a `base_tokenizer` other than `"code"` or the default `"simple"` is rejected, `analyzer="text"` with `base_tokenizer="code"` is rejected, and turning on a code flag without the code analyzer is rejected. Passing a flag as `False` is always allowed. - `analyzer="code"` with the default `base_tokenizer="simple"` selects the code tokenizer. `FTS` can't tell an explicit `"simple"` from the default, so that case is accepted too. - The deprecated `create_fts_index` is left alone; it points users at `create_index(config=FTS(...))`. - Rust `FtsIndexBuilder` is Lance's `InvertedIndexParams`, so the Rust SDK already had these options. The remote client serializes the same struct, so the options reach LanceDB Distributed in the `create_index` request body. A new case in the Rust `test_create_index` checks that body. ## Tests - `python/tests/test_fts.py`: tokenizer output for each flag, validation errors for `tokenize` and `FTS`, `False` flags on a text index, and local sync and async indexes that match `http`, `request` and `parseHTTPRequest` with splitting on and not match subwords with it off. - `cargo test --features remote remote::table::tests::test_create_index` |
||
 Xuanwo
|
63aa06ef7d |
feat(python): expose Function GPU classes and placement (#4791)
## Summary - Add sync/async `DBConnection.list_gpu_classes()` returning typed `GpuClass` records for `GET /v1/gpu/classes`. Records include the class name, Kubernetes resource, GPU model and memory, per-worker GPU maximum, and worker CPU/memory/ephemeral-storage limits. - Add `udf(gpu_class=...)` for immutable Function-level GPU placement. It requires `gpu=True`; Function versions expose their registered `gpu_class`. - Add `gpu_class` and `gpu_count` to `refresh_column` / `refresh_column_async` and Rust `RefreshColumnOptions`. Both are sent as top-level `backfill_column` fields and validated before any request. `gpu_count` is the aggregate GPU budget across Function workers. - Validate GPU class names with the server lowercase 63-character DNS-label rule. - Document the discovery-to-registration flow, configured hardware versus free capacity, System Usage permission, and per-refresh GPU placement in the API and web guides. Older servers without `/v1/gpu/classes` produce a clear `NotImplementedError` instead of a bare 404. Part of ENT-3470. The client follows the in-flight Sophon contract on `xuanwo/ent-3470-fix` for GPU class discovery and backfill placement options. ## Breaking change `PythonRuntimeSpec::PythonV2` adds `gpu_class`, and `RefreshColumnOptions` adds `gpu_class` and `gpu_count`. Rust callers constructing either value with all fields explicitly must supply the new fields. ## Validation At `7685c968`: Rust GPU tests (5 passed), Rust/PyO3 clippy, Python extension build, pyright, ruff, and Python Function/runtime, discovery, refresh placement, and refresh regressions (147 passed, 6 skipped). At `b22428fd`: Rust GPU tests (5 passed), rustfmt, ruff, pyright, Function slice 2 (85 passed), and remote GPU/refresh tests (20 passed). The final commit only updates documentation and Rust doc comments. |
||
|
Xuanwo
|
6a255926d7 |
fix(python): improve function failure diagnostics (#4790)
## Summary - Make `AsyncJob.wait(timeout)` and `Job.wait(timeout)` raise a descriptive `TimeoutError` containing the job ID and elapsed timeout. Registry jobs explain that work continues on the server and include the `db.open_job(id).wait()` reattach call. SQL query handles instead say to keep the original handle because query IDs cannot be reopened through the registry jobs API. - Render `JobFailureInfo` and `FunctionErrorRecord` with Python values instead of Rust `Option` wrappers. - Decode and expose the server's `row_address` on `FunctionErrorRecord`, and document how to retry recorded rows with `REFRESH COLUMN ... WHERE _rowaddr IN (...)` before compaction or rewrite. - Warn when an import inside a packaged Function body resolves to undeclared local source. The `udf` documentation explains why arbitrary base-image coverage and distribution-to-import-name mapping cannot be validated reliably by the SDK. `row_address` is optional, so records from older servers still decode with `None`. Rust code that constructs `FunctionErrorRecord` literals must now supply this new field. Part of ENT-3448 and ENT-3442. This consumes the server-side diagnostics from lancedb/sophon#9509 and logical row addresses from lancedb/sophon#9512. ## Validation - Rust `function_errors` remote decoding and legacy-missing-`row_address` tests. - Rust/PyO3 clippy for `lancedb` and `lancedb-python`. - Python extension build, pyright, and ruff. - Targeted Python job, Function packaging, Function error, and job detail tests: 62 passed, plus timeout-handle tests: 3 passed. |
||
 lancedb-gatefixer[bot]
|
9804f26bc3 |
fix: support batch size options for column refreshes (#4777)
Python column refresh methods rejected `batch_size`, and the Rust remote backend never included it in the backfill request. Callers could not reduce the rows passed to a slow Function to stay within the executor's per-call deadline. Add optional, keyword-only `batch_size` arguments throughout the synchronous and asynchronous Python APIs and bindings. Add compatible Rust `RefreshColumnOptions` APIs while preserving existing Rust method signatures. Reject non-positive sizes before submitting a job, send positive sizes in `backfill_column`, and omit the field when unset while retaining branch scoping. Batch size controls remote Function calls. Embedded expression refreshes accept and validate the option without changing execution, and blocking remote refreshes retain their existing unsupported behavior; remote callers use `refresh_column_async` and wait on the returned job. Validation: - Reproduced the original Python `TypeError` from the committed method. - Five focused Rust tests passed, covering request contents, omission, branch scoping, and rejection before any request. - `cargo check --quiet --features remote --tests --examples` passed. - `cargo clippy --quiet --features remote --tests --examples -- -D warnings` passed. - Nineteen targeted Python tests passed against the rebuilt extension, including synchronous and asynchronous remote requests, invalid sizes, and default-build behavior. Eight embedded refresh cases require the experimental-functions build and were skipped. - Rust formatting and Python formatting/lint checks passed, including CI's pinned Ruff version. Repository-wide Ruff reports 48 existing findings in untouched documentation files; none are in changed files. Fixes #4776 <!-- lance-gatekeeper-fix:v1 agent=4906ff3f5f7797a1d49a13803302021c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Bruno Ramirez
|
5f7ec4e0a2 |
feat(sdk): expose table overrides for remote tables
Expose table-level maintenance override read, update, and reset APIs for remote
Rust and Python tables. Local tables continue to return `NotSupported`.
Updates recursively merge the caller-provided JSON object into the current
override document so unrelated and unknown service fields are preserved. Reset
uses the explicit reset endpoint.
The integrated change keeps current main's remote database identity, namespace,
branch, SQL, WAL, and threaded mock-server behavior.
Validated with focused Rust and Python override tests and the complete applicable
CI run for
|
||
  Will JonesandClaude Opus 5.5
|
4d7a64712f |
docs: adopt Embedded/Distributed terminology across docs and SDKs (#4713)
## Summary Adopts the product terminology guide across the docs site pages, SDK docstrings and error messages. "LanceDB OSS" and local mode become **LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB Distributed**, which LanceDB Enterprise provides. Where free vs paid is the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB Enterprise". This is a first trial of the Embedded name, so reviewers can judge it in context. - **Embedded vs Distributed** for where execution runs (`optimize()`, consistency, branching, maintenance). Main pages define it once: "LanceDB Enterprise runs LanceDB Distributed: …" - **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://` region, the brute-force KNN limit) and marked legacy. "Cloud and Enterprise" becomes Enterprise or Distributed. - **Plan executor** becomes "executor" or the fleet name. - **Error messages** change, e.g. `optimize is not supported for remote tables.` → `... in LanceDB Distributed.` Tests that match on them are updated. - **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`, `DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the terms in prose. Unchanged: identifiers (`availability.oss`, the `remote` module and feature, `RemoteTable`, `db://`), heading anchors, internal code comments, and Geneva. ## Evidence - Rust lib tests for the changed messages and the Python tests in `test_remote_db.py`, `test_materialized_views.py` and `test_table.py` pass locally. - Not run locally: - The Node tests: `remote.test.ts` and `table.test.ts` have updated assertions. - Vale. - The `experimental_functions`-marked materialized-view tests. - `docs/src` API reference was hand-edited to match the docstrings rather than regenerated. ## Merge Risk **Door:** two-way; prose and message text only. **Blast Radius:** `docs`, `error-messages`: callers matching on old error strings would break. Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon (Enterprise pages). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
2cd635d41c |
fix: correct remote materialized view refresh results and options (#4672)
Remote materialized-view refreshes drained the SQL result and then fabricated `mode=rebuild`, `rows_written=<total view rows>`, and `source_version=0`. This made incremental refreshes report incorrect statistics. Make these three result fields optional across Rust, Python, and TypeScript, and leave them unset for remote SQL refreshes. Remove the post-refresh row count request. Local refreshes continue to populate all metrics, and `version` remains the view version observed after completion. Remote per-operation statistics remain unavailable until the SQL job adapter can retrieve them. Expose the forced-rebuild option through the Rust builder, synchronous/asynchronous Python refresh and job methods, and TypeScript refresh. Remote `full=true` sends `WITH (full = true)`; the default lets the server choose its refresh mode. Local source-version pinning and remote job cancellation are preserved. Local refresh regression tests respect the `experimental-functions` build gate, while remote refresh coverage also runs in default builds. Validation: - Rust: 20 materialized-view tests and 21 remote SQL tests passed with `remote,experimental-functions`; `cargo fmt --all` passed. - Python: rebuilt the extension in the uv-managed environment. The default build passed 23 tests with 27 feature-dependent skips. The opt-in build passed 61 tests with 3 expected default-build-only skips, covering the materialized-view suite and 14 async query-serialization cases from current main. Python Ruff lint and formatting checks passed. - TypeScript: default and opt-in native builds and TypeScript compilation passed. The default build passed 5 tests with 10 expected feature-dependent skips; the opt-in build passed all 14 materialized-view tests with the default-build-only test skipped. Lint/format checks and documentation generation passed. - Repository-wide `ruff check .` reports 48 pre-existing violations in unchanged `docs/web-tests` files; `ruff check python` passes. Fixes #4670 <!-- lance-gatekeeper-fix:v1 agent=ece43ec4b88a48d1789f60005fa3ecfc generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
lancedb-gatefixer[bot]andXuanwo
|
3352bf49df |
fix(python): expose optimize compaction options (#3870)
## Summary - expose typed compaction options on synchronous and asynchronous table optimization - translate validated Python options into native Lance compaction settings - document the row-based steady-state limit and first-pass/source-budget caveats - carry the Lance beta.15 fragment-exclusion option through the typed Python API ## Root cause The native Python optimize binding always constructed default compaction settings, and the public AsyncTable API offered no way to lower the fragment row target. Wide-row fragments therefore remained compaction candidates on every later optimize call, repeatedly rewriting the accumulated table. ## Limitations This is an explicit row-based mitigation, not byte-native task sizing or a hard memory cap. Callers must derive target_rows_per_fragment from expected row size. Existing oversized fragments are rewritten whole on the first pass; max_bytes_per_file limits output file size rather than compaction memory; and source budgets admit whole planned tasks, so a budget below the first task performs no compaction. Splitting tasks by a byte budget or returning a planner error requires a Lance-side change. ## Validation - cargo check --release -p lancedb-python - cargo clippy --quiet -p lancedb-python - cargo fmt --all --check - ruff format --check python tests - ruff check python tests - uv run --no-sync pytest python/tests/test_table.py -k optimize_compaction -q (16 passed) Fixes #2325 <!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io> |
||
 Jack Ye
|
dcfef7c5ab |
feat(python): open lazy blob files from SQL descriptors (#4668)
SQL blob descriptors need pinned source identity to reopen a value after
the query finishes. Adds `open_blob_file(descriptor)` and
`open_blob_files(descriptors)` so Python can read the original blob
incrementally without materializing it in the SQL result.
### Python experience
```python
rows = db.execute_query("""
SELECT f.timestamp, blob_descriptor(v.video) AS video
FROM videos v JOIN frames f ON v.id = f.video_id
""").read_all()
videos = db.open_table("videos")
for row in rows.to_pylist():
if row["video"] is not None:
with videos.open_blob_file(row["video"]) as video:
frame = extract_frame(video, row["timestamp"])
```
The singular API accepts the descriptor dictionary or an Arrow struct
scalar directly and returns a seekable `BlobFile`.
`videos.open_blob_files(rows["video"])` accepts an Arrow array/chunked
array or another iterable, opens handles as iteration advances, and
preserves order, duplicates, and nulls as `None`. Async tables provide
`await videos.open_blob_file(...)` and `async for video in
videos.open_blob_files(...)`.
The iterable API can be used directly with the Arrow result column. Zip
it with the rows to keep each frame timestamp:
```python
for row, video in zip(rows.to_pylist(), videos.open_blob_files(rows["video"])):
if video is not None:
with video:
frame = extract_frame(video, row["timestamp"])
```
Each iteration opens one handle. Null descriptors yield `None`; handles
preserve input order and duplicates, and the `with` block closes each
non-null file.
For an async table:
```python
async for video in async_videos.open_blob_files(rows["video"]):
if video is not None:
with video:
payload = await video.aread()
```
### Execution
The descriptor retains the canonical `kind`, `position`, `size`,
`blob_id`, and `blob_uri` fields and carries a `source` struct with its
dataset URI, version, commit timestamp, branch, column, row reference,
and whether that reference is a physical address. Native opening
resolves that exact snapshot without changing the table handle's
checkout and rejects a descriptor belonging to another table. Remote
handles send authenticated byte-range requests with the captured
version/branch, commit timestamp, and source URI, so an old physical
address is never interpreted against the latest table.
Known-size remote descriptors open without a payload request. External
blobs with an unspecified size use a metadata-only HEAD request. Payload
bytes load only when the file is read; full reads still allocate the
full value. Existing row-ID-based `fetch_blob_files` remains available
when a descriptor is not already present.
Coverage includes native snapshot preservation after overwrite, both row
IDs and addresses, dictionary/Arrow inputs, null/empty values, reordered
and duplicate descriptors, lazy iterable behavior, async iteration, and
remote range pinning for all six storage kinds.
Descriptor reads isolate their pinned snapshot from the live table
handle's freshness state. Both native and remote reads verify the source
commit timestamp to reject a replaced snapshot after table recreation.
|
||
 adenzhou1350
|
3d883af674 |
fix(python): propagate invalid index wait_timeout errors (#4495)
Invalid `wait_timeout` values currently reach `Duration::extract(...).unwrap()` in both native index-creation bindings. A normal Python argument error becomes `pyo3_runtime.PanicException`, which is not caught by `except Exception`. Propagate the extraction error with `map(...).transpose()?` before constructing the index operation. Valid `None` and `timedelta` behavior is unchanged. Tests cover both `create_index` and `create_index_async`, three invalid types, valid `None`/`timedelta`, and no index created after rejection. Validation against main `0aa842b`, Python 3.12, uv-locked dependencies, freshly compiled PyO3 extension for each arm: - Unmodified source: 6 new regressions fail with `PanicException`; 6 controls pass. - Fixed source: 12/12 pass; broader index-creation regression selection: 18/18 pass. - `cargo fmt --all --check`, Ruff check/format, and diff check pass. The broader selection includes the 12 cases above. No cloud, full-suite, or performance claim. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
adenzhou1350
|
a12aef2a8c |
fix: return a Python error when querying a closed table (#4455)
`AsyncTable.query()` unwraps the native closed-table error, so querying a closed table raises `PanicException` instead of the documented `RuntimeError`. `AsyncTable.head()` reaches the same path. Propagate `inner_ref()`'s error through `PyResult` instead. Add regressions for closed `query()` and `head()`, with `count_rows()`, repeated close, and normal open-table queries as controls. Validation against separately built native bindings: - Before: 2 regression failures (`PanicException`), 2 controls passed. - After: all 4 regressions passed. - Existing query tests plus the basic database test: 102 passed, 1 skipped on each arm (Torch is not installed). - `cargo fmt --all -- --check`, Ruff, and `git diff --check` passed. The native build used the locked Cargo dependencies with `remote,lancedb/metrics-otel,pyo3/extension-module`. This is source-tree native testing, not a complete wheel-packaging or cloud-service test. Signed-off-by: Xucheng Zhou <aden1350@outlook.com> |
||
|
Jack Ye
|
a4544c5472 |
feat: configure automatic refresh when declaring computed columns (#4607)
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
|
||
 Dan RammerandClaude Opus 5.5
|
92825f6f13 |
refactor!: rename MemWAL APIs to RTE (realtime engine) (#4598)
## What Renames the user-facing MemWAL surface to **RTE** (realtime engine) across Rust, Python, TypeScript, and Java, including the REST wire format. This follows the LSM → MemWAL rename in #4075. | Before | After | |---|---| | `MemWalWriteSpec` / `MemWalStats` | `RTEWriteSpec` / `RTEStats` | | `enable_memwal` / `disable_memwal` / `get_memwal_spec` | `enable_rte` / `disable_rte` / `get_rte_spec` | | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | `flush_rte` / `compact_rte` / `checkpoint_rte` | | `get_memwal_stats` / `close_memwal_writers` | `get_rte_stats` / `close_rte_writers` | | `use_memwal` (query, merge insert) | `use_rte` | | TS/Java `enableMemWal`, `useMemWal`, ... | `enableRTE`, `useRTE`, ... (napi `js_name` set explicitly) | | Java `LanceDbTableMemWal` | `LanceDbTableRTE` | | `table/memwal_stats.rs` | `table/rte_stats.rs` | Docs and comments now say "realtime engine (RTE)". Text about Lance's underlying mechanism keeps "MemWAL" ("Lance's MemWAL", "MemWAL index"). **Left alone:** Lance-owned names (`MemWalIndexDetails`, `DatasetMemWalExt`, `InitializeMemWal*`, `mem_wal` paths), internal `Lsm*` names and modules, and the `does not support with_row_id` refusal text that the hybrid fallback matches on. ## Compatibility Breaking, for an experimental API. The wire format changes, so this must roll out together with the matching sophon change (lancedb/sophon#8870): - Routes: `/enable_memwal/` → `/enable_rte/`, `/disable_memwal/` → `/disable_rte/`, `/get_memwal_spec/` → `/get_rte_spec/`, `/get_memwal_stats/` → `/get_rte_stats/`, `/flush_memwal/` → `/flush_rte/`, `/compact_memwal/` → `/compact_rte/` - Response keys: `memwal_spec` → `rte_spec`, `memwal_stats` → `rte_stats` - Request field: `use_memwal` → `use_rte` on query and merge-insert bodies The SQL keyword becomes `ALTER TABLE ... SET RTE` server-side. The guides under `docs/web` will be updated separately. ## Validation - Rust: `cargo clippy --features remote --tests --examples` clean; `cargo test --features remote -p lancedb --lib`: 1533 passed - Java: `mvn test`: 37 passed (Spotless applied with JDK 11) - Python: `ruff format` / `ruff check` clean; targeted tests (`test_rte_spec`, `test_merge_insert_lsm`, `test_remote_db`, `test_wal_hybrid`, `test_query`, `test_permutation`, `test_table`): 708 passed - TypeScript: `pnpm build`; `table`, `query` tests: 606 passed; `biome check` clean; `docs/src/js` regenerated with `pnpm run docs` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Dan RammerandClaude Opus 5
|
fe34eb3549 |
refactor!: rename LSM APIs to MemWAL and align SSTable/table shard naming (#4075)
## What Renames the MemWAL surface in two passes, across Rust, Python, TypeScript, and Java, including the wire format. ### 1. MemWAL APIs drop "LSM" The write path these APIs drive is Lance's MemWAL; "LSM" described its shape rather than naming it. Turning it on and off now reads as enabling a feature rather than installing a spec. | Before | After | |---|---| | `set_lsm_write_spec` | `enable_memwal` | | `unset_lsm_write_spec` | `disable_memwal` | | `get_lsm_write_spec` | `get_memwal_spec` | | `flush_lsm` / `compact_lsm` / `checkpoint_lsm` | `flush_memwal` / `compact_memwal` / `checkpoint_memwal` | | `get_lsm_stats` | `get_memwal_stats` | | `close_lsm_writers` | `close_memwal_writers` | | `use_lsm` (query, merge insert) | `use_memwal` | | `LsmWriteSpec` / `LsmStats` | `MemWalWriteSpec` / `MemWalStats` | | Java `LanceDbTableLsm` | `LanceDbTableMemWal` | TypeScript and Java use camelCase (`enableMemWal`, `getMemWalStats`, ...); the napi methods set `js_name` so the generated names keep the `MemWal` casing. flush/compact/checkpoint keep a `_memwal` suffix so `compact_memwal` is not confused with `compact_files` / `optimize`, which compact the base table. Internal names (`LsmMode`, `LsmScanner`, the `merge::lsm` / `query::lsm` modules) are unchanged. ### 2. Stats use SSTable and table shard Aligns the stats surface with the MemWAL naming settled upstream in lance-format/lance#7943 and #7957, where the persisted unit became an **SSTable**. **Generation → SSTable.** A `GenerationStats` describes one flushed MemTable, which is an SSTable. The generation *number* is kept — an SSTable is identified by its generation — so only the noun moved. **Bucket → table shard.** Each entry is one MemWAL shard. "Bucket" names only the hash sharding *transform*, so it was already wrong for `identity` and `year` sharding, which produce shards and no buckets at all. | Before | After | |---|---| | `GenerationStats` | `SsTableStats` | | `BucketStats` | `TableShardStats` | | `stats.buckets` | `stats.table_shards` | | `BucketStats.generations` | `TableShardStats.sstables` | | `include_generation_rows` | `include_sstable_rows` | | `newest_generation` | `newest_sstable_generation` | | `outstanding_generations` | `outstanding_sstables` | Docs and comments follow suit: "seal" → "freeze", "L0" / "flushed generations" → SSTables, "bucket" (meaning a shard) → table shard. `MemWalWriteSpec::bucket`, `num_buckets`, and the `"Sealed"` shard status value are **unchanged** — those name the sharding transform or a server value, not the shard. ## Compatibility Breaking, for an experimental API. The wire format changes with the SDKs, so this must roll out together with the matching server change: - Routes: `/set_lsm_write_spec/` → `/enable_memwal/`, `/unset_lsm_write_spec/` → `/disable_memwal/`, `/get_lsm_write_spec/` → `/get_memwal_spec/`, `/flush_lsm/` → `/flush_memwal/`, `/compact_lsm/` → `/compact_memwal/`, `/get_lsm_stats/` → `/get_memwal_stats/` - Response keys: `lsm_write_spec` → `memwal_spec`, `lsm_stats` → `memwal_stats`, `buckets` → `table_shards`, `generations` → `sstables` - Request fields: `include_generation_rows` → `include_sstable_rows`; `use_lsm` → `use_memwal` on query and merge-insert bodies ## Validation - Rust: `cargo test --features remote -p lancedb --lib --tests` — 1467 lib tests and all integration test targets pass; `cargo clippy` clean - Python: rebuilt the extension; `test_remote_db`, `test_memwal_spec`, `test_merge_insert_lsm`, `test_query`, `test_permutation`, `test_table` — 655 passed; `ruff format` / `ruff check` clean - TypeScript: `pnpm build`; `table`, `merge`, `query` tests — 593 passed; `biome check` clean; `docs/src/js` regenerated with `npm run docs` - Java: `mvn test` — 37 passed (Spotless applied with JDK 11; the pinned google-java-format does not run on JDK 21) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
lancedb-gatefixer[bot]
|
a1e4dc9f91 | fix(python): align BlobFile seek and read behavior (#4290) | ||
|
lancedb-gatefixer[bot]
|
0212cfd3d6 | fix(python): normalize blob row ID inputs and errors (#4293) | ||
|
lancedb-gatefixer[bot]
|
0acb61471c |
fix(python): route async remote blob reads through server (#4306)
## Cause Async remote pandas conversion could call `AsyncTable.to_lance()`, which opened the server's table storage URI in the client process. This bypassed server query and authorization APIs. The first repair routed blob reads through the server, but a query and subsequent blob fetch could resolve different table snapshots after replacement or compaction. ## Fix - Reject `AsyncTable.to_lance()` for remote tables before requesting a table URI. - Convert remote nonblob and blob description results through the server Arrow path. - Reject automatic remote byte and lazy blob materialization until the server can bind the query and fetches to one verified table identity and version. The default lazy mode now raises `NotImplementedError` when a blob column is projected; callers can request `blob_mode="descriptions"`. ## Validation - Mocked remote tests cover direct storage rejection, description results, byte and lazy rejection before any blob fetch, and nonblob projections. - Focused local and remote blob tests passed (19 tests). `ruff check .`, `ruff format .`, and pyright passed. Fixes #4272 <!-- lance-gatekeeper-fix:v1 agent=35937e6e1d968f62eea59adcc56c84e9 generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
bcf5ee263f |
fix(python): preserve code tokenizer filter defaults (#4370)
Python's FTS bindings always forwarded `stem=True` and `remove_stop_words=True`, overriding Lance's code tokenizer defaults. As a result, code indexes stemmed identifiers and discarded stop words, and `lancedb.tokenize` reproduced the same behavior. Use `None` for omitted filter options and only apply explicit booleans in the Rust bindings. This preserves tokenizer defaults across local and remote index creation, the deprecated `create_fts_index` helpers, and standalone tokenization. Update the signatures and documentation to describe the inherited defaults. Regression tests cover stored local settings, token output, stop-word searches, remote request payloads, explicit overrides, and the existing simple tokenizer behavior. The regression suite produced 16 failures before the fix and passes all 28 cases after it. Validation: - Rebuilt the Python extension using the uv-managed development environment. - Full Python FTS and index test files plus relevant remote index tests: 105 passed. - `cargo clippy --quiet --features remote --tests --examples -p lancedb-python` - `cargo fmt --all`, Ruff format/lint with CI's pinned version 0.9.9, and Pyright (zero errors). Fixes #4365 <!-- lance-gatekeeper-fix:v1 agent=9c1db2d3fc25c49c5b340f8debf43a0f generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
lancedb-gatefixer[bot]
|
1e4662ac23 |
fix(remote): unify unsupported operation errors and warnings (#4414)
Remote SDKs report Cloud-specific errors against self-hosted servers, and some Python wrappers raise different exception types for the same unsupported operation. The sync index wrapper also warns that `replace` is ignored even though Rust forwards it to the server. Use backend-neutral table/server diagnostics and the existing `NotSupported` error conversions for unsupported manifest operations, blob ranges, primary keys, optimization, and HNSW-PQ indexes. Sync Python delegates to the async paths, so manifest operations and `optimize()` consistently raise `NotImplementedError`. Legacy and config-based HNSW-PQ creation share the same error and alternative-index advice. Remove the false replacement warning and simplify the ignored GPU-accelerator warning. Keep the deprecated cleanup/compaction no-ops, with backend-neutral warnings that do not assume automatic server maintenance. Embedding configuration forwarding and stale warning removal are now supplied by #4408 in `main`. Retain the regression that verifies full embedding metadata and warning-free remote creation alongside the diagnostics tests. Validation against the merged `main` revision `bbc4aebe` and Lance 14.0.0-beta.1: - Rebuilt the Python extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`; all 122 remote Python tests passed. Root Ruff checks and formatting passed using the uv-managed environment. - `pnpm build`, `pnpm lint`, formatting, and `pnpm run docs` passed; all 47 remote Node tests and the focused local vector-index regression passed. - All 583 Rust remote tests and 54 query unit tests passed, along with `cargo check --quiet --features remote --tests --examples`, `cargo clippy --quiet --features remote --tests --examples`, and `cargo fmt --all`. Fixes #4393 <!-- lance-gatekeeper-fix:v1 agent=10bafd337999f4910855c3a9b71e7dcf generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Yuval LifshitzandClaude Opus 5.0
|
069b2f7d31 |
feat: add delete rows to table stats (#3783)
add to table stats the number of rows that has been marked for deletion, but not yet deleted. note that when a table fragment is removed due to deletion of all of its rows, these rows are not counted. since the main use of this feature is to determine when table compaction is needed, the fact that these rows are not reported should not impact that decision (deleted table fragments don't need compaction). Fixes: https://github.com/lancedb/lancedb/issues/3761 Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com> |
||
 XY Zhan
|
2bf70758a7 |
fix: read back the maintained-index selection a table was given (#4447)
`maintained_indexes` on a MemWAL table has three values: a named list, `[]` for none, and `null` for every index the table has, including ones created later. This PR makes every client read all three back as they were set. - **Rust remote client** (also used by the Python and Node SDKs) reads `null` as `None`. - **Java client** reads `null` as `null`, not as an empty list. - **Docs** for the setter, the field and the getter describe the three values the same way in Rust, Python, Node and Java. The generated JS docs are regenerated to match. ## Testing - `rust/lancedb`: 73 LSM tests pass, including one case per value. With the decoder typed as a list, the `null` case fails. - Java: two cases for `null` and `[]` on the read path, run by the Java CI job. - `cargo fmt`, `cargo clippy -D warnings` and `ruff format` are clean. |
||
|
Jack Ye
|
7e8f0d4982 |
refactor: always fully refresh materialized views (#4373)
Simplify experimental materialized views to one output-only storage
layout and one full-rebuild refresh implementation.
## Current layout
Local materialized views currently maintain incremental-refresh state
alongside the materialized values:
- an internal `__source_row_id` column for source provenance;
- `mv.definition` plus `mv.incarnation`;
- source version/timestamp, view version, and last-refresh watermarks;
- optional `mv.staging` state for table-function sources; and
- stable-row-ID requirements on the source table.
Refresh classifies source changes, appends or evicts rows when possible,
and falls back to a rebuild for unsupported deltas. Remote
create/refresh also use separate REST mutation paths.
## New layout
An MV is an ordinary Lance backing table containing only:
- the defining query's output columns;
- its current data fragments; and
- `mv.definition`, encoded as a versioned SQL envelope such as
`{"kind":"query","format":1,"query":"SELECT ..."}`.
There is no provenance column, incarnation, source/view watermark,
staging metadata, or stable-row-ID requirement. Local refresh may still
pin a source version for reproducibility, but it always recomputes the
complete result.
Full refresh streams the new result into uncommitted fragments and then
publishes a single transaction that removes every old fragment and
installs the new set. A shared conflict token prevents concurrent
replacements from both committing. This fragment-replacement primitive
is also used by Sophon's SQL worker.
## New remote create workflow
```text
Connection::create_materialized_view(...)
-> builder renders the defining SELECT
-> remote Database receives a CreateMaterializedViewRequest
-> submit CREATE MATERIALIZED VIEW ... AS ... through Flight SQL
-> retain the SQL Query handle while the Job is running
-> consume the durable SQL-job result
-> open the created table and read mv.definition
-> return MaterializedView
```
Remote refresh similarly submits `REFRESH MATERIALIZED VIEW` through
Flight SQL. Cancelling either wrapper forwards cancellation to the
retained SQL query before aborting the local task, so the server
operation is not silently left running.
## API changes
- remove incremental/no-op modes and the `full` switch;
- remove distributed grouped/partitioned refresh APIs and native
vector-source MV APIs;
- keep full-refresh results across Rust, Python, and TypeScript;
- preserve list/describe/drop as object-management operations while
routing create/refresh mutations through SQL.
The storage contract is experimental and has no compatibility shim;
views using the removed layout must be recreated.
|
||
|
lancedb-gatefixer[bot]
|
de19ecbf63 |
fix: reject empty column alterations and unknown Python keys (#4417)
Fixes #4390 Lance accepts column alterations with every change field unset, so a path-only Python call could commit a new local version without changing the schema. The Python binding also silently discarded unknown keys such as `nulable`. Validate every alteration in the shared Rust `Table` API before dispatching to either backend, and reject unknown dictionary keys in the Python binding while preserving the legacy `name` alias. Invalid batches leave local schema and version unchanged and send no remote request. Validation: - Reproduced both reported calls on the original extension: versions advanced from 1 to 2 to 3 with an unchanged schema; all 13 new invalid-input Python cases failed before the fix. - Rebuilt the extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`; all 18 focused Python cases pass across sync/async and local/mocked remote tables. - `cargo test --quiet --features remote -p lancedb --lib alter_column`: all eight tests pass, including both new regressions. - `cargo clippy --quiet --features remote --tests --examples` passes across the workspace. - `cargo fmt --all`, `ruff check .`, and `ruff format .` pass. - `typos` 1.26.0 passes across the repository; intentional misspellings in regression fixtures use the existing line-specific spelling exemptions. <!-- lance-gatekeeper-fix:v1 agent=36a547cd11108898893456042e38628b generation=1 --> --------- Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
Dan RammerandClaude Opus 5
|
2216586023 |
feat(query): dedup hybrid search by primary key on MemWAL tables (#4268)
Implementing a PK-based dedup to support WAL in hybrid search. Basically, the existing code just calls a vector query + FTS query and dedups locally using the row ID. Since row ID is unsupported in WAL (currently), we update the algorithm to use the tables PK (which is required in WAL, and uniquely linked to a PK on updates). To enable PK-dedup branch, we've chosen to fallback on error rather than fire an exploratory query. Specifically, we issue the regular hybrid query, using row IDs, and when that fails with "WAL does not support row IDs" we fallback to the PK-based variant, and then cache the fallback for future calls from the same client. The alternative is that we chec if WAL is enabled before issuing the query. This would require a `get_lsm_write_spec` call, slowing down all the existing non-WAL paths unnecessarily. The ultimate goal is to get row ID supported in WAL, but that is pending stable row ID GA and subsequent integration into the enterprise stack. Once that lands, we will rip out the PK-based dedup that is only for WAL-enabled tables and use the existing row ID code. This is the main reason for building this adaptively. I had spec-ed out a refactor so that rather than duplicating code between python and rust we build this in rust and have python call that through PyO3. However, that made the reverse path (ie. ) more difficult. So I think the tradeoff of maintaining duplicate code in the short term for the ease of reverting this in the medium term (hopefully 2-3 months). --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
lancedb-gatefixer[bot]
|
75e71176cb |
fix(python): prevent BlobFile finalizer panics on Tokio workers (#4291)
## Cause `io.RawIOBase.__del__` checks `closed` and calls `close()` during GC. Both PyO3 methods entered `block_on`, which panics when GC runs on a LanceDB Tokio worker. ## Fix Track Python blob handle closure atomically so `closed` is synchronous. `close()` marks the handle closed immediately and, when called on a Tokio thread, schedules the underlying async cleanup without blocking that thread. Reads and seeks after close fail immediately; ordinary callers still wait for cleanup. ## Validation - Rebuilt the Python extension with `uv run --extra tests --extra dev maturin develop --extras tests,dev`. - Ran the blob test file and remote blob handle test: 69 passed, 1 skipped. - Ran `cargo fmt --all`, `ruff format .`, and `ruff check .`. Fixes #4283 <!-- lance-gatekeeper-fix:v1 agent=15a7bd545e63dea4b69faa0429f0a684 generation=1 --> Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
 Wyatt Alt
|
01ee01dbc8 |
feat: define a materialized view by its query, with Functions in FROM position (#4190)
A view definition was a structured record under a `kind` tag, one kind per query shape, and a Function returning `list<struct>` was about to add a third. That names shapes instead of describing a relation. A materialized view is now a relation defined by a query, stored as one canonical SQL string under a format number: ```sql SELECT columns FROM [ns.]table [, function(args) AS alias | , UNNEST(column) AS alias] [WHERE predicate] [LIMIT n] ``` Any other clause is refused at parse time. Older readers report the view as unrefreshable, the pre-format layouts still read, and a legacy view is rewritten on its next refresh that commits, rebuilt only where its raw text meant something else under lance's parser. A Function in FROM position yields one row per element it returns, as a table function does in any dialect. The server stages its list output in a hidden table and records that binding beside the query, which stays as the user wrote it; refresh scans the staging and unnests the column, the same operator as UNNEST over a list column the table already holds. Row ids repeat per element, so eviction and incremental append are unchanged. A local database refuses a Function in FROM position, since it has no executor. |
||
|
Wyatt Alt
|
5231d37f8d |
feat: list a table's per-row Function errors from the client (#4208)
A refresh running under a skip policy records each row it skipped, with the failing input and the error, but the client could not read that store: the server exposes it over SQL and, since recently, a REST route. A user who hit per-row failures still had to open a SQL session. `Table::function_errors` calls the route. The listing is table-addressed with optional job and column filters, the same addressing the SQL surface uses, so the two cannot disagree about what a table's errors are. The two non-record signals come back as their own fields rather than as rows: capped-fragment summaries, and whether the listing stopped at its limit. Local tables refuse rather than answer with an empty list. Python and Node expose the same call, with the same optional filters. |
||
|
Jack Ye
|
f12996557f |
feat(remote): support materialized view APIs (#4180)
## Summary
Align the experimental materialized-view HTTP transport with the
equivalent Table API shape and add remote materialized-view support
across Rust, Python, and TypeScript. This is an intentional breaking
change to the experimental materialized-view surface.
Materialized-view creation performs an initial refresh by default. The
create endpoint returns `202 Accepted` with `{ "job_id": "..." }`;
blocking SDK creation waits for that job before returning a populated
view. `with_no_data` / `withNoData` explicitly creates only the
definition and empty backing table.
## Route comparison
| Operation | Materialized-view API | Equivalent Table API |
| --- | --- | --- |
| Create | `POST /v1/materialized_view/{id}/create` | `POST
/v1/table/{id}/create` |
| Describe/open | `POST /v1/materialized_view/{id}/describe` | `POST
/v1/table/{id}/describe` |
| List | `GET /v1/namespace/{id}/materialized_view/list` | `GET
/v1/namespace/{id}/table/list` |
| Refresh | `POST /v1/materialized_view/{id}/refresh` | asynchronous
Table mutation pattern |
| Drop | `POST /v1/materialized_view/{id}/drop` | `POST
/v1/table/{id}/drop` |
Create, describe, refresh, and drop identify the target in the singular
item path instead of duplicating it in the request body. Create and drop
require `202 Accepted` with a valid job ID. List is a namespace-scoped
GET with opaque pagination tokens. The Rust list API now returns view
names, matching Table listing and the existing Python and TypeScript
APIs.
## Python API changes
| Operation | Synchronous API | Asynchronous API | Table/job pattern |
| --- | --- | --- | --- |
| Create and wait | `DBConnection.create_materialized_view(...)` |
`await AsyncConnection.create_materialized_view(...)` | Returns a
materialized-view handle after its initial-population job finishes |
| Submit create | `DBConnection.create_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.create_materialized_view_async(...)
-> AsyncJob[None]` | Matches job-returning Table mutations such as
`create_index_async` |
| Open | `DBConnection.open_materialized_view(...)` | `await
AsyncConnection.open_materialized_view(...)` | Opens the backing Table
plus its definition |
| List | `DBConnection.list_materialized_views()` | `await
AsyncConnection.list_materialized_views()` | Returns names like Table
listing |
| Refresh and wait | `MaterializedView.refresh(...)` | `await
AsyncMaterializedView.refresh(...)` | Returns the typed refresh result
after the job finishes |
| Submit refresh | `MaterializedView.refresh_async(...) ->
Job[RefreshMaterializedViewResult]` | `await
AsyncMaterializedView.refresh_async(...) ->
AsyncJob[RefreshMaterializedViewResult]` | Matches
`Table.refresh_column_async`; remote job handles expose the server job
ID |
| Drop | `DBConnection.drop_materialized_view(...)` | `await
AsyncConnection.drop_materialized_view(...)` | Matches blocking
`drop_table` |
| Submit drop | `DBConnection.drop_materialized_view_async(...) ->
Job[None]` | `await AsyncConnection.drop_materialized_view_async(...) ->
AsyncJob[None]` | Matches `drop_table_async`; remote handles expose the
server cleanup job ID |
The materialized-view handle exposes its backing Table through `.table`,
so normal Table query, search, and index APIs apply. Definition lookup
and refresh are backend-aware rather than depending on local schema
metadata. TypeScript exposes the equivalent blocking/job drop pair as
`dropMaterializedView` and `dropMaterializedViewAsync`.
|
||
 DrewandXuanwo
|
a614400755 |
feat: accept blob URI writes (#3954)
#3528 added blob declarations and binary coercion. String values were still rejected. They now coerce to the blob `uri` child. ```python table.add([{"id": 1, "image": "s3://bucket/media/cat.jpg"}]) payload = table.fetch_blobs("image", table.search().to_arrow()) ``` A URI under a registered base writes with no extra options. An unregistered URI fails. `allow_external_blob_outside_bases` is a local escape hatch that stores an absolute URI. It does not register a base. Remote `add` rejects that flag before making a request. String input still coerces and is sent as a `uri` struct. `add_bases` is a follow-up. `merge_insert` does not coerce string blob input. ### Testing - `cargo test -p lancedb --test blob_integration` - `cargo test -p lancedb blob_coerce` - `cargo test -p lancedb --features remote --lib add_rejects_external_blob_flag add_string_blob_becomes_uri_struct` - `cd python && uv run --extra tests pytest python/tests/test_blob.py -k uri -q` Co-authored-by: Xuanwo <github@xuanwo.io> |
||
|
Xuanwo
|
b0dae5eb0b |
feat: return typed refresh job results (#4013)
## Problem `refresh_column_async` returned a unit-result job even though durable refresh jobs carry a canonical terminal result. Python callers could not obtain row counts or source and published versions through the public `Job` API, and local and remote refresh jobs exposed different result semantics. ## Behavior `refresh_column_async` now returns `Job[RefreshColumnResult]` for local and remote tables. The general typed-job bridge binds each endpoint to its public result model while preserving unit-result jobs and existing status, wait, cancel, and timeout behavior. A local no-op refresh reports no published version. The Node.js API continues to resolve `wait()` as `void`; its binding erases the Rust result type internally to preserve the existing public contract. ## Ownership and integration boundary LanceDB owns the language-neutral `Job<T>` contract and language-binding decode. Sophon owns production and durable persistence of terminal payloads. Sophon #7348 and #7378 now publish the canonical refresh result for Function-backed and expression-backed refresh jobs, respectively. The remote client fixture matches the merged server schema; live deployment and end-to-end demo acceptance remain separate rollout checks. |
||
|
Drew Gallardo
|
e98d8ac685 |
feat!: rename branch merge to cherry_pick (#3986)
This PR is a **breaking** rename of #3686. merge reads like git merge w/ three-way, replay history, combine two lines of work. That is not this API. This call takes one additive change on a branch and lands it on main. New column, including a blob column. Main's existing columns are not rewritten. If it cannot land, you get `status="failed"` and `diff.errors`, not a merge conflict to resolve. Cherry-pick is terminology that aligns more with that. ```python table = db.open_table("images") table.branches.create("exp") exp = table.branches.checkout("exp") exp.add_columns({"tag": "cast('draft' as string)"}) diff = table.branches.diff("exp") preview = table.branches.cherry_pick("exp", dry_run=True) result = table.branches.cherry_pick("exp") if result["status"] == "cherryPicked": print("landed at", result["mainVersionAfter"]) elif result["status"] == "failed": print(result["diff"]["errors"]) ``` ### Behavior - Remote / Enterprise only. Local still NotSupported. - HTTP 409 is not an exception. It is Ok with status="failed" and diff.errors (CherryPickError). - Unknown error / status codes still parse as Unknown. - Requests are not retried. 409 is final and carries the body. - Endpoint is POST /v1/table/{id}/branches/cherry_pick/. - merge_insert and Table.merge are unchanged. ### Testing - `cargo test -p lancedb --features remote diff_branch` - `cargo test -p lancedb --features remote cherry_pick` - `pytest python/python/tests/test_remote_db.py -k cherry_pick` - node `remote.test.ts` diffs / cherry-picks path |
||
|
Wyatt Alt
|
851fa16b47 |
feat(python): materialized view bindings (#3933)
Exposes materialized views to Python in both the async and sync clients: create_materialized_view / open_materialized_view / list_materialized_views on the connections, and MaterializedView / AsyncMaterializedView handles carrying the parsed definition and refresh(full=, source_version=), which returns the typed refresh result. select accepts column names, (alias, expression) pairs, or a dict of the same; the definition reads back off the stored schema, so a reopened handle needs no side channel. Remote connections raise NotImplementedError up front rather than failing deep in a request, matching the computed-column convention. <sub>Stack created with <a href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub> |
||
|
Xuanwo
|
685cb01d6d |
feat: add grouped function column bindings (#3994)
Function applications from the canonical remote contract cannot currently declare scalar or grouped computed-column outputs atomically. This adds the remote-only declaration contract for scalar, struct-as-one-column, and expanded named-struct outputs. It validates result mappings, fixes exact input/output Arrow schemas in the request, persists grouped sibling metadata, and keeps local Function execution unsupported. Unknown newer application or binding metadata remains readable, while schema-changing mutations fail closed instead of rewriting it. Stable Lance field IDs are deliberately not a declaration prerequisite in this slice. Inputs bind by parameter name and field path; Sophon remains responsible for exact-version validation, atomic all-NULL sibling creation, binding identity and revision allocation, and persisted output identities. |
||
|
Xuanwo
|
7adcffc2b4 |
fix(python): set LsmWriteSpec module metadata (#3995)
PyO3 exposed `LsmWriteSpec` with its default `builtins` module, causing mkdocstrings to resolve the public `lancedb.LsmWriteSpec` re-export as `builtins.LsmWriteSpec` and fail the documentation build. Declare the native extension module and pin the public re-export with a regression test. This also applies the repository's current Ruff formatter to seven previously unformatted Python scripts. |
||
|
Wyatt Alt
|
c429863122 |
feat: refresh_column_async returns a job handle (#3939)
Mirrors create_index's dual surface: the blocking refresh_column keeps
returning {rows_filled, version}, and refresh_column_async returns the
same
Job handle create_index uses, running the refresh as an in-process task.
Invalid input is reported by the submitting call rather than by the job.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
|
||
|
Wyatt Alt
|
fc0d917d32 |
feat: refresh computed columns (#3938)
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
|
||
|
Wyatt Alt
|
def869bb78 |
feat: declare computed columns by SQL expression (#3937)
add_columns().computed("doubled", "x * 2") stores the expression in
field
metadata and commits the column empty; a later refresh fills it. Type
and
inputs are derived from the expression.
The declaration stays authoritative for its lifetime: writes that would
give
the column a value (append, update, merge, SQL insert), schema changes
that
would break the stored expression or reshape its output, metadata edits,
volatile expressions, declaration metadata arriving through any path but
the
validated declare call, and LSM write specs in either order against
latest
committed state are all refused. The LSM check also refuses on the
mem-wal
catch-up feature flag, which outlives unset and marks retained SSTable
rows.
Simultaneous declare/install interleavings conflict at commit via
lance's
mem-wal rule (lance#8539). Local tables only.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
|
||
|
Xuanwo
|
920fc0e455 |
fix(python): set native module metadata (#3913)
PyO3 defaults native extension classes to `builtins`, so mkdocstrings/Griffe could not resolve the newly documented `lancedb.Session` alias and `Deploy docs to Pages` failed on `main`. Declare the extension module for the public native types referenced by the Python API docs so Griffe resolves them through `lancedb._lancedb` and Pages can build again. Validated with the docs toolchain used by CI (`griffe==0.49.0`, `mkdocstrings==0.25.2`, and `mkdocs==1.6.1`); `PYTHONPATH=. mkdocs build` succeeds. |
||
|
Dan RammerandClaude Opus 5
|
706a9c327f |
feat: infer maintained indexes when an LsmWriteSpec omits them (#3748)
## What `LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`: | value | meaning | |---|---| | `None` (new default) | every index the MemWAL supports, resolved when the spec is installed | | `Some([])` | maintain nothing — a scan/filter-only WAL table | | `Some([..])` | exactly these, taken verbatim | `with_maintained_indexes` keeps its signature; `with_no_maintained_indexes()` is new. Surfaced through the remote path (null on the wire), Python, and Node. ## Why Callers had to state the maintained set by hand every time, which is both tedious and easy to get wrong — the common case is "maintain what I already built." Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating to lance's `is_maintainable_index_type`. This is load-bearing rather than cosmetic: lance does **not** skip an index type its memtable cannot build, it errors when the shard writer opens, so sweeping up a bitmap index would fail every memtable claim and leave the table unwritable. The inferred set excludes those, and an explicit list naming one is now rejected at spec time instead of at claim time. ## Behavior change A freshly constructed spec used to maintain **nothing**; it now maintains **everything supported**. This flipped because napi collapses `undefined` and `null` to `None`, so TypeScript cannot express "absent means nothing, null means all" — any other choice makes the bindings disagree with the wire. The error direction also favors it: an unwanted maintained index costs memory, while a silently unmaintained one degrades FTS to an unscored scan. Three existing tests encoded the old default and are updated rather than worked around. ## Caveat The resolved set is a snapshot, not a subscription. An index created after the spec is installed is not maintained until the spec is unset and set again. `get_lsm_write_spec` therefore always reports a concrete list — `None` never round-trips. ## Dependency Needs a lance release carrying `is_maintainable_index_type` (lance-format/lance#8095) before this builds against the pinned tag. Draft until then. ## Testing 38 Rust LSM tests and 10 Python tests pass against a local lance build, including new coverage that a bitmap index is excluded from inference and rejected when named, and that `[]` stays distinguishable from null on the wire. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
79ba076429 |
feat(table): checkpoint_lsm, flush_lsm, compact_lsm, get_lsm_stats (#3736)
Converge a table's LSM write path into its base table, and inspect it. `checkpoint_lsm` is `flush` then `compact`, repeated until the fresh tier is empty — and the loop runs **client-side**. Putting it on the server would mean a background task, which means a single-flight intent, an intent that leaks on panic, a bounded-iteration policy, an "is it done" observable, and a story for every way a client can vanish mid-operation. None of that exists in this shape: each request does a bounded unit of work and reports what is left, so completion is *carried in the responses* rather than inferred from a shared counter that cannot distinguish "converged" from "hasn't started yet". Best-effort by construction. Nothing is frozen, so `converged` means L0 was empty as of the last pass. It is idempotent, abandonable at any point with zero consequence, and safe to run on a cadence — an already-converged table costs one round trip and zero compaction passes, because `flush` reports `generations_remaining` and the loop is never entered. ## The failure taxonomy is the load-bearing part Five distinct conditions used to arrive at a client as one 503. `Error::LsmRoute` carries a classification read from the response body's namespace error code **at the point of receipt** — before any generic helper folds the body into a string and keeps only the status. | condition | wire | client action | |---|---|---| | contention (latch held / pool saturated) | 429, code 21 | retry with backoff | | owning node draining | 503, code 19 `InvalidTableState` | **stop** | | fenced / no slot / transport | 503, code 17 | retry with backoff | | registry entry vanished | 404 | re-issue from `flush` (capped) | | table being dropped / not WAL-backed | 409 / 400 | stop | Draining is terminal because the drain gate is a one-way latch — retrying spins until the deadline to report a failure that was knowable on the first response. Transport retry is disabled on these routes for the same reason: it treats every 503 alike and would burn its budget before the classifier ever saw the body. `get_lsm_stats` returns `Option<LsmStats>`, matching `get_lsm_write_spec` — `None` only when the table has no LSM write path, since a struct of zeros would read as measurements. Python bindings mirror all four, preserving per-bucket detail rather than flattening to a table-level summary. ## Testing Six new unit tests against the mocked endpoint, plus the taxonomy round-trip: - flush into an empty L0 issues **zero** compact calls (asserts the call count — `generations_consumed: 0` is also true of a loop that ran a pointless pass) - the loop drives compact until the server reports zero remaining - **contention is not draining**: a 429 retries and converges; asserts the retry count - a draining node stops after **exactly one** request, no retries - stats round-trips fully populated; `include_generation_rows` off by default - every `(status, code)` pair classifies correctly, including unparseable 503 bodies falling back to *retryable* rather than terminal `cargo test -p lancedb --features remote --lib`: 723 passed. ## Notes for review - Depends on the sibling lance change returning `SealedGeneration` from `force_seal_active` only at the *server* level — no lance API is used here. - The branch is based on `codex/update-lance-10-0-0-beta-5`, so it carries one extra commit (`chore: update lance dependency to v10.0.0-beta.5`) that is not part of this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: lancedb automation <robot@lancedb.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
lancedb-gatefixer[bot]
|
7357d63e87 |
fix(python): guard concurrent table deletes (#3787)
<!-- lance-gatekeeper-fix:v1 agent=5c80c44c083b3b8ad0da595419d468fc generation=1 --> ## Root cause The legacy synchronous Python table called `delete` on a shared, mutable `lance.Dataset`. Concurrent table operations could hold a PyO3 borrow while delete requested an exclusive borrow, producing `RuntimeError: Already borrowed`. The current async-backed binding fixes this by cloning its thread-safe Rust table handle before awaiting, but that concurrency contract had no regression coverage. ## Fix - Document why delete must clone the Rust table handle before entering its async future. - Add a barrier-synchronized regression test that deletes distinct rows through one shared table from eight Python threads. - Verify every delete commits exactly one row, every commit gets a distinct version, and no rows remain. ## Validation - `cargo check --quiet --features remote --tests --examples` - `cargo fmt --all -- --check` - `uv run --extra tests --extra dev ruff format --check python/tests/test_table.py` - `uv run --extra tests --extra dev ruff check python/tests/test_table.py` - `uv run --extra tests --extra dev pytest python/tests/test_table.py::test_concurrent_deletes_are_thread_safe python/tests/test_table.py::test_delete python/tests/test_table.py::test_delete_expr python/tests/test_table.py::test_delete_expr_async -q` (4 passed) - Manual stress reproduction: 100 concurrent deletes on one table completed at versions 2–101 with zero rows remaining. Fixes #530 Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com> |
||
|
Wyatt Alt
|
8e24dd3828 |
feat(rust)!: make add_columns a builder (#3778)
Table::add_columns now takes no arguments and returns AddColumnsBuilder, so calls become .add_columns().transform(t).execute(). read_columns was the second positional argument but reaches only one of the five transform variants. In lance's add_columns_to_fragments only BatchUDF receives the caller's value: SqlExpressions replaces it with the columns its expressions reference, Stream and Reader pass None, and AllNulls reads nothing. So it was mandatory on every call -- all eighteen call sites here passed None -- and silently discarded four times out of five. As a builder method it is optional, and setting it where lance would discard it is now an error, which does reject a call that previously succeeded while ignoring the argument. Matches the builders add, update, and merge_insert already use. |
||
|
3dd9c598e9 |
feat(remote): add seekable blob range reads (#3750)
## Summary - Implements Cloud `fetch_blob_files`: returns real seekable `BlobFile` handles over HTTP Range instead of `NotSupported`. - Completes the second Cloud blob read verb after #3684 (`fetch_blobs` = eager whole bytes; this = lazy / partial / sequential reads). - Same public handle API as local (`read_range`, `read_up_to`, `seek`, `tell`, `close`), so one code path works for local and Cloud. Large blobs (video, audio, PDFs) should not require downloading the whole object to inspect a header or stream a slice. After search, callers open a handle and read only what they need: ```python hits = table.search(vec).select(["id", "video"]).limit(5).to_arrow() with table.fetch_blob_files("video", hits)[0] as f: header = f.read_range(0, 256) f.seek(keyframe_offset) chunk = f.read_up_to(1 << 20) ``` ### Behavior - Handle creation probes size with `bytes=0-0` (bounded concurrency, input order preserved). - `204` → null (`None`); `416` with `bytes */0` → valid empty blob; other `416` → error. - `read_range` validates `Content-Range` and body length; OOB ranges fail with `invalid_input` before the request (aligned with Lance). - `read_up_to` reuses one open-ended Range response across sequential reads; `seek` drops it. - Servers older than 0.5.0 get a clear `NotSupported` (does not suggest `fetch_blobs`, which they also lack). ## Testing - `cargo test --features remote -p lancedb remote_blob` - `cargo test --features remote -p lancedb test_blob` - `cargo clippy --features remote --tests --examples` (no new warnings from this change) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Wyatt AltandClaude Fable 5
|
a6418b6cb9 |
feat: create_index returns a Job handle (#3742)
IndexBuilder::execute now returns a Job with wait and cancel methods. Local tables build the index synchronously and return an already-done job. Remote tables read the job id the server returns from create_index and track it through the /v1/jobs API: wait polls describe until the job reaches a terminal state and cancel posts a cancellation. Servers that return no job id yield a done job, so behavior against older servers is unchanged. The job id is not exposed on the handle. The Python and TypeScript bindings keep their current signatures and discard the handle; exposing Job there is left to follow-ups. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
  Yang CenandYang Cen
|
f7feed48c3 |
feat(fts): support custom stop-word lists (#3734)
## What Expose custom FTS stop-word lists in the Python and TypeScript public APIs, including their standalone tokenize helpers and remote index creation. This PR supports concrete string lists only. It does not add file or LanceDB-table stop-word sources. ## Why Rust already exposes Lance's custom stop-word list option. The Python and TypeScript APIs did not pass it through, and local index details did not retain the full tokenizer parameters needed by index-backed tokenization after reopening a table. ## How - Add `custom_stop_words` / `customStopWords` to the Python and TypeScript FTS and tokenize options. - Preserve `None` / `undefined`, empty lists, and list contents without normalization. - Load the persisted FTS segment parameters when returning local index details. - Serialize the concrete list in remote create-index requests. - Keep Python and TypeScript tests thin; behavior, persistence, query tokenization, and remote JSON coverage live primarily in Rust. ## Validation - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo test --quiet --features remote --tests` - Python extension rebuild with `uv` and `maturin` - Targeted Python tests: 4 passed - Python `ruff format --check` and `ruff check` - TypeScript build, typecheck, Biome lint, generated docs, and targeted tests --------- Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local> |
||
|
Xuanwo
|
ff6ff09998 |
feat: support batched blob range reads (#3703)
## Summary
Lance can now plan multiple byte ranges for the same blob in one
`read_blob_ranges` operation, but LanceDB users currently cannot expose
a complete set of logical ranges to that planner.
This complements `BlobFile`: file-like consumers such as PyAV can
continue to discover ranges dynamically, while callers that already know
the ranges for a batch can submit them together.
## Motivating example
A training table may store a large video blob together with a small
application-level clip index:
```text
video: blob
clips: [{offset, length}, ...]
```
The caller can select the videos and clips for a batch, obtain their row
IDs from the query, and read all of the selected windows together:
```python
rows = (
table.search()
.select(["clips"])
.with_row_id(True)
.limit(64)
.to_arrow()
.to_pylist()
)
requests = []
for row in rows:
clip = sample_clip(row["clips"])
requests.append(
(row["_rowid"], clip["offset"], clip["length"])
)
chunks = table.fetch_blob_ranges("video", requests)
```
Here, `_rowid` comes from the LanceDB query, while `offset` and `length`
come from the application's clip index and are relative to that row's
video blob. The caller describes only the logical reads; Lance still
handles validation, source grouping, coalescing, scheduling, and byte
backpressure.
Lance v10.0.0-beta.5 returns one logical result per blob selector or
range request and explicitly distinguishes null blobs from valid empty
values. LanceDB consumes that aligned result contract directly and only
adds a cardinality check for unresolved row IDs.
This PR exposes batched blob-range reads on local Rust and Python
tables. Results preserve request identity, duplicates, null slots, and
valid empty ranges while allowing Lance to execute the physical reads
out of order. Scheduler buffer sizing remains an internal Lance concern,
so the LanceDB API does not expose `io_buffer_size`.
Cloud tables continue to report this operation as unsupported until
there is a corresponding remote API.
|
||
 Heng Ge
|
f655f62e09 |
feat(query): add use_lsm to read MemWAL LSM data (#3489)
## What
MemWAL LSM **read** support. When a table has an LSM write spec
(`set_lsm_write_spec`), `merge_insert` upserts live in the MemWAL
active/frozen memtables and flushed SSTables until an external
compaction merges them into the base table, so a normal scan returns
**stale** data. This routes reads through Lance's `LsmScanner` so
queries also surface that in-flight data, deduplicated by primary key
(newest generation wins).
## How
- Adds a **`use_lsm: Option<bool>`** query flag, symmetric with the
`merge_insert` flag:
- **unset** — auto-route through the LSM scanner when the table carries
a write spec
- **`use_lsm(true)`** — force the LSM path; error if there is no spec
- **`use_lsm(false)`** — read the base table only (the escape hatch)
- Plain scan, single-column full-text search, and single-vector ANN all
run through one `LsmScanner` (assembled from on-disk shard manifests
plus the cached writer's in-memory memtables), so a `where` predicate is
honored as a **prefilter** uniformly — including for vector search.
- **Compaction-aware snapshots:** an SSTable generation is dropped only
once it is both compacted into the base table and covered by the arm's
base-index catch-up (`index_catchup`); plain scans use the compaction
watermark alone.
- Query shapes the scanner cannot honor hard-error with guidance to set
`use_lsm(false)`: hybrid, multi/binary vectors, `with_row_id`,
reranking, `order_by`, dynamic/Substrait projection or filters,
`distance_range`, `use_index(false)`, postfilter, take-by-row-id/offset,
reads from a time-traveled version, and an unmaintained or ambiguous
(multiple) FTS/vector index. Namespace-pushdown queries fall back to
local execution when a spec is present; WAL-only writers are handled.
- Exposed across the Rust core and the Python (`use_lsm`) and TypeScript
(`useLsm`) bindings, including `TakeQuery`.
Rebased from Lance `7.2.0-beta.3` to `10.0.0-beta.3`.
|
||
 Drew GallardoandCursor
|
65cd142c7e |
feat: add remote branch diff and merge client APIs (#3686)
This PR adds some support for `diff` / `merge` in the remote client as for local tables we stay `NotSupported` until https://github.com/lance-format/lance/issues/7263. This wires the two review-and-land calls against the remote REST API: - `POST /v1/table/{id}/branches/diff` - `POST /v1/table/{id}/branches/merge` Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python returns the wire JSON, same shape as the REST response. Merge here means promoting a branch's added columns onto `main`. ### Behavior - Remote only. Local raises `NotSupported`. - A rejected merge is not an exception. HTTP 409 still returns `Ok` / a dict with `status="rejected"` and blockers in `diff.mergeBlockers`. - Unknown blocker / status codes parse as `Unknown` so a newer server does not break older clients. - `MergePreview` tolerates missing fields for the same reason. - Merge requests are not retried. 409 is final and carries the body you need. ### Example ```python table = db.open_table("images") table.branches.create("exp") exp = table.branches.checkout("exp") exp.add_columns({"tag": "cast('draft' as string)"}) diff = table.branches.diff("exp") preview = table.branches.merge("exp", dry_run=True) result = table.branches.merge("exp", dry_run=False) if result["status"] == "merged": print("landed at", result["mainVersionAfter"]) elif result["status"] == "rejected": print(result["diff"]["mergeBlockers"]) ``` ### Testing cargo test -p lancedb --features remote diff_branch cargo test -p lancedb --features remote merge_branch --------- Co-authored-by: Cursor <cursoragent@cursor.com> |