## Problem
Function-backed columns need an exact `fresh`, `stale`,
`never_computed`, and
`failed` partition for the table version a caller is reading. The
existing
error listing is capped, and composing it with job and schema listings
cannot
prove which rows currently failed.
## Behavior
- Adds a strict freshness reader beside the existing permissive planner.
It
distinguishes an absent certificate from legacy evidence that omits
failures, and from a current certificate carrying failure coverage.
- Extends the immutable freshness sidecar with per-fragment known and
failed
row bitmaps. Partial or `WHERE` publications cover only observed rows;
untouched rows stay unknown. Deleted rows do not block classification.
- Adds failure-aware whole-fragment and row-repair writers. A rejected
older
attempt cannot overwrite failure evidence already recorded for a newer
input
signature.
- Adds `column_status`, `column_statuses`, and `wait_fresh` to Python
and the
corresponding Node methods. The client calls the server's exact status
endpoint, validates the four-way partition and one-version guarantee,
and
preserves the complete Function version, last-job, and automatic-refresh
metadata.
- Adds an MDX-commented draft usage section to the Enterprise
computed-column
refresh guide, covering Python, Rust, and TypeScript and remaining
hidden
until the server integration ships.
- Sends `column`, `branch`, and exact `version` selectors to the status
route.
A pinned handle rejects a nonempty response from any other table
version;
latest handles keep the existing read-freshness headers.
- Retries `wait_fresh` only for HTTP 503 with
`function_column_status_finalization_pending`. Existing failed rows
remain
visible and do not keep the wait pending; other errors propagate.
- Models the shared scheduler projection, including `up_to_date`, real
`blocked` states, their reason, failure streak, and effective cooldown.
## Validation
This is the SDK side of ENT-3563. Sophon's failure-coverage publication
and
status endpoint are a separate integration; deployed end-to-end and
whole-ticket
acceptance remain pending.
- Rust: 9 exact freshness/status tests passed, plus 1 exact
latest-handle
freshness-header test. The pinned transport test was extended with a
wrong-version response and rerun at `ebd6649a` (1 passed). Coverage
includes untouched partial rows,
selected-row failure clearing, cross-generation evidence, typed and
non-retried errors, `up_to_date`, blocked scheduler fields, and
exact-version
requests.
- Python: 2 native PyO3 remote-transport tests passed together,
including
typed retry; one local facade test reaches the existing unsupported
boundary.
- Node: the NAPI addon built successfully, TypeScript and Biome passed,
and the
selected native remote-transport test passed 1/0, including
exact-version
selection and blocked scheduler metadata.
- `cargo fmt`, `git diff --check`, Ruff format/lint, Python compile,
Biome, and
TypeScript checks passed.
Validation is mapped by the relevant unchanged source rather than
treated as one
final-head run: the nine Rust tests ran at `e4d51fc8`, the latest-header
and Python
remote tests at `31e68b7d`, the local Python test and Node build at
`6f213584`, and
the final TypeScript/Jest delta at `4f2d5c27`. The final product tree
excludes those five private EC2 runner files and adds
six generated TypeDoc files plus one hidden draft usage section in the
computed-column refresh guide. The Node delta corrected a private
TypeScript
command and reused the unchanged native addon.
## Summary
Adds an optional `max_concurrency` setting to Function registration and
explicit refresh:
- A registered Function version can define the default maximum number of
user-code calls in flight for one logical execution.
- A refresh can override that default for that execution only.
- The value must be a positive `u32` and is independent from
`resource_budget`; it is not an account-wide quota.
- Omitting the option preserves the previous request payload. Local
embedded refreshes reject this Enterprise-only option.
Rust, Python sync/async, PyO3, and the existing Node refresh surface
forward the option. Node has no Function registration surface today, so
no registration API is added there. The generated TypeDoc pages are
included for the new public option.
## Validation
The native-tested source covers Function version decoding, registration
validation and serialization, the Flight SQL registration ticket and
returned version, remote refresh forwarding, local rejection, and Node's
numeric boundary before conversion. Exact stage results are Rust `1+3`
plus four remote cases at `1` each; Python `14/1/49/24`; Node `57`
remote and `1` local rejection. Ruff, standard repository Pyright, and
Biome passed.
The tested Python extension was
`21aa256a970f4a16ea8253d68401d0fc573d946412e8752155dfcad63f30654d`; the
tested Node addon and generated native declaration were
`f6089f861981f96f8ce0f6dbef43de0907b9eeb4b63e84cdd5da14e00d6cbd30` and
`4c4ceb205824075d706897c726c215dcf839880c757f0d0582d24cec5f938416`. The
final 24-file diff preserves the 22 tested
product/test/fixture/source-doc blobs and adds only two generated
TypeDoc pages.
## Depends on merged #4951
Phase 1 merged as `4d22c0ce2b6825c880c818e737b079db494b4053`, so this PR
now targets `main`. It reuses the canonical
`FunctionWorkerResourceBudget` introduced there rather than copying the
type.
For Function `add_columns` declarations, Python and Rust now persist
`auto_refresh_resource_budget` in computed-column metadata and send it
as `auto_refresh_resource_budget` in the remote request. CPU
milliseconds and memory bytes are required positive integers. Omitting
`gpu_count` is accepted as zero and the canonical outgoing JSON still
contains `"gpu_count": 0`. Unknown fields, missing CPU or memory, wrong
types, zero CPU or memory, negative values, and overflow are rejected.
The same budget is applied to every public output binding of a
multi-output Function declaration. Existing metadata is retained, and
SQL-expression declarations or local execution reject the budget instead
of silently ignoring it. Node is unchanged because its current
`addColumns` surface only accepts SQL computed expressions and has no
Function declaration path.
This change does not implement fleet call concurrency limits or enforce
the budget in the runtime. The Enterprise server must support the nested
Function worker resource-budget contract before these values can act as
execution limits.
## Validation
- Python extension build passed.
- Remote Function declaration and local unsupported-path rejection each
passed.
- Canonical metadata round-trip passed, including omitted GPU decoding
to zero and explicit `gpu_count: 0` on encode.
- Function multi-output remote wire test passed with `--features
remote`.
- Repository-scoped Python type check passed with 0 errors.
- All runtime source blobs match the native-tested source. The
post-native deltas are one documentation example line and an existing
routing-test stub update for the new optional argument; the affected
parameterized test passed 3/3 against the existing extension build. No
private test runners are part of this PR.
## Summary
- Add the canonical Rust `FunctionWorkerResourceBudget` and
`RefreshColumnOptions::resource_budget()` for explicit Enterprise
Function refreshes.
- Forward the nested budget through Python sync/async, PyO3, Rust remote
requests, Node NAPI, and TypeScript.
- Require positive CPU and memory values; normalize an omitted nested
GPU count to numeric zero; reject the nested budget when combined with
the top-level `gpu_count` shorthand.
- Reject `resource_budget` on LanceDB Embedded refresh paths instead of
silently ignoring it.
- Document the Rust, Python, and TypeScript APIs, including that SQL
servers currently require all three budget fields.
The Rust, Python, and TypeScript additions use `cpu_millis` /
`cpuMillis`, `memory_bytes` / `memoryBytes`, and `gpu_count` /
`gpuCount`. The top-level GPU shorthand keeps its existing behavior and
remains mutually exclusive with the complete nested budget.
## Scope and compatibility
This change configures only the explicit refresh call. It does not
configure `add_columns` automatic refresh and does not introduce a
fleet-wide call-cap interface.
A LanceDB Enterprise server must expose the nested `resource_budget`
backfill contract. LanceDB Embedded continues to support its existing
expression refresh defaults and rejects the new complete budget
explicitly.
## Validation
Validated against the final signed product source:
- Rust: resource-budget remote serialization test `2 passed`; validation
test `1 passed`.
- Python: actual PyO3 build; remote-request matrix `22 passed`;
invalid-option matrix `20 passed`; local rejection `22 passed`; Ruff
check and format clean.
- Repository-standard Pyright entry point: `0 errors`, within the
repository's configured include set.
- Node: native build and Biome lint passed; the existing remote-request
test `1 passed`, including fractional, negative, overflow, NaN, and
infinity nested GPU cases; the existing local-rejection test `1 passed`.
- TypeDoc generation and generated-document cleanliness passed.
- The native-tested product source differs from this head only in two
semantically equivalent Node range checks required by Clippy. Current CI
compiles and lints the final blobs. No dependency or lockfile changes
are included.
Python refresh APIs currently reject `publication_interval_rows`, so
callers cannot request incremental Function output publication through
the SDK. Add the optional typed parameter through the sync/async Python
APIs, PyO3, and Rust `RefreshColumnOptions`, and serialize it as a
top-level Enterprise refresh option.
An omitted value preserves the existing wire request and whole-fragment
default. Positive values set a publication threshold; zero and negative
Python values fail before a request is sent. Embedded refreshes reject
the Enterprise-only option. Documentation describes a threshold rather
than an exact number of rows per publication.
Validation on the designated development host:
- A matching baseline build reproduced the missing public keyword
argument in both sync and async submission tests.
- The updated PyO3 build passed 54 focused Python tests; Rust payload,
invalid-option, and default-wire tests each passed. Rust formatting and
Python Ruff checks passed.
- The single signed delivery commit differs from that tested
implementation only in a Rustdoc correction describing Embedded
rejection; formatting passed again. No dependency pins or lockfiles
changed.
The matching SDK is prepared for the separate server validation of
failure/cancellation retention and missing-only refill. Those end-to-end
results are still pending and are not implied by these SDK tests.
Refs [ENT-3733](https://linear.app/lancedb/issue/ENT-3733).
The server's backfill request takes a row filter, failure limits,
per-call timeouts and worker sizing, and SQL exposes all of them through
`REFRESH COLUMN ... WHERE ... WITH (...)`, but the typed SDKs could send
only batch_size, GPU placement and the retry and error policies. Anyone
needing a filtered refresh or a bounded skip policy had to drop to SQL.
This adds `where`, `max_failed_fraction`, `max_failed_rows`,
`row_timeout_seconds`, `init_timeout_seconds`, `num_workers` and
`max_executor_processes_per_worker` to Python, Rust and Node, passed
through unchanged. `RefreshColumnOptions` now serializes straight into
the request body, so the three SDKs share one producer. Node gets a
`RefreshColumnOptions` object, which also carries the options it lacked
before. A LanceDB Embedded refresh rejects every option other than
batch_size and GPU placement, naming the option.
Expose `FunctionRetryPolicy` on Function declarations and refresh
overrides, together with `error_policy` and recorded attempt counts.
Python uses the Rust validation rules before sending a request; local
refreshes reject these Enterprise options.
The policy bounds calls and backoff, supports
class/message/provider-code matching, and can disable retries with
`max_attempts=1`. Batch retry and skip operate on the whole failed call.
Requires Enterprise server support for the new policy.
Validation: rebuilt the Python extension; 47 targeted Python tests and
10 Rust tests passed. Python `make check`, Rust formatting, and `cargo
clippy --features remote --tests --examples -- -D warnings` passed. The
additional repository-root Ruff sweep reports existing errors under
`docs/web-tests`; the Python workspace checks pass.
Draft user guidance covers the supporting server requirement, safe
repetition of partial work and whole-batch exhaustion. The guide
sections stay hidden until release. Vale 3.24.0 passed for the added
draft prose; no paired deployed SDK/server retry acceptance was run.
Companion server: https://github.com/lancedb/sophon/pull/9812.
BREAKING CHANGE: Rust callers constructing `FunctionRegistrationRequest`
or `FunctionErrorRecord` with struct literals must supply the new
`retry_policy` or `attempts` field, respectively.
Expand the Linux Python test run without pandas and pylance from two files to
the suite while retaining the regular dependency run for optional-package
modules.
Move optional imports into tests that need them and use Arrow assertions where
pandas is not required, preserving the existing behavioral assertions.
Validation:
- Physical no-pandas/no-pylance broad run: 2884 passed, 316 skipped, 8 existing
marker cases deselected.
- Regular affected suite: 1687 passed, 39 skipped.
- Current-head hosted integration: all 15 checks and 4 workflows passed.
Co-authored-by: importcpp <57519936+importcpp@users.noreply.github.com>
Add a Python regression test for merging metadata-free data into a table whose target schema has metadata. The test calls the native `execute_merge_insert` binding directly so it exercises the native partial-schema comparison without the public merge path normalizing source metadata.
The test verifies that merge insert succeeds with the expected rows and preserves the target schema metadata.
Fixes#2058
Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader.
Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them.
SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits.
Fixes#1959.
Fixes#4761
`Table.update` does not compute embeddings, so changing an embedding source
column could leave the previous embedding in place. Source-column updates now
raise `ValueError` unless every associated vector column is supplied in the same
call.
The guard reads Python embedding metadata and native `lancedb::column_definitions`
without loading embedding models. It uses Lance's field-path parser for quoted
targets and covers synchronous and asynchronous local and remote tables. Supplying
every associated vector column, vector-only updates, and unrelated updates remain
supported.
This is a breaking change for source-only updates. The API documentation and
Update rows guide explain the requirement and the `merge_insert` alternative for
regenerating embeddings.
Validation at the reviewed head covered 44 targeted public update cases and the
full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable
exact-head checks and workflows completed successfully.
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.
- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.
Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.
Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
Python column refresh methods rejected `batch_size`, and the Rust remote
backend never included it in the backfill request. Callers could not
reduce the rows passed to a slow Function to stay within the executor's
per-call deadline.
Add optional, keyword-only `batch_size` arguments throughout the
synchronous and asynchronous Python APIs and bindings. Add compatible
Rust `RefreshColumnOptions` APIs while preserving existing Rust method
signatures. Reject non-positive sizes before submitting a job, send
positive sizes in `backfill_column`, and omit the field when unset while
retaining branch scoping.
Batch size controls remote Function calls. Embedded expression refreshes
accept and validate the option without changing execution, and blocking
remote refreshes retain their existing unsupported behavior; remote
callers use `refresh_column_async` and wait on the returned job.
Validation:
- Reproduced the original Python `TypeError` from the committed method.
- Five focused Rust tests passed, covering request contents, omission,
branch scoping, and rejection before any request.
- `cargo check --quiet --features remote --tests --examples` passed.
- `cargo clippy --quiet --features remote --tests --examples -- -D
warnings` passed.
- Nineteen targeted Python tests passed against the rebuilt extension,
including synchronous and asynchronous remote requests, invalid sizes,
and default-build behavior. Eight embedded refresh cases require the
experimental-functions build and were skipped.
- Rust formatting and Python formatting/lint checks passed, including
CI's pinned Ruff version. Repository-wide Ruff reports 48 existing
findings in untouched documentation files; none are in changed files.
Fixes#4776
<!-- lance-gatekeeper-fix:v1 agent=4906ff3f5f7797a1d49a13803302021c
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Expose table-level maintenance override read, update, and reset APIs for remote
Rust and Python tables. Local tables continue to return `NotSupported`.
Updates recursively merge the caller-provided JSON object into the current
override document so unrelated and unknown service fields are preserved. Reset
uses the explicit reset endpoint.
The integrated change keeps current main's remote database identity, namespace,
branch, SQL, WAL, and threaded mock-server behavior.
Validated with focused Rust and Python override tests and the complete applicable
CI run for 43bccbadc5.
Updates Rust and Java Lance dependencies to
[v15.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.6),
pins Python tests to published pylance 15.0.0b6, and refreshes both
lockfiles.
Aligns Arrow 59, DataFusion 55, and PyO3 0.29; adapts execution plans
and MemWAL directory handling, updates the ABI metadata test, and
removes the obsolete Lance Arrow patch now fixed upstream.
Validation: workspace clippy with all features/tests and warnings
denied, cargo fmt, Python/CI Ruff checks, uv lock check, 16 CI-script
tests, and targeted Python regressions (54 passed, 28 skipped);
repository-wide Ruff retains existing documentation-example failures,
and the upstream macOS wheel build failed while Linux and Windows wheels
are published.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Adopts the product terminology guide across the docs site pages, SDK
docstrings and error messages. "LanceDB OSS" and local mode become
**LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB
Distributed**, which LanceDB Enterprise provides. Where free vs paid is
the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB
Enterprise". This is a first trial of the Embedded name, so reviewers
can judge it in context.
- **Embedded vs Distributed** for where execution runs (`optimize()`,
consistency, branching, maintenance). Main pages define it once:
"LanceDB Enterprise runs LanceDB Distributed: …"
- **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://`
region, the brute-force KNN limit) and marked legacy. "Cloud and
Enterprise" becomes Enterprise or Distributed.
- **Plan executor** becomes "executor" or the fleet name.
- **Error messages** change, e.g. `optimize is not supported for remote
tables.` → `... in LanceDB Distributed.` Tests that match on them are
updated.
- **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`,
`DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the
terms in prose.
Unchanged: identifiers (`availability.oss`, the `remote` module and
feature, `RemoteTable`, `db://`), heading anchors, internal code
comments, and Geneva.
## Evidence
- Rust lib tests for the changed messages and the Python tests in
`test_remote_db.py`, `test_materialized_views.py` and `test_table.py`
pass locally.
- Not run locally:
- The Node tests: `remote.test.ts` and `table.test.ts` have updated
assertions.
- Vale.
- The `experimental_functions`-marked materialized-view tests.
- `docs/src` API reference was hand-edited to match the docstrings
rather than regenerated.
## Merge Risk
**Door:** two-way; prose and message text only.
**Blast Radius:** `docs`, `error-messages`: callers matching on old
error strings would break.
Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon
(Enterprise pages).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Published SDK builds currently expose local SQL computed columns and
materialized-view execution that are not ready for release. Add the
opt-in `experimental-functions` Cargo feature to the Rust core and
forward it through the Python and Node bindings; published packages
leave it disabled.
Local declarations and refreshes now return `NotSupported` before
mutating tables or spawning jobs, including prepared materialized-view
creation. Remote Cloud/Enterprise client operations, Python UDF
packaging, ordinary eager `add_columns`, and access to existing data
remain available. Stored metadata and write protections stay active, and
shared server commit helpers remain usable.
CI covers both feature configurations. Existing local execution tests
run with the feature enabled; default-build regressions cover early
rejection, unchanged table versions, existing metadata protections, and
normal eager transforms. API and source-build documentation explain the
opt-in.
Validation: Rust library tests pass with the feature disabled (1,869
tests) and enabled (1,947 tests), plus three default-build integration
regressions. Python and TypeScript bindings were rebuilt and tested in
both configurations, including mocked remote requests; Rust formatting,
Python package lint, TypeScript lint/build, and generated docs were
checked. Repository-wide Ruff still reports 48 pre-existing errors in
unchanged `docs/web-tests` files.
## Summary
- expose typed compaction options on synchronous and asynchronous table
optimization
- translate validated Python options into native Lance compaction
settings
- document the row-based steady-state limit and first-pass/source-budget
caveats
- carry the Lance beta.15 fragment-exclusion option through the typed
Python API
## Root cause
The native Python optimize binding always constructed default compaction
settings, and the public AsyncTable API offered no way to lower the
fragment row target. Wide-row fragments therefore remained compaction
candidates on every later optimize call, repeatedly rewriting the
accumulated table.
## Limitations
This is an explicit row-based mitigation, not byte-native task sizing or
a hard memory cap. Callers must derive target_rows_per_fragment from
expected row size. Existing oversized fragments are rewritten whole on
the first pass; max_bytes_per_file limits output file size rather than
compaction memory; and source budgets admit whole planned tasks, so a
budget below the first task performs no compaction. Splitting tasks by a
byte budget or returning a planner error requires a Lance-side change.
## Validation
- cargo check --release -p lancedb-python
- cargo clippy --quiet -p lancedb-python
- cargo fmt --all --check
- ruff format --check python tests
- ruff check python tests
- uv run --no-sync pytest python/tests/test_table.py -k
optimize_compaction -q (16 passed)
Fixes#2325
<!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Keep FP16 vectors through IVF-PQ index creation and search in the
synchronous and asynchronous quickstarts. Add native and conditional
CUDA regression coverage using a float64 query against an explicitly
typed FP16 column. This is test and documentation-example coverage for
dtype preservation already present in Lance; it does not change
production code.
Related #2120.
Validation on `bbad20940632b21a9ee85bdcb46b4cc19dbd65e3` with Python
3.12.12, Lance 15.0.0-beta.1, PyArrow 24.0.0, and Torch 2.14.1:
- The native FP16 parameter, Arrow FP16 ingestion test, and both
quickstarts passed: 4 passed, 1 CUDA-only skip.
- The two affected test files passed: 386 passed, 1 CUDA-only skip (`-m
"not slow and not s3_test"`).
- Ruff 0.9.9 lint and formatting checks passed for the Python package.
Real CUDA execution has not been performed. Torch is installed, but
`torch.cuda.is_available()` is false; `test_create_f16_table[cuda]`
skips with `CUDA not available`. The native CPU result does not
establish that the original GPU failure is resolved. Issue #2120 remains
open for verification on an actual CUDA device.
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
`IvfRq()` forced `num_bits=1` through Python's Rust binding, overriding
Lance's five-bit default. The shared native index creation path also
substituted one bit when the Rust or Node builder left the value unset.
Make Python's bit count optional and forward only explicit values. Use
`RQBuildParams::default()` for an unset native bit count, and omit unset
values when dispatching to pylance for accelerator builds. Update the
Python, Rust, Node, and vector-index documentation.
New indexes with an unset bit count now inherit Lance's default,
currently five bits, across Python, Rust, and Node. Five-bit vector
codes use roughly five times the storage of one-bit codes, plus other
index overhead. This also changes reader compatibility: indexes created
with `num_bits >= 2` use a newer on-disk layout that some older LanceDB
versions cannot read. Upgrade readers to versions that support multi-bit
IVF_RQ, or explicitly set `num_bits=1` (for example, `IvfRq(num_bits=1)`
in Python) when sharing indexes with readers limited to one-bit IVF_RQ.
Validation:
- Reproduced six regression failures before the fix: one-bit default
metadata, rejected `None`, and incorrect remote/accelerator forwarding.
- After merging `main` through `b762ec22`, rebuilt both native bindings
against Lance 15.0.0-beta.1. Python validation passed with 106 tests and
15 optional Ollama skips. Coverage includes the full index suite,
omitted and explicit bit counts, reading Lance-created IVF_RQ indexes,
accelerator forwarding, CUDA-unavailability handling, timeout
validation, remote requests, and table/namespace listing behavior.
- Rust formatting and Python package Ruff lint/format checks pass. Node
`pnpm lint`, `pnpm build` (including TypeScript checks), `pnpm run
docs`, the full connection suite, and the existing IVF_RQ regression
pass. Generated documentation adds no further changes.
The merge preserves upstream's explicit RaBitQ heading anchor alongside
the `IvfRq()` example and five-bit default/reader-compatibility
guidance.
Repository-wide `ruff check .` still reports 49 existing errors in
unchanged documentation examples under `docs/web-tests`; root formatting
also finds 11 unchanged documentation example files. The Python package
checks pass.
Fixes#4582
<!-- lance-gatekeeper-fix:v1 agent=6ef9406f0142b1ab66d53d9d56098223
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Pandas converts NaN vector components to Arrow nulls. Both Python's NaN
mask and Rust's default append validator ignored null children, so
`add()` silently stored incomplete vectors despite
`on_bad_vectors="error"`.
Reject null components in the existing Rust ingestion validator, and
include null components in Python's bad-vector detection and filling.
`drop` removes affected rows, `fill` repairs missing components while
preserving valid ones, and `null` replaces affected vectors. Explicit
null components in Arrow and list inputs follow the same policies. Rust
allows whole null vectors and retains the explicit
`NaNVectorBehavior::Keep` opt-out.
The ingestion guide and synchronous/asynchronous `create_table()` option
docs describe null components, all four policies, and filling that
preserves valid components.
Validation:
- Reproduced the default pandas append failure with the regression test
before changing the Rust validator.
- Rebuilt the native Python extension from the merged sources. The
Python table and utility suites passed: 481 passed, 1 skipped. Coverage
includes create/append/overwrite, pandas/rows/Arrow inputs, every
bad-vector policy, float16/32/64, sliced and chunked arrays, all-null
components, and hidden children of null lists.
- All 43 focused Rust tests passed: 7 validator tests and 36 ingestion
tests.
- Rust Clippy with `remote`, tests, and examples passed. Ruff lint
passes for the Python package, formatting checks pass for modified
Python files, and `cargo fmt --all` is clean.
- Documentation follow-up: all 56 focused vector-policy tests passed;
Python package lint and creation-docstring formatting checks passed.
Repository-wide Ruff reports 49 existing errors in untouched
documentation examples. Create/overwrite retain their existing
`RuntimeError` wrapping for errors raised while consuming lazy Arrow
readers.
Fixes#4435
<!-- lance-gatekeeper-fix:v1 agent=b5108e27d28ef613a78c1e4354c03c48
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Arrow map scalars returned by a table query cannot currently be passed
back to `update`: the generic Arrow scalar conversion turns them into a
list of tuples, which the SQL literal encoder cannot handle. Empty map
scalars become an untyped SQL list and also fail to update a map column.
Add a dedicated `MapScalar` conversion that renders its keys and values
as a SQL map while preserving the existing null conversion. The
regression tests exercise real synchronous and asynchronous table
updates, quoted keys, nullable values, empty typed maps, and nested
maps.
Validation: the complete documented unit-test scope passed locally
(1,515 passed, 37 skipped; unchanged baseline: 1,507 passed, 37
skipped). Full doctests passed (71 passed, 6 skipped), and
repository-wide Python formatting/lint, Rust formatting/clippy, and
Python 3.13 type checking passed. The locally built native wheel also
passed all eight regression cases in a separate Python 3.13 environment.
Full Linux/S3 integration and all platform matrices remain for CI.
The SQL map constructor's existing backend constraints still apply:
duplicate keys are rejected, and a nested map containing both nonempty
and empty inner maps can fail SQL type unification. This change does not
claim arbitrary Arrow map roundtrip support.
AI assistance: implementation and local verification used Codex with
gpt-6.1-sol agents; the diff and evidence were reviewed before
submission.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
`optimize()` fails once a tagged version is older than the cleanup
cutoff, although the docs say tagged versions
are exempt from cleanup, and the failed prune also skips the
index-optimize step (repro and numbers in the issue).
Change: `OptimizeAction::Prune` already exposes
`error_if_tagged_old_versions: Option<bool>`, so no new API.
`cleanup_old_versions` and `cleanup_old_versions_before` in
`rust/lancedb/src/table/optimize.rs` now use
`unwrap_or(false)`, which covers `OptimizeAction::Prune`,
`OptimizeAction::All` and `optimize_prune_before`. The
Python and Node bindings pass `None`, which now means "retain tagged
versions"; I left `python/src/table.rs` and
`nodejs/src/table.rs` untouched (this also avoids a conflict with #3870,
which edits that region). `Some(true)`
still opts in to the error. Doc comments updated; the sentence in
`docs/web/management/table.mdx` is accurate once
this lands.
Tests: Rust `test_optimize_prune_retains_tagged_versions` (`Some(true)`
still errors; the default retains the tag;
`optimize_prune_before`; `checkout_tag` plus a data read), a Python sync
test with a scalar index (index step runs,
a second optimize works, untagged versions are pruned,
`checkout("release-1")` works), a Python async test and a
Node test.
**What I could and could not verify (please read):**
- I could NOT compile Rust here (not enough disk), so `cargo check`,
`cargo test`, the Python/Node binding builds
and therefore the fixed-path assertions of the new Rust, Python and Node
tests have not been run. This is why the
PR is a draft; I will mark it ready once CI is green and fix whatever it
finds.
- Ran: both new Python tests fail on the released 0.40.0 wheel with the
exact error above (so they do catch the
bug); pylance 13.0.0 confirms the semantics
(`cleanup_old_versions(error_if_tagged_old_versions=False)` removes
the untagged versions and keeps the tagged one, still readable with
`lance.dataset(version="keep")`); in Lance
v14.0.0-beta.7 `cleanup.rs` the policy field is `bool`, the builder
default is `true`, and Lance's own
`auto_cleanup_hook` uses `Some(false)`; `rustfmt --edition 2024 --check`
passes on `optimize.rs`.
- The Rust and Node tests rely on timestamp ordering
(`nextMillisecond()`, `days(0)`/`now()`), so they could be
flaky: tell me if you prefer another approach.
Behaviour change: anyone who relied on the error to learn that a tag
blocks cleanup now gets a silent retain.
Not changed: the deprecated sync `LanceTable.cleanup_old_versions` calls
pylance directly and still errors on
tagged versions (the keyword may not exist in the oldest supported
pylance); `freshness.rs:1377` and
`refresh.rs:1077` still pass `None` (I did not check what those paths do
with tagged versions).
Fixes#4543
## Summary
`create_table(data=iter(()), schema=...)` currently raises
`RuntimeError: coroutine raised StopIteration`, although an empty list
with the same schema works. A tuple of batches also fails because the
helper calls `next()` before obtaining an iterator.
Normalize the iterable once and use the supplied schema for an empty
reader. Without a schema, raise a clear `ValueError`. Nonempty schema
inference and batch casting are unchanged.
Tests cover synchronous/asynchronous creation, empty/nonempty iterators
and tuples, and schema-aware `add()` sanitization. The default
schema-less `add()` rejection is deliberately unchanged.
## Validation
- Built the current Rust/PyO3 extension from base `9f37f392` with the
repository's `ci` profile; both Python-only arms use the same verified
extension and locked Python dependency versions.
- Complete `python/tests/test_table.py` +
`python/tests/test_scannable.py`: **268 passed, 0 skipped**. The
unchanged production code with identical tests has **24 failures / 244
passes**.
- Both arms used four-CPU affinity. An initial unrestricted-affinity run
exhausted thread creation under the private memory limit; that failed
attempt is not counted as a pass.
- Package-wide Ruff check/format pass. Repository-wide checks still
report pre-existing findings in unchanged `docs/web-tests`; those files
are not modified here.
AI-assisted change, reproduced against unchanged production code and
validated with a native source build. This does not claim the full
repository/platform matrix or a performance improvement.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Invalid `wait_timeout` values currently reach
`Duration::extract(...).unwrap()` in both native index-creation
bindings. A normal Python argument error becomes
`pyo3_runtime.PanicException`, which is not caught by `except
Exception`.
Propagate the extraction error with `map(...).transpose()?` before
constructing the index operation. Valid `None` and `timedelta` behavior
is unchanged.
Tests cover both `create_index` and `create_index_async`, three invalid
types, valid `None`/`timedelta`, and no index created after rejection.
Validation against main `0aa842b`, Python 3.12, uv-locked dependencies,
freshly compiled PyO3 extension for each arm:
- Unmodified source: 6 new regressions fail with `PanicException`; 6
controls pass.
- Fixed source: 12/12 pass; broader index-creation regression selection:
18/18 pass.
- `cargo fmt --all --check`, Ruff check/format, and diff check pass.
The broader selection includes the 12 cases above. No cloud, full-suite,
or performance claim.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Creating a table from an Arrow `list` or `large_list` vector field
infers a fixed-size vector type, but currently drops that field's
metadata. Descriptions and tags such as `lancedb:description` and
`lancedb:tag:model` disappear even though the input field carries them.
Preserve the original field metadata while replacing its inferred type.
Tests use public synchronous and asynchronous table creation, reopening,
and reads for floating-point and integer inference, checking the
complete schema and row values. Existing dtype inference and nullability
remain unchanged.
Validation on macOS arm64 with an official native extension built from
main `d864e719` using Rust 1.97.0:
- Python 3.11.14 and 3.13.11: full `pytest -m "not slow and not s3_test"
-q -rs python/tests`: **1,533 passed, 37 skipped, 68 deselected** each,
wall times 331.464 s and 416.308 s.
- Added public SDK cases on unchanged main: **12 failed**; after the
fix: **12 passed** on each runtime. Final installed distributions passed
**8 public SDK round trips** each, including fixed-size fields and
fields without metadata.
- Documentation tests on Python 3.11.14: **77 passed, 6 skipped**. Ruff
lint/format, Pyright (**0 errors**), applicable pre-commit hooks (**4
passed, 2 without matching files**), compile checks and `git diff
--check` passed.
- Comparable unchanged-main full baselines from the same day, native
extension and dependency versions: **1,521 passed, 37 skipped, 68
deselected** on each runtime. These were reused rather than rerun.
The final local wheel is an explicitly verified repack of the official
main wheel, replacing only `table.py` and `RECORD`; all production
Python files match the tested source and the native extension is
unchanged. No candidate Rust rebuild was performed. Slow/S3 integration
tests, the remaining OS/runtime and minimum-dependency matrices, and the
workflow's Linux/Python 3.13 doctest job were not run locally.
The existing Rust migration draft #2970 and issues #3216/#3222 do not
fix this metadata loss in the current Python path; a future migration
should preserve the same metadata. During final verification main
advanced to `81d52a98` through #4373 and the beta.13 version bump.
Neither modified the two files here; the local native and full-suite
validation above remain explicitly based on `d864e719`, and the updated
native changes have not been tested locally.
AI assistance: developed and verified with **gpt-6.1-sol**.
## Summary
- preflight CUDA availability and initialization before accelerated
index training
- direct unavailable or misconfigured CUDA environments to the explicit
CPU path
- cover both legacy and unified Python index APIs
## Root cause
LanceDB delegated accelerator requests directly to PyLance. When PyTorch
lacked CUDA support or CUDA initialization failed, users received a
low-level assertion or runtime error with no indication that omitting
`accelerator` or setting `accelerator=None` creates the index on CPU.
The repair validates CUDA before training and preserves explicit
accelerator semantics without silently changing the requested backend.
## Scope
This PR addresses only the separately reproduced CUDA availability and
initialization error. It does not close#1766; the reported query
failure after successful CUDA index creation remains open pending
reproducible evidence.
## Validation
- `uv run --extra tests pytest python/tests/test_table.py -k
create_index -q` (10 passed)
- `ruff format .`
- `ruff check .`
- CPU-only integration check confirms the actionable fallback message
Related to #1766
<!-- lance-gatekeeper-fix:v1 agent=d05c37237b04c2c8f88395a82e00bc67
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4033
`create_table(data=...)` without an explicit schema raised `IndexError:
index out of bounds` when a `vector`/`embedding`-named list column was
all-null. `_modal_list_size` infers the dimension with
`pc.mode(pc.list_value_length(arr))[0]`, and pyarrow's `mode` skips
nulls, so an all-null column produces a zero-length result and the `[0]`
subscript blows up.
The sibling `_infer_vector_dim` already guards the identical
`pc.mode(...)` with a `len == 0` check and returns `None`;
`_modal_list_size` just didn't. This makes `_modal_list_size` return
`Optional[int]` and return `None` when every value is null, and the call
site then leaves the column as a variable-length list instead of
inventing a fixed size — the dimension is genuinely unknowable when
there are no non-null rows.
Added a regression test that creates a table from an all-null
`list<float32>` vector column and asserts it succeeds, the column stays
a `list<float32>`, and the rows are queryable. It fails on `main` with
the `IndexError` and passes with this change.
## Summary
Sync `Table.search()` declares tuple, `pyarrow.Array`, and
`pyarrow.ChunkedArray` query vectors as supported, and the async query
path already accepts them. The sync query builder only classified lists
and NumPy arrays as vectors, however, so these inputs were routed
through embedding/FTS resolution and then rejected. Arrow and tuple
inputs also did not contribute a query dimension when selecting among
multiple vector columns.
This change:
- recognizes the supported tuple and Arrow containers as sync vector
queries for both `auto` and explicit `vector` query types;
- converts them to the existing internal `float32` NumPy representation
at the sync builder boundary;
- infers dimensions from tuple, primitive Arrow, and list-valued Arrow
queries so schemas with differently sized vector columns select the
matching column.
The regression test uses two vector columns with dimensions 2 and 3. It
verifies tuple, Arrow array, and chunked Arrow queries through both sync
query modes and confirms the dimension-2 column is selected.
## Test plan
- `pytest
python/tests/test_table.py::test_sync_search_accepts_supported_vector_containers
-q` (6 passed)
- `pytest python/tests/test_util.py python/tests/test_query.py
python/tests/test_table.py -q -m "not slow and not s3_test"` (308
passed, 1 skipped, 1 deselected)
- `pytest python/tests -q -m "not slow and not s3_test"
--ignore=python/tests/docs` (981 passed, 25 skipped, 65 deselected)
- `ruff format --check python`
- `ruff check python`
- `cargo fmt --all --check`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- move synchronous iterator preparation off LanceDB's singleton
background event loop
- keep the loop available when an input iterator executes a synchronous
table query
- cover query-backed iterators for both `Table.add` and `create_table`
## Root cause
The async implementations prepared input data on the singleton
background loop and peeked at the first item of iterators there. If that
iterator called a synchronous LanceDB query, `LOOP.run()` scheduled the
query on the same loop and blocked waiting for it, so neither operation
could progress.
## Validation
- `ruff format` on the changed Python files
- `ruff check .`
- `cd python && uv run --frozen --extra tests pytest
python/tests/test_table.py -q` (138 passed)
- the broader non-slow suite was attempted but hit a native segmentation
fault in the external DataFusion `register_table_provider` path; an
isolated control using the pre-change inline preparation behavior
reproduced the same unrelated crash
Fixes#2107
<!-- lance-gatekeeper-fix:v1 agent=8b4aaf556014ac0e457c9d63495697e4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
\`inf_vector_column_query\`'s \`dim\` filter (added in #3423) is meant
to disambiguate between multiple vector columns of different sizes. When
a schema has exactly one vector column and the query's dimension doesn't
match it, the filter excluded that column too, raising "There is no
vector column in the data" instead of resolving the column and letting
the dimension mismatch surface at query execution.
This was latent until #4409 wired client-side vector-column inference
into remote search: a mismatched-dimension query against a
single-vector-column remote table now fails client-side with the wrong
error, before ever reaching the server's own dimension validation.
Fall back to the unfiltered candidate set when the dimension filter
leaves zero matches, and only raise when that is still ambiguous or
genuinely empty.
Python `Decimal` values and Arrow decimal scalars raise
`NotImplementedError` in `table.update(values=...)`. Register a Decimal
converter that emits quoted fixed-point text so the native cast to the
target column preserves every digit, including decimal128 and decimal256
values. Non-finite Decimal values raise `ValueError` with a static
message.
Closes#4360.
Validation with the native extension built from the current source at
7f593359, on macOS arm64 and Python 3.12.14:
- All 33 new regression cases failed before the fix and passed
afterward. They cover Python and Arrow scalar updates, signs, zero,
exponents, full 38/76-digit precision, and a Decimal context precision
of 6.
- 16 existing converter, update, and Decimal expression tests passed.
- Repository Ruff formatting and lint passed; formatting left all 156
files unchanged.
- A native round-trip check confirmed that an unquoted SQL number rounds
a high-precision value while the quoted Decimal update preserves it
exactly.
The full suite, other Python versions, and remote API were not
exercised.
AI assistance: Codex helped implement and validate the Decimal
conversion and its regression tests.
add to table stats the number of rows that has
been marked for deletion, but not yet deleted.
note that when a table fragment is removed due to
deletion of all of its rows, these rows are not counted. since the main
use of this feature is to determine when table compaction is needed, the
fact that these rows are not reported should not impact that decision
(deleted table fragments don't need compaction).
Fixes: https://github.com/lancedb/lancedb/issues/3761
Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com>
Refs #4404.
AsyncTable.list_versions split the nanosecond timestamp with float
division/modulo (`ts // 1e9`, `ts % 1e9`).
At ~1.8e18 a float can't hold every integer, so the microseconds drifted
(.274000 came back as .273999).
Use `divmod(ts, 1_000_000_000)` and integer // 1000 instead.
Only the precision fix is included here. Timezone-aware UTC (option 1 in
#4404) is on hold for now,
and I can add it here or in a follow-up depending on what they prefer.
Added test_list_versions_timestamp_precision using the value from the
issue (1790989429274000000), expecting microsecond == 274000.
Empty list and NumPy vector queries were validated too late: local
search inferred a vector column first, producing misleading schema
errors, while remote search indexed `vector[0]` and raised `IndexError`.
Add a shared empty-vector check before column inference and query
creation, and validate direct vector-builder inputs. Local and remote
sync search now raise `ValueError: Query vector must not be empty` for
these inputs.
Validation:
- All 26 new regression cases failed on the original code and pass with
the fix, covering list/NumPy inputs, automatic/explicit vector search,
inferred/explicit column names, one/two local vector columns, and remote
search before query submission.
- After merging current `main`, all 32 focused cases passed, covering
these regressions and upstream async searches without a query.
- The query, table, remote, utility, hybrid-query, and database suites
passed in the rebuilt uv environment: 541 passed, 1 skipped, 1 slow test
deselected.
- `ruff check .`, `ruff format .`, and `cargo fmt --all` passed from the
repository root.
Fixes#4389
<!-- lance-gatekeeper-fix:v1 agent=f1f4875090ef391bb1aa912dd54026dc
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4390
Lance accepts column alterations with every change field unset, so a
path-only Python call could commit a new local version without changing
the schema. The Python binding also silently discarded unknown keys such
as `nulable`.
Validate every alteration in the shared Rust `Table` API before
dispatching to either backend, and reject unknown dictionary keys in the
Python binding while preserving the legacy `name` alias. Invalid batches
leave local schema and version unchanged and send no remote request.
Validation:
- Reproduced both reported calls on the original extension: versions
advanced from 1 to 2 to 3 with an unchanged schema; all 13 new
invalid-input Python cases failed before the fix.
- Rebuilt the extension with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`; all 18 focused Python cases pass across
sync/async and local/mocked remote tables.
- `cargo test --quiet --features remote -p lancedb --lib alter_column`:
all eight tests pass, including both new regressions.
- `cargo clippy --quiet --features remote --tests --examples` passes
across the workspace.
- `cargo fmt --all`, `ruff check .`, and `ruff format .` pass.
- `typos` 1.26.0 passes across the repository; intentional misspellings
in regression fixtures use the existing line-specific spelling
exemptions.
<!-- lance-gatekeeper-fix:v1 agent=36a547cd11108898893456042e38628b
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4383
`LanceTable.migrate_v2_manifest_paths()` raised `AttributeError` because
it called a nonexistent `AsyncTable.migrate_v2_manifest_paths()` method.
Delegate to the existing async method, `migrate_manifest_paths_v2()`,
while preserving the public sync API.
Add a local regression test for tables starting with either v1 or v2
manifest paths. It checks repeated migration, v2 manifest filenames on
disk, and preservation of data and version history after reopening the
table.
Validation:
- Both new regression cases failed with the reported `AttributeError`
before the fix.
- `cd python && uv run --extra tests pytest
python/tests/test_table.py::test_migrate_v2_manifest_paths
python/tests/test_db.py::test_create_table_v2_manifest_paths_async -q` —
3 passed.
- Repository-wide `ruff check .` and `ruff format .` passed using Ruff
0.15.20 from `python/uv.lock`.
- `git diff --check` passed.
<!-- lance-gatekeeper-fix:v1 agent=bb500f0d1ddefeba3c7d7d8390a2fb74
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Why
`optimize(cleanup_older_than=timedelta(0))` deletes the data files of
every version except the latest. Any other handle still on an older
version then fails with a missing-file error, and nothing tells the
caller this can happen. #2470 asks for a docs note and a runtime
warning.
Fixes#2470
## Scope
- `AsyncTable.optimize` emits a `UserWarning` when `cleanup_older_than`
rounds to 0 ms or less. `LanceTable.optimize` runs through it, so sync
callers get the warning too.
- The `cleanup_older_than` docstring on `Table.optimize`,
`LanceTable.optimize`, and `AsyncTable.optimize` gets a `.. warning::`
block, in the same style as the existing `delete_unverified` one.
- Removed the function-local `import warnings` in `AsyncTable.optimize`.
`warnings` is already imported at module level, and the local import
would shadow it for the new check (`UnboundLocalError`).
- `test_optimize` and `test_optimize_delete_unverified` intentionally
use a 0 cleanup, so they now assert the warning with `pytest.warns`.
Nodejs `cleanupOlderThan` is out of scope, since the issue is filed
against the Python SDK.
## Tradeoffs
Like the existing `retrain` deprecation warning, the warning is raised
inside the async coroutine. For sync callers it is attributed to
`asyncio/events.py` rather than the caller's line. Warning from both the
sync and async entry points would duplicate the check for a small gain.
## Verification
- Repro: a second handle checked out at an older version, then
`optimize(cleanup_older_than=timedelta(0))` on the first. The second
handle's read fails with `LanceError(IO) ... Not found:
.../data/<file>.lance`. Before this change no warning is emitted. After
it, the `UserWarning` fires.
- New tests `test_optimize_warns_on_zero_cleanup` (async, and checks
that `timedelta(days=1)` and the default don't warn) and
`test_optimize_warns_on_zero_cleanup_sync` fail on main (`DID NOT WARN`)
and pass with the fix. The failing tests are in their own commit before
the fix.
- `pytest python/tests/test_table.py -m "not slow and not s3_test"`: all
pass except `test_merge_insert`. That test's `timeout=timedelta(0)`
assertion fails the same way on unmodified main on my machine.
- `ruff format --check` and `ruff check` are clean on the changed files.
Tests ran against the source tree with the `_lancedb` extension from the
matching `0.40.0b11` wheel. The change touches no Rust.
This PR was written by an AI coding agent (Claude Code) and reviewed
before submission.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Jay Zhou <190699450+jayzhou2309@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Closes: #4060
- Convert Python dict/list objects to JSON strings during ingestion.
- Support JSON fields nested inside structs and lists.
- Cover `add` and `merge_insert` JSON ingestion paths.
- Add regression coverage for nested struct and list JSON fields.
## Testing
- `git diff --check`
- Python compilation passed.
- Focused pytest was attempted but could not complete because the native
extension build stalled during `uv` bootstrap.
A refresh running under a skip policy records each row it skipped, with
the failing input and the error, but the client could not read that
store: the server exposes it over SQL and, since recently, a REST route.
A user who hit per-row failures still had to open a SQL session.
`Table::function_errors` calls the route. The listing is table-addressed
with optional job and column filters, the same addressing the SQL
surface uses, so the two cannot disagree about what a table's errors
are. The two non-record signals come back as their own fields rather
than as rows: capped-fragment summaries, and whether the listing stopped
at its limit. Local tables refuse rather than answer with an empty list.
Python and Node expose the same call, with the same optional filters.
Two ways of writing to a `json` column failed or silently corrupted
data.
**All-null batches were rejected.** `add()` refused a batch whose values
for a `json` column were all null, while every plain Arrow type accepted
the same batch. This bites row-at-a-time inserts hardest: a one-row
batch with no value for an optional column is trivially all-null, so
most such writes failed. pyarrow infers `null` as the column's type, and
the write path had no handling for it — casting to the table's type
dropped the field metadata that identifies the column as `lance.json`,
so lance rejected the batch (`` `val` should have type json but type was
large_binary ``). A null-typed input column now becomes typed nulls
matching the table's field exactly, metadata included.
**Unlabelled JSON text was stored raw.** JSON supplied as plain strings
(what pyarrow infers for a column of `str`) was cast to the column's
`LargeBinary` storage type and relabelled `lance.json`, putting unparsed
text where JSONB was expected. Reads returned the text unnormalized and
`json_extract` failed with `InvalidJsonb`. Lance-core does the JSONB
encoding, but only for input labelled `arrow.json`, so string input is
now labelled rather than cast — at the top level and inside structs.
Both fixes are in the shared Rust write path, so they apply to any
binding, including hand-built Arrow tables that never pass through
Python's list-of-dicts type inference. `_align_field` gets the same
JSON-string fix for the legacy Python `_sanitize_data` path, which
`on_bad_vectors` and embedding functions still route through.
The blob v2 half of the issue landed separately in #4065, which added a
`DataType::Null` arm to blob coercion. This PR keeps that implementation
and adds end-to-end add-path coverage for it.
The tests from #4066 are included here and pass, so that PR's
Python-layer inference changes are no longer needed to close the issue.
Fixes#3759
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
refresh_column fills nulls, so once a row has a value nothing revisits
it: an update to one of its inputs, or a definition change, leaves the
computed value stale for good.
This stamps the column's field metadata with the definition it was
computed under and a per-fragment signature of the input storage it was
read from (input data files and overlays; not the deletion file, since a
delete changes no surviving value). A refresh recomputes every live row
of a fragment whose stamp disagrees with the manifest, then records what
it computed from in a second commit after the fill. A compacted fragment
inherits freshness through the Rewrite lineage when every fragment it
was built from was signed, or was appended since the stamp, never had an
input moved, and left its rows of the product unfilled (a raw append may
supply a value; the product's data is the evidence, and the null fill
covers those rows); otherwise it recomputes. A column declared before
the stamps existed keeps the null-fill contract on its first refresh,
which enrolls it as it stood.
The map is one entry per fragment per column, so it is kept out of the
manifest: each stamp writes an immutable sidecar under `_computed/`,
named by its content digest, and the field metadata holds the digest.
Pruning old versions also drops the sidecars no remaining version
references, keeping any younger than seven days as lance keeps
unverified files, since a sidecar is put before the commit that
references it. The stamp commit is metadata-only, so a materialized
view's drift check treats it like the fill. The core lives in
`table::freshness` so a remote refresh can share the contract.
Adds [typos](https://github.com/crate-ci/typos) as a CI check and
pre-commit hook, the same way Lance does it, so misspellings like the
ones fixed in #4146 get caught automatically going forward.
This also fixes the misspellings `typos` found across the repo (Rust,
Python, TypeScript source, comments, and generated docs), and adds a
small `.typos.toml` with `extend-words` entries for terms that are
correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ`
(a real quantization algorithm name), `mmaped` (the actual name of a
`candle-core` API we call), and `Writeable` (from Python's
`_typeshed.WriteableBuffer`). Third-party license files are excluded.
Fixes#4147
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core from v12.0.0-beta.14 to
[v12.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.15).
No compatibility fixes were required; `cargo clippy --quiet --workspace
--tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and
`git diff --check` passed.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
Merge insert has always taken a list of columns to match on, and local
tables have always joined on all of them. Remote tables did not: any
list longer than one was rejected with `MergeInsertBuilder only supports
a single 'on' column`, so a composite-key upsert was impossible against
LanceDB Cloud and Enterprise from Rust, Python or TypeScript.
The remote request now carries `on` as a list and sends it as one
repeated query parameter per column — `?on=shard_key&on=id`. That is how
the lance-namespace spec encodes an array-valued `on`, so the server
receives a composite key in the shape it expects. A single column still
serializes to `?on=id`, exactly what clients sent before, so existing
callers are unaffected. A column repeated within `on` is now rejected
client-side rather than sent for the server to reject with a 400.
No binding changes were needed: `Table.merge_insert` in Python and
`Table.mergeInsert` in TypeScript already accepted a list, it just could
not reach a remote table. Both gain a test for composite keys, and the
doc comments now say what passing several columns means.
Part of
[ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the).
## Example
```python
table.merge_insert(["shard_key", "id"]) \
.when_matched_update_all() \
.when_not_matched_insert_all() \
.execute(new_data)
```
A row whose `id` matches an existing row but whose `shard_key` differs
is an insert, not an update.
## Not included
Java. Java callers reach merge insert through
`org.lance.namespace.LanceNamespace`, whose
`MergeInsertIntoTableRequest.on` is a single string until
lance-namespace 0.12
([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363),
[lance#8915](https://github.com/lance-format/lance/pull/8915)). There is
nothing in this repo's Java SDK to change until the `lance-core` pin can
move.
Sending more than one column requires a server that accepts the repeated
parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571));
an older server returns a 400 rather than silently merging on one
column.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Computed-column planning currently sees Blob v2 storage descriptors, so
expressions cannot consume payload bytes or preserve Blob semantics in
their outputs.
A computed declaration now derives its output field from its expression.
A direct projection of a Blob v2 field inherits the source field's Blob
metadata; other expressions retain their ordinary Arrow-inferred type.
Declarations remain ordered, so the same rule applies across chained
projections.
Refresh materializes referenced Blob inputs as `LargeBinary` payload
bytes and publishes inherited Blob outputs through Lance's Blob
conversion path. Remote requests remain within the shared namespace
contract as `{name, computed}`; the server planner is being updated in
tandem to implement the same Blob-aware planning semantics, and remote
enablement must be aligned with that server rollout.
The existing null-as-unfilled contract remains unchanged. Row-level
freshness and cell flags remain follow-up work.
## Summary
- allow Python sync, async, and remote table updates to accept type-safe
`Expr` filters
- serialize expression filters before invoking the existing update
implementation
- cover numeric-looking text and apostrophe-containing text in sync and
async regression tests
## Root cause
`Table.update` was the remaining Python write path that required callers
to construct a raw SQL predicate. Dynamic text interpolated without SQL
literal encoding could therefore be parsed as an integer, float, or
unterminated string instead of Utf8. The expression API already encodes
literals safely for query and delete filters.
## Validation
- `cd python && .venv/bin/pytest
python/tests/test_table.py::test_update_async
python/tests/test_table.py::test_update_expr_filter_literals_async
python/tests/test_table.py::test_update
python/tests/test_table.py::test_update_expr_filter_literals -q`
- `cd python && .venv/bin/pytest python/tests/test_expr.py -q`
- `cd python && .venv/bin/ruff format --check .`
- `cd python && .venv/bin/ruff check .`
Fixes#1869
<!-- lance-gatekeeper-fix:v1 agent=01f1e7b69c65e8b6d3b3c1e1a7918179
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>