Python's `add_columns` can now accept precomputed values from pandas or Polars DataFrames, Arrow Tables, RecordBatches, readers, and iterables of batches on local tables. The sync and async APIs pass an Arrow stream to the existing Rust reader transform without materializing the whole reader.
Rows attach positionally to the current live rows in storage scan order. Input must contain only new columns and exactly one row per live row; pandas indexes are ignored. Readers are consumed once, and their batch boundaries can differ from table fragments. Keep the table unchanged between computing values and adding them.
SQL mappings, Function applications, computed columns, and all-null schema inputs keep their existing behavior. Data inputs cannot be combined with computed columns, and remote tables reject reader inputs with `NotImplementedError`. Tests cover deleted rows, row counts, empty batches and tables, reader errors, collisions, reserved metadata, Arrow types, and the existing input forms. Both guides describe the supported inputs and limits.
Fixes#1959.
Fixes#4761
`Table.update` does not compute embeddings, so changing an embedding source
column could leave the previous embedding in place. Source-column updates now
raise `ValueError` unless every associated vector column is supplied in the same
call.
The guard reads Python embedding metadata and native `lancedb::column_definitions`
without loading embedding models. It uses Lance's field-path parser for quoted
targets and covers synchronous and asynchronous local and remote tables. Supplying
every associated vector column, vector-only updates, and unrelated updates remain
supported.
This is a breaking change for source-only updates. The API documentation and
Update rows guide explain the requirement and the `merge_insert` alternative for
regenerating embeddings.
Validation at the reviewed head covered 44 targeted public update cases and the
full Python suite (3107 passed, 162 skipped, 70 deselected). All applicable
exact-head checks and workflows completed successfully.
Use “LanceDB Enterprise” throughout the staged website guides, SDK
reference sources, generated API pages, and user-facing SDK messages. A
separate branded “LanceDB Distributed” name adds terminology without
clarifying behavior; describe where operations run directly.
- Update `docs/web`, including the architecture diagram title and alt
text. Use Enterprise in headings, table labels, and prose; simplify
redundant product/execution-model wording and name the product only once
in the diagram alt text.
- Update Python docstrings and the Python/Java reference pages,
TypeScript comments and native declaration comments, and Rust API
documentation. Regenerate the TypeScript reference, including 24
replacements across six pages.
- Update SDK warnings/errors and existing test expectations to match,
along with contributor guidance and Vale rules. Remove the obsolete rule
distinguishing Enterprise from Distributed.
- Keep lowercase “distributed” for technical descriptions. API
identifiers, anchors, asset paths, and execution behavior are unchanged.
No branded-name exceptions are proposed.
Validation:
- Python and Node native builds succeeded; focused Python tests passed
(36 passed, 4 skipped), and focused Node tests passed (14).
- TypeDoc generation and the complete MkDocs reference build passed.
MkDocs retains an existing unresolved
`lancedb.functions.SecretReference` cross-reference.
- Rust formatting, Node lint/format, changed-file Python lint/format, 32
Vale tests, and whitespace checks passed. Repository-wide Python lint
has 48 existing findings in six untouched files.
- Combined documentation assembly passed. Mintlify validation and
link/anchor/redirect checks previously passed for both site builds; no
code snippets changed.
Companion PRs: [saved editions, terminology guide, and shared
rules](https://github.com/lancedb/docs/pull/413), [Enterprise overview
and writing rules](https://github.com/lancedb/sophon/pull/9511).
---------
Co-authored-by: Brendan Clement <brendan.clement@lancedb.com>
Python column refresh methods rejected `batch_size`, and the Rust remote
backend never included it in the backfill request. Callers could not
reduce the rows passed to a slow Function to stay within the executor's
per-call deadline.
Add optional, keyword-only `batch_size` arguments throughout the
synchronous and asynchronous Python APIs and bindings. Add compatible
Rust `RefreshColumnOptions` APIs while preserving existing Rust method
signatures. Reject non-positive sizes before submitting a job, send
positive sizes in `backfill_column`, and omit the field when unset while
retaining branch scoping.
Batch size controls remote Function calls. Embedded expression refreshes
accept and validate the option without changing execution, and blocking
remote refreshes retain their existing unsupported behavior; remote
callers use `refresh_column_async` and wait on the returned job.
Validation:
- Reproduced the original Python `TypeError` from the committed method.
- Five focused Rust tests passed, covering request contents, omission,
branch scoping, and rejection before any request.
- `cargo check --quiet --features remote --tests --examples` passed.
- `cargo clippy --quiet --features remote --tests --examples -- -D
warnings` passed.
- Nineteen targeted Python tests passed against the rebuilt extension,
including synchronous and asynchronous remote requests, invalid sizes,
and default-build behavior. Eight embedded refresh cases require the
experimental-functions build and were skipped.
- Rust formatting and Python formatting/lint checks passed, including
CI's pinned Ruff version. Repository-wide Ruff reports 48 existing
findings in untouched documentation files; none are in changed files.
Fixes#4776
<!-- lance-gatekeeper-fix:v1 agent=4906ff3f5f7797a1d49a13803302021c
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Expose table-level maintenance override read, update, and reset APIs for remote
Rust and Python tables. Local tables continue to return `NotSupported`.
Updates recursively merge the caller-provided JSON object into the current
override document so unrelated and unknown service fields are preserved. Reset
uses the explicit reset endpoint.
The integrated change keeps current main's remote database identity, namespace,
branch, SQL, WAL, and threaded mock-server behavior.
Validated with focused Rust and Python override tests and the complete applicable
CI run for 43bccbadc5.
Updates Rust and Java Lance dependencies to
[v15.0.0-beta.6](https://github.com/lance-format/lance/releases/tag/v15.0.0-beta.6),
pins Python tests to published pylance 15.0.0b6, and refreshes both
lockfiles.
Aligns Arrow 59, DataFusion 55, and PyO3 0.29; adapts execution plans
and MemWAL directory handling, updates the ABI metadata test, and
removes the obsolete Lance Arrow patch now fixed upstream.
Validation: workspace clippy with all features/tests and warnings
denied, cargo fmt, Python/CI Ruff checks, uv lock check, 16 CI-script
tests, and targeted Python regressions (54 passed, 28 skipped);
repository-wide Ruff retains existing documentation-example failures,
and the upstream macOS wheel build failed while Linux and Windows wheels
are published.
---------
Co-authored-by: Daniel Rammer <hamersaw@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Adopts the product terminology guide across the docs site pages, SDK
docstrings and error messages. "LanceDB OSS" and local mode become
**LanceDB Embedded**. "Remote" as the execution model becomes **LanceDB
Distributed**, which LanceDB Enterprise provides. Where free vs paid is
the point, pages contrast "LanceDB Embedded (open source)" with "LanceDB
Enterprise". This is a first trial of the Embedded name, so reviewers
can judge it in context.
- **Embedded vs Distributed** for where execution runs (`optimize()`,
consistency, branching, maintenance). Main pages define it once:
"LanceDB Enterprise runs LanceDB Distributed: …"
- **LanceDB Cloud** is kept only for legacy-Cloud specifics (`db://`
region, the brute-force KNN limit) and marked legacy. "Cloud and
Enterprise" becomes Enterprise or Distributed.
- **Plan executor** becomes "executor" or the fleet name.
- **Error messages** change, e.g. `optimize is not supported for remote
tables.` → `... in LanceDB Distributed.` Tests that match on them are
updated.
- **Vale rules** (`EmbeddedName`, `OSSProduct`, `RemoteExecution`,
`DistributedEnterprise`, `LegacyCloud`, `InternalNames`) enforce the
terms in prose.
Unchanged: identifiers (`availability.oss`, the `remote` module and
feature, `RemoteTable`, `db://`), heading anchors, internal code
comments, and Geneva.
## Evidence
- Rust lib tests for the changed messages and the Python tests in
`test_remote_db.py`, `test_materialized_views.py` and `test_table.py`
pass locally.
- Not run locally:
- The Node tests: `remote.test.ts` and `table.test.ts` have updated
assertions.
- Vale.
- The `experimental_functions`-marked materialized-view tests.
- `docs/src` API reference was hand-edited to match the docstrings
rather than regenerated.
## Merge Risk
**Door:** two-way; prose and message text only.
**Blast Radius:** `docs`, `error-messages`: callers matching on old
error strings would break.
Companion PRs: lancedb/docs (assembler labels, STYLE.md) and sophon
(Enterprise pages).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Published SDK builds currently expose local SQL computed columns and
materialized-view execution that are not ready for release. Add the
opt-in `experimental-functions` Cargo feature to the Rust core and
forward it through the Python and Node bindings; published packages
leave it disabled.
Local declarations and refreshes now return `NotSupported` before
mutating tables or spawning jobs, including prepared materialized-view
creation. Remote Cloud/Enterprise client operations, Python UDF
packaging, ordinary eager `add_columns`, and access to existing data
remain available. Stored metadata and write protections stay active, and
shared server commit helpers remain usable.
CI covers both feature configurations. Existing local execution tests
run with the feature enabled; default-build regressions cover early
rejection, unchanged table versions, existing metadata protections, and
normal eager transforms. API and source-build documentation explain the
opt-in.
Validation: Rust library tests pass with the feature disabled (1,869
tests) and enabled (1,947 tests), plus three default-build integration
regressions. Python and TypeScript bindings were rebuilt and tested in
both configurations, including mocked remote requests; Rust formatting,
Python package lint, TypeScript lint/build, and generated docs were
checked. Repository-wide Ruff still reports 48 pre-existing errors in
unchanged `docs/web-tests` files.
## Summary
- expose typed compaction options on synchronous and asynchronous table
optimization
- translate validated Python options into native Lance compaction
settings
- document the row-based steady-state limit and first-pass/source-budget
caveats
- carry the Lance beta.15 fragment-exclusion option through the typed
Python API
## Root cause
The native Python optimize binding always constructed default compaction
settings, and the public AsyncTable API offered no way to lower the
fragment row target. Wide-row fragments therefore remained compaction
candidates on every later optimize call, repeatedly rewriting the
accumulated table.
## Limitations
This is an explicit row-based mitigation, not byte-native task sizing or
a hard memory cap. Callers must derive target_rows_per_fragment from
expected row size. Existing oversized fragments are rewritten whole on
the first pass; max_bytes_per_file limits output file size rather than
compaction memory; and source budgets admit whole planned tasks, so a
budget below the first task performs no compaction. Splitting tasks by a
byte budget or returning a planner error requires a Lance-side change.
## Validation
- cargo check --release -p lancedb-python
- cargo clippy --quiet -p lancedb-python
- cargo fmt --all --check
- ruff format --check python tests
- ruff check python tests
- uv run --no-sync pytest python/tests/test_table.py -k
optimize_compaction -q (16 passed)
Fixes#2325
<!-- lance-gatekeeper-fix:v1 agent=9b58eee0c79040b19540e259af0e676c
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Keep FP16 vectors through IVF-PQ index creation and search in the
synchronous and asynchronous quickstarts. Add native and conditional
CUDA regression coverage using a float64 query against an explicitly
typed FP16 column. This is test and documentation-example coverage for
dtype preservation already present in Lance; it does not change
production code.
Related #2120.
Validation on `bbad20940632b21a9ee85bdcb46b4cc19dbd65e3` with Python
3.12.12, Lance 15.0.0-beta.1, PyArrow 24.0.0, and Torch 2.14.1:
- The native FP16 parameter, Arrow FP16 ingestion test, and both
quickstarts passed: 4 passed, 1 CUDA-only skip.
- The two affected test files passed: 386 passed, 1 CUDA-only skip (`-m
"not slow and not s3_test"`).
- Ruff 0.9.9 lint and formatting checks passed for the Python package.
Real CUDA execution has not been performed. Torch is installed, but
`torch.cuda.is_available()` is false; `test_create_f16_table[cuda]`
skips with `CUDA not available`. The native CPU result does not
establish that the original GPU failure is resolved. Issue #2120 remains
open for verification on an actual CUDA device.
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
`IvfRq()` forced `num_bits=1` through Python's Rust binding, overriding
Lance's five-bit default. The shared native index creation path also
substituted one bit when the Rust or Node builder left the value unset.
Make Python's bit count optional and forward only explicit values. Use
`RQBuildParams::default()` for an unset native bit count, and omit unset
values when dispatching to pylance for accelerator builds. Update the
Python, Rust, Node, and vector-index documentation.
New indexes with an unset bit count now inherit Lance's default,
currently five bits, across Python, Rust, and Node. Five-bit vector
codes use roughly five times the storage of one-bit codes, plus other
index overhead. This also changes reader compatibility: indexes created
with `num_bits >= 2` use a newer on-disk layout that some older LanceDB
versions cannot read. Upgrade readers to versions that support multi-bit
IVF_RQ, or explicitly set `num_bits=1` (for example, `IvfRq(num_bits=1)`
in Python) when sharing indexes with readers limited to one-bit IVF_RQ.
Validation:
- Reproduced six regression failures before the fix: one-bit default
metadata, rejected `None`, and incorrect remote/accelerator forwarding.
- After merging `main` through `b762ec22`, rebuilt both native bindings
against Lance 15.0.0-beta.1. Python validation passed with 106 tests and
15 optional Ollama skips. Coverage includes the full index suite,
omitted and explicit bit counts, reading Lance-created IVF_RQ indexes,
accelerator forwarding, CUDA-unavailability handling, timeout
validation, remote requests, and table/namespace listing behavior.
- Rust formatting and Python package Ruff lint/format checks pass. Node
`pnpm lint`, `pnpm build` (including TypeScript checks), `pnpm run
docs`, the full connection suite, and the existing IVF_RQ regression
pass. Generated documentation adds no further changes.
The merge preserves upstream's explicit RaBitQ heading anchor alongside
the `IvfRq()` example and five-bit default/reader-compatibility
guidance.
Repository-wide `ruff check .` still reports 49 existing errors in
unchanged documentation examples under `docs/web-tests`; root formatting
also finds 11 unchanged documentation example files. The Python package
checks pass.
Fixes#4582
<!-- lance-gatekeeper-fix:v1 agent=6ef9406f0142b1ab66d53d9d56098223
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Pandas converts NaN vector components to Arrow nulls. Both Python's NaN
mask and Rust's default append validator ignored null children, so
`add()` silently stored incomplete vectors despite
`on_bad_vectors="error"`.
Reject null components in the existing Rust ingestion validator, and
include null components in Python's bad-vector detection and filling.
`drop` removes affected rows, `fill` repairs missing components while
preserving valid ones, and `null` replaces affected vectors. Explicit
null components in Arrow and list inputs follow the same policies. Rust
allows whole null vectors and retains the explicit
`NaNVectorBehavior::Keep` opt-out.
The ingestion guide and synchronous/asynchronous `create_table()` option
docs describe null components, all four policies, and filling that
preserves valid components.
Validation:
- Reproduced the default pandas append failure with the regression test
before changing the Rust validator.
- Rebuilt the native Python extension from the merged sources. The
Python table and utility suites passed: 481 passed, 1 skipped. Coverage
includes create/append/overwrite, pandas/rows/Arrow inputs, every
bad-vector policy, float16/32/64, sliced and chunked arrays, all-null
components, and hidden children of null lists.
- All 43 focused Rust tests passed: 7 validator tests and 36 ingestion
tests.
- Rust Clippy with `remote`, tests, and examples passed. Ruff lint
passes for the Python package, formatting checks pass for modified
Python files, and `cargo fmt --all` is clean.
- Documentation follow-up: all 56 focused vector-policy tests passed;
Python package lint and creation-docstring formatting checks passed.
Repository-wide Ruff reports 49 existing errors in untouched
documentation examples. Create/overwrite retain their existing
`RuntimeError` wrapping for errors raised while consuming lazy Arrow
readers.
Fixes#4435
<!-- lance-gatekeeper-fix:v1 agent=b5108e27d28ef613a78c1e4354c03c48
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Arrow map scalars returned by a table query cannot currently be passed
back to `update`: the generic Arrow scalar conversion turns them into a
list of tuples, which the SQL literal encoder cannot handle. Empty map
scalars become an untyped SQL list and also fail to update a map column.
Add a dedicated `MapScalar` conversion that renders its keys and values
as a SQL map while preserving the existing null conversion. The
regression tests exercise real synchronous and asynchronous table
updates, quoted keys, nullable values, empty typed maps, and nested
maps.
Validation: the complete documented unit-test scope passed locally
(1,515 passed, 37 skipped; unchanged baseline: 1,507 passed, 37
skipped). Full doctests passed (71 passed, 6 skipped), and
repository-wide Python formatting/lint, Rust formatting/clippy, and
Python 3.13 type checking passed. The locally built native wheel also
passed all eight regression cases in a separate Python 3.13 environment.
Full Linux/S3 integration and all platform matrices remain for CI.
The SQL map constructor's existing backend constraints still apply:
duplicate keys are rejected, and a nested map containing both nonempty
and empty inner maps can fail SQL type unification. This change does not
claim arbitrary Arrow map roundtrip support.
AI assistance: implementation and local verification used Codex with
gpt-6.1-sol agents; the diff and evidence were reviewed before
submission.
---------
Co-authored-by: Xuanwo <github@xuanwo.io>
`optimize()` fails once a tagged version is older than the cleanup
cutoff, although the docs say tagged versions
are exempt from cleanup, and the failed prune also skips the
index-optimize step (repro and numbers in the issue).
Change: `OptimizeAction::Prune` already exposes
`error_if_tagged_old_versions: Option<bool>`, so no new API.
`cleanup_old_versions` and `cleanup_old_versions_before` in
`rust/lancedb/src/table/optimize.rs` now use
`unwrap_or(false)`, which covers `OptimizeAction::Prune`,
`OptimizeAction::All` and `optimize_prune_before`. The
Python and Node bindings pass `None`, which now means "retain tagged
versions"; I left `python/src/table.rs` and
`nodejs/src/table.rs` untouched (this also avoids a conflict with #3870,
which edits that region). `Some(true)`
still opts in to the error. Doc comments updated; the sentence in
`docs/web/management/table.mdx` is accurate once
this lands.
Tests: Rust `test_optimize_prune_retains_tagged_versions` (`Some(true)`
still errors; the default retains the tag;
`optimize_prune_before`; `checkout_tag` plus a data read), a Python sync
test with a scalar index (index step runs,
a second optimize works, untagged versions are pruned,
`checkout("release-1")` works), a Python async test and a
Node test.
**What I could and could not verify (please read):**
- I could NOT compile Rust here (not enough disk), so `cargo check`,
`cargo test`, the Python/Node binding builds
and therefore the fixed-path assertions of the new Rust, Python and Node
tests have not been run. This is why the
PR is a draft; I will mark it ready once CI is green and fix whatever it
finds.
- Ran: both new Python tests fail on the released 0.40.0 wheel with the
exact error above (so they do catch the
bug); pylance 13.0.0 confirms the semantics
(`cleanup_old_versions(error_if_tagged_old_versions=False)` removes
the untagged versions and keeps the tagged one, still readable with
`lance.dataset(version="keep")`); in Lance
v14.0.0-beta.7 `cleanup.rs` the policy field is `bool`, the builder
default is `true`, and Lance's own
`auto_cleanup_hook` uses `Some(false)`; `rustfmt --edition 2024 --check`
passes on `optimize.rs`.
- The Rust and Node tests rely on timestamp ordering
(`nextMillisecond()`, `days(0)`/`now()`), so they could be
flaky: tell me if you prefer another approach.
Behaviour change: anyone who relied on the error to learn that a tag
blocks cleanup now gets a silent retain.
Not changed: the deprecated sync `LanceTable.cleanup_old_versions` calls
pylance directly and still errors on
tagged versions (the keyword may not exist in the oldest supported
pylance); `freshness.rs:1377` and
`refresh.rs:1077` still pass `None` (I did not check what those paths do
with tagged versions).
Fixes#4543
## Summary
`create_table(data=iter(()), schema=...)` currently raises
`RuntimeError: coroutine raised StopIteration`, although an empty list
with the same schema works. A tuple of batches also fails because the
helper calls `next()` before obtaining an iterator.
Normalize the iterable once and use the supplied schema for an empty
reader. Without a schema, raise a clear `ValueError`. Nonempty schema
inference and batch casting are unchanged.
Tests cover synchronous/asynchronous creation, empty/nonempty iterators
and tuples, and schema-aware `add()` sanitization. The default
schema-less `add()` rejection is deliberately unchanged.
## Validation
- Built the current Rust/PyO3 extension from base `9f37f392` with the
repository's `ci` profile; both Python-only arms use the same verified
extension and locked Python dependency versions.
- Complete `python/tests/test_table.py` +
`python/tests/test_scannable.py`: **268 passed, 0 skipped**. The
unchanged production code with identical tests has **24 failures / 244
passes**.
- Both arms used four-CPU affinity. An initial unrestricted-affinity run
exhausted thread creation under the private memory limit; that failed
attempt is not counted as a pass.
- Package-wide Ruff check/format pass. Repository-wide checks still
report pre-existing findings in unchanged `docs/web-tests`; those files
are not modified here.
AI-assisted change, reproduced against unchanged production code and
validated with a native source build. This does not claim the full
repository/platform matrix or a performance improvement.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Invalid `wait_timeout` values currently reach
`Duration::extract(...).unwrap()` in both native index-creation
bindings. A normal Python argument error becomes
`pyo3_runtime.PanicException`, which is not caught by `except
Exception`.
Propagate the extraction error with `map(...).transpose()?` before
constructing the index operation. Valid `None` and `timedelta` behavior
is unchanged.
Tests cover both `create_index` and `create_index_async`, three invalid
types, valid `None`/`timedelta`, and no index created after rejection.
Validation against main `0aa842b`, Python 3.12, uv-locked dependencies,
freshly compiled PyO3 extension for each arm:
- Unmodified source: 6 new regressions fail with `PanicException`; 6
controls pass.
- Fixed source: 12/12 pass; broader index-creation regression selection:
18/18 pass.
- `cargo fmt --all --check`, Ruff check/format, and diff check pass.
The broader selection includes the 12 cases above. No cloud, full-suite,
or performance claim.
Signed-off-by: Xucheng Zhou <aden1350@outlook.com>
Creating a table from an Arrow `list` or `large_list` vector field
infers a fixed-size vector type, but currently drops that field's
metadata. Descriptions and tags such as `lancedb:description` and
`lancedb:tag:model` disappear even though the input field carries them.
Preserve the original field metadata while replacing its inferred type.
Tests use public synchronous and asynchronous table creation, reopening,
and reads for floating-point and integer inference, checking the
complete schema and row values. Existing dtype inference and nullability
remain unchanged.
Validation on macOS arm64 with an official native extension built from
main `d864e719` using Rust 1.97.0:
- Python 3.11.14 and 3.13.11: full `pytest -m "not slow and not s3_test"
-q -rs python/tests`: **1,533 passed, 37 skipped, 68 deselected** each,
wall times 331.464 s and 416.308 s.
- Added public SDK cases on unchanged main: **12 failed**; after the
fix: **12 passed** on each runtime. Final installed distributions passed
**8 public SDK round trips** each, including fixed-size fields and
fields without metadata.
- Documentation tests on Python 3.11.14: **77 passed, 6 skipped**. Ruff
lint/format, Pyright (**0 errors**), applicable pre-commit hooks (**4
passed, 2 without matching files**), compile checks and `git diff
--check` passed.
- Comparable unchanged-main full baselines from the same day, native
extension and dependency versions: **1,521 passed, 37 skipped, 68
deselected** on each runtime. These were reused rather than rerun.
The final local wheel is an explicitly verified repack of the official
main wheel, replacing only `table.py` and `RECORD`; all production
Python files match the tested source and the native extension is
unchanged. No candidate Rust rebuild was performed. Slow/S3 integration
tests, the remaining OS/runtime and minimum-dependency matrices, and the
workflow's Linux/Python 3.13 doctest job were not run locally.
The existing Rust migration draft #2970 and issues #3216/#3222 do not
fix this metadata loss in the current Python path; a future migration
should preserve the same metadata. During final verification main
advanced to `81d52a98` through #4373 and the beta.13 version bump.
Neither modified the two files here; the local native and full-suite
validation above remain explicitly based on `d864e719`, and the updated
native changes have not been tested locally.
AI assistance: developed and verified with **gpt-6.1-sol**.
## Summary
- preflight CUDA availability and initialization before accelerated
index training
- direct unavailable or misconfigured CUDA environments to the explicit
CPU path
- cover both legacy and unified Python index APIs
## Root cause
LanceDB delegated accelerator requests directly to PyLance. When PyTorch
lacked CUDA support or CUDA initialization failed, users received a
low-level assertion or runtime error with no indication that omitting
`accelerator` or setting `accelerator=None` creates the index on CPU.
The repair validates CUDA before training and preserves explicit
accelerator semantics without silently changing the requested backend.
## Scope
This PR addresses only the separately reproduced CUDA availability and
initialization error. It does not close#1766; the reported query
failure after successful CUDA index creation remains open pending
reproducible evidence.
## Validation
- `uv run --extra tests pytest python/tests/test_table.py -k
create_index -q` (10 passed)
- `ruff format .`
- `ruff check .`
- CPU-only integration check confirms the actionable fallback message
Related to #1766
<!-- lance-gatekeeper-fix:v1 agent=d05c37237b04c2c8f88395a82e00bc67
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4033
`create_table(data=...)` without an explicit schema raised `IndexError:
index out of bounds` when a `vector`/`embedding`-named list column was
all-null. `_modal_list_size` infers the dimension with
`pc.mode(pc.list_value_length(arr))[0]`, and pyarrow's `mode` skips
nulls, so an all-null column produces a zero-length result and the `[0]`
subscript blows up.
The sibling `_infer_vector_dim` already guards the identical
`pc.mode(...)` with a `len == 0` check and returns `None`;
`_modal_list_size` just didn't. This makes `_modal_list_size` return
`Optional[int]` and return `None` when every value is null, and the call
site then leaves the column as a variable-length list instead of
inventing a fixed size — the dimension is genuinely unknowable when
there are no non-null rows.
Added a regression test that creates a table from an all-null
`list<float32>` vector column and asserts it succeeds, the column stays
a `list<float32>`, and the rows are queryable. It fails on `main` with
the `IndexError` and passes with this change.
## Summary
Sync `Table.search()` declares tuple, `pyarrow.Array`, and
`pyarrow.ChunkedArray` query vectors as supported, and the async query
path already accepts them. The sync query builder only classified lists
and NumPy arrays as vectors, however, so these inputs were routed
through embedding/FTS resolution and then rejected. Arrow and tuple
inputs also did not contribute a query dimension when selecting among
multiple vector columns.
This change:
- recognizes the supported tuple and Arrow containers as sync vector
queries for both `auto` and explicit `vector` query types;
- converts them to the existing internal `float32` NumPy representation
at the sync builder boundary;
- infers dimensions from tuple, primitive Arrow, and list-valued Arrow
queries so schemas with differently sized vector columns select the
matching column.
The regression test uses two vector columns with dimensions 2 and 3. It
verifies tuple, Arrow array, and chunked Arrow queries through both sync
query modes and confirms the dimension-2 column is selected.
## Test plan
- `pytest
python/tests/test_table.py::test_sync_search_accepts_supported_vector_containers
-q` (6 passed)
- `pytest python/tests/test_util.py python/tests/test_query.py
python/tests/test_table.py -q -m "not slow and not s3_test"` (308
passed, 1 skipped, 1 deselected)
- `pytest python/tests -q -m "not slow and not s3_test"
--ignore=python/tests/docs` (981 passed, 25 skipped, 65 deselected)
- `ruff format --check python`
- `ruff check python`
- `cargo fmt --all --check`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- move synchronous iterator preparation off LanceDB's singleton
background event loop
- keep the loop available when an input iterator executes a synchronous
table query
- cover query-backed iterators for both `Table.add` and `create_table`
## Root cause
The async implementations prepared input data on the singleton
background loop and peeked at the first item of iterators there. If that
iterator called a synchronous LanceDB query, `LOOP.run()` scheduled the
query on the same loop and blocked waiting for it, so neither operation
could progress.
## Validation
- `ruff format` on the changed Python files
- `ruff check .`
- `cd python && uv run --frozen --extra tests pytest
python/tests/test_table.py -q` (138 passed)
- the broader non-slow suite was attempted but hit a native segmentation
fault in the external DataFusion `register_table_provider` path; an
isolated control using the pre-change inline preparation behavior
reproduced the same unrelated crash
Fixes#2107
<!-- lance-gatekeeper-fix:v1 agent=8b4aaf556014ac0e457c9d63495697e4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Add `auto_refresh_enabled` and `auto_refresh_interval_seconds` to the
Rust builder and Python sync/async add-column APIs for computed columns
and registered Function outputs.
Python users can configure refresh when declaring a column:
```python
table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
Omit the interval to let the server determine when to refresh:
```python
table.add_columns(computed={"discounted_total": "total * 0.9"})
```
The async API accepts the same options:
```python
await async_table.add_columns(
computed={"total": "price * quantity"},
auto_refresh_enabled=True,
auto_refresh_interval_seconds=300,
)
```
For an existing registered `FunctionApplication`, the options also apply
to Function columns. Disabling automatic refresh still permits explicit
refresh:
```python
table.add_columns(
{"features": function_application},
auto_refresh_enabled=False,
auto_refresh_interval_seconds=300,
)
table.refresh_column("features")
```
Options apply to every output declared by a call. Automatic refresh is
enabled by default; when the interval is omitted, the server determines
when to refresh. An explicit interval sets a minimum delay after a
successful refresh. The interval does not force recomputation of
unchanged inputs. Invalid intervals and options on one-time transforms
are rejected before declaration.
Native declarations persist `computed_column.auto_refresh_enabled` and
an explicitly supplied `computed_column.auto_refresh_interval_seconds`
in each output field’s metadata in the same commit as the declaration.
An omitted interval is neither stored nor sent in remote requests.
Remote declarations forward the settings to Cloud/Enterprise servers.
Local tables store these settings; automatic execution requires a
supporting server with scheduling enabled.
\`inf_vector_column_query\`'s \`dim\` filter (added in #3423) is meant
to disambiguate between multiple vector columns of different sizes. When
a schema has exactly one vector column and the query's dimension doesn't
match it, the filter excluded that column too, raising "There is no
vector column in the data" instead of resolving the column and letting
the dimension mismatch surface at query execution.
This was latent until #4409 wired client-side vector-column inference
into remote search: a mismatched-dimension query against a
single-vector-column remote table now fails client-side with the wrong
error, before ever reaching the server's own dimension validation.
Fall back to the unfiltered candidate set when the dimension filter
leaves zero matches, and only raise when that is still ambiguous or
genuinely empty.
Python `Decimal` values and Arrow decimal scalars raise
`NotImplementedError` in `table.update(values=...)`. Register a Decimal
converter that emits quoted fixed-point text so the native cast to the
target column preserves every digit, including decimal128 and decimal256
values. Non-finite Decimal values raise `ValueError` with a static
message.
Closes#4360.
Validation with the native extension built from the current source at
7f593359, on macOS arm64 and Python 3.12.14:
- All 33 new regression cases failed before the fix and passed
afterward. They cover Python and Arrow scalar updates, signs, zero,
exponents, full 38/76-digit precision, and a Decimal context precision
of 6.
- 16 existing converter, update, and Decimal expression tests passed.
- Repository Ruff formatting and lint passed; formatting left all 156
files unchanged.
- A native round-trip check confirmed that an unquoted SQL number rounds
a high-precision value while the quoted Decimal update preserves it
exactly.
The full suite, other Python versions, and remote API were not
exercised.
AI assistance: Codex helped implement and validate the Decimal
conversion and its regression tests.
add to table stats the number of rows that has
been marked for deletion, but not yet deleted.
note that when a table fragment is removed due to
deletion of all of its rows, these rows are not counted. since the main
use of this feature is to determine when table compaction is needed, the
fact that these rows are not reported should not impact that decision
(deleted table fragments don't need compaction).
Fixes: https://github.com/lancedb/lancedb/issues/3761
Co-authored-by: Claude Opus 5.0 <noreply@anthropic.com>
Refs #4404.
AsyncTable.list_versions split the nanosecond timestamp with float
division/modulo (`ts // 1e9`, `ts % 1e9`).
At ~1.8e18 a float can't hold every integer, so the microseconds drifted
(.274000 came back as .273999).
Use `divmod(ts, 1_000_000_000)` and integer // 1000 instead.
Only the precision fix is included here. Timezone-aware UTC (option 1 in
#4404) is on hold for now,
and I can add it here or in a follow-up depending on what they prefer.
Added test_list_versions_timestamp_precision using the value from the
issue (1790989429274000000), expecting microsecond == 274000.
Empty list and NumPy vector queries were validated too late: local
search inferred a vector column first, producing misleading schema
errors, while remote search indexed `vector[0]` and raised `IndexError`.
Add a shared empty-vector check before column inference and query
creation, and validate direct vector-builder inputs. Local and remote
sync search now raise `ValueError: Query vector must not be empty` for
these inputs.
Validation:
- All 26 new regression cases failed on the original code and pass with
the fix, covering list/NumPy inputs, automatic/explicit vector search,
inferred/explicit column names, one/two local vector columns, and remote
search before query submission.
- After merging current `main`, all 32 focused cases passed, covering
these regressions and upstream async searches without a query.
- The query, table, remote, utility, hybrid-query, and database suites
passed in the rebuilt uv environment: 541 passed, 1 skipped, 1 slow test
deselected.
- `ruff check .`, `ruff format .`, and `cargo fmt --all` passed from the
repository root.
Fixes#4389
<!-- lance-gatekeeper-fix:v1 agent=f1f4875090ef391bb1aa912dd54026dc
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4390
Lance accepts column alterations with every change field unset, so a
path-only Python call could commit a new local version without changing
the schema. The Python binding also silently discarded unknown keys such
as `nulable`.
Validate every alteration in the shared Rust `Table` API before
dispatching to either backend, and reject unknown dictionary keys in the
Python binding while preserving the legacy `name` alias. Invalid batches
leave local schema and version unchanged and send no remote request.
Validation:
- Reproduced both reported calls on the original extension: versions
advanced from 1 to 2 to 3 with an unchanged schema; all 13 new
invalid-input Python cases failed before the fix.
- Rebuilt the extension with `uv run --extra tests --extra dev maturin
develop --extras tests,dev`; all 18 focused Python cases pass across
sync/async and local/mocked remote tables.
- `cargo test --quiet --features remote -p lancedb --lib alter_column`:
all eight tests pass, including both new regressions.
- `cargo clippy --quiet --features remote --tests --examples` passes
across the workspace.
- `cargo fmt --all`, `ruff check .`, and `ruff format .` pass.
- `typos` 1.26.0 passes across the repository; intentional misspellings
in regression fixtures use the existing line-specific spelling
exemptions.
<!-- lance-gatekeeper-fix:v1 agent=36a547cd11108898893456042e38628b
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Fixes#4383
`LanceTable.migrate_v2_manifest_paths()` raised `AttributeError` because
it called a nonexistent `AsyncTable.migrate_v2_manifest_paths()` method.
Delegate to the existing async method, `migrate_manifest_paths_v2()`,
while preserving the public sync API.
Add a local regression test for tables starting with either v1 or v2
manifest paths. It checks repeated migration, v2 manifest filenames on
disk, and preservation of data and version history after reopening the
table.
Validation:
- Both new regression cases failed with the reported `AttributeError`
before the fix.
- `cd python && uv run --extra tests pytest
python/tests/test_table.py::test_migrate_v2_manifest_paths
python/tests/test_db.py::test_create_table_v2_manifest_paths_async -q` —
3 passed.
- Repository-wide `ruff check .` and `ruff format .` passed using Ruff
0.15.20 from `python/uv.lock`.
- `git diff --check` passed.
<!-- lance-gatekeeper-fix:v1 agent=bb500f0d1ddefeba3c7d7d8390a2fb74
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Why
`optimize(cleanup_older_than=timedelta(0))` deletes the data files of
every version except the latest. Any other handle still on an older
version then fails with a missing-file error, and nothing tells the
caller this can happen. #2470 asks for a docs note and a runtime
warning.
Fixes#2470
## Scope
- `AsyncTable.optimize` emits a `UserWarning` when `cleanup_older_than`
rounds to 0 ms or less. `LanceTable.optimize` runs through it, so sync
callers get the warning too.
- The `cleanup_older_than` docstring on `Table.optimize`,
`LanceTable.optimize`, and `AsyncTable.optimize` gets a `.. warning::`
block, in the same style as the existing `delete_unverified` one.
- Removed the function-local `import warnings` in `AsyncTable.optimize`.
`warnings` is already imported at module level, and the local import
would shadow it for the new check (`UnboundLocalError`).
- `test_optimize` and `test_optimize_delete_unverified` intentionally
use a 0 cleanup, so they now assert the warning with `pytest.warns`.
Nodejs `cleanupOlderThan` is out of scope, since the issue is filed
against the Python SDK.
## Tradeoffs
Like the existing `retrain` deprecation warning, the warning is raised
inside the async coroutine. For sync callers it is attributed to
`asyncio/events.py` rather than the caller's line. Warning from both the
sync and async entry points would duplicate the check for a small gain.
## Verification
- Repro: a second handle checked out at an older version, then
`optimize(cleanup_older_than=timedelta(0))` on the first. The second
handle's read fails with `LanceError(IO) ... Not found:
.../data/<file>.lance`. Before this change no warning is emitted. After
it, the `UserWarning` fires.
- New tests `test_optimize_warns_on_zero_cleanup` (async, and checks
that `timedelta(days=1)` and the default don't warn) and
`test_optimize_warns_on_zero_cleanup_sync` fail on main (`DID NOT WARN`)
and pass with the fix. The failing tests are in their own commit before
the fix.
- `pytest python/tests/test_table.py -m "not slow and not s3_test"`: all
pass except `test_merge_insert`. That test's `timeout=timedelta(0)`
assertion fails the same way on unmodified main on my machine.
- `ruff format --check` and `ruff check` are clean on the changed files.
Tests ran against the source tree with the `_lancedb` extension from the
matching `0.40.0b11` wheel. The change touches no Rust.
This PR was written by an AI coding agent (Claude Code) and reviewed
before submission.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Jay Zhou <190699450+jayzhou2309@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary
Closes: #4060
- Convert Python dict/list objects to JSON strings during ingestion.
- Support JSON fields nested inside structs and lists.
- Cover `add` and `merge_insert` JSON ingestion paths.
- Add regression coverage for nested struct and list JSON fields.
## Testing
- `git diff --check`
- Python compilation passed.
- Focused pytest was attempted but could not complete because the native
extension build stalled during `uv` bootstrap.
A refresh running under a skip policy records each row it skipped, with
the failing input and the error, but the client could not read that
store: the server exposes it over SQL and, since recently, a REST route.
A user who hit per-row failures still had to open a SQL session.
`Table::function_errors` calls the route. The listing is table-addressed
with optional job and column filters, the same addressing the SQL
surface uses, so the two cannot disagree about what a table's errors
are. The two non-record signals come back as their own fields rather
than as rows: capped-fragment summaries, and whether the listing stopped
at its limit. Local tables refuse rather than answer with an empty list.
Python and Node expose the same call, with the same optional filters.
Two ways of writing to a `json` column failed or silently corrupted
data.
**All-null batches were rejected.** `add()` refused a batch whose values
for a `json` column were all null, while every plain Arrow type accepted
the same batch. This bites row-at-a-time inserts hardest: a one-row
batch with no value for an optional column is trivially all-null, so
most such writes failed. pyarrow infers `null` as the column's type, and
the write path had no handling for it — casting to the table's type
dropped the field metadata that identifies the column as `lance.json`,
so lance rejected the batch (`` `val` should have type json but type was
large_binary ``). A null-typed input column now becomes typed nulls
matching the table's field exactly, metadata included.
**Unlabelled JSON text was stored raw.** JSON supplied as plain strings
(what pyarrow infers for a column of `str`) was cast to the column's
`LargeBinary` storage type and relabelled `lance.json`, putting unparsed
text where JSONB was expected. Reads returned the text unnormalized and
`json_extract` failed with `InvalidJsonb`. Lance-core does the JSONB
encoding, but only for input labelled `arrow.json`, so string input is
now labelled rather than cast — at the top level and inside structs.
Both fixes are in the shared Rust write path, so they apply to any
binding, including hand-built Arrow tables that never pass through
Python's list-of-dicts type inference. `_align_field` gets the same
JSON-string fix for the legacy Python `_sanitize_data` path, which
`on_bad_vectors` and embedding functions still route through.
The blob v2 half of the issue landed separately in #4065, which added a
`DataType::Null` arm to blob coercion. This PR keeps that implementation
and adds end-to-end add-path coverage for it.
The tests from #4066 are included here and pass, so that PR's
Python-layer inference changes are no longer needed to close the issue.
Fixes#3759
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
refresh_column fills nulls, so once a row has a value nothing revisits
it: an update to one of its inputs, or a definition change, leaves the
computed value stale for good.
This stamps the column's field metadata with the definition it was
computed under and a per-fragment signature of the input storage it was
read from (input data files and overlays; not the deletion file, since a
delete changes no surviving value). A refresh recomputes every live row
of a fragment whose stamp disagrees with the manifest, then records what
it computed from in a second commit after the fill. A compacted fragment
inherits freshness through the Rewrite lineage when every fragment it
was built from was signed, or was appended since the stamp, never had an
input moved, and left its rows of the product unfilled (a raw append may
supply a value; the product's data is the evidence, and the null fill
covers those rows); otherwise it recomputes. A column declared before
the stamps existed keeps the null-fill contract on its first refresh,
which enrolls it as it stood.
The map is one entry per fragment per column, so it is kept out of the
manifest: each stamp writes an immutable sidecar under `_computed/`,
named by its content digest, and the field metadata holds the digest.
Pruning old versions also drops the sidecars no remaining version
references, keeping any younger than seven days as lance keeps
unverified files, since a sidecar is put before the commit that
references it. The stamp commit is metadata-only, so a materialized
view's drift check treats it like the fill. The core lives in
`table::freshness` so a remote refresh can share the contract.
Adds [typos](https://github.com/crate-ci/typos) as a CI check and
pre-commit hook, the same way Lance does it, so misspellings like the
ones fixed in #4146 get caught automatically going forward.
This also fixes the misspellings `typos` found across the repo (Rust,
Python, TypeScript source, comments, and generated docs), and adds a
small `.typos.toml` with `extend-words` entries for terms that are
correct but look like typos: `AKS` (Azure Kubernetes Service), `RabitQ`
(a real quantization algorithm name), `mmaped` (the actual name of a
`candle-core` API we call), and `Writeable` (from Python's
`_typeshed.WriteableBuffer`). Third-party license files are excluded.
Fixes#4147
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Updates the Rust workspace Lance dependencies, Cargo lockfile, and Java
lance-core from v12.0.0-beta.14 to
[v12.0.0-beta.15](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.15).
No compatibility fixes were required; `cargo clippy --quiet --workspace
--tests --all-features -- -D warnings`, `cargo fmt --all --quiet`, and
`git diff --check` passed.
---------
Co-authored-by: Jack Ye <yezhaoqin@gmail.com>
Merge insert has always taken a list of columns to match on, and local
tables have always joined on all of them. Remote tables did not: any
list longer than one was rejected with `MergeInsertBuilder only supports
a single 'on' column`, so a composite-key upsert was impossible against
LanceDB Cloud and Enterprise from Rust, Python or TypeScript.
The remote request now carries `on` as a list and sends it as one
repeated query parameter per column — `?on=shard_key&on=id`. That is how
the lance-namespace spec encodes an array-valued `on`, so the server
receives a composite key in the shape it expects. A single column still
serializes to `?on=id`, exactly what clients sent before, so existing
callers are unaffected. A column repeated within `on` is now rejected
client-side rather than sent for the server to reject with a 400.
No binding changes were needed: `Table.merge_insert` in Python and
`Table.mergeInsert` in TypeScript already accepted a list, it just could
not reach a remote table. Both gain a test for composite keys, and the
doc comments now say what passing several columns means.
Part of
[ENT-2084](https://linear.app/lancedb/issue/ENT-2084/mergeinsertintotablerequest-support-multiple-columns-for-the).
## Example
```python
table.merge_insert(["shard_key", "id"]) \
.when_matched_update_all() \
.when_not_matched_insert_all() \
.execute(new_data)
```
A row whose `id` matches an existing row but whose `shard_key` differs
is an insert, not an update.
## Not included
Java. Java callers reach merge insert through
`org.lance.namespace.LanceNamespace`, whose
`MergeInsertIntoTableRequest.on` is a single string until
lance-namespace 0.12
([lance-namespace#363](https://github.com/lance-format/lance-namespace/pull/363),
[lance#8915](https://github.com/lance-format/lance/pull/8915)). There is
nothing in this repo's Java SDK to change until the `lance-core` pin can
move.
Sending more than one column requires a server that accepts the repeated
parameter ([sophon#7571](https://github.com/lancedb/sophon/pull/7571));
an older server returns a 400 rather than silently merging on one
column.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Computed-column planning currently sees Blob v2 storage descriptors, so
expressions cannot consume payload bytes or preserve Blob semantics in
their outputs.
A computed declaration now derives its output field from its expression.
A direct projection of a Blob v2 field inherits the source field's Blob
metadata; other expressions retain their ordinary Arrow-inferred type.
Declarations remain ordered, so the same rule applies across chained
projections.
Refresh materializes referenced Blob inputs as `LargeBinary` payload
bytes and publishes inherited Blob outputs through Lance's Blob
conversion path. Remote requests remain within the shared namespace
contract as `{name, computed}`; the server planner is being updated in
tandem to implement the same Blob-aware planning semantics, and remote
enablement must be aligned with that server rollout.
The existing null-as-unfilled contract remains unchanged. Row-level
freshness and cell flags remain follow-up work.
## Summary
- allow Python sync, async, and remote table updates to accept type-safe
`Expr` filters
- serialize expression filters before invoking the existing update
implementation
- cover numeric-looking text and apostrophe-containing text in sync and
async regression tests
## Root cause
`Table.update` was the remaining Python write path that required callers
to construct a raw SQL predicate. Dynamic text interpolated without SQL
literal encoding could therefore be parsed as an integer, float, or
unterminated string instead of Utf8. The expression API already encodes
literals safely for query and delete filters.
## Validation
- `cd python && .venv/bin/pytest
python/tests/test_table.py::test_update_async
python/tests/test_table.py::test_update_expr_filter_literals_async
python/tests/test_table.py::test_update
python/tests/test_table.py::test_update_expr_filter_literals -q`
- `cd python && .venv/bin/pytest python/tests/test_expr.py -q`
- `cd python && .venv/bin/ruff format --check .`
- `cd python && .venv/bin/ruff check .`
Fixes#1869
<!-- lance-gatekeeper-fix:v1 agent=01f1e7b69c65e8b6d3b3c1e1a7918179
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Problem
`refresh_column_async` returned a unit-result job even though durable
refresh jobs carry a canonical terminal result. Python callers could not
obtain row counts or source and published versions through the public
`Job` API, and local and remote refresh jobs exposed different result
semantics.
## Behavior
`refresh_column_async` now returns `Job[RefreshColumnResult]` for local
and remote tables. The general typed-job bridge binds each endpoint to
its public result model while preserving unit-result jobs and existing
status, wait, cancel, and timeout behavior. A local no-op refresh
reports no published version.
The Node.js API continues to resolve `wait()` as `void`; its binding
erases the Rust result type internally to preserve the existing public
contract.
## Ownership and integration boundary
LanceDB owns the language-neutral `Job<T>` contract and language-binding
decode. Sophon owns production and durable persistence of terminal
payloads. Sophon #7348 and #7378 now publish the canonical refresh
result for Function-backed and expression-backed refresh jobs,
respectively. The remote client fixture matches the merged server
schema; live deployment and end-to-end demo acceptance remain separate
rollout checks.
## Problem
Generic job result propagation exposed the PyO3 representation of Rust's
unit value as `()` in Python. Unit jobs therefore returned an empty
tuple instead of `None`, breaking the documented `Job.wait()` contract
and the Python doctest workflow.
## Behavior
Unit job completion now converts explicitly to Python `None`. Typed job
results continue to pass through unchanged, with synchronous and
asynchronous regression coverage.
<!-- lance-gatekeeper-fix:v1 agent=40e5cf476a59265c71653574eda834d2
generation=1 -->
## Summary
- preserve incoming PyArrow `arrow.json` fields while schema
sanitization aligns input to a stored `lance.json` schema
- let Lance perform the required JSONB encoding instead of relabeling
raw JSON bytes as encoded storage
- cover both merge insert and the conditional add sanitization path with
end-to-end regression tests
## Root cause
Python schema sanitization aligns incoming data to the table schema
before passing it to Lance. Merge insert always takes this path, while
add takes it conditionally for preprocessing such as non-default
bad-vector handling or embedding functions. For JSON columns, the cast
changed logical `arrow.json` strings into the table's JSONB-backed
`lance.json` storage type without encoding the bytes, so Lance treated
raw JSON text as JSONB.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_table.py
-k 'merge_insert or add_sanitization_encodes_json' -q`
- targeted schema-cast and JSON encoding tests
- `ruff check .`
- `ruff format --check python/python/lancedb/table.py
python/python/tests/test_table.py`
Fixes#3923
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Mirrors create_index's dual surface: the blocking refresh_column keeps
returning {rows_filled, version}, and refresh_column_async returns the
same
Job handle create_index uses, running the refresh as an in-process task.
Invalid input is reported by the submitting call rather than by the job.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
add_columns().computed("doubled", "x * 2") stores the expression in
field
metadata and commits the column empty; a later refresh fills it. Type
and
inputs are derived from the expression.
The declaration stays authoritative for its lifetime: writes that would
give
the column a value (append, update, merge, SQL insert), schema changes
that
would break the stored expression or reshape its output, metadata edits,
volatile expressions, declaration metadata arriving through any path but
the
validated declare call, and LSM write specs in either order against
latest
committed state are all refused. The LSM check also refuses on the
mem-wal
catch-up feature flag, which outlives unset and marks retained SSTable
rows.
Simultaneous declare/install interleavings conflict at commit via
lance's
mem-wal rule (lance#8539). Local tables only.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
Some issues:
- file_size_bytes is optional in the manifest, so if it's not there (old
writer I guess) it'll under-report the table size.
- it changes results a little bit from the old way by including per-file
footers and metadata (probably not a big difference at real scale)
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
## Summary
- add an end-to-end regression for indexed vector search after merging a
pandas column
- verify unmatched rows retain a null merged value instead of failing
Arrow batch assembly
## Root cause
Historical Lance readers could assemble schema-evolved columns in
physical data-file order. Indexed row-ID reads after a merge could
therefore omit or misorder the newly merged column for unmatched rows.
The currently pinned Lance release contains the reader correction, but
LanceDB did not cover the reported merge-then-search path.
## Validation
- uv run --extra tests pytest python/tests/test_table.py::test_merge
python/tests/test_table.py::test_search_after_merge -q
- uv run --project python --extra dev ruff check .
- uv run --project python --extra dev ruff format --check
python/python/tests/test_table.py
Fixes#599
<!-- lance-gatekeeper-fix:v1 agent=4e17331e0542c132eae31e86da508629
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Root cause
`Table.to_polars()` disabled PyArrow predicate pushdown by selecting the
non-PyArrow Polars scan callback. Polars 1.32.3 invokes that callback
with `batch_size` both positionally and through its partial, so
collecting the returned lazy frame raises `TypeError:
_scan_pyarrow_dataset_impl() got multiple values for argument
batch_size`.
## Fix
- Keep the compatible PyArrow callback path.
- Add an identity `map_batches` barrier so predicates stay in Polars
instead of reaching the LanceDB adapter as unsupported PyArrow
expressions.
- Extend the tested Polars range through 1.32.3 and retain lazy-frame
regression coverage.
## Validation
- `python/tests/test_table.py::test_polars` with Polars 1.32.3
- `python/tests/test_table.py::test_polars` with the locked Polars 1.3.0
baseline
- `ruff format --check` on the changed Python files
- `ruff check .`
- `uv lock --check`
Fixes#2619
<!-- lance-gatekeeper-fix:v1 agent=0d42bcda944ac42765b25f2c19ff729f
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- exercise float16 sanitization through the reported direct Arrow-data
table creation path
- assert that the inferred fixed-size vector schema remains float16
- retain end-to-end index creation and vector search coverage
## Root cause and fix
PyArrow 16 does not provide an is_nan kernel for half-float arrays, so
passing float16 vector values directly to that kernel raises
ArrowNotImplementedError. LanceDB's sanitizer already carries the
compatibility fix from #837: it casts float16 values to float32 only for
NaN detection while preserving the stored vector type.
The existing end-to-end regression created an empty schema-defined table
and added data afterward. This change aligns that regression with the
issue reproduction by creating a table directly from a
FixedSizeList<float16> Arrow table and verifying the persisted schema.
## Validation
- uv run --extra tests pytest
python/tests/test_table.py::test_create_f16_table_from_arrow_data -q
- direct 1,000-row by 128-dimension float16 Arrow-table reproduction
- PyArrow 16.1 half-float is_nan kernel reproduction
- uvx ruff@0.15.20 format --check python/python/tests/test_table.py
- uvx ruff@0.15.20 check .
Fixes#835
<!-- lance-gatekeeper-fix:v1 agent=dd0a32a959f691f49de958d4333fb29d
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Srijan Keshri
lancedb-gatefixer[bot]
TaylorandBrendan Clement
Bruno Ramirez
Will JonesandClaude Opus 5.5
Xuanwo
LeonSGPandXuanwo
Avanish Garg
adenzhou1350
Madan Kumar
Zhewen TanandXuanwo
Jack Ye
Kevin Dong
Yuval LifshitzandClaude Opus 5.0
Aviral Saxena
Rudra Prasad Bhuyan
Wyatt Alt
Dan TasseandWill Jones