Function applications from the canonical remote contract cannot
currently declare scalar or grouped computed-column outputs atomically.
This adds the remote-only declaration contract for scalar,
struct-as-one-column, and expanded named-struct outputs. It validates
result mappings, fixes exact input/output Arrow schemas in the request,
persists grouped sibling metadata, and keeps local Function execution
unsupported. Unknown newer application or binding metadata remains
readable, while schema-changing mutations fail closed instead of
rewriting it.
Stable Lance field IDs are deliberately not a declaration prerequisite
in this slice. Inputs bind by parameter name and field path; Sophon
remains responsible for exact-version validation, atomic all-NULL
sibling creation, binding identity and revision allocation, and
persisted output identities.
LanceDB's Python SDK now requires Pydantic `>=2.7.4,<3` and uses the v2
APIs throughout. This removes dual-version behavior from schema
conversion, query serialization, embedding models, and Function wire
models while preserving their existing public and canonical-wire
behavior.
The minimum-dependencies CI job pins Pydantic 2.7.4 so the declared
compatibility floor remains covered.
PyO3 exposed `LsmWriteSpec` with its default `builtins` module, causing
mkdocstrings to resolve the public `lancedb.LsmWriteSpec` re-export as
`builtins.LsmWriteSpec` and fail the documentation build. Declare the
native extension module and pin the public re-export with a regression
test.
This also applies the repository's current Ruff formatter to seven
previously unformatted Python scripts.
## Problem
Enterprise Function-backed computed columns need a stable SDK contract
before Sophon catalog and execution endpoints can be added. The existing
`Job` API can only represent unit terminal results, and there is no
shared Rust/Python wire definition for immutable Function versions,
applications, bindings, or refresh results.
## Behavior
This introduces remote-only canonical Function values in Rust and
Python, evolves `Job<T = ()>` to decode typed remote terminal results
while keeping local spawned operations unit-typed, and fixes the
cross-language contract with shared JSON golden fixtures. Unknown fields
and discriminator values remain forward-decodable, while canonical
output contains only fields known to the client. Function models contain
secret names only.
Sophon remains the sole owner of catalog persistence, environment bake,
secret resolution, execution, and publication. This PR does not add
authoring/catalog endpoints, local execution, refresh runners, or live
Sophon E2E coverage.
<!-- lance-gatekeeper-fix:v1 agent=40e5cf476a59265c71653574eda834d2
generation=1 -->
## Summary
- preserve incoming PyArrow `arrow.json` fields while schema
sanitization aligns input to a stored `lance.json` schema
- let Lance perform the required JSONB encoding instead of relabeling
raw JSON bytes as encoded storage
- cover both merge insert and the conditional add sanitization path with
end-to-end regression tests
## Root cause
Python schema sanitization aligns incoming data to the table schema
before passing it to Lance. Merge insert always takes this path, while
add takes it conditionally for preprocessing such as non-default
bad-vector handling or embedding functions. For JSON columns, the cast
changed logical `arrow.json` strings into the table's JSONB-backed
`lance.json` storage type without encoding the bytes, so Lance treated
raw JSON text as JSONB.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_table.py
-k 'merge_insert or add_sanitization_encodes_json' -q`
- targeted schema-cast and JSON encoding tests
- `ruff check .`
- `ruff format --check python/python/lancedb/table.py
python/python/tests/test_table.py`
Fixes#3923
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Why
Four of the eight LSM methods are **remote-only in the core**. `impl
BaseTable for NativeTable` implements only
`set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`,
`compact_lsm` and `get_lsm_stats` fall through to trait defaults
returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and
`checkpoint_lsm` is built on all three.
That explains the state of the bindings: Node had bound the four that
work against a local table and stopped, so a Cloud user could install an
LSM write spec but had no way to observe fresh-tier state or drive a
checkpoint. Java had none of it at all.
| SDK | set/unset/get spec | closeWriters | flush | compact | getStats |
checkpoint |
|---|---|---|---|---|---|---|
| Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Node *(before)* | ✅ | ✅ | — | — | — | — |
| **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** |
| Java *(before)* | — | — | — | — | — | — |
| **Java (after)** | **new** | n/a | **new** | **new** | **new** |
**new** |
Go and C are separate repos and are out of scope here. `closeLsmWriters`
drains cached in-process shard writers, so it has no meaning for Java,
which is a pure REST client.
## Node
Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and
`getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats`
/ `MemtableStats` objects — typed rather than a JSON blob, matching the
existing `LsmWriteSpec` object in the same file, with `u64` cast to
`i64` per that file's convention.
Because these four are remote-only, the new tests assert each binding
reaches the core and surfaces `NotSupported` against a local table. That
covers the wiring; behavior against a real endpoint stays covered by the
mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`.
## Python
No new methods. All eight are on `LanceTable`, `AsyncTable` and
`RemoteTable` — the last four landed on the sync `RemoteTable` in #3961,
which is merged into this branch.
What was missing here was reachability. `LsmWriteSpec` was importable
only from the private `lancedb._lancedb`, appearing in `table.py` solely
under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no
mention of it, which per the repo's docs guidance means it rendered
nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in
`__all__`, and documented.
## Java
Java reaches LanceDB purely over REST through the generated Lance
Namespace client, and these routes are not in that spec, so they are
issued through a small dedicated client rather than added to the spec.
That call is revisitable — LSM is one of four unspecified route families
alongside `multipart_write`, `page_cache/prewarm` and
`branches/diff|merge`. If those are ever regularized into the spec as a
group, `LanceDbTableLsm` is one file that gets deleted.
`LsmWriteSpec` here is deliberately **not**
`org.lance.memwal.InitializeMemWalParams`. That type defaults to
maintaining *no* indexes where a spec here defaults to maintaining
*every* index, and it cannot express the `null` that asks the server to
resolve the set:
| Value | On the wire | Meaning |
|---|---|---|
| unset (null) | `null` | Server resolves **every** maintainable index |
| `Collections.emptyList()` | `[]` | Maintain **none** |
| `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those |
A dedicated test pins null and `[]` as distinct on the wire, since
collapsing them is the failure mode that motivated a LanceDB-owned type.
`checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs`
with its constants and status semantics intact: 429/503 retried in place
against an 8-budget, 421 restarting from flush against a 3-budget, 5s
poll, and a target watermark fixed after the seal so it terminates under
write load.
`getLsmStats` returns typed `LsmStats` / `BucketStats` /
`GenerationStats` / `MemtableStats`, mirroring the Rust structs in
`rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes.
Decoding is strict — see below.
## Review feedback
Both gatekeeper findings were real. Each was reproduced against the
scripted test server first, and each fix ships with the reproducer as a
regression test.
**The transport was doubling every checkpoint retry budget.**
`HttpClients.createDefault()` installs Apache's default response retry
strategy, whose retryable-status list is exactly 429 and 503 — the two
statuses `isRetryable` owns. A 429 held against `flush_lsm` issued
**18** wire requests where the loop intends 9, and `compact_lsm` was
retried in place despite the loop being built to fall through to a fresh
stats poll instead. Timing confirmed the mechanism: that run took 25.4s
≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry
interval.
Automatic retries are now disabled, so the checkpoint loop is the sole
owner of the 421/429/503 transitions. A side effect worth noting:
`testCheckpointRetriesRetryableStatusInPlace` was passing on a
transport-absorbed 429 and never reaching `issue()`'s retry branch at
all. It now exercises the real path.
**Stats decoding failed open.** `getLsmStats` read the response with
Jackson's `path()`, which yields a missing node that iterates as an
empty array — making "malformed" indistinguishable from "no buckets",
which is indistinguishable from "drained". Four separate payloads made
`checkpointLsm()` report convergence for a checkpoint that never ran:
| Response | Before | Now |
|---|---|---|
| `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ |
| `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` |
| empty response body | **reported success** | `IllegalStateException` |
| bucket missing required fields | **reported success** |
`IllegalStateException` |
The empty-body row is the one to weight: a proxy 200 with no body is a
realistic production event, and it silently reported a checkpoint that
never happened.
Decoding is now strict and fails closed, matching the serde contract on
the Rust side exactly. One deliberate deviation from the review comment,
which asked that *only* explicit JSON `null` count as disabled: Rust has
`#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to
`None` there too. Java now matches that. It is an absent-or-malformed
**`buckets`** that fails closed, which is the case the comment was
actually protecting.
## Testing
- Java: **33 passing** (8 existing + 25 LSM) against a scripted
`com.sun.net.httpserver.HttpServer` — no new test dependency. Wire
assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`;
checkpoint tests cover convergence, not piling onto a latched bucket,
421 restart-from-flush, 429 retry-in-place, terminal-status propagation,
reissue exhaustion, the exact wire-request count against the retry
budget, and five malformed stats payloads.
- Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm
run tsc`, `npm run lint`, `npm run docs` all clean.
- Python: `ruff format --check` and `ruff check` clean.
- Java formatting: `./mvnw -pl lancedb-core spotless:apply` and
`spotless:check` both clean under a JDK 11 toolchain.
## Note: spotless needs a pre-16 JDK
`./mvnw spotless:apply` fails on JDK 16+ with
`JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7,
pinned at `java/pom.xml:34`, predates JDK 16's compiler API change.
**This is pre-existing** and reproduces on a pristine `main` checkout.
It is not a blocker, just a toolchain requirement. Spotless was run
against these sources under JDK 11 and both `spotless:apply` and
`spotless:check` pass on the whole module:
```shell
JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply
```
Bumping the plugin so it works on modern JDKs is still worth doing, but
separately from this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
The sync `RemoteTable` carried `set_lsm_write_spec`,
`unset_lsm_write_spec`, `get_lsm_write_spec`, and `close_lsm_writers`,
but not `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, or
`get_lsm_stats`.
That left the four LSM control methods reachable from `AsyncTable` only.
They are also the four that *only* work against a remote table —
`NativeTable` does not override the `BaseTable` defaults, so on a local
table they return `NotSupported` (`rust/lancedb/src/table.rs:679-701`).
The net effect for sync users:
| | `checkpoint_lsm` / `get_lsm_stats` |
|---|---|
| `LanceTable` (sync, local) | present, but always `NotSupported` |
| `RemoteTable` (sync, remote) | `AttributeError` — method absent |
| `AsyncTable` (remote) | works |
So there was no working sync path at all, despite the Rust `RemoteTable`
implementing every one of these against real endpoints.
## Changes
* Add `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, and `get_lsm_stats`
to `lancedb.remote.table.RemoteTable`, mirroring the delegation style of
their neighbours.
* Correct the docstrings on `set_lsm_write_spec` /
`unset_lsm_write_spec`, which read `"""Not supported on LanceDB
Cloud."""` although `rust/lancedb/src/remote/table.rs:2549-2601`
implements both against `/v1/table/{}/set_lsm_write_spec/` and
`/unset_lsm_write_spec/`. They appear to have been copy-pasted from
`set_unenforced_primary_key` directly above.
No Rust or PyO3 changes — the bindings and the `AsyncTable` methods
already existed. The `Table` ABC is left alone, matching how the
existing `*_lsm_write_spec` methods are declared on the concrete classes
only.
## Tests
Four new tests in `python/python/tests/test_remote_db.py`, against the
existing mock HTTP server:
* `test_get_lsm_stats_sync` — the server payload round-trips into the
dict, and `include_generation_rows` defaults to `False` and is forwarded
when set.
* `test_get_lsm_stats_sync_returns_none_when_lsm_disabled` — a
`{"lsm_stats": null}` envelope yields `None` rather than an error.
* `test_flush_and_compact_lsm_sync` — both are one-shot POSTs answered
`202` with no body.
* `test_checkpoint_lsm_sync` — pins the binding to the endpoints it
drives (`flush_lsm` then `get_lsm_stats`); the convergence loop itself
is already covered in Rust.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
`LanceHybridQueryBuilder` (sync hybrid search,
`table.search(query_type="hybrid")`) silently ignored `.offset()`.
`self._offset` was never forwarded to the vector/FTS sub-queries and
never applied when slicing the final combined/reranked result, so
`.offset(N)` behaved identically to `.offset(0)` — no error, just wrong
pagination.
Fixes#3765
## Changes
- `_create_query_builders()`: each sub-query now fetches `limit +
offset` rows so there's enough data to slice the correct window out of
after combining/reranking.
- `_combine_hybrid_results()` / `to_arrow()`: the final table is sliced
with `offset=self._offset` instead of always starting at 0.
## Test plan
- [x] New regression test `test_hybrid_query_offset` in
`python/python/tests/test_hybrid_query.py`
- [x] `uv run --extra tests pytest python/tests/test_hybrid_query.py
-vv` — 13 passed
- [x] `uv run --extra dev ruff format` / `ruff check` — clean
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
## Summary
Fixes naive `lit(datetime)` equality filters against table timestamp
columns on non-UTC hosts, and adds the integration matrix from #3262.
## Failure (before)
On a machine in US Eastern (UTC−4 / EDT), with PyPI `lancedb==0.36.0`:
```python
from datetime import datetime
import lancedb
from lancedb.expr import col, lit
db = lancedb.connect("memory://")
ts = datetime(2024, 7, 1, 10, 0, 0) # naive
table = db.create_table("t", [{"id": 1, "ts": ts}])
rows = table.search().where(col("ts") == lit(ts)).to_list()
# actual: [] (0 rows)
# expected: 1 row
```
### Root cause
In `python/src/expr.rs`, `expr_lit` converted every `datetime` via
Python's `.timestamp()`:
- **naive** `.timestamp()` = local wall → UTC epoch (shifted by host
offset)
- **PyArrow naive** storage = UTC wall-clock microseconds (no local
shift)
So `lit(naive)` became `CAST('2024-07-01 14:00:00' AS TIMESTAMP)` on EDT
while the table held `10:00:00`.
## After
Naive datetimes are interpreted as UTC wall clock
(`replace(tzinfo=timezone.utc).timestamp()`), matching Arrow storage.
Aware datetimes still use `.timestamp()` (correct epoch).
Same repro on this branch: **1 matching row**.
## Tests
Added `TestExprDatetimeTimezoneIntegration` covering:
| Case | Result |
|------|--------|
| both naive | match |
| both same TZ (UTC) | match |
| different TZs, same instant | match |
| table TZ + naive lit | match (wall clock) |
| table naive + aware lit | match |
| naive lit SQL is wall clock, not local-shifted | asserts `10:00:00` in
SQL |
### Verification
```bash
cd python
maturin develop
pytest python/tests/test_expr.py -v
```
**102 passed** (full `test_expr.py`, including the 6 new cases).
Closes#3262
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
LanceDB Cloud and Enterprise support computed columns through the REST
API,
so declaration dispatches per backend: local tables plan the expression
themselves, remote ones send {name, computed} entries for the server to
plan. A remote refresh is the server's backfill job --
refresh_column_async
submits it and returns a handle whose successful wait establishes a
read-freshness baseline on the submitting handle, unless a checkout has
pinned the handle by the time the job completes; the blocking form
refuses
rather than invent a fill count the server does not report.
Declaration entries are built from the namespace client's
AddColumnsEntry
model (lance-namespace 0.11.0, via the lance beta.13 pin), so the
payload
shape is compile-checked against the published contract.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
Mirrors create_index's dual surface: the blocking refresh_column keeps
returning {rows_filled, version}, and refresh_column_async returns the
same
Job handle create_index uses, running the refresh as an in-process task.
Invalid input is reported by the submitting call rather than by the job.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
add_columns().computed("doubled", "x * 2") stores the expression in
field
metadata and commits the column empty; a later refresh fills it. Type
and
inputs are derived from the expression.
The declaration stays authoritative for its lifetime: writes that would
give
the column a value (append, update, merge, SQL insert), schema changes
that
would break the stored expression or reshape its output, metadata edits,
volatile expressions, declaration metadata arriving through any path but
the
validated declare call, and LSM write specs in either order against
latest
committed state are all refused. The LSM check also refuses on the
mem-wal
catch-up feature flag, which outlives unset and marks retained SSTable
rows.
Simultaneous declare/install interleavings conflict at commit via
lance's
mem-wal rule (lance#8539). Local tables only.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
## Summary
- add `drop_table_async` and return a job handle while preserving
`drop_table`
- consume remote 202 responses with cleanup job IDs and retain
older-server compatibility
- expose the API through Python and TypeScript connection wrappers
Closes#3704
## Problem
Transforms can fail on bad data (e.g. nulls/NaNs from incomplete user
surveys). Today any transform exception aborts iteration, and there is
no way to skip invalid rows during loading.
## Solution
New `on_transform_error` parameter on `StreamingDataset`:
- `"raise"` (default, matches current behavior and the convention in
tf.data / WebDataset / Ray Data)
- `"skip"` — drop the failing rows and continue
- `"warn"` — like skip, plus a logged warning per failing batch
- a WebDataset-style callable `handler(exc) -> bool`, so users can skip
only expected error types
Key design points:
- **Row-granular skipping**: when a batch fails, the transform is re-run
on single-row slices so only the rows that actually fail are dropped
(avoids Ray-style whole-block loss). Skips are counted in a new
`rows_skipped` property.
- **No crash on uneven skips**: the round-robin loop now ends the epoch
at the last cycle where every split still has a row, instead of hitting
`IndexError` when a split runs dry early.
- **Exact resumability under skips**: checkpoints are now
position-based. `state_dict` gains `positions_consumed_per_split` (exact
for owned splits), and a new `merge_state_dicts` static method combines
per-rank states via elementwise max for elastic resume across topology
changes. Old checkpoints without the new key still load. Positions equal
sample counts when nothing is skipped, so existing behavior is
unchanged.
- **Guardrail**: transforms returning the wrong number of rows now raise
a clear `ValueError` instead of silently corrupting split accounting.
### Answers to the issue's open questions
- *Can we do this?* Yes — all transforms funnel through one guarded call
in the Stage 2 pipeline.
- *What do other libraries do?* tf.data `ignore_errors()`, WebDataset
`handler=`, Ray `max_errored_blocks`; MosaicML StreamingDataset offers
nothing (skipping conflicts with its determinism model). This design
follows the common conventions: raise by default, opt-in skipping,
count/log drops.
- *Error handling or pre-filtering?* Both: the existing `filter=`
remains the recommended tool for predictable bad data (splits are built
post-filter, so all guarantees hold — now documented);
`on_transform_error` covers failures not expressible as a predicate.
- *Impact on splits / elastic determinism?* Per-split sample sequences
stay deterministic (skips are data-dependent, not topology-dependent).
With unequal bad-row counts across splits the last few global steps of
an epoch can differ across topologies (bounded by the skew), which is
documented on the parameter. With equal counts per split, full
determinism is preserved — covered by a test.
## Testing
15 new tests in `test_elastic_dataloader.py` covering: default raise,
invalid values, uniform and uneven skips (including epoch-end
truncation), warn logging, selective callable handlers, wrong-row-count
guardrail, determinism across runs and across world sizes (1/2/3/4) with
skips, exact mid-epoch resume with skips on the same topology, elastic
resume via `merge_state_dicts` (ws=2 → ws=1), merge validation, and
backward-compat loading of old checkpoints.
Note: relying on CI for the test run — my local machine OOMs during the
final link of the native extension. The change itself is pure Python.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Some issues:
- file_size_bytes is optional in the manifest, so if it's not there (old
writer I guess) it'll under-report the table size.
- it changes results a little bit from the old way by including per-file
footers and metadata (probably not a big difference at real scale)
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
## What
`JinaEmbeddings._generate_image_input_dict()` crashes with
`AttributeError: 'function' object has no attribute 'urlparse'` on any
image given as a URL string, local path string, or `pathlib.Path` — i.e.
every documented `jina-clip-v1` image-embedding use case except raw
`bytes`.
## Why
```python
from urllib.parse import urlparse
...
parsed = urlparse.urlparse(image)
```
`urlparse` is imported as a function, then called as if it were the
`urllib.parse` module (`urlparse.urlparse(...)`). The module-level
`is_valid_url()` a few lines above does it correctly (`urlparse(text)`),
which is why this reads as a typo rather than intentional. Fixed to
`urlparse(str(image))` — `str()` is needed because `urlparse()` only
accepts `str`/`bytes` and raises a different `AttributeError` on a raw
`Path`.
## Testing
Added `test_jina_generate_image_input_dict_local_path`, which fails with
the original `AttributeError` before the fix and passes after, covering
both a `str` path and a `pathlib.Path`. Verified locally (built the Rust
extension, ran red→green, then the full `test_embeddings.py` file: 15
passed / 8 skipped, no regressions) and with `ruff check`/`ruff format`.
---
Disclosure: this PR was drafted with AI assistance (Claude); I reviewed,
tested, and take responsibility for the change.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
`LsmWriteSpec::maintained_indexes` becomes `Option<Vec<String>>`:
| value | meaning |
|---|---|
| `None` (new default) | every index the MemWAL supports, resolved when
the spec is installed |
| `Some([])` | maintain nothing — a scan/filter-only WAL table |
| `Some([..])` | exactly these, taken verbatim |
`with_maintained_indexes` keeps its signature;
`with_no_maintained_indexes()` is new. Surfaced through the remote path
(null on the wire), Python, and Node.
## Why
Callers had to state the maintained set by hand every time, which is
both tedious and easy to get wrong — the common case is "maintain what I
already built."
Resolution filters on `IndexConfig::is_memwal_maintainable`, delegating
to lance's `is_maintainable_index_type`. This is load-bearing rather
than cosmetic: lance does **not** skip an index type its memtable cannot
build, it errors when the shard writer opens, so sweeping up a bitmap
index would fail every memtable claim and leave the table unwritable.
The inferred set excludes those, and an explicit list naming one is now
rejected at spec time instead of at claim time.
## Behavior change
A freshly constructed spec used to maintain **nothing**; it now
maintains **everything supported**. This flipped because napi collapses
`undefined` and `null` to `None`, so TypeScript cannot express "absent
means nothing, null means all" — any other choice makes the bindings
disagree with the wire. The error direction also favors it: an unwanted
maintained index costs memory, while a silently unmaintained one
degrades FTS to an unscored scan.
Three existing tests encoded the old default and are updated rather than
worked around.
## Caveat
The resolved set is a snapshot, not a subscription. An index created
after the spec is installed is not maintained until the spec is unset
and set again. `get_lsm_write_spec` therefore always reports a concrete
list — `None` never round-trips.
## Dependency
Needs a lance release carrying `is_maintainable_index_type`
(lance-format/lance#8095) before this builds against the pinned tag.
Draft until then.
## Testing
38 Rust LSM tests and 10 Python tests pass against a local lance build,
including new coverage that a bitmap index is excluded from inference
and rejected when named, and that `[]` stays distinguishable from null
on the wire.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Converge a table's LSM write path into its base table, and inspect it.
`checkpoint_lsm` is `flush` then `compact`, repeated until the fresh
tier is empty — and the loop runs **client-side**. Putting it on the
server would mean a background task, which means a single-flight intent,
an intent that leaks on panic, a bounded-iteration policy, an "is it
done" observable, and a story for every way a client can vanish
mid-operation. None of that exists in this shape: each request does a
bounded unit of work and reports what is left, so completion is *carried
in the responses* rather than inferred from a shared counter that cannot
distinguish "converged" from "hasn't started yet".
Best-effort by construction. Nothing is frozen, so `converged` means L0
was empty as of the last pass. It is idempotent, abandonable at any
point with zero consequence, and safe to run on a cadence — an
already-converged table costs one round trip and zero compaction passes,
because `flush` reports `generations_remaining` and the loop is never
entered.
## The failure taxonomy is the load-bearing part
Five distinct conditions used to arrive at a client as one 503.
`Error::LsmRoute` carries a classification read from the response body's
namespace error code **at the point of receipt** — before any generic
helper folds the body into a string and keeps only the status.
| condition | wire | client action |
|---|---|---|
| contention (latch held / pool saturated) | 429, code 21 | retry with
backoff |
| owning node draining | 503, code 19 `InvalidTableState` | **stop** |
| fenced / no slot / transport | 503, code 17 | retry with backoff |
| registry entry vanished | 404 | re-issue from `flush` (capped) |
| table being dropped / not WAL-backed | 409 / 400 | stop |
Draining is terminal because the drain gate is a one-way latch —
retrying spins until the deadline to report a failure that was knowable
on the first response. Transport retry is disabled on these routes for
the same reason: it treats every 503 alike and would burn its budget
before the classifier ever saw the body.
`get_lsm_stats` returns `Option<LsmStats>`, matching
`get_lsm_write_spec` — `None` only when the table has no LSM write path,
since a struct of zeros would read as measurements.
Python bindings mirror all four, preserving per-bucket detail rather
than flattening to a table-level summary.
## Testing
Six new unit tests against the mocked endpoint, plus the taxonomy
round-trip:
- flush into an empty L0 issues **zero** compact calls (asserts the call
count — `generations_consumed: 0` is also true of a loop that ran a
pointless pass)
- the loop drives compact until the server reports zero remaining
- **contention is not draining**: a 429 retries and converges; asserts
the retry count
- a draining node stops after **exactly one** request, no retries
- stats round-trips fully populated; `include_generation_rows` off by
default
- every `(status, code)` pair classifies correctly, including
unparseable 503 bodies falling back to *retryable* rather than terminal
`cargo test -p lancedb --features remote --lib`: 723 passed.
## Notes for review
- Depends on the sibling lance change returning `SealedGeneration` from
`force_seal_active` only at the *server* level — no lance API is used
here.
- The branch is based on `codex/update-lance-10-0-0-beta-5`, so it
carries one extra commit (`chore: update lance dependency to
v10.0.0-beta.5`) that is not part of this change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: lancedb automation <robot@lancedb.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- add an end-to-end regression for indexed vector search after merging a
pandas column
- verify unmatched rows retain a null merged value instead of failing
Arrow batch assembly
## Root cause
Historical Lance readers could assemble schema-evolved columns in
physical data-file order. Indexed row-ID reads after a merge could
therefore omit or misorder the newly merged column for unmatched rows.
The currently pinned Lance release contains the reader correction, but
LanceDB did not cover the reported merge-then-search path.
## Validation
- uv run --extra tests pytest python/tests/test_table.py::test_merge
python/tests/test_table.py::test_search_after_merge -q
- uv run --project python --extra dev ruff check .
- uv run --project python --extra dev ruff format --check
python/python/tests/test_table.py
Fixes#599
<!-- lance-gatekeeper-fix:v1 agent=4e17331e0542c132eae31e86da508629
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Root cause
`Table.to_polars()` disabled PyArrow predicate pushdown by selecting the
non-PyArrow Polars scan callback. Polars 1.32.3 invokes that callback
with `batch_size` both positionally and through its partial, so
collecting the returned lazy frame raises `TypeError:
_scan_pyarrow_dataset_impl() got multiple values for argument
batch_size`.
## Fix
- Keep the compatible PyArrow callback path.
- Add an identity `map_batches` barrier so predicates stay in Polars
instead of reaching the LanceDB adapter as unsupported PyArrow
expressions.
- Extend the tested Polars range through 1.32.3 and retain lazy-frame
regression coverage.
## Validation
- `python/tests/test_table.py::test_polars` with Polars 1.32.3
- `python/tests/test_table.py::test_polars` with the locked Polars 1.3.0
baseline
- `ruff format --check` on the changed Python files
- `ruff check .`
- `uv lock --check`
Fixes#2619
<!-- lance-gatekeeper-fix:v1 agent=0d42bcda944ac42765b25f2c19ff729f
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- exercise float16 sanitization through the reported direct Arrow-data
table creation path
- assert that the inferred fixed-size vector schema remains float16
- retain end-to-end index creation and vector search coverage
## Root cause and fix
PyArrow 16 does not provide an is_nan kernel for half-float arrays, so
passing float16 vector values directly to that kernel raises
ArrowNotImplementedError. LanceDB's sanitizer already carries the
compatibility fix from #837: it casts float16 values to float32 only for
NaN detection while preserving the stored vector type.
The existing end-to-end regression created an empty schema-defined table
and added data afterward. This change aligns that regression with the
issue reproduction by creating a table directly from a
FixedSizeList<float16> Arrow table and verifying the persisted schema.
## Validation
- uv run --extra tests pytest
python/tests/test_table.py::test_create_f16_table_from_arrow_data -q
- direct 1,000-row by 128-dimension float16 Arrow-table reproduction
- PyArrow 16.1 half-float is_nan kernel reproduction
- uvx ruff@0.15.20 format --check python/python/tests/test_table.py
- uvx ruff@0.15.20 check .
Fixes#835
<!-- lance-gatekeeper-fix:v1 agent=dd0a32a959f691f49de958d4333fb29d
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- publish the PEP 561 `py.typed` marker so downstream type checkers
consume the inline public annotations
- add a Pyright contract test that distinguishes synchronous `connect`
from awaited `connect_async`
- verify the marker is present in the installed package
## Root cause
The public Python module already annotated `lancedb.connect` as
synchronous and `lancedb.connect_async` as asynchronous. The private
native `_lancedb.connect` stub is intentionally awaitable because it
backs `connect_async`. However, the distribution did not include a PEP
561 marker, so downstream tools such as mypy could ignore the public
inline annotations and expose misleading or incomplete type information.
## Validation
- `python/.venv/bin/ruff format --check python/python/tests/test_db.py
python/python/type_tests/connect.py`
- `python/.venv/bin/ruff check .`
- `cd python && .venv/bin/pytest
python/tests/test_db.py::test_package_includes_pep_561_marker -q`
- `cd python && .venv/bin/pyright --pythonpath .venv/bin/python`
- downstream mypy contract check for both public connection functions
Fixes#2159
<!-- lance-gatekeeper-fix:v1 agent=b07901451487187fc03f61890d3aa6bb
generation=1 -->
Co-authored-by: lancedb-gatefixer[bot] <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- raise a clear `TypeError` when `Vector` is used without a dimension
- preserve normal `Vector(dim)` behavior across Pydantic v1 and v2
- add a regression test that defines a model without importing PyArrow
## Root cause
Pydantic interpreted the bare `Vector` factory as a callable field type
and inspected its postponed annotations in the user model's namespace.
Because that namespace did not define LanceDB's internal `pa` alias,
model construction failed with the misleading `NameError: name 'pa' is
not defined` instead of explaining that `Vector` must be parameterized.
The factory now exposes Pydantic's v1 and v2 schema hooks and rejects
bare use before signature introspection with guidance to use
`Vector(dim)`.
## Validation
- `uvx --from 'ruff==0.15.20' ruff check .`
- `uvx --from 'ruff==0.15.20' ruff format --check
python/python/lancedb/pydantic.py python/python/tests/test_pydantic.py`
- `cd python && uv run --extra tests pytest
python/tests/test_pydantic.py::test_bare_vector_raises_clear_error -q`
- `cd python && uv run --extra tests pytest
python/tests/test_pydantic.py -q`
- compatibility checks with Pydantic 1.10.22, 2.11.4, and 2.13.4
Fixes#2384
<!-- lance-gatekeeper-fix:v1 agent=71e7473e18c91db5137a3c0d3bb73640
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add regression coverage for adding dictionary rows with a nullable
fixed-size-list column
- verify ordinary list columns remain aligned alongside the null
fixed-size-list value
## Root cause
PyArrow infers an all-`None` dictionary column as the generic `null`
type. The original schema-alignment path treated the target
fixed-size-list type as proof that the inferred source was also
list-like and unconditionally accessed `value_field`, which raised
`AttributeError`. Current alignment logic correctly falls back to the
target type when the source is not list-like; this test locks in that
repair for the reported ingestion path.
## Validation
- `uv run --extra tests pytest
python/tests/test_table.py::test_add_with_empty_fixed_size_list_drops_bad_rows
python/tests/test_table.py::test_add_nullable_fixed_size_list_with_none
python/tests/test_table.py::test_add_nullable_struct_with_none -q`
- `uv run --with pyarrow==19.0.1 --extra tests pytest
python/tests/test_table.py::test_add_nullable_fixed_size_list_with_none
-q`
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_table.py`
- `uv run --project python --extra dev ruff check .`
Fixes#2340
<!-- lance-gatekeeper-fix:v1 agent=cb0475e85e764f79bd03b35eb8955ec4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add regression coverage for multiple query vectors in the local
synchronous Python API
- verify that each query vector receives its own limited
nearest-neighbor result and `query_index`
## Root cause
In LanceDB v0.16, the local synchronous scanner passed a nested vector
array as one query, unlike the async and remote implementations. The
subsequent sync-to-async table migration supplied the correct shared
runtime path, but this local sync behavior was never regression-tested
and issue #1857 remained open.
## Validation
- `uv run --extra tests pytest
python/tests/test_query.py::test_query_multiple_vectors -q`
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/tests/test_query.py`
- `uv run --project python --extra tests --extra dev ruff check .`
Fixes#1857
<!-- lance-gatekeeper-fix:v1 agent=6b25bc529d76813c3db7627c8be947ef
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- convert PyArrow scalar values through their Python representation
before SQL literal rendering
- add an end-to-end regression for updating a fixed-size-list vector
from a queried FixedSizeListScalar
## Root cause
Python update literal conversion used single dispatch for native Python
and NumPy values but had no PyArrow Scalar registration. A
FixedSizeListScalar returned by a query therefore reached the
unsupported generic conversion instead of the existing recursive list
converter.
## Validation
- uv run --extra tests pytest python/tests/test_table.py::test_update
python/tests/test_table.py::test_update_with_arrow_scalar
python/tests/test_table.py::test_update_types -q
- uv run --extra tests pytest python/tests/test_util.py -q
- uv run --project python --extra tests --extra dev ruff format --check
python/python/lancedb/util.py python/python/tests/test_table.py
- uv run --project python --extra tests --extra dev ruff check .
Fixes#1228
<!-- lance-gatekeeper-fix:v1 agent=950dd892194e53b61c203d5e3715cac7
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- validate the generated LanceDB Cloud hostname during connection setup
- return a clear invalid-input error for empty, overlong, or oversized
DNS names before network resolution
- add Rust and Python regression coverage for malformed `db://`
authorities
## Root cause
The `db://` authority and region were interpolated into the Cloud API
hostname without DNS length validation. Empty or overlong labels
therefore reached the resolver and surfaced as an opaque IDNA
`UnicodeError` instead of a useful connection error.
## Validation
- `cargo test --quiet --features remote -p lancedb
test_rejects_invalid_cloud_dns_hostname --lib`
- `cargo check --quiet --features remote --tests --examples`
- `uv run --no-sync --extra tests pytest
python/tests/test_remote_db.py::test_async_remote_db
python/tests/test_remote_db.py::test_connect_rejects_invalid_cloud_dns_hostname
-q`
- `cargo fmt --all -- --check`
- `ruff check .`
- `ruff format --check python/python/tests/test_remote_db.py`
Fixes#799
<!-- lance-gatekeeper-fix:v1 agent=4d1597b3d244b58f0603ed40a8a59cf9
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a Python regression test for two partial-schema merge inserts
against the same BTree-indexed rows
- verify repeated updates retain one copy of every row and the final
update values
## Root cause
Lance 4.0, used by LanceDB 0.30.2, removed a rewritten fragment from the
index bitmap while stale BTree entries for that fragment remained
searchable. The next merge found each target through both the stale
index and the unindexed-fragment scan, producing the ambiguous-match
error. Lance fixed the root cause in lance-format/lance#6563 by applying
the fragment-bitmap allow-list to index results, and the Lance release
pinned by current LanceDB includes that fix. This test preserves the
corrected behavior through the Python API.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_table.py
-k merge_insert -q` (9 passed)
- `cd python && uv run --extra tests --extra dev ruff format --check
python/tests/test_table.py`
- `cd python && uv run --extra tests --extra dev ruff check
python/tests/test_table.py`
Repository-wide Ruff also reports 20 pre-existing violations in
untouched CI and plugin scripts.
Fixes#3280
<!-- lance-gatekeeper-fix:v1 agent=ee6b9565f9780712026076930566f116
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a minimized regression for mostly-null `list<float32>` data at the
v2.2 structural page boundary
- verify scans preserve all 64,885 rows, including 64,668 null list
values
## Root cause
Lance 3.0.0 sliced repetition/definition state using top-level row
offsets in the complex all-null decoder. At this page boundary, the list
and validity children were materialized at different lengths. The
current Lance dependency contains the upstream decoder repair; this test
locks that behavior into the LanceDB Python suite without duplicating
decoder logic.
## Validation
- reproduced the attached 1,892,466-row case on `lancedb==0.30.0` with
`expected 1024 got 285`
- verified the full attachment reads on the current branch
- `python/.venv/bin/ruff format --check
python/python/tests/test_table.py`
- `python/.venv/bin/ruff check .`
- `cd python && uv run --extra tests pytest
python/tests/test_table.py::test_read_mostly_null_list_v2_2_page_boundary
-q`
- `cd python && uv run --extra tests pytest python/tests/test_table.py
-q` (137 passed)
Fixes#3194
<!-- lance-gatekeeper-fix:v1 agent=0445adc5303a3302152cea3d2110bed1
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add deterministic regression coverage that `Table.add()` releases
backing Arrow buffers without cyclic garbage collection
- track the foreign buffer owner rather than RSS, separating live input
retention from allocator high-water behavior
- preserve the bounded-lifetime behavior of the Scannable writer that
superseded the historical preprocessing path
## Root cause
The historical Python preprocessing/write path produced a high allocator
RSS while ingesting very wide IPC batches. The current Scannable writer
releases each input buffer when `Table.add()` completes; remaining RSS
is allocator high-water rather than a live Arrow reference. The resolved
behavior had no regression coverage, so a future native lifetime
regression could silently reintroduce the original failure mode.
## Validation
- `uv run --extra tests --extra dev maturin develop`
- `uv run --project python --extra tests pytest
python/python/tests/test_table.py::test_add
python/python/tests/test_table.py::test_add_releases_arrow_buffers_without_gc
-q`
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_table.py`
- `uv run --project python --extra dev ruff check .`
Fixes#2512
<!-- lance-gatekeeper-fix:v1 agent=29226408a8d07da592daf341d5384e37
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add an end-to-end Python regression for pandas DataFrame inputs merged
into a table created from a Pydantic model
- verify reordered, nullable Arrow source fields can update and insert
into a non-nullable target schema when the values contain no nulls
## Root cause
Lance merge_insert previously compared source schema nullability with
the target, unlike add. The upstream fix now pinned by LanceDB ignores
declared nullability during schema compatibility and validates actual
null values at write time. LanceDB lacked regression coverage for the
full pandas-to-Pydantic path, so this test locks in the correct behavior
without falsifying the input schema nullability.
## Validation
- 5 focused merge-insert tests passed
- Ruff lint passed for the repository
- Ruff format check passed for the changed file
- git diff --check passed
Fixes#2366
<!-- lance-gatekeeper-fix:v1 agent=f897fccfa206620c8a2acdc3bcd1c21f
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- capture the stale-index state behind the reported fixed-size-binary
panic: the vector and FTS indices cover newer fragments while the BTree
prefilter does not
- verify vector, FTS, and hybrid searches return matches from both
scalar-indexed and unindexed fragments without panicking
- preserve binding-level coverage for the Lance fix in
https://github.com/lance-format/lance/pull/3768, which restricts
incomplete scalar prefilters when search indices are further ahead
The production root cause is in Lance and the current LanceDB dependency
already contains that fix, so this change adds the missing LanceDB
Python regression coverage.
## Validation
- `cd python && uv run --no-sync pytest
python/tests/test_hybrid_query.py::test_hybrid_query_with_stale_fixed_size_binary_prefilter
-q`
- `cd python && uv run --no-sync pytest
python/tests/test_hybrid_query.py -q`
- `python/.venv/bin/ruff check .`
- `python/.venv/bin/ruff format --check
python/python/tests/test_hybrid_query.py`
Fixes#2370
<!-- lance-gatekeeper-fix:v1 agent=5d16e59b9e513fd9247e0698732fa283
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add fast regression coverage for VoyageAI `voyage-3` source embeddings
- verify table text uses `client.embed` and never
`client.multimodal_embed`
## Root cause
The original VoyageAI source-embedding path treated table source values
as images and always invoked the multimodal API. Production routing was
corrected by later merged changes, but the table regression was covered
only by API-gated slow tests. This test locks the corrected text routing
into the regular unit suite.
## Validation
- `cd python && uv run --extra tests pytest
python/tests/test_voyageai_embeddings.py -q`
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/tests/test_voyageai_embeddings.py`
- `uv run --project python --extra tests --extra dev ruff check .`
Fixes#2059
<!-- lance-gatekeeper-fix:v1 agent=49b9e2daeed95a78ce827e2bf90abda0
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- pass an Instructor-compatible `[instruction, text]` pair when
detecting embedding dimensions
- add a regression test that verifies the dimension probe uses the
configured source instruction
## Root cause
`InstructorEmbeddingFunction.ndims()` encoded a bare string even though
Instructor models require instruction/text pairs. With affected
`sentence-transformers` versions, the bare input omitted
`instruction_mask` and raised `KeyError` while defining the LanceDB
schema.
## Validation
- `uv run --extra tests pytest python/tests/test_embeddings.py -q` (`14
passed, 9 skipped`)
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/lancedb/embeddings/instructor.py
python/python/tests/test_embeddings.py`
- `uv run --project python --extra tests --extra dev ruff check .`
Fixes#2041
<!-- lance-gatekeeper-fix:v1 agent=4b05e0d9f3eef17bccfb446e788294f4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a Python regression for vector search over a sliced Arrow table
with nullable scalar columns
- verify the nearest row retains its non-null score values after the
table is written
## Root cause
Lance 0.19.2 deep-copied a validity bitmap without preserving its
non-zero bit offset. For a sliced nullable table, scalar values and
vectors began at the slice while the copied validity bitmap began at the
parent table's first row. That made valid score values appear null even
though the corresponding vector stayed intact. The upstream Lance repair
is already present in the current dependency; this adds a LanceDB-level
guard for the reported create/search path.
## Validation
- reproduced on Python 3.12 with LanceDB 0.16.0, pylance 0.19.2, PyArrow
18.0.0, and Polars 1.14.0
- `uv run --project python --extra dev ruff format --check
python/python/tests/test_table.py`
- `uv run --project python --extra dev ruff check .`
- `cd python && uv run --extra tests pytest
python/tests/test_table.py::test_search_preserves_nulls_from_sliced_arrow_table
-q`
Fixes#1879
<!-- lance-gatekeeper-fix:v1 agent=bfa0551793f8e3cf3980cf64ad89908a
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- make the existing #1968 regression explicitly assert that schema-only
table creation succeeds
- verify the new table has zero rows and preserves the requested
fixed-size vector schema before accepting subsequent data
## Root cause
In v0.16.0, schema-only table creation sent an empty table through
vector sanitization, which calculated a remainder using `len(data)` and
raised `ZeroDivisionError`. Later refactors removed that runtime path,
but the issue-specific regression only asserted the final row count
after a subsequent add. This change makes the reported operation and its
expected empty-table state explicit so the original defect remains
directly covered.
## Validation
- `uv run --extra tests pytest
python/tests/test_table.py::test_create_table_without_data_with_vector_schema
-q`
- `uv --project python run --extra tests --extra dev ruff format --check
python/python/tests/test_table.py`
- `uv --project python run --extra tests --extra dev ruff check .`
- `git diff --check`
Fixes#1968
<!-- lance-gatekeeper-fix:v1 agent=b8ec6f40f4bba2f9beeaaae12233e5c4
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- select rustls with native certificate roots explicitly for LanceDB's
remote HTTP client
- add a Linux regression test that rejects `libssl` or `libcrypto`
dependencies in the built Python extension
## Root cause
The Python remote client originally enabled reqwest's native TLS
backend. During manylinux wheel repair, that caused OpenSSL 1.1
libraries to be bundled into the wheel. Loading those libraries on RHEL
9 with FIPS enabled aborts during the OpenSSL self-test before `import
lancedb` can complete.
LanceDB has since moved away from native TLS, but its own reqwest
dependency relied on transitive rustls feature selection and the built
extension had no regression guard. This change makes rustls selection
explicit and tests the produced Linux native module's dynamic
dependencies.
## Validation
- `uv run --no-sync pytest python/tests/test_import.py -q`
- `ruff format --check python`
- `ruff check .`
- `cargo fmt --all -- --check`
- `cargo check --quiet --features remote --tests --examples`
- `ldd python/lancedb/_lancedb.abi3.so` (no `libssl` or `libcrypto`
dependency)
- verified the resolved Python Rust dependency graph contains rustls and
no `openssl-sys` or `native-tls`
Fixes#1884
<!-- lance-gatekeeper-fix:v1 agent=31f916c7ac5c072bbbd54f3539d24f71
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- align the PyO3 runtime and build ABI floor with the declared Python
3.10 minimum
- add a regression test that keeps both ABI features synchronized with
`requires-python`
## Root cause
The Python 3.10 support-floor update originally changed PyO3 to
`abi3-py310`, but a later dependency update reverted both PyO3 features
to `abi3-py39`. Published Windows wheels were consequently tagged
`cp39-abi3` while importing `PyCMethod_New`, a stable-ABI procedure
absent from CPython 3.9.0 and 3.9.1. Windows reports that mismatch as
“The specified procedure could not be found” while loading `_lancedb`.
Restoring `abi3-py310` makes the wheel tag and native imports agree with
the package metadata and prevents future wheels from advertising
unsupported Python 3.9 compatibility.
## Validation
- `uv run --extra tests pytest python/tests/test_package_metadata.py -q`
- `uv run --extra tests --extra dev ruff format --check .`
- `uv run --extra tests --extra dev ruff check .`
- `cargo fmt --all`
- `cargo check --quiet -p lancedb-python`
- `uvx --from maturin==1.12.4 maturin build --profile ci` (built
`lancedb-0.37.1b0-cp310-abi3-manylinux_2_34_x86_64.whl`)
Fixes#2051
<!-- lance-gatekeeper-fix:v1 agent=d66c984498190d2207d1c5126cba5047
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- expand the synchronous debugger regression to enumerate every exposed
connection attribute while the Python background loop is unavailable
- retain direct representation checks for connections and tables
## Root cause
VS Code debugpy suspends Python threads at a breakpoint and inspects
local variables. Connection representation and property access
previously dispatched asynchronous work to LanceDBBackgroundEventLoop
and waited for the suspended loop thread, deadlocking the debugger. The
production safeguards landed in #3620 and #3788; this regression
exercises debugger-style whole-object expansion so a newly exposed
property cannot reintroduce the original failure.
## Validation
- uv run --no-sync pytest
python/tests/test_db.py::test_sync_debugger_inspection_does_not_use_background_loop
python/tests/test_db.py::test_read_consistency_interval_does_not_use_background_loop
-q (2 passed)
- uv run --no-sync pytest python/tests/test_db.py -q (48 passed)
- python/.venv/bin/ruff format --check python/python/tests/test_db.py
- python/.venv/bin/ruff check .
- git diff --check
Fixes#3611
<!-- lance-gatekeeper-fix:v1 agent=cdf4b39b2ce2ccb3eb5fe501acae77bb
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- replace the synthetic registry-variable metadata test with the OpenAI
embedding function reported in #2387
- verify the resolved API key survives table metadata reconstruction
- assert the OpenAI client receives the resolved key while serialized
metadata retains the variable reference
## Root cause
LanceDB 0.22.0 reconstructed embedding functions from table metadata
with the model constructor, bypassing EmbeddingFunction.create and
leaving the literal $var:api_key placeholder in OpenAI configuration.
The production path was corrected for duplicate #2181 by #2640; this
change gives that fix direct, network-free OpenAI regression coverage
for #2387.
## Validation
- uv run --extra tests pytest python/tests/test_embeddings.py -q (13
passed, 9 skipped)
- uv run --project python --extra dev ruff check .
- uv run --project python --extra dev ruff format --check
python/python/tests/test_embeddings.py
- git diff --check
Fixes#2387
<!-- lance-gatekeeper-fix:v1 agent=d453b1b9b2a298a776f2e4ea1b1449b5
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add Python regression coverage for an IVF build that cannot form all
requested non-empty partitions
- verify hierarchical k-means returns an actionable RuntimeError instead
of panicking or silently creating a degenerate index
- exercise the current Lance v10.1.0-beta.1 dependency, which contains
the upstream error-return fix
## Root cause
Hierarchical k-means previously guarded a shortfall in generated
clusters with only a debug assertion. Debug builds panicked, while
release builds could silently publish an index with many empty
partitions. The upstream Lance fix now returns a descriptive error and
is already included in the dependency pinned on main; this test locks in
propagation through the LanceDB Python API.
## Validation
- uv run --extra tests pytest python/tests/test_index.py -q (24 passed)
- uv run --extra tests pytest
python/tests/test_index.py::test_create_ivf_index_reports_unsplittable_partitions
-q (1 passed)
- python/.venv/bin/ruff format python/python/tests/test_index.py
- python/.venv/bin/ruff check .
- git diff --check
Fixes#3649
<!-- lance-gatekeeper-fix:v1 agent=a4d34448a9d350a3e2e659f33f5db6f2
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
<!-- lance-gatekeeper-fix:v1 agent=5c80c44c083b3b8ad0da595419d468fc
generation=1 -->
## Root cause
The legacy synchronous Python table called `delete` on a shared, mutable
`lance.Dataset`. Concurrent table operations could hold a PyO3 borrow
while delete requested an exclusive borrow, producing `RuntimeError:
Already borrowed`. The current async-backed binding fixes this by
cloning its thread-safe Rust table handle before awaiting, but that
concurrency contract had no regression coverage.
## Fix
- Document why delete must clone the Rust table handle before entering
its async future.
- Add a barrier-synchronized regression test that deletes distinct rows
through one shared table from eight Python threads.
- Verify every delete commits exactly one row, every commit gets a
distinct version, and no rows remain.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo fmt --all -- --check`
- `uv run --extra tests --extra dev ruff format --check
python/tests/test_table.py`
- `uv run --extra tests --extra dev ruff check
python/tests/test_table.py`
- `uv run --extra tests --extra dev pytest
python/tests/test_table.py::test_concurrent_deletes_are_thread_safe
python/tests/test_table.py::test_delete
python/tests/test_table.py::test_delete_expr
python/tests/test_table.py::test_delete_expr_async -q` (4 passed)
- Manual stress reproduction: 100 concurrent deletes on one table
completed at versions 2–101 with zero rows remaining.
Fixes#530
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- cache the immutable read consistency interval on synchronous
connection wrappers
- keep debugger property expansion from dispatching to the background
event loop
- cover direct connections and wrappers reconstructed from native
connections
## Root cause
The debugger expands connection variables by evaluating properties after
suspending all Python threads.
`LanceDBConnection.read_consistency_interval` dispatched a coroutine to
`LanceDBBackgroundEventLoop` and synchronously waited for it, but that
loop thread was also suspended, causing a deadlock.
## Validation
- `uv run --no-sync pytest python/tests/test_db.py -q` (48 passed)
- `ruff format --check python/python/lancedb/db.py
python/python/tests/test_db.py`
- `ruff check .`
- `git diff --check`
Fixes#3773
<!-- lance-gatekeeper-fix:v1 agent=e2e612236d722d926f64245d3f682bbc
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
`LanceMergeInsertBuilder.when_not_matched_by_source_delete()` didn't
clear a previously-set condition when called again with no argument (or
a different condition type). Per the docstring, `condition=None` means
"delete all unmatched rows," but if the builder had already been
configured with a string/Expr condition, a later no-arg call left the
stale condition in place instead of widening the delete to
unconditional.
Fixes#3767
## Change
Each call now unconditionally sets both
`_when_not_matched_by_source_condition` and
`_when_not_matched_by_source_condition_expr` (one to the new value, the
other to `None`), so the latest call always wins — consistent with every
other setter on this builder (e.g.
`when_matched_update_all(where=...)`).
## Test plan
- [x] New regression test
`test_merge_insert_by_source_delete_reconfigure` in
`python/python/tests/test_table.py`
- [x] `uv run --extra tests pytest
python/tests/test_table.py::test_merge_insert_by_source_delete_reconfigure
python/tests/test_table.py::test_merge_insert_by_source_delete_expr
python/tests/test_table.py::test_merge_insert_by_source_delete_expr_async
-vv` — 3 passed
- [x] `uv run --extra dev ruff format` / `ruff check` — clean
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
`LanceHybridQueryBuilder._create_query_builders()` checked
`self._minimum_nprobes` for truthiness instead of `is not None` — the
very next line correctly checks `is not None` for
`self._maximum_nprobes`. Since `0` is falsy in Python,
`.minimum_nprobes(0)` on a hybrid query silently dropped the value
instead of forwarding it to the vector sub-query, where it would raise
the same `ValueError` a plain vector query raises for the same input
(`minimum_nprobes must be greater than 0`, validated in
`rust/lancedb/src/query.rs` and covered for the plain-query path by
`test_invalid_nprobes_sync`).
Fixes#3766
## Change
One-line fix: `if self._minimum_nprobes:` → `if self._minimum_nprobes is
not None:`, matching the existing `maximum_nprobes` check right below
it.
## Test plan
- [x] New regression test
`test_hybrid_query_minimum_nprobes_zero_raises` in
`python/python/tests/test_hybrid_query.py`
- [x] `uv run --extra tests pytest python/tests/test_hybrid_query.py
-vv` — 13 passed
- [x] `uv run --extra dev ruff format` / `ruff check` — clean
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
## Summary
- Implements Cloud `fetch_blob_files`: returns real seekable `BlobFile`
handles over HTTP Range instead of `NotSupported`.
- Completes the second Cloud blob read verb after #3684 (`fetch_blobs` =
eager whole bytes; this = lazy / partial / sequential reads).
- Same public handle API as local (`read_range`, `read_up_to`, `seek`,
`tell`, `close`), so one code path works for local and Cloud.
Large blobs (video, audio, PDFs) should not require downloading the
whole object to inspect a header or stream a slice. After search,
callers open a handle and read only what they need:
```python
hits = table.search(vec).select(["id", "video"]).limit(5).to_arrow()
with table.fetch_blob_files("video", hits)[0] as f:
header = f.read_range(0, 256)
f.seek(keyframe_offset)
chunk = f.read_up_to(1 << 20)
```
### Behavior
- Handle creation probes size with `bytes=0-0` (bounded concurrency,
input order preserved).
- `204` → null (`None`); `416` with `bytes */0` → valid empty blob;
other `416` → error.
- `read_range` validates `Content-Range` and body length; OOB ranges
fail with `invalid_input` before the request (aligned with Lance).
- `read_up_to` reuses one open-ended Range response across sequential
reads; `seek` drops it.
- Servers older than 0.5.0 get a clear `NotSupported` (does not suggest
`fetch_blobs`, which they also lack).
## Testing
- `cargo test --features remote -p lancedb remote_blob`
- `cargo test --features remote -p lancedb test_blob`
- `cargo clippy --features remote --tests --examples` (no new warnings
from this change)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Adds job operations to the connection surface, building on the Job
handle from #3742: job(id), list_jobs, get_job, cancel_job, and
job_history, plus a non-blocking Job.status(). Implemented on the
Database trait (defaulting to NotSupported), the remote backend
(/v1/jobs), and the Python and Node bindings; job_history returns Arrow
batches.
errors() and progress() are not included.
Tested with mocked endpoints in all three languages.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
IndexBuilder::execute now returns a Job with wait and cancel methods.
Local tables build the index synchronously and return an already-done
job. Remote tables read the job id the server returns from create_index
and track it through the /v1/jobs API: wait polls describe until the job
reaches a terminal state and cancel posts a cancellation. Servers that
return no job id yield a done job, so behavior against older servers is
unchanged. The job id is not exposed on the handle.
The Python and TypeScript bindings keep their current signatures and
discard the handle; exposing Job there is left to follow-ups.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`RemoteDBConnection.open_table` accepts `storage_options` and never uses
it:
```python
def open_table(
self,
name: str,
*,
namespace_path: Optional[List[str]] = None,
storage_options: Optional[Dict[str, str]] = None,
index_cache_size: Optional[int] = None,
...
) -> Table:
...
if index_cache_size is not None:
logging.info("index_cache_size is ignored in LanceDb Cloud ...")
table = LOOP.run(self._conn.open_table(name, namespace_path=namespace_path))
```
The value is never passed down and never mentioned. `index_cache_size`
is ignored on Cloud in the
same way, but it says so.
I checked this at runtime on 0.34.0, not just by reading it: swapping
the inner connection for a
recorder, `open_table("t", storage_options={...})` hands the layer below
`['namespace_path']` and
nothing else, no log record is emitted, and the same probe shows
`index_cache_size` producing its
message as expected.
This adds the matching log line, so the two ignored parameters behave
the same way. `ruff check` and
`ruff format --check` are clean on the file.
A note on severity. This is not a security hole and nothing is exposed.
Someone passing credentials
there gets silence instead of an error, and finds out later.
One thing I am unsure about, and it changes the fix. I have assumed
per-table storage options are
meaningless on Cloud, which is what the `index_cache_size` line next to
it implies about managed
storage. If they are supposed to work, then the right change is to pass
them through to
`self._conn.open_table` instead and this patch is the wrong one. Happy
to redo it that way.
I did not check whether `create_table` or the async connection have the
same gap.
Xuanwo
lancedb-gatefixer[bot]
Dan Rammer
Adityaj0
Igor Ganapolsky
Wyatt Alt
Jack Ye
Sravan Avvaru
Dan Tasse
Andrew Chen
Drew Gallardo
Cohen Karnell