The Windows wheel job is the slowest job in the PyPI release workflow.
Fat LTO of the cdylib is single-threaded and the peak-memory step of the
build, so it does not get faster with more cores — and it has already
caused rustc-LLVM OOM on the Windows runners for the nodejs builds.
Switch the job to thin LTO with 16 codegen units on a
`windows-2025-8x-x64` runner, trading some runtime performance on our
least performance-sensitive platform for build time. This matches what
the nodejs Windows builds in `npm-publish.yml` already do.
`pypi-publish.yml` is in this workflow's `pull_request` paths filter, so
this PR triggers a dry-run build that shows the new timing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arrow-rs and datafusion crates are released in lockstep, but
Dependabot has been opening one PR per sub-crate for them — the 58.3.0
to 58.4.0 wave produced four separate PRs for `arrow`, `arrow-array`,
`arrow-schema`, and `arrow-buffer`. The existing `rust-minor-patch`
group did not catch them because it only filters on `update-types` and
declares no patterns.
This PR adds an explicit `arrow-datafusion` group matching `arrow*`,
`parquet*`, `datafusion*`, and `object_store`, so those bumps arrive as
a single PR. It is listed before `rust-minor-patch` because a dependency
joins the first group it matches, and it deliberately omits
`update-types` so major bumps are grouped too.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bugs in Node's reading of the embedding_functions schema metadata.
First, parseFunctions keyed its result map by function name, so a table
whose metadata configures the same function for two vector columns came
back with only the last one. It now keys by the vector column, the
convention Python's parser already uses.
Second, Node could not read metadata written by the Python bindings at
all, which spell the keys snake_case: configs parsed with both columns
undefined, breaking embedding application on add() and leaving only
query-side embedding working. The parse now accepts both spellings.
Both fixes land in one shared parser used by every reader --
parseFunctions and the makeArrowTable schema validator, which had its
own private camelCase-only parse -- so the wire contract cannot fork
between entry points. A config naming no source or vector column is an
error at the boundary rather than a default downstream, as are two
configs claiming one column. The "vector" fallback remains only on the
optional field of user-supplied configs.
Breaking: parseFunctions is exported and its map keys change from
function name to vector column.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Remote Function bindings can expose a nullable parameter schema even
when the source table column is non-nullable. Binding validation rebuilt
the exact input schema from table nullability and rejected this safe
widening.
Accept non-null table columns for nullable Function parameters while
continuing to reject nullable table columns for non-null parameters. All
other input schema fields remain exact, including named multi-input
ordering, names, types, and metadata.
Remote Function catalog requests used singular endpoints that are not
exposed by Phalanx. Route registration to `POST /v1/functions/create`
and exact-version lookup to `POST /v1/functions/get`, while preserving
the existing typed Job submission and wait behavior.
As in https://github.com/lancedb/lancedb/pull/3977, we're trying to
reduce anything in the lancedb skill that duplicates other docs. So this
shrinks the branch-ops logic down to a few lines that mostly just point
the agent to fetch the branching docs from lancedb.github.io.
Run stats (2 runs each):
<img width="1001" height="232" alt="Screenshot 2026-08-20 at 5 02 34 PM"
src="https://github.com/user-attachments/assets/a3f4d305-278e-4093-b153-07f0af57b251"
/>
This is out of order, rearranged:
|condition|time (sec)|cost|
|---|---|---|
|No branch_ops.md|250|1.33|
|No branch_ops.md|227|1.26|
|Old branch_ops.md|116|0.83|
|Old branch_ops.md|127|0.87|
|New branch_ops.md|135|0.86|
|New branch_ops.md|147|0.93|
Averaged between each of the two runs:
<img width="775" height="337" alt="Screenshot 2026-08-20 at 5 34 15 PM"
src="https://github.com/user-attachments/assets/4f2fb2a8-3112-4614-87d9-8dbf807f3b75"
/>
It seems helpful to have *some* doc about branching; otherwise the model
gets a little confused about our branch model and what methods to call.
But it looks like the new one (in this PR; all just references to
current docs) is basically as good as the old one (lots of duplicative
text).
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Problem
Generic job result propagation exposed the PyO3 representation of Rust's
unit value as `()` in Python. Unit jobs therefore returned an empty
tuple instead of `None`, breaking the documented `Job.wait()` contract
and the Python doctest workflow.
## Behavior
Unit job completion now converts explicitly to Python `None`. Typed job
results continue to pass through unchanged, with synchronous and
asynchronous regression coverage.
Follows lance-format/lance#8680, which removes
`FLAG_MEM_WAL_INDEX_CATCHUP`.
With one set of semantics there is no mode to switch into.
## Removed
`require_mem_wal_index_catchup` — the activation entry point — from the
trait,
from `Table`, and from the LSM merge module.
## The read path
`exclusion_watermarks` loses its `catchup_required` argument and keeps
the
conservative branch: an index with no entry is not known to hold these
rows, so
every generation stays readable from its SSTable. Nothing is excluded
until an
index records that it covers those generations, so a table that has
never
recorded catch-up reads every row from its SSTables rather than assuming
the
base covers them.
## One guard needed a replacement, not deletion
`refresh_column` and computed-column declaration refuse a table whose
rows sit
in un-compacted tiers, since refresh enumerates base fragments and would
silently omit them. They keyed on the feature bit because
`unset_lsm_write_spec` **drops the MemWAL index** — after an unset the
write
spec no longer describes such a table, and the bit was the only marker
that
outlived it. Two tests covered this, so deleting the term would have
dropped a
tested property.
Both guards now check for MemWAL shard directories on storage, which
outlive
the index. That is strictly wider than the bit ever was: the bit only
marked
tables where activation had run.
## Two tests conflated two different things
An index that is *caught up* and one that is *untracked* both fell back
to the
compaction watermark, because absence carried no information without the
bit.
Absence now means "not caught up", so untracked retains everything.
`an_untracked_index_does_not_widen_a_lagging_sibling` becomes
`an_untracked_index_retains_everything`, with the genuinely-caught-up
case
asserted separately.
## Testing
933 `lancedb` lib tests. `cargo fmt` clean. (The pre-existing
`Error::Http`
build failure in `job.rs` without the `remote` feature is unrelated and
untouched.)
The Linux Rust job can exhaust its disk after restoring a large fallback
target cache and compiling multiple feature graphs into one target
directory.
Run remote tests in an independent job with registry-only caching, and
run the simple example with all features so it reuses the preceding
build artifacts. This preserves remote coverage and fork behavior while
preventing all-features and remote-only artifacts from accumulating
together.
Failure evidence:
https://github.com/lancedb/lancedb/actions/runs/32467317540/job/96726650990
## Problem
The canonical Function wire values and typed remote Job contract do not
yet provide a Python authoring surface or catalog client, so users
cannot package a scalar callable, register it, or reopen the exact
immutable Function version.
## Behavior
This adds scalar-only `@udf` authoring with deterministic annotation or
explicit Arrow schema validation, content-addressed Python artifacts,
and an internal scalar-to-Arrow-batch adapter descriptor. Registration
payloads model non-secret environment values and secret names only.
Remote connections can submit `create_function_async` and receive a
typed `Job<FunctionVersion>`, then reopen that exact version by name and
version ID. Synchronous connections can call `create_function` to submit
and wait for the immutable version in one operation. Local Function
catalog operations return a stable `NotSupported` error. Shared
Rust/Python golden payloads and mocked catalog responses freeze the
request, typed terminal result, and exact lookup contract.
## Validation
- Rust formatting, remote check, clippy, and focused LDB-1/LDB-2 tests
- Python formatting, lint, and focused LDB-1/LDB-2 tests
- Python API documentation build
Function applications from the canonical remote contract cannot
currently declare scalar or grouped computed-column outputs atomically.
This adds the remote-only declaration contract for scalar,
struct-as-one-column, and expanded named-struct outputs. It validates
result mappings, fixes exact input/output Arrow schemas in the request,
persists grouped sibling metadata, and keeps local Function execution
unsupported. Unknown newer application or binding metadata remains
readable, while schema-changing mutations fail closed instead of
rewriting it.
Stable Lance field IDs are deliberately not a declaration prerequisite
in this slice. Inputs bind by parameter name and field path; Sophon
remains responsible for exact-version validation, atomic all-NULL
sibling creation, binding identity and revision allocation, and
persisted output identities.
LanceDB's Python SDK now requires Pydantic `>=2.7.4,<3` and uses the v2
APIs throughout. This removes dual-version behavior from schema
conversion, query serialization, embedding models, and Function wire
models while preserving their existing public and canonical-wire
behavior.
The minimum-dependencies CI job pins Pydantic 2.7.4 so the declared
compatibility floor remains covered.
Local benchmarks currently inherit the release profile's fat LTO and
single codegen unit, making local iteration pay release-artifact build
costs.
Provide repository-defined profiles for no-LTO local work and cheaper
benchmark builds, and document when each profile is appropriate. Release
artifacts continue to use fat LTO.
PyO3 exposed `LsmWriteSpec` with its default `builtins` module, causing
mkdocstrings to resolve the public `lancedb.LsmWriteSpec` re-export as
`builtins.LsmWriteSpec` and fail the documentation build. Declare the
native extension module and pin the public re-export with a regression
test.
This also applies the repository's current Ruff formatter to seven
previously unformatted Python scripts.
## Problem
Enterprise Function-backed computed columns need a stable SDK contract
before Sophon catalog and execution endpoints can be added. The existing
`Job` API can only represent unit terminal results, and there is no
shared Rust/Python wire definition for immutable Function versions,
applications, bindings, or refresh results.
## Behavior
This introduces remote-only canonical Function values in Rust and
Python, evolves `Job<T = ()>` to decode typed remote terminal results
while keeping local spawned operations unit-typed, and fixes the
cross-language contract with shared JSON golden fixtures. Unknown fields
and discriminator values remain forward-decodable, while canonical
output contains only fields known to the client. Function models contain
secret names only.
Sophon remains the sole owner of catalog persistence, environment bake,
secret resolution, execution, and publication. This PR does not add
authoring/catalog endpoints, local execution, refresh runners, or live
Sophon E2E coverage.
Background: if we keep adding stuff to the lancedb skill that repeats
other knowledge, we're basically creating a whole new docs site, which
means one more thing that can get out of date. Worse, if it gets out of
date, it will tell agents to do the wrong thing.
These files were added without a ton of analysis of whether they'd be
improving agent performance at all. It looks like they don't really:
<img width="644" height="90" alt="Screenshot 2026-08-20 at 5 21 03 PM"
src="https://github.com/user-attachments/assets/44e60436-b7ad-498b-8e73-0181385c7c60"
/>
(top run is without these docs, bottom run is with them - arguably these
docs might even make the agent a little slower! that's probably noise
though; I'd just say at least they're unnecessary.)
So this PR just removes them. We'll more judiciously add bits we need
and/or point to preexisting docs, to avoid duplication.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`cargo deny` did not check crate-level dependency declarations against
`[workspace.dependencies]`, so a crate used by both the core crate and
the bindings could be declared independently in each one and drift. For
example `tokio` was pinned at `1.23` in `rust/lancedb` and `1.40` in
`python`, and `pin-project` at `1.0.7` in the workspace table but
`1.1.5` in `python`.
This PR turns on cargo-deny's `bans.workspace-dependencies` lint, which
fails when a dependency is used by more than one member without going
through `workspace = true`, and when a `[workspace.dependencies]` entry
is used by nobody.
Enabling it surfaced 12 violations. Fixing them means adding `bytes`,
`lancedb`, `serde`, `serde_json`, `tempfile`, `tokio`, and `uuid` to
`[workspace.dependencies]`, and pointing the `arrow`, `arrow-buffer`,
`async-trait`, `chrono`, and `pin-project` declarations at the entries
that already existed. `Cargo.lock` is unchanged, so resolution is the
same as before.
The shared `chrono` entry now carries `default-features = false,
features = ["clock"]`, matching what `nodejs` and `python` already asked
for — cargo ignores a member's `default-features = false` unless the
workspace entry sets it too. On the targets we build, `clock` covers
everything `rust/lancedb` was getting from chrono's defaults.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
<!-- lance-gatekeeper-fix:v1 agent=40e5cf476a59265c71653574eda834d2
generation=1 -->
## Summary
- preserve incoming PyArrow `arrow.json` fields while schema
sanitization aligns input to a stored `lance.json` schema
- let Lance perform the required JSONB encoding instead of relabeling
raw JSON bytes as encoded storage
- cover both merge insert and the conditional add sanitization path with
end-to-end regression tests
## Root cause
Python schema sanitization aligns incoming data to the table schema
before passing it to Lance. Merge insert always takes this path, while
add takes it conditionally for preprocessing such as non-default
bad-vector handling or embedding functions. For JSON columns, the cast
changed logical `arrow.json` strings into the table's JSONB-backed
`lance.json` storage type without encoding the bytes, so Lance treated
raw JSON text as JSONB.
## Validation
- `cd python && uv run --extra tests pytest python/tests/test_table.py
-k 'merge_insert or add_sanitization_encodes_json' -q`
- targeted schema-cast and JSON encoding tests
- `ruff check .`
- `ruff format --check python/python/lancedb/table.py
python/python/tests/test_table.py`
Fixes#3923
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Why
Four of the eight LSM methods are **remote-only in the core**. `impl
BaseTable for NativeTable` implements only
`set`/`unset`/`get_lsm_write_spec` and `close_lsm_writers`; `flush_lsm`,
`compact_lsm` and `get_lsm_stats` fall through to trait defaults
returning `NotSupported` (`rust/lancedb/src/table.rs:679,687,696`), and
`checkpoint_lsm` is built on all three.
That explains the state of the bindings: Node had bound the four that
work against a local table and stopped, so a Cloud user could install an
LSM write spec but had no way to observe fresh-tier state or drive a
checkpoint. Java had none of it at all.
| SDK | set/unset/get spec | closeWriters | flush | compact | getStats |
checkpoint |
|---|---|---|---|---|---|---|
| Rust core | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Python | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Node *(before)* | ✅ | ✅ | — | — | — | — |
| **Node (after)** | ✅ | ✅ | **new** | **new** | **new** | **new** |
| Java *(before)* | — | — | — | — | — | — |
| **Java (after)** | **new** | n/a | **new** | **new** | **new** |
**new** |
Go and C are separate repos and are out of scope here. `closeLsmWriters`
drains cached in-process shard writers, so it has no meaning for Java,
which is a pure REST client.
## Node
Adds napi bindings for `flushLsm`, `compactLsm`, `checkpointLsm` and
`getLsmStats`, plus typed `LsmStats` / `BucketStats` / `GenerationStats`
/ `MemtableStats` objects — typed rather than a JSON blob, matching the
existing `LsmWriteSpec` object in the same file, with `u64` cast to
`i64` per that file's convention.
Because these four are remote-only, the new tests assert each binding
reaches the core and surfaces `NotSupported` against a local table. That
covers the wiring; behavior against a real endpoint stays covered by the
mocked-endpoint tests in `rust/lancedb/src/remote/table.rs`.
## Python
No new methods. All eight are on `LanceTable`, `AsyncTable` and
`RemoteTable` — the last four landed on the sync `RemoteTable` in #3961,
which is merged into this branch.
What was missing here was reachability. `LsmWriteSpec` was importable
only from the private `lancedb._lancedb`, appearing in `table.py` solely
under `if TYPE_CHECKING:`, and `docs/src/python/python.md` had no
mention of it, which per the repo's docs guidance means it rendered
nowhere in the API reference. It is now `lancedb.LsmWriteSpec`, in
`__all__`, and documented.
## Java
Java reaches LanceDB purely over REST through the generated Lance
Namespace client, and these routes are not in that spec, so they are
issued through a small dedicated client rather than added to the spec.
That call is revisitable — LSM is one of four unspecified route families
alongside `multipart_write`, `page_cache/prewarm` and
`branches/diff|merge`. If those are ever regularized into the spec as a
group, `LanceDbTableLsm` is one file that gets deleted.
`LsmWriteSpec` here is deliberately **not**
`org.lance.memwal.InitializeMemWalParams`. That type defaults to
maintaining *no* indexes where a spec here defaults to maintaining
*every* index, and it cannot express the `null` that asks the server to
resolve the set:
| Value | On the wire | Meaning |
|---|---|---|
| unset (null) | `null` | Server resolves **every** maintainable index |
| `Collections.emptyList()` | `[]` | Maintain **none** |
| `Arrays.asList("id_idx")` | `["id_idx"]` | Exactly those |
A dedicated test pins null and `[]` as distinct on the wire, since
collapsing them is the failure mode that motivated a LanceDB-owned type.
`checkpointLsm` is ported from `rust/lancedb/src/table/checkpoint.rs`
with its constants and status semantics intact: 429/503 retried in place
against an 8-budget, 421 restarting from flush against a 3-budget, 5s
poll, and a target watermark fixed after the seal so it terminates under
write load.
`getLsmStats` returns typed `LsmStats` / `BucketStats` /
`GenerationStats` / `MemtableStats`, mirroring the Rust structs in
`rust/lancedb/src/table/lsm_stats.rs` and the objects Node exposes.
Decoding is strict — see below.
## Review feedback
Both gatekeeper findings were real. Each was reproduced against the
scripted test server first, and each fix ships with the reproducer as a
regression test.
**The transport was doubling every checkpoint retry budget.**
`HttpClients.createDefault()` installs Apache's default response retry
strategy, whose retryable-status list is exactly 429 and 503 — the two
statuses `isRetryable` owns. A 429 held against `flush_lsm` issued
**18** wire requests where the loop intends 9, and `compact_lsm` was
retried in place despite the loop being built to fall through to a fresh
stats poll instead. Timing confirmed the mechanism: that run took 25.4s
≈ 16.3s of the loop's own backoff plus 9 × the transport's 1s retry
interval.
Automatic retries are now disabled, so the checkpoint loop is the sole
owner of the 421/429/503 transitions. A side effect worth noting:
`testCheckpointRetriesRetryableStatusInPlace` was passing on a
transport-absorbed 429 and never reaching `issue()`'s retry branch at
all. It now exercises the real path.
**Stats decoding failed open.** `getLsmStats` read the response with
Jackson's `path()`, which yields a missing node that iterates as an
empty array — making "malformed" indistinguishable from "no buckets",
which is indistinguishable from "drained". Four separate payloads made
`checkpointLsm()` report convergence for a checkpoint that never ran:
| Response | Before | Now |
|---|---|---|
| `{"lsm_stats": null}` or absent key | disabled ✓ | disabled ✓ |
| `{"lsm_stats": {}}` | **reported success** | `IllegalStateException` |
| empty response body | **reported success** | `IllegalStateException` |
| bucket missing required fields | **reported success** |
`IllegalStateException` |
The empty-body row is the one to weight: a proxy 200 with no body is a
realistic production event, and it silently reported a checkpoint that
never happened.
Decoding is now strict and fails closed, matching the serde contract on
the Rust side exactly. One deliberate deviation from the review comment,
which asked that *only* explicit JSON `null` count as disabled: Rust has
`#[serde(default)]` on `lsm_stats`, so an **absent key** decodes to
`None` there too. Java now matches that. It is an absent-or-malformed
**`buckets`** that fails closed, which is the case the comment was
actually protecting.
## Testing
- Java: **33 passing** (8 existing + 25 LSM) against a scripted
`com.sun.net.httpserver.HttpServer` — no new test dependency. Wire
assertions mirror `rust/lancedb/src/remote/table.rs:6581-6748`;
checkpoint tests cover convergence, not piling onto a latched bucket,
421 restart-from-flush, 429 retry-in-place, terminal-status propagation,
reissue exhaustion, the exact wire-request count against the retry
budget, and five malformed stats payloads.
- Node: **19 LSM tests passing**; `cargo check`, `npm run build`, `npm
run tsc`, `npm run lint`, `npm run docs` all clean.
- Python: `ruff format --check` and `ruff check` clean.
- Java formatting: `./mvnw -pl lancedb-core spotless:apply` and
`spotless:check` both clean under a JDK 11 toolchain.
## Note: spotless needs a pre-16 JDK
`./mvnw spotless:apply` fails on JDK 16+ with
`JCTree$JCImport.getQualifiedIdentifier()` — google-java-format 1.7,
pinned at `java/pom.xml:34`, predates JDK 16's compiler API change.
**This is pre-existing** and reproduces on a pristine `main` checkout.
It is not a blocker, just a toolchain requirement. Spotless was run
against these sources under JDK 11 and both `spotless:apply` and
`spotless:check` pass on the whole module:
```shell
JAVA_HOME=/path/to/jdk11 ./mvnw -pl lancedb-core spotless:apply
```
Bumping the plugin so it works on modern JDKs is still worth doing, but
separately from this PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
The sync `RemoteTable` carried `set_lsm_write_spec`,
`unset_lsm_write_spec`, `get_lsm_write_spec`, and `close_lsm_writers`,
but not `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, or
`get_lsm_stats`.
That left the four LSM control methods reachable from `AsyncTable` only.
They are also the four that *only* work against a remote table —
`NativeTable` does not override the `BaseTable` defaults, so on a local
table they return `NotSupported` (`rust/lancedb/src/table.rs:679-701`).
The net effect for sync users:
| | `checkpoint_lsm` / `get_lsm_stats` |
|---|---|
| `LanceTable` (sync, local) | present, but always `NotSupported` |
| `RemoteTable` (sync, remote) | `AttributeError` — method absent |
| `AsyncTable` (remote) | works |
So there was no working sync path at all, despite the Rust `RemoteTable`
implementing every one of these against real endpoints.
## Changes
* Add `checkpoint_lsm`, `flush_lsm`, `compact_lsm`, and `get_lsm_stats`
to `lancedb.remote.table.RemoteTable`, mirroring the delegation style of
their neighbours.
* Correct the docstrings on `set_lsm_write_spec` /
`unset_lsm_write_spec`, which read `"""Not supported on LanceDB
Cloud."""` although `rust/lancedb/src/remote/table.rs:2549-2601`
implements both against `/v1/table/{}/set_lsm_write_spec/` and
`/unset_lsm_write_spec/`. They appear to have been copy-pasted from
`set_unenforced_primary_key` directly above.
No Rust or PyO3 changes — the bindings and the `AsyncTable` methods
already existed. The `Table` ABC is left alone, matching how the
existing `*_lsm_write_spec` methods are declared on the concrete classes
only.
## Tests
Four new tests in `python/python/tests/test_remote_db.py`, against the
existing mock HTTP server:
* `test_get_lsm_stats_sync` — the server payload round-trips into the
dict, and `include_generation_rows` defaults to `False` and is forwarded
when set.
* `test_get_lsm_stats_sync_returns_none_when_lsm_disabled` — a
`{"lsm_stats": null}` envelope yields `None` rather than an error.
* `test_flush_and_compact_lsm_sync` — both are one-shot POSTs answered
`202` with no body.
* `test_checkpoint_lsm_sync` — pins the binding to the endpoints it
drives (`flush_lsm` then `get_lsm_stats`); the convergence loop itself
is already covered in Rust.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Context
Java users opening catalog-backed tables with vended credentials
currently lack a documented workflow. Opening the catalog-returned URI
directly drops the namespace-provided storage options and automatic
credential refresh.
Document the namespace-backed `Dataset.open()` path so temporary object
store credentials are applied and refreshed transparently.
## Summary
`LanceHybridQueryBuilder` (sync hybrid search,
`table.search(query_type="hybrid")`) silently ignored `.offset()`.
`self._offset` was never forwarded to the vector/FTS sub-queries and
never applied when slicing the final combined/reranked result, so
`.offset(N)` behaved identically to `.offset(0)` — no error, just wrong
pagination.
Fixes#3765
## Changes
- `_create_query_builders()`: each sub-query now fetches `limit +
offset` rows so there's enough data to slice the correct window out of
after combining/reranking.
- `_combine_hybrid_results()` / `to_arrow()`: the final table is sliced
with `offset=self._offset` instead of always starting at 0.
## Test plan
- [x] New regression test `test_hybrid_query_offset` in
`python/python/tests/test_hybrid_query.py`
- [x] `uv run --extra tests pytest python/tests/test_hybrid_query.py
-vv` — 13 passed
- [x] `uv run --extra dev ruff format` / `ruff check` — clean
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Will Jones <willjones127@gmail.com>
## Summary
Fixes naive `lit(datetime)` equality filters against table timestamp
columns on non-UTC hosts, and adds the integration matrix from #3262.
## Failure (before)
On a machine in US Eastern (UTC−4 / EDT), with PyPI `lancedb==0.36.0`:
```python
from datetime import datetime
import lancedb
from lancedb.expr import col, lit
db = lancedb.connect("memory://")
ts = datetime(2024, 7, 1, 10, 0, 0) # naive
table = db.create_table("t", [{"id": 1, "ts": ts}])
rows = table.search().where(col("ts") == lit(ts)).to_list()
# actual: [] (0 rows)
# expected: 1 row
```
### Root cause
In `python/src/expr.rs`, `expr_lit` converted every `datetime` via
Python's `.timestamp()`:
- **naive** `.timestamp()` = local wall → UTC epoch (shifted by host
offset)
- **PyArrow naive** storage = UTC wall-clock microseconds (no local
shift)
So `lit(naive)` became `CAST('2024-07-01 14:00:00' AS TIMESTAMP)` on EDT
while the table held `10:00:00`.
## After
Naive datetimes are interpreted as UTC wall clock
(`replace(tzinfo=timezone.utc).timestamp()`), matching Arrow storage.
Aware datetimes still use `.timestamp()` (correct epoch).
Same repro on this branch: **1 matching row**.
## Tests
Added `TestExprDatetimeTimezoneIntegration` covering:
| Case | Result |
|------|--------|
| both naive | match |
| both same TZ (UTC) | match |
| different TZs, same instant | match |
| table TZ + naive lit | match (wall clock) |
| table naive + aware lit | match |
| naive lit SQL is wall clock, not local-shifted | asserts `10:00:00` in
SQL |
### Verification
```bash
cd python
maturin develop
pytest python/tests/test_expr.py -v
```
**102 passed** (full `test_expr.py`, including the 6 new cases).
Closes#3262
---------
Co-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
LanceDB Cloud and Enterprise support computed columns through the REST
API,
so declaration dispatches per backend: local tables plan the expression
themselves, remote ones send {name, computed} entries for the server to
plan. A remote refresh is the server's backfill job --
refresh_column_async
submits it and returns a handle whose successful wait establishes a
read-freshness baseline on the submitting handle, unless a checkout has
pinned the handle by the time the job completes; the blocking form
refuses
rather than invent a fill count the server does not report.
Declaration entries are built from the namespace client's
AddColumnsEntry
model (lance-namespace 0.11.0, via the lance beta.13 pin), so the
payload
shape is compile-checked against the published contract.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
Updates the Lance Rust workspace dependencies and Java lance-core
dependency to
[v11.0.0-beta.13](https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.13).
Adds the required `ListTablesResponse.context` compatibility field and
validates the workspace with Clippy warnings denied.
Mirrors create_index's dual surface: the blocking refresh_column keeps
returning {rows_filled, version}, and refresh_column_async returns the
same
Job handle create_index uses, running the refresh as an in-process task.
Invalid input is reported by the submitting call rather than by the job.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
table.refresh_column("doubled") fills the rows of a declared column that
hold no value, in two passes per fragment: the first scans only the
unfilled
live rows to count exact gains and decide staging, the second streams
the
fragment's physical rows into a standalone column file published in one
DataReplacement -- committed under the dataset's own session -- so peak
memory is bounded by a scan batch. A row that holds a value keeps it;
deleted and already-filled rows never reach the expression, so a poison
value in them cannot fail the refresh. Refresh refuses under an LSM
write
spec, including the mem-wal catch-up flag that outlives unset and marks
retained SSTable rows.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
add_columns().computed("doubled", "x * 2") stores the expression in
field
metadata and commits the column empty; a later refresh fills it. Type
and
inputs are derived from the expression.
The declaration stays authoritative for its lifetime: writes that would
give
the column a value (append, update, merge, SQL insert), schema changes
that
would break the stored expression or reshape its output, metadata edits,
volatile expressions, declaration metadata arriving through any path but
the
validated declare call, and LSM write specs in either order against
latest
committed state are all refused. The LSM check also refuses on the
mem-wal
catch-up feature flag, which outlives unset and marks retained SSTable
rows.
Simultaneous declare/install interleavings conflict at commit via
lance's
mem-wal rule (lance#8539). Local tables only.
---
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
> Stacked on #3780. Blocked only on #3922 (`lance` → `v11.0.0-beta.6`),
so CI
> stays red until that lands.
## Missing coverage must mean "not known to be covered"
#3780 caps the SSTable exclusion watermark at an index's recorded
catch-up when
there is one, and silently ignores the case where there is none. On a
table that
requires catch-up, an absent entry means the index is *not* known to
hold the
compacted rows — and the LSM base arm reads base through the index
(`fast_search`, no brute-force tail), so dropping that SSTable loses
those rows
for that query.
```rust
Some(caught_up) => watermark = watermark.min(caught_up),
None if catchup_required => watermark = 0, // retain everything
None => {}
```
`catchup_required` reads the manifest feature bit directly, and requires
both
words: a half-set manifest is treated as legacy, which is the
conservative side.
Without the bit the field is not maintained at all, so absence carries
no
information and behaviour is unchanged.
## Activation, as a table-level entry point
`Table::require_mem_wal_index_catchup()` performs the one-way switch,
separate
from `set_lsm_write_spec`: a table carrying the bit retains every
generation
until something records catch-up, so it has to follow the deployment of
whatever
repairs coverage, not the creation of the table.
This is a convenience, not the only path — a writer holding the dataset
calls
the equivalent on `DatasetMemWalExt`, which is what the WAL pod does.
Lance
enforces the preconditions either way: the MemWAL index must exist, and
the
table must not already carry `compacted_sstables` from before this
protocol,
since those numbers cannot be validated.
## Still correct after the Lance rework
lance-format/lance#8481 replaced the transmitted `IndexCatchupAdvance`
with a
position derived at commit time from the version a transaction read.
That
changed how a writer earns coverage; it did not change what a reader may
conclude from its absence. The rule here, and the field it reads, are
unchanged.
## Tests
Existing `exclusion_watermarks` unit tests carry the new argument.
Coverage
against a real dataset follows once #3922 lands and this can build.
Updates the Rust workspace Lance dependencies and Java lance-core
dependency to v11.0.0-beta.10. No compatibility fixes were required;
workspace clippy with all features and Rust formatting pass.
Lance tag:
https://github.com/lance-format/lance/releases/tag/v11.0.0-beta.10
## Summary
- add `drop_table_async` and return a job handle while preserving
`drop_table`
- consume remote 202 responses with cleanup job IDs and retain
older-server compatibility
- expose the API through Python and TypeScript connection wrappers
Say what resolving "latest" actually does: pick the newest release,
preferring stable over pre-release, and skip the run if it is not newer
than the version pinned in Cargo.toml.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
`exclusion_watermarks` resolved a single index and capped SSTable
exclusion at that index's catch-up watermark. It now takes every index
the query relies on and retains to the **lowest** of them, and the
resolver collects arms together rather than returning at the first
match.
This is groundwork, not a fix for a reachable bug: `reject_unsupported`
refuses hybrid search, so the vector and full-text arms are mutually
exclusive and the list never holds more than one entry today. The
generalisation is what the remaining work below plugs into.
Unchanged: a plain scan uses the compaction watermark alone, an index
with no catch-up entry contributes no cap, and a caught-up index falls
back to the compaction watermark. Taking a minimum over more indexes can
only lower a watermark, so the failure direction is "read an SSTable
unnecessarily", never "miss rows".
## Tests
Three in `lsm`: the existing lagging-index test updated for the new
signature;
`exclusion_watermark_takes_the_minimum_across_every_index_used` (two
indexes at 7 and 4 against compaction at 9 — each alone stops at its own
watermark, together the lower governs, order-independent); and
`an_untracked_index_does_not_widen_a_lagging_sibling`.
`cargo test -p lancedb --lib` — 45 lsm tests, 484 in the crate. `cargo
fmt --check` clean.
## Follow-ups
This crate pins lance to a released tag, so anything needing unreleased
Lance symbols waits for a bump.
1. **Select legacy versus strict semantics from the feature bit.** On a
table with `FLAG_MEM_WAL_INDEX_CATCHUP` set, a *missing* entry must mean
"not caught up" and retain the SSTables, instead of leaving the
compaction watermark unchanged. Needs the bit from
lance-format/lance#8263. **This must land before any table is
activated** — otherwise the bit is set while queries still read
permissively.
2. **Collect scalar and bitmap-family prefilter indexes.** The genuinely
multi-index query is a vector search with a scalar prefilter, and it is
gated on the vector index alone today. Identifying the others needs the
planner's chosen indexes, not the columns the filter names, so it needs
a Lance-side helper.
3. **Verify a retained SSTable can actually answer.** Both base and
SSTable arms use `fast_search`; a source without a compatible index
contributes nothing, so retention alone does not guarantee its rows are
returned. Needs a flat-search fallback or an explicit error in Lance's
`LsmScanner`.
4. **Planner-level integration tests.** Current tests exercise the
watermark arithmetic directly. End-to-end coverage over real queries —
prefilter forms, legacy versus activated, missing index and missing
shard entries — depends on 1–3.
## What is the bug?
#3731 tries to distinguish a missing table from a corrupt table after
Lance returns `DatasetNotFound`. It does that by listing the database
parent and treating a physical `<name>.lance` entry as evidence that the
table exists.
That premise is not sound for a listing database. Table creation writes
data before atomically committing the first manifest, so the same
physical prefix can represent a live concurrent create, abandoned
uncommitted data, or an old empty directory. It is not evidence of a
committed table. The parent listing also makes every missing-table open,
including the create-on-miss path, perform work proportional to the
number of sibling tables. Cloud `list_with_delimiter` exhausts all pages
before returning.
## How does this PR fix the problem?
This PR makes the committed Lance manifest the sole table-existence
authority for listing-database opens:
- `DatasetNotFound` maps directly to `TableNotFound`; no parent or
target storage probe runs.
- Other Lance load errors continue to propagate unchanged.
- A physical directory, object prefix, or uncommitted data file alone
does not block `Create`.
- Concurrent `Create` requests are arbitrated by the conditional
version-1 manifest commit: one succeeds and the loser receives
`TableAlreadyExists`.
- `table_names` is documented as physical discovery, not an atomic
table-existence check. Its snapshot can contain an entry that is still
being created, has only uncommitted storage, or is concurrently dropped.
This removes the need for a new Lance object-store capability. LanceDB
remains on the official Lance `v11.0.0-beta.6` dependency from `main`;
the merge commit for lance-format/lance#7722 is an ancestor of that tag,
so the ambiguous-GCS-500 corruption-prevention fix is retained.
## Performance evidence
Lower is better. The benchmark uses real `.lance` directories with
marker objects on the local filesystem; fixture creation and teardown
are outside the timed region. Baseline is `origin/main` at `6fb976cf`,
candidate is `e1240751`. Both were built from the same lockfile on the
same macOS arm64 machine with the repository's `release` profile (fat
LTO), then executed in alternating baseline/candidate order for three
pairs. Each run used 10 warmups and 100 distinct missing-table opens per
scale. The table reports the median of the three run-level percentiles.
| Scenario / metric | Baseline | This PR | Benefit |
| --- | ---: | ---: | ---: |
| 1,000 real sibling directories, p50 | 11.905 ms | 21.042 us | 566x
speedup |
| 10,000 real sibling directories, p50 | 143.630 ms | 18.375 us | 7,817x
speedup |
| 100,000 real sibling directories, p50 | 1.991 s | 19.917 us | 99,984x
speedup |
| 100,000 real sibling directories, p95 | 2.346 s | 25.792 us | 90,965x
speedup |
These results validate removal of the sibling-cardinality dependency in
this local-filesystem workload; they are not an extrapolation to
production GCS latency. A structural object-store regression test
separately asserts that opening one missing table performs zero
parent-scoped `list`, `list_with_offset`, or `list_with_delimiter`
calls.
Run with:
```bash
BENCH_SIBLINGS=1000,10000,100000 BENCH_WARMUPS=10 BENCH_TRIALS=100 \
cargo run --locked --release --quiet -p lancedb --example bench_open_missing_table
```
## Correctness and compatibility boundaries
- An empty `.lance` directory or orphan data without a committed
manifest now opens as `TableNotFound` and may be replaced by a
successful `Create`.
- Two synchronized creators sharing one object store deterministically
produce one success and one conditional-manifest conflict mapped to
`TableAlreadyExists`.
- A readable manifest remains authoritative; non-`DatasetNotFound`
corruption, external-manifest, authorization, and object-store errors
are not folded into `TableNotFound`.
- `TableCorrupted` remains in the public error enum for compatibility,
but this listing-database fallback no longer synthesizes it from an
ambiguous physical footprint.
- Reliably distinguishing `Missing`, `Creating`, and `Corrupt` would
require explicit authoritative lifecycle/catalog metadata (for example a
leased creation record). It cannot be inferred from a directory or
prefix, and is outside this incident fix.
## Validation
- `cargo fmt --all -- --check`
- `cargo check --quiet --locked -p lancedb --features remote --tests
--examples`
- `cargo clippy --quiet --locked -p lancedb --features remote --tests
--examples -- -D warnings`
- `cargo test --quiet --locked -p lancedb --features remote --tests`
- library: 843 passed, 1 ignored
- integration groups: 39 passed, 6 passed, 5 passed
- focused coverage for empty directories, orphan data, physical listing
snapshots, zero parent listings, and concurrent manifest arbitration
Gatefixer
Dan Tasse
Will Jones
Wyatt Alt
Xuanwo
LanceDB Robot
XY Zhan
Lance Release
dependabot[bot]
Dan Rammer
Adityaj0
Igor Ganapolsky
Jack Ye
Yang Cen