A caller that queues a refresh and executes it later can only tell the
view it captured from a drop-and-recreate by comparing the definition
and
version. A recreated view with the same definition and an equal or
higher
version passes that check, and the check runs before the refresh reloads
the view, so it never sees the state it commits against.
This mints an `mv.incarnation` token in the schema metadata at each
physical creation of a view table.
`RefreshMaterializedViewBuilder::expect_incarnation` carries the
captured
token into the refresh, which compares it against the latest stored
manifest before planning and again immediately before each commit
(publish, fragment swap, watermark stamp), refusing to land in a
different
incarnation. A view with no token -- declared before tokens existed, or
its metadata replaced wholesale -- is refused under a bound refresh with
its own wording and is minted one by its next unbound refresh. The token
is exposed through `MaterializedView::incarnation`; refreshes without an
expectation are unchanged.
This is best effort: the token is not part of lance's commit condition,
so a recreation landing between the final pre-commit read and the commit
itself is not caught. Closing that window needs a base-manifest
precondition in lance's commit path.
`table_names` paginates using a `start_after` table name. This works for
the `/v1/table` endpoint, which guarantees table-order. But the
`/v1/namespace/{id}/table/list` does not. We change that caller to
instead collect all table names, sort, and apply the pagination locally.
We are deprecating this API, so this is just an interim fix. For good
performance, users should move to the `list_tables` API instead, which
uses opaque tokens that don't rely on lexical sorting.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- add `StreamingDataLoader`, which transports worker snapshots with
prefetched batches and commits them to the parent dataset only when the
trainer receives each batch
- preserve exact non-uniform per-split progress and resume lagging
splits without replaying already-consumed rows
- reject stale parent checkpoints after a standard multi-process
`DataLoader` has started, with guidance to use the consumer-aware loader
- document the new public loader and merge non-uniform state across
ranks
## Root cause
PyTorch runs `StreamingDataset.__iter__` in private worker-process
copies, while callers invoke `state_dict()` on the parent dataset.
Sharing producer counters would still be incorrect because DataLoader
prefetch can advance workers beyond batches returned to the trainer.
## Validation
- `uv run --extra tests pytest python/tests/test_elastic_dataloader.py
-q` (154 passed)
- focused non-uniform merge regression (1 passed)
- `uv run --project python --extra tests --extra dev ruff format .`
- `uv run --project python --extra tests --extra dev ruff check .`
- `cd docs && PYTHONPATH=. ../python/.venv/bin/mkdocs build`
Fixes#3967
<!-- lance-gatekeeper-fix:v1 agent=572be272619660b97e87fd5c85188341
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Listing tables a page at a time against a local database silently
skipped one table at every page boundary. `ListingDatabase::list_tables`
returned the first name of the *next* page as that page's token, but
resuming from a token drops every name at or before it — so the table
the token named was never handed to the caller. Walking `[a, b, c, d,
e]` with a limit of 2 returned `[a, b, d, e]`.
This PR returns the last name of the page as the token instead, which is
what resuming after the token expects.
This is reachable from Python today through
`db.list_tables(page_token=...)` on a local connection; it also affects
`len(db)` and `name in db`, which walk the pages. Remote and
namespace-backed connections page on the server and were never affected.
## Example
```python
db = lancedb.connect(tmp_path)
for name in ["a", "b", "c", "d", "e"]:
db.create_table(name, [{"id": 1}])
names, token = [], None
while True:
page = db.list_tables(page_token=token, limit=2)
names += page.tables
token = page.page_token
if not token:
break
# before: ['a', 'b', 'd', 'e']
# after: ['a', 'b', 'c', 'd', 'e']
```
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Problem
`refresh_column_async` returned a unit-result job even though durable
refresh jobs carry a canonical terminal result. Python callers could not
obtain row counts or source and published versions through the public
`Job` API, and local and remote refresh jobs exposed different result
semantics.
## Behavior
`refresh_column_async` now returns `Job[RefreshColumnResult]` for local
and remote tables. The general typed-job bridge binds each endpoint to
its public result model while preserving unit-result jobs and existing
status, wait, cancel, and timeout behavior. A local no-op refresh
reports no published version.
The Node.js API continues to resolve `wait()` as `void`; its binding
erases the Rust result type internally to preserve the existing public
contract.
## Ownership and integration boundary
LanceDB owns the language-neutral `Job<T>` contract and language-binding
decode. Sophon owns production and durable persistence of terminal
payloads. Sophon #7348 and #7378 now publish the canonical refresh
result for Function-backed and expression-backed refresh jobs,
respectively. The remote client fixture matches the merged server
schema; live deployment and end-to-end demo acceptance remain separate
rollout checks.
## How packing works
Consider four tokenized documents:
[1]
[2]
[10, 11, 12, 13, 14, 15, 16, 17]
[20]
With:
```
StreamingDataset(
table,
shuffle=False,
columns=["tokens"],
num_splits=2,
pack_sequences=5,
eos_id=9,
pad_id=0,
blocks_per_epoch=6,
)
```
the documents are assigned to two fixed logical splits. Each split
maintains an independent token buffer, appends eos_id after every
document, and emits blocks of five tokens.
Because blocks_per_epoch=6, each split emits exactly three blocks:
Cycl/e 1:
Split 0: [1, 9, 2, 9, 0] # 9 is eos, 0 is padding
Split 1: [10, 11, 12, 13, 14]
Cycle 2:
Split 0: [0, 0, 0, 0, 0]
Split 1: [15, 16, 17, 9, 20]
Cycle 3:
Split 0: [0, 0, 0, 0, 0]
Split 1: [9, 0, 0, 0, 0]
If a split runs out of tokens early, it emits padded blocks through the
fixed budget. This prevents one rank from finishing before another.
Logical splits are independent of rank and worker ownership. A
checkpoint records each split’s consumed-document count, emitted-block
count, remaining tokens, and document boundaries. Merging
those per-split states allows the same packed stream to resume after the
topology changes.
doc_ids identifies document segments, including continuations across
block boundaries. It is not a padding mask: padding retains the
preceding document ID, so callers must mask padding using a
reserved pad_id.
blocks_per_epoch="auto" is also available. It estimates the budget from
a deterministic bounded sample and warns that the result is approximate.
WIP pre-training tests:
```
┌────────────────────────────────────┬────────────────────────┬─────────────────────────────────────┐
│ │ GPT-2 124M │ GPT-2 medium 354M │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Corpus │ 2.4M docs / 12GB table │ 9.67M docs / 45GB table │
│ Tokens (Chinchilla) │ 2.43B │ 7.0B │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Data prep (ingest→curate→tokenize) │ ~12 min │ ~51 min │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Training wall time │ ~50 min │ 3h 06m │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Throughput / MFU │ 1.60M tok/s / 35% │ 684k tok/s / 42.0%, │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Final val loss │ 3.230 │ 2.840 │
└────────────────────────────────────┴────────────────────────┴─────────────────────────────────────┘
```
---------
Co-authored-by: OpenAI Codex <codex@openai.com>
Rename prefetch_batches → io_queue_depth and introduce
transform_queue_depth as a symmetric pair: both express "number of
batches to buffer per split at this pipeline stage." The old names are
still accepted as keyword arguments but log a deprecation warning
redirecting callers to the new names.
transform_queue_depth caps how many transform-result batches can
accumulate per split in the post-transform queue. Without this limit a
slow consumer (e.g. a GPU training step) causes cooked rows to pile up
unboundedly. The backpressure check in _try_submit_tx counts both
already-cooked rows and rows expected from in-flight transforms; it
skips proactive transform submission when the combined total reaches the
limit. The reactive _ensure_cooked path bypasses the check so the
consumer never stalls.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Exposes materialized views to TypeScript: createMaterializedView,
openMaterializedView and listMaterializedViews on Connection, and a
MaterializedView handle carrying the parsed definition and
refresh({full, sourceVersion}), which returns the typed refresh result.
select accepts column names, [alias, expression] pairs, or a record of
the
same; the definition reads back off the stored schema, so a reopened
handle
needs no side channel. Remote connections surface the core's
not-supported
error up front.
The napi crate needed the same recursion-limit raise as the core crate:
the
refresh future's type graph overflows the default trait-recursion depth.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
This PR is a **breaking** rename of #3686.
merge reads like git merge w/ three-way, replay history, combine two
lines of work. That is not this API.
This call takes one additive change on a branch and lands it on main.
New column, including a blob column. Main's existing columns are not
rewritten. If it cannot land, you get `status="failed"` and
`diff.errors`, not a merge conflict to resolve.
Cherry-pick is terminology that aligns more with that.
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.cherry_pick("exp", dry_run=True)
result = table.branches.cherry_pick("exp")
if result["status"] == "cherryPicked":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "failed":
print(result["diff"]["errors"])
```
### Behavior
- Remote / Enterprise only. Local still NotSupported.
- HTTP 409 is not an exception. It is Ok with status="failed" and
diff.errors (CherryPickError).
- Unknown error / status codes still parse as Unknown.
- Requests are not retried. 409 is final and carries the body.
- Endpoint is POST /v1/table/{id}/branches/cherry_pick/.
- merge_insert and Table.merge are unchanged.
### Testing
- `cargo test -p lancedb --features remote diff_branch`
- `cargo test -p lancedb --features remote cherry_pick`
- `pytest python/python/tests/test_remote_db.py -k cherry_pick`
- node `remote.test.ts` diffs / cherry-picks path
Exposes materialized views to Python in both the async and sync clients:
create_materialized_view / open_materialized_view /
list_materialized_views
on the connections, and MaterializedView / AsyncMaterializedView handles
carrying the parsed definition and refresh(full=, source_version=),
which
returns the typed refresh result. select accepts column names, (alias,
expression) pairs, or a dict of the same; the definition reads back off
the
stored schema, so a reopened handle needs no side channel. Remote
connections raise NotImplementedError up front rather than failing deep
in
a request, matching the computed-column convention.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
Example tests pin behaviors; the refresh contract is a property: after
any
sequence of source mutations, a view maintained by default refreshes
equals
the definition evaluated against the source directly, and so does a
forced
rebuild. This drives every mutation sequence up to length three --
appends,
deletes, updates crossing the filter, compactions, unrelated column adds
--
over an identity and a filtered view shape, checking against an oracle
that
shares nothing with the refresh path: a plain column scan with the
filter
applied in Rust. The oracle runs after every step because a later
rebuild-forcing mutation silently heals an incremental error; end-state
checks miss exactly the transient bugs that matter. A length-four sweep
runs behind
ignore.
Named regressions additionally assert the refresh mode, which value
comparison cannot: a wrongly rebuilding classifier still matches the
oracle, so the append, unrelated-column and compaction cases pin that
the
incremental path actually ran.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
A declared view holds no rows; refresh computes them. It pins one source
version, brings the view to exactly the definition's result at that
version,
and records the version as a watermark in the view's schema metadata.
It is incremental when it can reconcile what changed: appended rows are
computed and appended, and rows the source deleted or updated are found
by
the lance delta and evicted by their __source_row_id provenance, the
updated
ones recomputed in the same commit. Compaction rearranges rows without
changing
them, so its outputs cost nothing -- which is what keeps routine
background
compaction from rebuilding the view. A vacuumed watermark, a
delta the transaction-log walk cannot classify, a Legacy-storage source,
or
more staged ids than a fixed cap all fall back to a rebuild; rebuilding
an
indexed view swaps every fragment in one Update, so readers never see it
unindexed or empty.
Concurrent refreshes serialize at commit -- each carries the
same sentinel row id in its inserted-rows filter, so the loser lands
nothing. On the append path the watermark moves in a follow-up commit,
so a
crash between the two re-appends those rows. Bumps lance
to v11.0.0-beta.19 for the delta reader.
## Summary
- reject whole-document `lance.json` fields during native BITMAP index
preparation
- preserve BITMAP support for raw `LargeBinary` fields
- return guidance to use a JSON-path scalar index or FTS instead
- add regression coverage for the logical JSON type while retaining the
existing raw binary coverage
## Root cause
Native scalar-index validation resolved the complete Arrow field but
checked BITMAP compatibility only against its physical data type.
Because `lance.json` is stored as `LargeBinary`, it was incorrectly
accepted under the raw binary compatibility rule.
The fix reuses Lance’s `lance_arrow::json::is_json_field` helper before
physical type validation. Remote serialization is unchanged, so remote
clients continue to send the requested BITMAP type for server-side
validation.
## Validation
- `cargo fmt --all -- --check`
- `cargo test --quiet --features remote -p lancedb
test_create_bitmap_index -- --nocapture`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
Fixes#3889
<!-- lance-gatekeeper-fix:v1 agent=e097fc02a548edc0d0be2e18c65c03a3
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a merge-insert regression test whose fixed-size-list child count
crosses `u32::MAX`
- verify delete-by-source updates the matching row, deletes every other
row, and completes without an Arrow panic
- use a null child array so the boundary case avoids allocating a real
vector payload
## Root cause and fix
The affected Lance merge fallback carried the target payload through a
full outer hash join. Arrow's fixed-size-list take kernel uses `u32`
child indices, so taking a target row whose child offset crossed
`u32::MAX` wrapped the offset and produced child data shorter than the
parent array, triggering the reported `ArrayData::slice` assertion.
The projection-aware merge path in the Lance version now used by `main`
avoids materializing the target fixed-size-list payload in that join.
This regression test locks in that production behavior at the exact
child-index boundary.
## Validation
- `cargo fmt --all`
- `cargo test --quiet --features remote -p lancedb
test_merge_insert_fixed_size_list_above_u32_child_count`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
Fixes#2874
<!-- lance-gatekeeper-fix:v1 agent=582e68bcad65739e189352cb3cbf144c
generation=3 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- expose `memory_limit` and `num_workers` on the Python FTS
configuration for local builds
- forward both build-only settings to the Lance inverted-index builder
- add an end-to-end regression proving the configured memory budget
reaches the native build
## Root cause
LanceDB 0.26.1 pinned Lance 1.0.1. That Lance version used an FTS
partition-merge path whose retained data made memory grow with merge
progress on very large indexes. Upstream Lance
[#5754](https://github.com/lance-format/lance/pull/5754) changed
partition merging to stream its inputs, reducing peak memory by about
25%. Lance [#6174](https://github.com/lance-format/lance/pull/6174) then
removed the old merge phase, compressed posting lists during
construction, reduced indexing memory by about 60%, and introduced a
total build `memory_limit` for bounded workers.
Current `main` pins Lance 11.0.0-beta.3, which contains those
architectural fixes. This PR does not duplicate or claim the upstream
leak fix; it addresses the remaining Python API gap.
## This repair
LanceDB Python did not expose the native FTS builder resource controls.
`memory_limit` now sets the total local-build budget in MiB, divided
among effective workers, and `num_workers` controls build parallelism.
Both are build-only settings and do not affect remote builds or
persisted index configuration.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo fmt --all`
- `uv run --project python --extra tests --extra dev ruff check .`
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/lancedb/index.py python/python/tests/test_fts.py`
- `uv run --project python --extra tests pytest python/tests/test_fts.py
-q` (51 passed)
Fixes#2923
<!-- lance-gatekeeper-fix:v1 agent=a1ceedf74531e0212cb6f1ebf9390a26
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
A materialized view is a table whose contents are defined by a query
over
one source table and maintained by refresh rather than by writes.
The declaration half: create_materialized_view(name, source) resolves a
projected, filtered and limited definition against the source schema --
output types come from the DataFusion planner, never the caller -- and
commits an empty table carrying it as kind-tagged JSON in schema
metadata.
The tag lets a kind added later read back as a view this version cannot
refresh rather than as a plain table. Views open and list as ordinary
tables.
Sources must have stable row ids, checked here because the property
cannot
be enabled later: each view row records its source row in
__source_row_id,
and that provenance survives compactions, updates and deletes only when
row
ids are stable.
A view inherits the metadata describing its columns and none governing
how a
table is written, so blob markers carry through while declarations its
always-nullable fields would contradict are stripped. Embedding
configuration is rewritten to the view's column names, and dropped where
it
does not project both ends of a function.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
## Summary
- document that current LanceDB releases use one compute worker without
warning on two-vCPU containers
- distinguish compute-worker tuning from storage I/O concurrency
- direct users of affected LanceDB 0.21.1 installations to upgrade and
link the current threading guidance
## Root cause
The Lance version bundled with LanceDB 0.21.1 warned whenever the
detected CPU count was less than or equal to its default two-core I/O
reservation. A two-vCPU deployment therefore emitted the warning on
every query even though falling back to one compute worker was the
intended behavior. Lance fixed that warning condition upstream in
lance-format/lance#3710, and LanceDB current main already pins a version
containing the runtime fix; the Python package documentation did not
explain the corrected behavior or the distinct thread controls.
## Validation
- `git diff --check`
- verified the linked Lance threading-model documentation returns HTTP
200
Fixes#2326
<!-- lance-gatekeeper-fix:v1 agent=9f141242416a6dbeb43be0e80404dd4d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add cross-platform regression coverage for Azure table URI
construction
- assert that az:// database paths always produce forward-slash blob
keys
## Root cause
ListingDatabase previously used the host filesystem Path join operation
for object-store URIs, which inserted a backslash on Windows. The URI
construction was corrected in #2575, but the original Azure report had
no regression coverage and remained open.
## Validation
- cargo fmt --all
- cargo test --quiet --features remote -p lancedb
test_table_uri_uses_forward_slashes_for_azure
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples
Fixes#2283
<!-- lance-gatekeeper-fix:v1 agent=dd96adfd0fcf303c11e873300663d8f6
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`StreamingDataset`, `PermutationBuilder`, and `Permutation` now work
against a `RemoteTable` (LanceDB Cloud and Enterprise), which unblocks
benchmarking the loader against the enterprise cluster cache.
```python
db = lancedb.connect("db://my-db", api_key=..., host_override=...)
ds = StreamingDataset(db.open_table("training"), world_size=8, rank=r)
```
Rows are addressed by `_rowid` exactly as before —
`PermutationReader::load_batch` already built the same `_rowid IN (...)`
filter that `Table::take_row_ids` sends, so the loader's fetch was
always the take path. It just was never allowed to run.
### The guard
`PermutationBuilder.__init__` rejected anything without `_inner`, so a
`RemoteTable` raised `TypeError` before reaching the PyO3 layer — which
already unwraps one via `_table._inner`.
### A bounded schema lookup
`PermutationReader::output_schema` reads the schema off a query plan,
and building a plan on a remote table *executes* the query
(`create_plan` → `execute_query`). With no limit that is `k =
isize::MAX`, so asking a remote table for its output schema pulled the
whole table over HTTP and threw it away — once per assigned split, on
every epoch, since `StreamingDataset.__iter__` constructs a
`Permutation` per split.
One row rather than zero, deliberately: lance gates its limit node on
`self.limit.unwrap_or(0) > 0`, so `Some(0)` means *no limit*.
### Tables with an LSM write spec are refused
A permutation references rows by row id, and rows that have not been
flushed to the base table do not have one yet. The loader could read
around them, but they would then be missing from training with nothing
said about it, so the build refuses such a table up front instead of
half supporting it.
### Fallible identity construction
`PermutationReader::identity` resolved `inner_new` with `unwrap`. That
was near total against a local dataset, but construction counts the base
table — an HTTP round trip for a remote one — so a transient network or
auth failure became a panic across the PyO3 boundary.
### Tests
End-to-end `permutation_builder` and `StreamingDataset` runs against a
mock server, the former torch-free so it runs wherever the suite does,
plus a test that a build succeeds without an LSM write spec and is
refused once one is installed.
A registered `FunctionVersion` has an exact identity and grouped output
contract, but the Python SDK cannot currently bind it to table columns
without manually constructing wire models.
Calling a `FunctionVersion` with named `col(...)` references now returns
one immutable `FunctionApplication` pinned to that exact version. The
application preserves named-struct outputs as one sibling group, while
`rename(columns=...)` defines the result-field to table-column mapping
consumed by `Table.add_columns`. Derived expressions and incomplete or
unknown input names fail before declaration.
A remote backfill submission validates its target column against a table
snapshot, but it did not carry the existing read-after-write freshness
headers. Immediately after `add_columns`, a stale query node could
therefore reject the newly committed column.
Route backfill submission through the remote table read fence so it
carries the version returned by the preceding write. The shared remote
submission path gives synchronous and asynchronous client surfaces the
same freshness guarantee.
The Windows wheel job is the slowest job in the PyPI release workflow.
Fat LTO of the cdylib is single-threaded and the peak-memory step of the
build, so it does not get faster with more cores — and it has already
caused rustc-LLVM OOM on the Windows runners for the nodejs builds.
Switch the job to thin LTO with 16 codegen units on a
`windows-2025-8x-x64` runner, trading some runtime performance on our
least performance-sensitive platform for build time. This matches what
the nodejs Windows builds in `npm-publish.yml` already do.
`pypi-publish.yml` is in this workflow's `pull_request` paths filter, so
this PR triggers a dry-run build that shows the new timing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arrow-rs and datafusion crates are released in lockstep, but
Dependabot has been opening one PR per sub-crate for them — the 58.3.0
to 58.4.0 wave produced four separate PRs for `arrow`, `arrow-array`,
`arrow-schema`, and `arrow-buffer`. The existing `rust-minor-patch`
group did not catch them because it only filters on `update-types` and
declares no patterns.
This PR adds an explicit `arrow-datafusion` group matching `arrow*`,
`parquet*`, `datafusion*`, and `object_store`, so those bumps arrive as
a single PR. It is listed before `rust-minor-patch` because a dependency
joins the first group it matches, and it deliberately omits
`update-types` so major bumps are grouped too.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bugs in Node's reading of the embedding_functions schema metadata.
First, parseFunctions keyed its result map by function name, so a table
whose metadata configures the same function for two vector columns came
back with only the last one. It now keys by the vector column, the
convention Python's parser already uses.
Second, Node could not read metadata written by the Python bindings at
all, which spell the keys snake_case: configs parsed with both columns
undefined, breaking embedding application on add() and leaving only
query-side embedding working. The parse now accepts both spellings.
Both fixes land in one shared parser used by every reader --
parseFunctions and the makeArrowTable schema validator, which had its
own private camelCase-only parse -- so the wire contract cannot fork
between entry points. A config naming no source or vector column is an
error at the boundary rather than a default downstream, as are two
configs claiming one column. The "vector" fallback remains only on the
optional field of user-supplied configs.
Breaking: parseFunctions is exported and its map keys change from
function name to vector column.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Remote Function bindings can expose a nullable parameter schema even
when the source table column is non-nullable. Binding validation rebuilt
the exact input schema from table nullability and rejected this safe
widening.
Accept non-null table columns for nullable Function parameters while
continuing to reject nullable table columns for non-null parameters. All
other input schema fields remain exact, including named multi-input
ordering, names, types, and metadata.
Remote Function catalog requests used singular endpoints that are not
exposed by Phalanx. Route registration to `POST /v1/functions/create`
and exact-version lookup to `POST /v1/functions/get`, while preserving
the existing typed Job submission and wait behavior.
As in https://github.com/lancedb/lancedb/pull/3977, we're trying to
reduce anything in the lancedb skill that duplicates other docs. So this
shrinks the branch-ops logic down to a few lines that mostly just point
the agent to fetch the branching docs from lancedb.github.io.
Run stats (2 runs each):
<img width="1001" height="232" alt="Screenshot 2026-08-20 at 5 02 34 PM"
src="https://github.com/user-attachments/assets/a3f4d305-278e-4093-b153-07f0af57b251"
/>
This is out of order, rearranged:
|condition|time (sec)|cost|
|---|---|---|
|No branch_ops.md|250|1.33|
|No branch_ops.md|227|1.26|
|Old branch_ops.md|116|0.83|
|Old branch_ops.md|127|0.87|
|New branch_ops.md|135|0.86|
|New branch_ops.md|147|0.93|
Averaged between each of the two runs:
<img width="775" height="337" alt="Screenshot 2026-08-20 at 5 34 15 PM"
src="https://github.com/user-attachments/assets/4f2fb2a8-3112-4614-87d9-8dbf807f3b75"
/>
It seems helpful to have *some* doc about branching; otherwise the model
gets a little confused about our branch model and what methods to call.
But it looks like the new one (in this PR; all just references to
current docs) is basically as good as the old one (lots of duplicative
text).
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Problem
Generic job result propagation exposed the PyO3 representation of Rust's
unit value as `()` in Python. Unit jobs therefore returned an empty
tuple instead of `None`, breaking the documented `Job.wait()` contract
and the Python doctest workflow.
## Behavior
Unit job completion now converts explicitly to Python `None`. Typed job
results continue to pass through unchanged, with synchronous and
asynchronous regression coverage.
Follows lance-format/lance#8680, which removes
`FLAG_MEM_WAL_INDEX_CATCHUP`.
With one set of semantics there is no mode to switch into.
## Removed
`require_mem_wal_index_catchup` — the activation entry point — from the
trait,
from `Table`, and from the LSM merge module.
## The read path
`exclusion_watermarks` loses its `catchup_required` argument and keeps
the
conservative branch: an index with no entry is not known to hold these
rows, so
every generation stays readable from its SSTable. Nothing is excluded
until an
index records that it covers those generations, so a table that has
never
recorded catch-up reads every row from its SSTables rather than assuming
the
base covers them.
## One guard needed a replacement, not deletion
`refresh_column` and computed-column declaration refuse a table whose
rows sit
in un-compacted tiers, since refresh enumerates base fragments and would
silently omit them. They keyed on the feature bit because
`unset_lsm_write_spec` **drops the MemWAL index** — after an unset the
write
spec no longer describes such a table, and the bit was the only marker
that
outlived it. Two tests covered this, so deleting the term would have
dropped a
tested property.
Both guards now check for MemWAL shard directories on storage, which
outlive
the index. That is strictly wider than the bit ever was: the bit only
marked
tables where activation had run.
## Two tests conflated two different things
An index that is *caught up* and one that is *untracked* both fell back
to the
compaction watermark, because absence carried no information without the
bit.
Absence now means "not caught up", so untracked retains everything.
`an_untracked_index_does_not_widen_a_lagging_sibling` becomes
`an_untracked_index_retains_everything`, with the genuinely-caught-up
case
asserted separately.
## Testing
933 `lancedb` lib tests. `cargo fmt` clean. (The pre-existing
`Error::Http`
build failure in `job.rs` without the `remote` feature is unrelated and
untouched.)
The Linux Rust job can exhaust its disk after restoring a large fallback
target cache and compiling multiple feature graphs into one target
directory.
Run remote tests in an independent job with registry-only caching, and
run the simple example with all features so it reuses the preceding
build artifacts. This preserves remote coverage and fork behavior while
preventing all-features and remote-only artifacts from accumulating
together.
Failure evidence:
https://github.com/lancedb/lancedb/actions/runs/32467317540/job/96726650990
## Problem
The canonical Function wire values and typed remote Job contract do not
yet provide a Python authoring surface or catalog client, so users
cannot package a scalar callable, register it, or reopen the exact
immutable Function version.
## Behavior
This adds scalar-only `@udf` authoring with deterministic annotation or
explicit Arrow schema validation, content-addressed Python artifacts,
and an internal scalar-to-Arrow-batch adapter descriptor. Registration
payloads model non-secret environment values and secret names only.
Remote connections can submit `create_function_async` and receive a
typed `Job<FunctionVersion>`, then reopen that exact version by name and
version ID. Synchronous connections can call `create_function` to submit
and wait for the immutable version in one operation. Local Function
catalog operations return a stable `NotSupported` error. Shared
Rust/Python golden payloads and mocked catalog responses freeze the
request, typed terminal result, and exact lookup contract.
## Validation
- Rust formatting, remote check, clippy, and focused LDB-1/LDB-2 tests
- Python formatting, lint, and focused LDB-1/LDB-2 tests
- Python API documentation build
Function applications from the canonical remote contract cannot
currently declare scalar or grouped computed-column outputs atomically.
This adds the remote-only declaration contract for scalar,
struct-as-one-column, and expanded named-struct outputs. It validates
result mappings, fixes exact input/output Arrow schemas in the request,
persists grouped sibling metadata, and keeps local Function execution
unsupported. Unknown newer application or binding metadata remains
readable, while schema-changing mutations fail closed instead of
rewriting it.
Stable Lance field IDs are deliberately not a declaration prerequisite
in this slice. Inputs bind by parameter name and field path; Sophon
remains responsible for exact-version validation, atomic all-NULL
sibling creation, binding identity and revision allocation, and
persisted output identities.
LanceDB's Python SDK now requires Pydantic `>=2.7.4,<3` and uses the v2
APIs throughout. This removes dual-version behavior from schema
conversion, query serialization, embedding models, and Function wire
models while preserving their existing public and canonical-wire
behavior.
The minimum-dependencies CI job pins Pydantic 2.7.4 so the declared
compatibility floor remains covered.
Local benchmarks currently inherit the release profile's fat LTO and
single codegen unit, making local iteration pay release-artifact build
costs.
Provide repository-defined profiles for no-LTO local work and cheaper
benchmark builds, and document when each profile is appropriate. Release
artifacts continue to use fat LTO.
PyO3 exposed `LsmWriteSpec` with its default `builtins` module, causing
mkdocstrings to resolve the public `lancedb.LsmWriteSpec` re-export as
`builtins.LsmWriteSpec` and fail the documentation build. Declare the
native extension module and pin the public re-export with a regression
test.
This also applies the repository's current Ruff formatter to seven
previously unformatted Python scripts.
## Problem
Enterprise Function-backed computed columns need a stable SDK contract
before Sophon catalog and execution endpoints can be added. The existing
`Job` API can only represent unit terminal results, and there is no
shared Rust/Python wire definition for immutable Function versions,
applications, bindings, or refresh results.
## Behavior
This introduces remote-only canonical Function values in Rust and
Python, evolves `Job<T = ()>` to decode typed remote terminal results
while keeping local spawned operations unit-typed, and fixes the
cross-language contract with shared JSON golden fixtures. Unknown fields
and discriminator values remain forward-decodable, while canonical
output contains only fields known to the client. Function models contain
secret names only.
Sophon remains the sole owner of catalog persistence, environment bake,
secret resolution, execution, and publication. This PR does not add
authoring/catalog endpoints, local execution, refresh runners, or live
Sophon E2E coverage.
Background: if we keep adding stuff to the lancedb skill that repeats
other knowledge, we're basically creating a whole new docs site, which
means one more thing that can get out of date. Worse, if it gets out of
date, it will tell agents to do the wrong thing.
These files were added without a ton of analysis of whether they'd be
improving agent performance at all. It looks like they don't really:
<img width="644" height="90" alt="Screenshot 2026-08-20 at 5 21 03 PM"
src="https://github.com/user-attachments/assets/44e60436-b7ad-498b-8e73-0181385c7c60"
/>
(top run is without these docs, bottom run is with them - arguably these
docs might even make the agent a little slower! that's probably noise
though; I'd just say at least they're unnecessary.)
So this PR just removes them. We'll more judiciously add bits we need
and/or point to preexisting docs, to avoid duplication.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Lance Release
Wyatt Alt
Will Jones
lancedb-gatefixer[bot]
Xuanwo
Ayush Chaurasia
LanceDB Robot
Weston Pace
Drew Gallardo
Jack Ye
Dan Tasse
XY Zhan