A Function's remote environment can now be conda instead of pip.
`@udf(conda=[...], conda_channels=[...])` registers one; pip and conda are
exclusive, channels are priority-ordered and require conda. The Rust and
Python `PythonEnvironmentSpec` models gain `channels`, dropped from the
canonical JSON when empty so existing pip registrations keep their digests.
Updates the Lance Rust workspace dependencies and Java lance-core
dependency to
[v12.0.0-beta.1](https://github.com/lance-format/lance/releases/tag/v12.0.0-beta.1).
Includes compatibility updates for the renamed shard-manifest API and
paginated object-store wrappers.
## Summary
- lazily initialize built-in OpenAI and Hugging Face providers when
consumers call the public embedding registry API
- choose automatic vector versus FTS search from embedding metadata on a
fresh pinned table revision for every execution
- expose automatic string searches as an `AutoQuery` with only
operations common to both native query families
- keep the registry shared and built-in registration safe across
duplicated module graphs
## Root cause
Nitro treats dependency modules as side-effect-free and removes the bare
OpenAI provider import from its generated route. Registration therefore
never runs, so `getRegistry().get("openai")` remains undefined even when
the registry itself is shared globally. Bundlers may also duplicate the
provider and registry module graphs.
The public embedding entry point now initializes built-in providers only
when `getRegistry()` is explicitly called, keeping initialization on a
live path that Nitro retains. Each terminal automatic-search execution
pins the exact table revision visible at dispatch, reads embedding
metadata and computes an embedding from that snapshot, replays the
builder operations, and constructs and executes the selected native
query against the same snapshot. Pinned native snapshots execute locally
when namespace pushdown cannot carry their revision, while remote
snapshots are seeded directly from one version-and-schema response. The
public `AutoQuery` builder exposes only the operations shared by FTS and
vector search, so runtime class narrowing cannot expose invalid
vector-only methods. Repeated built-in registration replaces stale
constructors from duplicated module graphs while public `register()`
retains its duplicate-alias error.
## Validation
- `cargo fmt --all`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `pnpm build`
- `pnpm lint`
- `pnpm run docs`
- `pnpm test --runInBand` (783 passed, 5 skipped)
- serial examples suite with a local OpenAI mock (11 passed), including
`sentence-transformers.test.ts`
- packaged Nitro 2.13.4 server route using the reported imports returned
`{"registered":true}`
- fresh-process FTS fixture initialized both public built-ins and
confirmed automatic string search still returned the indexed row
- schema-consistency regressions cover read-consistency refresh,
checkout, checkoutLatest, restore, runtime class narrowing, concurrent
overwrite during embedding computation, and reused automatic-search
builders
- focused regressions confirm pinned native snapshots bypass unversioned
namespace pushdown and remote snapshots use one describe request
Fixes#2429
<!-- lance-gatekeeper-fix:v1 agent=2adf0f21b8bfb634606ed8897a849e30
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary
- document that Azure Blob Storage credentials can be passed directly
through `storage_options`
- provide valid quoted `account_name` and `account_key` examples for
both sync and async Python connections
- execute the option dictionaries during doctests so the original
unquoted-key mistake is caught
## Root cause
The historical Python storage guide used `account_name` and
`account_key` as bare identifiers in dictionary literals. Following that
example either raised `NameError` or, when those names were predefined,
produced incorrect option keys. The runtime already accepts direct Azure
credentials, but the current Python API reference did not contain a
corrected Azure example.
## Validation
- `python/.venv/bin/ruff format --check
python/python/lancedb/__init__.py`
- `python/.venv/bin/ruff check .`
- `cd python && uv run --no-sync pytest --doctest-modules
python/lancedb/__init__.py -q`
- `cd python && uv run --no-sync pytest python/tests/test_import.py -q`
Fixes#2236
<!-- lance-gatekeeper-fix:v1 agent=7a7a9e009eb2ebe52ac9b1adf2e8afb2
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
#3528 added blob declarations and binary coercion. String values were
still rejected. They now coerce to the blob `uri` child.
```python
table.add([{"id": 1, "image": "s3://bucket/media/cat.jpg"}])
payload = table.fetch_blobs("image", table.search().to_arrow())
```
A URI under a registered base writes with no extra options. An
unregistered URI fails. `allow_external_blob_outside_bases` is a local
escape hatch that stores an absolute URI. It does not register a base.
Remote `add` rejects that flag before making a request. String input
still coerces and is sent as a `uri` struct.
`add_bases` is a follow-up. `merge_insert` does not coerce string blob
input.
### Testing
- `cargo test -p lancedb --test blob_integration`
- `cargo test -p lancedb blob_coerce`
- `cargo test -p lancedb --features remote --lib
add_rejects_external_blob_flag add_string_blob_becomes_uri_struct`
- `cd python && uv run --extra tests pytest python/tests/test_blob.py -k
uri -q`
Co-authored-by: Xuanwo <github@xuanwo.io>
## Problem
`uv run ... maturin develop` synchronizes the project as an editable
package before running the command. Maturin's editable build otherwise
uses the release profile, which enables the repository's fat LTO
configuration during local bootstrap.
## Behavior
Editable Python builds now explicitly use Cargo's dev profile. The
minimum maturin version is raised to 1.10, where `editable-profile`
support was introduced.
## Problem
The unreleased First-Class Function authoring API exposed
`secrets=[...]` and serialized `required_secrets`, promising runtime
resolution and injection that Sophon does not implement.
## Behavior
Remove the secrets dimension from the public Python decorator, Python
and Rust registration/version models, and shared wire fixtures. Existing
successfully registered Functions retain stable identity: the field
could only be empty and empty values were already omitted from canonical
serialization.
A stable LanceDB release has not shipped this Function API, so this
contracts the surface before it becomes a published compatibility
commitment.
## Validation gap
The Rust shared-golden Function suites and Python formatting/lint checks
pass locally. Python pytest was not run because the local environment
lacks its runtime dependencies and the frozen native editable build did
not complete in practical time.
Function applications and bindings currently encode `group_id` and
binding `revision` even though `binding_id` already owns the complete
immutable binding lifecycle and `outputs` already defines the atomic
multi-output set.
Make `binding_id` the sole binding identity, remove the redundant fields
from the Rust and Python client contracts, and describe multi-output
declarations directly. This intentionally replaces the removed wire
fields without a compatibility path.
Registering a real (embedding) Function failed on the client for three
reasons:
- `_package_source` treated `inspect.getclosurevars().unbound` as
"unresolved globals"; CPython puts attribute names there, so any body
with `np.linalg.norm(...)` or `body.split()` was rejected. Module-scope
references now come from Python's own scope analysis (`symtable`) over
the function source, recursively, and each is resolved the way the
interpreter would: the function's globals first (a module global may
shadow a builtin), then builtins. Free variables of nested scopes stay
lexical; postponed annotations are not runtime loads. A genuinely
missing global still fails.
- `_canonical_arrow_type` emitted spellings the server's frozen grammar
rejects (`fixed_size_list<T>[n]`, `timestamp[us]`, `struct<...>`,
zero-sized lists). It now emits exactly the grammar, with the server's
`fixed_size_list<item, size>` form, and the Rust declaration planner
parses that form too.
A shared golden
(`tests/fixtures/first_class_functions/v1/arrow_types.json`)
enumerates every grammar type, nested forms and rejected spellings; the
Python emitter and Rust parser are tested against it, and the same file
is under test in sophon. Packaging tests execute the shipped artifact in
a fresh namespace.
Contract changes (hence `breaking-change`):
- `@udf` now rejects namespace acquisition structurally
(`globals()`/`eval`/... by name, plus `import
sys`/`builtins`/`importlib`/`inspect` inside the body), requires the
function's captured `__builtins__` to be the standard mapping itself
(identity, so neither lookups nor implicit hooks such as `__import__`
can differ), rejects module globals that are namespace-bearing modules
(`builtins`, `sys`, ...), and treats the function's own name as
recursion only when the module binds it to the function or to the exact
`UdfDefinition` the decorator produced; it resolves module globals
through the function's real namespace (a module global may shadow a
builtin) and ships importable classes/functions as imports.
- List outputs must declare a non-nullable, metadata-free child named
`item` (`pa.list_(pa.field("item", t, nullable=False))`); that is what
the grammar means, and pyarrow's default nullable child was being
silently collapsed into it.
Contract, stated in the `udf` docstring: the artifact is a snapshot of
the function source plus exactly the module names it references.
Reaching the module namespace by another route is rejected where a
static packager can see it and is otherwise unsupported; there is no
dynamic-access detection beyond that.
A permutation stores `_rowid`s, which are row addresses unless stable
row ids are enabled. Nothing in the data loader pinned a table version,
so a compaction between building a permutation and reading it can
resolve those ids to different rows.
The exposure differs by backend but exists on both:
- Remote never pins. `prepare_query_bodies` stamps `"version":
current_version()` on every request, but `current_version()` is `None`
unless `checkout` was called, so every request means "latest".
- Native pins implicitly by holding an `Arc<Dataset>` under
`ConsistencyMode::Lazy`, but `StreamingDataset.__setstate__` reopens the
table in each DataLoader worker, so each worker pins to whatever is
latest at fork time.
## Changes
`Table::at_version` returns an independent handle pinned to a version
without mutating the receiver. `checkout` cannot serve this: on remote
the version cell is an `Arc<RwLock<Option<u64>>>` shared across clones,
so pinning through it would silently pin the caller's table too.
`PermutationBuilder::build` pins for the whole build and records the
version in the permutation table's schema metadata, alongside the
existing split names. `PermutationReader` pins the base table to that
version before any take.
Because the reader pins on construction, the Python worker fork is
covered without touching the pickle format — `Permutation.__setstate__`
drops the reader and `_ensure_open` rebuilds it, which re-pins.
## Behaviour change
A permutation is now bound to the version it was built against, so rows
appended to the base table afterwards are not visible through an
existing permutation. That is the intended semantics — the permutation
only addresses rows that existed when it was built — but it is a change
worth flagging.
Permutations written before this carry no version key and read exactly
as they did before.
## Summary
- compare inferred Arrow types by their semantic representation across
records
- throw the schema inference error when a later record has an
incompatible type
- cover compatible and incompatible multi-record inference across
supported Arrow versions
## Root cause
Schema inference compared newly allocated Arrow DataType objects by
identity, so equivalent inferred types did not compare equal. The
mismatch path also constructed an Error without throwing it, which
silently accepted incompatible values.
## Validation
- pnpm test __test__/arrow.test.ts --runInBand (176 tests passed)
- pnpm lint
- pnpm build
- pnpm run docs
Fixes#3781
<!-- lance-gatekeeper-fix:v1 agent=00ec4f61a3fd82694fa4fb9bb2b37aa8
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`table_names` is being replaced by `list_tables` across the SDKs, but
TypeScript only had `tableNames`. This PR adds `listTables`, which
returns a page of table names together with the token that resumes after
it, and marks `tableNames` and `TableNamesOptions` deprecated in favor
of it.
It binds the `Connection::list_tables` that already exists, so nothing
in the Rust API changes and nothing existing breaks. `pageToken` is
documented as opaque rather than as a table name, since what resumes a
listing is the database's to decide — that keeps callers off a detail
that is going to change.
Stacked on #4040, which fixes a table being dropped at every page
boundary. The page-walking test here needs that fix to pass. Review the
last commit only until #4040 lands.
## Example
```ts
const names = [];
let pageToken = undefined;
do {
const page = await conn.listTables({ pageToken, limit: 100 });
names.push(...page.tables);
pageToken = page.pageToken;
} while (pageToken);
```
A namespace can be listed by passing its path first, mirroring
`tableNames`:
```ts
const page = await conn.listTables(["analytics"], { limit: 100 });
```
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes#4015
<!-- lance-gatekeeper-fix:v1 agent=1c6407eec60f0af2336ae2dee811e1be
generation=1 -->
## Root cause
PermutationBuilder issued count_rows and the projected row-ID scan
through an unpinned RemoteTable handle. Each request could independently
resolve latest, so a concurrent table update could make the count and
scanned rows come from different snapshots.
## Fix
- add a backend hook for obtaining an independent handle pinned to the
currently selected version
- resolve latest once for remote tables while preserving an explicit
checkout and leaving the caller handle unchanged
- build the count, filtered projection, and scan from that pinned handle
while retaining native-table behavior
- add a remote mock regression that advances latest between count and
scan and covers explicit checkout preservation
## Validation
- cargo test --quiet --features remote -p lancedb --lib
test_remote_permutation_builder_pins_snapshot
- cargo test --quiet --features remote -p lancedb --lib
dataloader::permutation::builder::tests
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples
- cargo fmt --all -- --check
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
A caller that queues a refresh and executes it later can only tell the
view it captured from a drop-and-recreate by comparing the definition
and
version. A recreated view with the same definition and an equal or
higher
version passes that check, and the check runs before the refresh reloads
the view, so it never sees the state it commits against.
This mints an `mv.incarnation` token in the schema metadata at each
physical creation of a view table.
`RefreshMaterializedViewBuilder::expect_incarnation` carries the
captured
token into the refresh, which compares it against the latest stored
manifest before planning and again immediately before each commit
(publish, fragment swap, watermark stamp), refusing to land in a
different
incarnation. A view with no token -- declared before tokens existed, or
its metadata replaced wholesale -- is refused under a bound refresh with
its own wording and is minted one by its next unbound refresh. The token
is exposed through `MaterializedView::incarnation`; refreshes without an
expectation are unchanged.
This is best effort: the token is not part of lance's commit condition,
so a recreation landing between the final pre-commit read and the commit
itself is not caught. Closing that window needs a base-manifest
precondition in lance's commit path.
`table_names` paginates using a `start_after` table name. This works for
the `/v1/table` endpoint, which guarantees table-order. But the
`/v1/namespace/{id}/table/list` does not. We change that caller to
instead collect all table names, sort, and apply the pagination locally.
We are deprecating this API, so this is just an interim fix. For good
performance, users should move to the `list_tables` API instead, which
uses opaque tokens that don't rely on lexical sorting.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- add `StreamingDataLoader`, which transports worker snapshots with
prefetched batches and commits them to the parent dataset only when the
trainer receives each batch
- preserve exact non-uniform per-split progress and resume lagging
splits without replaying already-consumed rows
- reject stale parent checkpoints after a standard multi-process
`DataLoader` has started, with guidance to use the consumer-aware loader
- document the new public loader and merge non-uniform state across
ranks
## Root cause
PyTorch runs `StreamingDataset.__iter__` in private worker-process
copies, while callers invoke `state_dict()` on the parent dataset.
Sharing producer counters would still be incorrect because DataLoader
prefetch can advance workers beyond batches returned to the trainer.
## Validation
- `uv run --extra tests pytest python/tests/test_elastic_dataloader.py
-q` (154 passed)
- focused non-uniform merge regression (1 passed)
- `uv run --project python --extra tests --extra dev ruff format .`
- `uv run --project python --extra tests --extra dev ruff check .`
- `cd docs && PYTHONPATH=. ../python/.venv/bin/mkdocs build`
Fixes#3967
<!-- lance-gatekeeper-fix:v1 agent=572be272619660b97e87fd5c85188341
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
Co-authored-by: Xuanwo <github@xuanwo.io>
Listing tables a page at a time against a local database silently
skipped one table at every page boundary. `ListingDatabase::list_tables`
returned the first name of the *next* page as that page's token, but
resuming from a token drops every name at or before it — so the table
the token named was never handed to the caller. Walking `[a, b, c, d,
e]` with a limit of 2 returned `[a, b, d, e]`.
This PR returns the last name of the page as the token instead, which is
what resuming after the token expects.
This is reachable from Python today through
`db.list_tables(page_token=...)` on a local connection; it also affects
`len(db)` and `name in db`, which walk the pages. Remote and
namespace-backed connections page on the server and were never affected.
## Example
```python
db = lancedb.connect(tmp_path)
for name in ["a", "b", "c", "d", "e"]:
db.create_table(name, [{"id": 1}])
names, token = [], None
while True:
page = db.list_tables(page_token=token, limit=2)
names += page.tables
token = page.page_token
if not token:
break
# before: ['a', 'b', 'd', 'e']
# after: ['a', 'b', 'c', 'd', 'e']
```
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Problem
`refresh_column_async` returned a unit-result job even though durable
refresh jobs carry a canonical terminal result. Python callers could not
obtain row counts or source and published versions through the public
`Job` API, and local and remote refresh jobs exposed different result
semantics.
## Behavior
`refresh_column_async` now returns `Job[RefreshColumnResult]` for local
and remote tables. The general typed-job bridge binds each endpoint to
its public result model while preserving unit-result jobs and existing
status, wait, cancel, and timeout behavior. A local no-op refresh
reports no published version.
The Node.js API continues to resolve `wait()` as `void`; its binding
erases the Rust result type internally to preserve the existing public
contract.
## Ownership and integration boundary
LanceDB owns the language-neutral `Job<T>` contract and language-binding
decode. Sophon owns production and durable persistence of terminal
payloads. Sophon #7348 and #7378 now publish the canonical refresh
result for Function-backed and expression-backed refresh jobs,
respectively. The remote client fixture matches the merged server
schema; live deployment and end-to-end demo acceptance remain separate
rollout checks.
## How packing works
Consider four tokenized documents:
[1]
[2]
[10, 11, 12, 13, 14, 15, 16, 17]
[20]
With:
```
StreamingDataset(
table,
shuffle=False,
columns=["tokens"],
num_splits=2,
pack_sequences=5,
eos_id=9,
pad_id=0,
blocks_per_epoch=6,
)
```
the documents are assigned to two fixed logical splits. Each split
maintains an independent token buffer, appends eos_id after every
document, and emits blocks of five tokens.
Because blocks_per_epoch=6, each split emits exactly three blocks:
Cycl/e 1:
Split 0: [1, 9, 2, 9, 0] # 9 is eos, 0 is padding
Split 1: [10, 11, 12, 13, 14]
Cycle 2:
Split 0: [0, 0, 0, 0, 0]
Split 1: [15, 16, 17, 9, 20]
Cycle 3:
Split 0: [0, 0, 0, 0, 0]
Split 1: [9, 0, 0, 0, 0]
If a split runs out of tokens early, it emits padded blocks through the
fixed budget. This prevents one rank from finishing before another.
Logical splits are independent of rank and worker ownership. A
checkpoint records each split’s consumed-document count, emitted-block
count, remaining tokens, and document boundaries. Merging
those per-split states allows the same packed stream to resume after the
topology changes.
doc_ids identifies document segments, including continuations across
block boundaries. It is not a padding mask: padding retains the
preceding document ID, so callers must mask padding using a
reserved pad_id.
blocks_per_epoch="auto" is also available. It estimates the budget from
a deterministic bounded sample and warns that the result is approximate.
WIP pre-training tests:
```
┌────────────────────────────────────┬────────────────────────┬─────────────────────────────────────┐
│ │ GPT-2 124M │ GPT-2 medium 354M │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Corpus │ 2.4M docs / 12GB table │ 9.67M docs / 45GB table │
│ Tokens (Chinchilla) │ 2.43B │ 7.0B │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Data prep (ingest→curate→tokenize) │ ~12 min │ ~51 min │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Training wall time │ ~50 min │ 3h 06m │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Throughput / MFU │ 1.60M tok/s / 35% │ 684k tok/s / 42.0%, │
├────────────────────────────────────┼────────────────────────┼─────────────────────────────────────┤
│ Final val loss │ 3.230 │ 2.840 │
└────────────────────────────────────┴────────────────────────┴─────────────────────────────────────┘
```
---------
Co-authored-by: OpenAI Codex <codex@openai.com>
Rename prefetch_batches → io_queue_depth and introduce
transform_queue_depth as a symmetric pair: both express "number of
batches to buffer per split at this pipeline stage." The old names are
still accepted as keyword arguments but log a deprecation warning
redirecting callers to the new names.
transform_queue_depth caps how many transform-result batches can
accumulate per split in the post-transform queue. Without this limit a
slow consumer (e.g. a GPU training step) causes cooked rows to pile up
unboundedly. The backpressure check in _try_submit_tx counts both
already-cooked rows and rows expected from in-flight transforms; it
skips proactive transform submission when the combined total reaches the
limit. The reactive _ensure_cooked path bypasses the check so the
consumer never stalls.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Exposes materialized views to TypeScript: createMaterializedView,
openMaterializedView and listMaterializedViews on Connection, and a
MaterializedView handle carrying the parsed definition and
refresh({full, sourceVersion}), which returns the typed refresh result.
select accepts column names, [alias, expression] pairs, or a record of
the
same; the definition reads back off the stored schema, so a reopened
handle
needs no side channel. Remote connections surface the core's
not-supported
error up front.
The napi crate needed the same recursion-limit raise as the core crate:
the
refresh future's type graph overflows the default trait-recursion depth.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
This PR is a **breaking** rename of #3686.
merge reads like git merge w/ three-way, replay history, combine two
lines of work. That is not this API.
This call takes one additive change on a branch and lands it on main.
New column, including a blob column. Main's existing columns are not
rewritten. If it cannot land, you get `status="failed"` and
`diff.errors`, not a merge conflict to resolve.
Cherry-pick is terminology that aligns more with that.
```python
table = db.open_table("images")
table.branches.create("exp")
exp = table.branches.checkout("exp")
exp.add_columns({"tag": "cast('draft' as string)"})
diff = table.branches.diff("exp")
preview = table.branches.cherry_pick("exp", dry_run=True)
result = table.branches.cherry_pick("exp")
if result["status"] == "cherryPicked":
print("landed at", result["mainVersionAfter"])
elif result["status"] == "failed":
print(result["diff"]["errors"])
```
### Behavior
- Remote / Enterprise only. Local still NotSupported.
- HTTP 409 is not an exception. It is Ok with status="failed" and
diff.errors (CherryPickError).
- Unknown error / status codes still parse as Unknown.
- Requests are not retried. 409 is final and carries the body.
- Endpoint is POST /v1/table/{id}/branches/cherry_pick/.
- merge_insert and Table.merge are unchanged.
### Testing
- `cargo test -p lancedb --features remote diff_branch`
- `cargo test -p lancedb --features remote cherry_pick`
- `pytest python/python/tests/test_remote_db.py -k cherry_pick`
- node `remote.test.ts` diffs / cherry-picks path
Exposes materialized views to Python in both the async and sync clients:
create_materialized_view / open_materialized_view /
list_materialized_views
on the connections, and MaterializedView / AsyncMaterializedView handles
carrying the parsed definition and refresh(full=, source_version=),
which
returns the typed refresh result. select accepts column names, (alias,
expression) pairs, or a dict of the same; the definition reads back off
the
stored schema, so a reopened handle needs no side channel. Remote
connections raise NotImplementedError up front rather than failing deep
in
a request, matching the computed-column convention.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
Example tests pin behaviors; the refresh contract is a property: after
any
sequence of source mutations, a view maintained by default refreshes
equals
the definition evaluated against the source directly, and so does a
forced
rebuild. This drives every mutation sequence up to length three --
appends,
deletes, updates crossing the filter, compactions, unrelated column adds
--
over an identity and a filtered view shape, checking against an oracle
that
shares nothing with the refresh path: a plain column scan with the
filter
applied in Rust. The oracle runs after every step because a later
rebuild-forcing mutation silently heals an incremental error; end-state
checks miss exactly the transient bugs that matter. A length-four sweep
runs behind
ignore.
Named regressions additionally assert the refresh mode, which value
comparison cannot: a wrongly rebuilding classifier still matches the
oracle, so the append, unrelated-column and compaction cases pin that
the
incremental path actually ran.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
A declared view holds no rows; refresh computes them. It pins one source
version, brings the view to exactly the definition's result at that
version,
and records the version as a watermark in the view's schema metadata.
It is incremental when it can reconcile what changed: appended rows are
computed and appended, and rows the source deleted or updated are found
by
the lance delta and evicted by their __source_row_id provenance, the
updated
ones recomputed in the same commit. Compaction rearranges rows without
changing
them, so its outputs cost nothing -- which is what keeps routine
background
compaction from rebuilding the view. A vacuumed watermark, a
delta the transaction-log walk cannot classify, a Legacy-storage source,
or
more staged ids than a fixed cap all fall back to a rebuild; rebuilding
an
indexed view swaps every fragment in one Update, so readers never see it
unindexed or empty.
Concurrent refreshes serialize at commit -- each carries the
same sentinel row id in its inserted-rows filter, so the loser lands
nothing. On the append path the watermark moves in a follow-up commit,
so a
crash between the two re-appends those rows. Bumps lance
to v11.0.0-beta.19 for the delta reader.
## Summary
- reject whole-document `lance.json` fields during native BITMAP index
preparation
- preserve BITMAP support for raw `LargeBinary` fields
- return guidance to use a JSON-path scalar index or FTS instead
- add regression coverage for the logical JSON type while retaining the
existing raw binary coverage
## Root cause
Native scalar-index validation resolved the complete Arrow field but
checked BITMAP compatibility only against its physical data type.
Because `lance.json` is stored as `LargeBinary`, it was incorrectly
accepted under the raw binary compatibility rule.
The fix reuses Lance’s `lance_arrow::json::is_json_field` helper before
physical type validation. Remote serialization is unchanged, so remote
clients continue to send the requested BITMAP type for server-side
validation.
## Validation
- `cargo fmt --all -- --check`
- `cargo test --quiet --features remote -p lancedb
test_create_bitmap_index -- --nocapture`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
Fixes#3889
<!-- lance-gatekeeper-fix:v1 agent=e097fc02a548edc0d0be2e18c65c03a3
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add a merge-insert regression test whose fixed-size-list child count
crosses `u32::MAX`
- verify delete-by-source updates the matching row, deletes every other
row, and completes without an Arrow panic
- use a null child array so the boundary case avoids allocating a real
vector payload
## Root cause and fix
The affected Lance merge fallback carried the target payload through a
full outer hash join. Arrow's fixed-size-list take kernel uses `u32`
child indices, so taking a target row whose child offset crossed
`u32::MAX` wrapped the offset and produced child data shorter than the
parent array, triggering the reported `ArrayData::slice` assertion.
The projection-aware merge path in the Lance version now used by `main`
avoids materializing the target fixed-size-list payload in that join.
This regression test locks in that production behavior at the exact
child-index boundary.
## Validation
- `cargo fmt --all`
- `cargo test --quiet --features remote -p lancedb
test_merge_insert_fixed_size_list_above_u32_child_count`
- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
Fixes#2874
<!-- lance-gatekeeper-fix:v1 agent=582e68bcad65739e189352cb3cbf144c
generation=3 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- expose `memory_limit` and `num_workers` on the Python FTS
configuration for local builds
- forward both build-only settings to the Lance inverted-index builder
- add an end-to-end regression proving the configured memory budget
reaches the native build
## Root cause
LanceDB 0.26.1 pinned Lance 1.0.1. That Lance version used an FTS
partition-merge path whose retained data made memory grow with merge
progress on very large indexes. Upstream Lance
[#5754](https://github.com/lance-format/lance/pull/5754) changed
partition merging to stream its inputs, reducing peak memory by about
25%. Lance [#6174](https://github.com/lance-format/lance/pull/6174) then
removed the old merge phase, compressed posting lists during
construction, reduced indexing memory by about 60%, and introduced a
total build `memory_limit` for bounded workers.
Current `main` pins Lance 11.0.0-beta.3, which contains those
architectural fixes. This PR does not duplicate or claim the upstream
leak fix; it addresses the remaining Python API gap.
## This repair
LanceDB Python did not expose the native FTS builder resource controls.
`memory_limit` now sets the total local-build budget in MiB, divided
among effective workers, and `num_workers` controls build parallelism.
Both are build-only settings and do not affect remote builds or
persisted index configuration.
## Validation
- `cargo check --quiet --features remote --tests --examples`
- `cargo fmt --all`
- `uv run --project python --extra tests --extra dev ruff check .`
- `uv run --project python --extra tests --extra dev ruff format --check
python/python/lancedb/index.py python/python/tests/test_fts.py`
- `uv run --project python --extra tests pytest python/tests/test_fts.py
-q` (51 passed)
Fixes#2923
<!-- lance-gatekeeper-fix:v1 agent=a1ceedf74531e0212cb6f1ebf9390a26
generation=1 -->
---------
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
A materialized view is a table whose contents are defined by a query
over
one source table and maintained by refresh rather than by writes.
The declaration half: create_materialized_view(name, source) resolves a
projected, filtered and limited definition against the source schema --
output types come from the DataFusion planner, never the caller -- and
commits an empty table carrying it as kind-tagged JSON in schema
metadata.
The tag lets a kind added later read back as a view this version cannot
refresh rather than as a plain table. Views open and list as ordinary
tables.
Sources must have stable row ids, checked here because the property
cannot
be enabled later: each view row records its source row in
__source_row_id,
and that provenance survives compactions, updates and deletes only when
row
ids are stable.
A view inherits the metadata describing its columns and none governing
how a
table is written, so blob markers carry through while declarations its
always-nullable fields would contradict are stripped. Embedding
configuration is rewritten to the view's column names, and dropped where
it
does not project both ends of a function.
<sub>Stack created with <a
href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a
href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
## Summary
- document that current LanceDB releases use one compute worker without
warning on two-vCPU containers
- distinguish compute-worker tuning from storage I/O concurrency
- direct users of affected LanceDB 0.21.1 installations to upgrade and
link the current threading guidance
## Root cause
The Lance version bundled with LanceDB 0.21.1 warned whenever the
detected CPU count was less than or equal to its default two-core I/O
reservation. A two-vCPU deployment therefore emitted the warning on
every query even though falling back to one compute worker was the
intended behavior. Lance fixed that warning condition upstream in
lance-format/lance#3710, and LanceDB current main already pins a version
containing the runtime fix; the Python package documentation did not
explain the corrected behavior or the distinct thread controls.
## Validation
- `git diff --check`
- verified the linked Lance threading-model documentation returns HTTP
200
Fixes#2326
<!-- lance-gatekeeper-fix:v1 agent=9f141242416a6dbeb43be0e80404dd4d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
## Summary
- add cross-platform regression coverage for Azure table URI
construction
- assert that az:// database paths always produce forward-slash blob
keys
## Root cause
ListingDatabase previously used the host filesystem Path join operation
for object-store URIs, which inserted a backslash on Windows. The URI
construction was corrected in #2575, but the original Azure report had
no regression coverage and remained open.
## Validation
- cargo fmt --all
- cargo test --quiet --features remote -p lancedb
test_table_uri_uses_forward_slashes_for_azure
- cargo check --quiet --features remote --tests --examples
- cargo clippy --quiet --features remote --tests --examples
Fixes#2283
<!-- lance-gatekeeper-fix:v1 agent=dd96adfd0fcf303c11e873300663d8f6
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
`StreamingDataset`, `PermutationBuilder`, and `Permutation` now work
against a `RemoteTable` (LanceDB Cloud and Enterprise), which unblocks
benchmarking the loader against the enterprise cluster cache.
```python
db = lancedb.connect("db://my-db", api_key=..., host_override=...)
ds = StreamingDataset(db.open_table("training"), world_size=8, rank=r)
```
Rows are addressed by `_rowid` exactly as before —
`PermutationReader::load_batch` already built the same `_rowid IN (...)`
filter that `Table::take_row_ids` sends, so the loader's fetch was
always the take path. It just was never allowed to run.
### The guard
`PermutationBuilder.__init__` rejected anything without `_inner`, so a
`RemoteTable` raised `TypeError` before reaching the PyO3 layer — which
already unwraps one via `_table._inner`.
### A bounded schema lookup
`PermutationReader::output_schema` reads the schema off a query plan,
and building a plan on a remote table *executes* the query
(`create_plan` → `execute_query`). With no limit that is `k =
isize::MAX`, so asking a remote table for its output schema pulled the
whole table over HTTP and threw it away — once per assigned split, on
every epoch, since `StreamingDataset.__iter__` constructs a
`Permutation` per split.
One row rather than zero, deliberately: lance gates its limit node on
`self.limit.unwrap_or(0) > 0`, so `Some(0)` means *no limit*.
### Tables with an LSM write spec are refused
A permutation references rows by row id, and rows that have not been
flushed to the base table do not have one yet. The loader could read
around them, but they would then be missing from training with nothing
said about it, so the build refuses such a table up front instead of
half supporting it.
### Fallible identity construction
`PermutationReader::identity` resolved `inner_new` with `unwrap`. That
was near total against a local dataset, but construction counts the base
table — an HTTP round trip for a remote one — so a transient network or
auth failure became a panic across the PyO3 boundary.
### Tests
End-to-end `permutation_builder` and `StreamingDataset` runs against a
mock server, the former torch-free so it runs wherever the suite does,
plus a test that a build succeeds without an LSM write spec and is
refused once one is installed.
A registered `FunctionVersion` has an exact identity and grouped output
contract, but the Python SDK cannot currently bind it to table columns
without manually constructing wire models.
Calling a `FunctionVersion` with named `col(...)` references now returns
one immutable `FunctionApplication` pinned to that exact version. The
application preserves named-struct outputs as one sibling group, while
`rename(columns=...)` defines the result-field to table-column mapping
consumed by `Table.add_columns`. Derived expressions and incomplete or
unknown input names fail before declaration.
A remote backfill submission validates its target column against a table
snapshot, but it did not carry the existing read-after-write freshness
headers. Immediately after `add_columns`, a stale query node could
therefore reject the newly committed column.
Route backfill submission through the remote table read fence so it
carries the version returned by the preceding write. The shared remote
submission path gives synchronous and asynchronous client surfaces the
same freshness guarantee.
The Windows wheel job is the slowest job in the PyPI release workflow.
Fat LTO of the cdylib is single-threaded and the peak-memory step of the
build, so it does not get faster with more cores — and it has already
caused rustc-LLVM OOM on the Windows runners for the nodejs builds.
Switch the job to thin LTO with 16 codegen units on a
`windows-2025-8x-x64` runner, trading some runtime performance on our
least performance-sensitive platform for build time. This matches what
the nodejs Windows builds in `npm-publish.yml` already do.
`pypi-publish.yml` is in this workflow's `pull_request` paths filter, so
this PR triggers a dry-run build that shows the new timing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arrow-rs and datafusion crates are released in lockstep, but
Dependabot has been opening one PR per sub-crate for them — the 58.3.0
to 58.4.0 wave produced four separate PRs for `arrow`, `arrow-array`,
`arrow-schema`, and `arrow-buffer`. The existing `rust-minor-patch`
group did not catch them because it only filters on `update-types` and
declares no patterns.
This PR adds an explicit `arrow-datafusion` group matching `arrow*`,
`parquet*`, `datafusion*`, and `object_store`, so those bumps arrive as
a single PR. It is listed before `rust-minor-patch` because a dependency
joins the first group it matches, and it deliberately omits
`update-types` so major bumps are grouped too.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bugs in Node's reading of the embedding_functions schema metadata.
First, parseFunctions keyed its result map by function name, so a table
whose metadata configures the same function for two vector columns came
back with only the last one. It now keys by the vector column, the
convention Python's parser already uses.
Second, Node could not read metadata written by the Python bindings at
all, which spell the keys snake_case: configs parsed with both columns
undefined, breaking embedding application on add() and leaving only
query-side embedding working. The parse now accepts both spellings.
Both fixes land in one shared parser used by every reader --
parseFunctions and the makeArrowTable schema validator, which had its
own private camelCase-only parse -- so the wire contract cannot fork
between entry points. A config naming no source or vector column is an
error at the boundary rather than a default downstream, as are two
configs claiming one column. The "vector" fallback remains only on the
optional field of user-supplied configs.
Breaking: parseFunctions is exported and its map keys change from
function name to vector column.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wyatt Alt
LanceDB Robot
lancedb-gatefixer[bot]
Drew
Xuanwo
Lance Release
Jack Ye
Will Jones
Ayush Chaurasia
Weston Pace
Dan Tasse